Multiple NICs and zbx 1.8.10

[cirrhus9.com]

Hello:

We have an AppLogic Grid. (C.L.O.U.D)
AppLogic Entities/hosts have a set of standard catalog appliances that handle
routine traffic between all of it using various 10.x.x.x IP networks.

The process goes like this...
1.) the INSSLR appliance at xx.xxx.224.36 (secure inbound gateway)
2.) the main host. <-- zabbix-agent installed here.
3.) the Net appliance at xx.xxx.224.37

Traffic in gets routed to the main host via 10.x.x.x routing (1>2)
Traffic out from the main host goes out the Net device. (2>3)

I access that host (actually the INSSLR device) using xx.xxx.224.36

The problem is that the Net appliance on xx.xxx.224.37 is the one sending info back to the zbxserver as far as I can tell.
The /var/log/zabbix-server/zabbix_server.log has NO entries for .36 but tons for .37

All firewalls are disabled atm.
telnet both ways to 10050 connects but "closed by foreign host" immediately.
netstat -plaunt | grep zabbix on both zbx and sFTP servers shows the what I expect (ugly details below)

How would/does zbx handle a server that has multiple NICs?
One for inbound Traffic and One for outbound traffic for example?

Sanitized network details:

-== ZabbixServer ==-
ifconfig | grep -A 1 "eth"
eth0      Link encap:Ethernet  HWaddr F2:01:01:00:48:01  
          inet addr:xx.xxx.224.62  Bcast:xx.xxx.224.63  Mask:255.255.255.224
--
eth1      Link encap:Ethernet  HWaddr F2:01:01:00:48:00  
          inet addr:10.8.72.1  Bcast:10.15.255.255  Mask:255.248.0.0

netstat -plaunt | grep zabbix
tcp        0      0 0.0.0.0:10050               0.0.0.0:*                   LISTEN      1484/zabbix_agentd  
tcp        0      0 0.0.0.0:10051               0.0.0.0:*                   LISTEN      1530/zabbix_server  
tcp        0      0 :::10050                    :::*                        LISTEN      1484/zabbix_agentd  
tcp        0      0 :::10051                    :::*                        LISTEN      1530/zabbix_server 


-== c9sFTP.INSSLR ==-
ifconfig | grep -A 1 "eth"
eth0      Link encap:Ethernet  HWaddr f2:01:01:00:58:0c  
          inet addr:10.8.88.4  Bcast:0.0.0.0  Mask:255.255.255.255
--
eth3      Link encap:Ethernet  HWaddr f2:01:01:00:58:0f  
          inet addr:10.8.88.1  Bcast:0.0.0.0  Mask:255.255.255.255
--
eth5      Link encap:Ethernet  HWaddr f2:01:01:00:58:11  
          inet addr:10.8.88.7  Bcast:10.15.255.255  Mask:255.248.0.0

netstat -plaunt | grep zabbix          
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1519/zabbix_agentd
tcp6       0      0 :::10050                :::*                    LISTEN      1519/zabbix_agentd


-== c9sFTP.Main host ==-
ifconfig | grep -A 1 "eth"
eth0      Link encap:Ethernet  HWaddr f2:01:01:00:58:0c
          inet addr:10.8.88.4  Bcast:0.0.0.0  Mask:255.255.255.255
--
eth3      Link encap:Ethernet  HWaddr f2:01:01:00:58:0f
          inet addr:10.8.88.1  Bcast:0.0.0.0  Mask:255.255.255.255
--
eth5      Link encap:Ethernet  HWaddr f2:01:01:00:58:11
          inet addr:10.8.88.7  Bcast:10.15.255.255  Mask:255.248.0.0

netstat -plaunt | grep zabbix          
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN1519/zabbix_agentd
tcp6       0      0 :::10050                :::*                    LISTEN1519/zabbix_agentd          


-== c9sFTP.NET ==-
ifconfig | grep -A 1 "eth"
eth0      Link encap:Ethernet  HWaddr F2:01:01:00:58:0A
          inet addr:xx.xxx.224.37  Bcast:xx.xxx.224.63  Mask:255.255.255.224
--
eth1      Link encap:Ethernet  HWaddr F2:01:01:00:58:08
          inet addr:10.8.88.2  Bcast:0.0.0.0  Mask:255.255.255.255
--
eth3      Link encap:Ethernet  HWaddr F2:01:01:00:58:0B
          inet addr:10.8.88.6  Bcast:10.15.255.255  Mask:255.248.0.0
          
netstat -plaunt | grep zabbix
Does NOT apply.

c9sFTP: agentd.conf and /etc/hosts:

grep -i "hostname=" /usr/share/zabbix-agent/zabbix_agentd.conf /etc/zabbix/zabbix_agentd.conf && grep c9sFTP /etc/hosts

/usr/share/zabbix-agent/zabbix_agentd.conf:Hostname=c9sFTP
/etc/zabbix/zabbix_agentd.conf:Hostname=c9sFTP
xx.xxx.224.36    c9sFTP

Yes, I see two darn agentd.conf files. Ubuntu does this when using apt-get.

I am not against installing zabbix-agent 1.8.10 from source but I have another issue if I do with /etc/init.d/ (from "cp /usr/src/zabbix-1.8.10/misc/init.d/debian/zabbix-agent /etc/init.d/" but that's a problem for another time).

Thank you all for your time and experience in this matter.

Habitual on #zabbix

[cirrhus9.com]

Update:

I sniffed the traffic on the c9sFTP server using

tcpdump dst xx.xx.224.62 -vv -i eth3 and port 10051

and simultaneously sniffed on the zbx_server using

tcpdump dst xx.xx.224.62 -vv -i eth0 and port 10051

NOTE: These are NOT the only interfaces I sniffed, but they are the ones that had activity on them.

and then from the c9sFTP host, I ran

telnet xx.xx.224.62 10051

and got output on both:

tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes

13:43:21.628300 IP (tos 0x10, ttl 64, id 54864, offset 0, flags [DF], proto TCP (6), length 60)
    10.8.88.1.50471 > xx.xx.224.62.zabbix-trapper: Flags [S], cksum 0x8989 (incorrect -> 0x09c3), seq 1564735493, win 14600, options [mss 1460,sackOK,TS val 198323797 ecr 0,nop,wscale 6], length 0
13:43:21.632322 IP (tos 0x10, ttl 64, id 54865, offset 0, flags [DF], proto TCP (6), length 40)
    10.8.88.1.50471 > xx.xx.224.62.zabbix-trapper: Flags [.], cksum 0x8975 (incorrect -> 0x8064), seq 1564735494, ack 2482098057, win 229, length 0
13:43:22.614201 IP (tos 0x10, ttl 64, id 54866, offset 0, flags [DF], proto TCP (6), length 43)
    10.8.88.1.50471 > xx.xx.224.62.zabbix-trapper: Flags [P.], cksum 0x8978 (incorrect -> 0x484c), seq 0:3, ack 1, win 229, length 3
13:43:22.614979 IP (tos 0x10, ttl 64, id 54867, offset 0, flags [DF], proto TCP (6), length 40)
    10.8.88.1.50471 > xx.xx.224.62.zabbix-trapper: Flags [F.], cksum 0x8975 (incorrect -> 0x805f), seq 3, ack 2, win 229, length 0

NOTES:
c9sFTP eth3 address: 10.8.88.1
zbx-server eth0 address: xx.xx.224.62

The 10.8.88.1 IP gets recycled after rebooting the application and another would be assigned, (it is also possible that it could get the same 10.x ip).

Thank you for your time.

[cirrhus9.com]

Edit/Update:
The issue stems from the inability of the Zabbix Server to communicate with 10.8.88.1.

Thank You.

[cirrhus9.com]

[SOLVED]Multiple NICs and zbx 1.8.10

Update:

The Book is so invaluable, I cannot express enough.

snmp agent walking and some pretty intensive deciphering...
but if it was easy, the Paper Boy would be doing it.