Monitoring hosts through a proxy between routable and unroutable IP addresses

[lstewart]

Some facts:

• Our zabbix server has a routable IP address.
• The server running zabbix_proxy has a routable IP address on bond1
• The server running zabbix_proxy has a non-routable IP address within the same network as the other servers for which it is proxying on bond0

Basically the problem, in a nutshell, is that I see a "green Z" next to each of the hosts being monitored by this proxy (master1) (See attached screen shot). But when attempting to view the latest data for any of these hosts there simply isn't any to be seen. In fact the hosts aren't even listed when selecting the group they are in in some cases. The proxy is listed since it's being monitored directly via zabbix_agentd.conf on 10050.

First the non-routable stuff:
The zabbix_proxy is on the server with 10.255.1.245. The other hosts have IP addresses within 10.255.1.0/24. I see packets to and from both the proxy and individual servers on non-routable IP addresses when running the following tcpdump:

[root@master1 ~]# tcpdump -c 15 -nn -i bond0 port 10050 or port 10051
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 96 bytes
10:16:31.425965 IP 10.255.1.17.33310 > 10.255.1.245.10051: S 1392752071:1392752071(0) win 5840 <mss 1460,sackOK,timestamp 3432872987 0,nop,wscale 7>
10:16:34.789859 IP 10.255.1.238.42831 > 10.255.1.245.10051: S 3779031933:3779031933(0) win 5840 <mss 1460,sackOK,timestamp 1559017234 0,nop,wscale 7>
10:16:36.628319 IP 10.255.1.243.37487 > 10.255.1.245.10051: S 2823046627:2823046627(0) win 5840 <mss 1460,sackOK,timestamp 2767155727 0,nop,wscale 7>
10:16:41.894809 IP 10.255.1.70.40922 > 10.255.1.245.10051: S 756638802:756638802(0) win 5840 <mss 1460,sackOK,timestamp 425820633 0,nop,wscale 7>
10:16:50.107406 IP 10.255.1.19.34922 > 10.255.1.245.10051: S 63061232:63061232(0) win 5840 <mss 1460,sackOK,timestamp 1730973505 0,nop,wscale 7>
10:16:58.883140 IP 10.255.1.20.58824 > 10.255.1.245.10051: S 109903095:109903095(0) win 5840 <mss 1460,sackOK,timestamp 1731381647 0,nop,wscale 7>
10:17:05.161733 IP 10.255.1.69.34200 > 10.255.1.245.10051: S 1854813038:1854813038(0) win 5840 <mss 1460,sackOK,timestamp 1531786540 0,nop,wscale 7>
10:17:05.784721 IP 10.255.1.18.51799 > 10.255.1.245.10051: S 1542462253:1542462253(0) win 5840 <mss 1460,sackOK,timestamp 1213561257 0,nop,wscale 7>
10:17:09.350871 IP 10.255.1.50.34093 > 10.255.1.245.10051: S 1075408094:1075408094(0) win 5840 <mss 1460,sackOK,timestamp 4140971333 0,nop,wscale 7>
10:17:13.622479 IP 10.255.1.246.50030 > 10.255.1.245.10051: S 1616641935:1616641935(0) win 5840 <mss 1460,nop,nop,sackOK>
10:17:17.173754 IP 10.255.1.22.56642 > 10.255.1.245.10051: S 87713421:87713421(0) win 5840 <mss 1460,sackOK,timestamp 1730592211 0,nop,wscale 7>
10:17:19.572655 IP 10.255.1.21.36264 > 10.255.1.245.10051: S 118303326:118303326(0) win 5840 <mss 1460,sackOK,timestamp 1731014073 0,nop,wscale 7>
10:17:29.819647 IP 10.255.1.237.39361 > 10.255.1.245.10051: S 2102639669:2102639669(0) win 5840 <mss 1460,sackOK,timestamp 230500691 0,nop,wscale 7>
10:17:30.531640 IP 10.255.1.49.55030 > 10.255.1.245.10051: S 3456481516:3456481516(0) win 5840 <mss 1460,sackOK,timestamp 490790285 0,nop,wscale 7>
10:17:31.487136 IP 10.255.1.17.38214 > 10.255.1.245.10051: S 1466331774:1466331774(0) win 5840 <mss 1460,sackOK,timestamp 3432933056 0,nop,wscale 7>
15 packets captured
15 packets received by filter
0 packets dropped by kernel

The routable stuff:
Zabbix server and the zabbix proxy are talking. This is no problem.
XXX.XXX.XX.245 = The zabbix proxy
YYY.YYY.YYY.5 = The zabbix server

[root@master1 ~]# tcpdump -c 15 -nn -i bond1 port 10050 or port 10051
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond1, link-type EN10MB (Ethernet), capture size 96 bytes
10:30:01.160319 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: S 3891045545:3891045545(0) win 5840 <mss 1460,sackOK,timestamp 1395159156 0,nop,wscale 7>
10:30:01.161472 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: S 2106121527:2106121527(0) ack 3891045546 win 5792 <mss 1460,sackOK,timestamp 706535709 1395159156,nop,wscale 7>
10:30:01.161487 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: . ack 1 win 46 <nop,nop,timestamp 1395159157 706535709>
10:30:01.161545 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: P 1:6(5) ack 1 win 46 <nop,nop,timestamp 1395159157 706535709>
10:30:01.162722 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: . ack 6 win 46 <nop,nop,timestamp 706535709 1395159157>
10:30:01.162753 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: P 6:79(73) ack 1 win 46 <nop,nop,timestamp 1395159158 706535709>
10:30:01.163222 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: . ack 79 win 46 <nop,nop,timestamp 706535709 1395159158>
10:30:01.204699 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: P 1:6(5) ack 79 win 46 <nop,nop,timestamp 706535713 1395159158>
10:30:01.204706 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: FP 6:38(32) ack 79 win 46 <nop,nop,timestamp 706535713 1395159158>
10:30:01.204736 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: . ack 6 win 46 <nop,nop,timestamp 1395159200 706535713>
10:30:01.204771 IP XXX.XXX.XX.245.56074 > YYY.YYY.YYY.5.10051: F 79:79(0) ack 39 win 46 <nop,nop,timestamp 1395159200 706535713>
10:30:01.205199 IP YYY.YYY.YYY.5.10051 > XXX.XXX.XX.245.56074: . ack 80 win 46 <nop,nop,timestamp 706535713 1395159200>
10:30:23.415528 IP YYY.YYY.YYY.5.41538 > XXX.XXX.XX.245.10050: S 2439014193:2439014193(0) win 5840 <mss 1460,sackOK,timestamp 706537934 0,nop,wscale 7>
10:30:23.415558 IP XXX.XXX.XX.245.10050 > YYY.YYY.YYY.5.41538: S 3923172907:3923172907(0) ack 2439014194 win 5792 <mss 1460,sackOK,timestamp 1395181414 706537934,nop,wscale 7>
10:30:23.416030 IP YYY.YYY.YYY.5.41538 > XXX.XXX.XX.245.10050: . ack 1 win 46 <nop,nop,timestamp 706537935 1395181414>
15 packets captured
17 packets received by filter
0 packets dropped by kernel

Proxy Configuration
[root@master1 ~]# cat /etc/zabbix/zabbix_proxy.conf | grep -v "^#" | grep -v ^$
Server=YYY.YYY.YYY.5
Hostname=master1.gip.XYZ.YYY
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=2
DebugLevel=3
DBName=zabbix
DBUser=yxyxyxyxyx
DBPassword=xyxyxyxyxy
HeartbeatFrequency=30
ConfigFrequency=300
Timeout=3

Zabbix Agents on hosts behind the proxy
These are working and I've confirmed that the "Hostname" field in each zabbix_agentd.conf file matches the hostname configured for the host in the Zabbix server. There are no log entries about connections being dropped because of this either.

[root@master1 ~]# zabbix_get -I10.255.1.245 -s10.255.1.17 -p10050 -k"system.cpu.load[,avg5]"
0.110000
[root@master1 ~]# /usr/local/bin/zabbix_get -s10.255.1.17 -kagent.ping
1
[root@master1 ~]# /usr/local/bin/zabbix_get -s10.255.1.18 -kagent.ping
1
...etc

Does zabbix_proxy have difficulties passing values when dealing with two interfaces? Again, bond1 uses a routable IP address and bond0 uses a non-routable IP address.

What can I check or consider when troubleshooting further?

Thank you for anything you can think of.

[EnigmA-X]

I suggest you try using the 'SourceIP' and 'ListenIP' variables in your config.

Keep us posted!