Workaround for Zabbix Server to use SourceIP without breaking SNMP

[kevind]

There is a bug which causes SNMP to break if you use the "SourceIP" parameter in zabbix_server.conf. Traffic to Zabbix Agents will use the specified SourceIP, but no SNMP packets are sent. I hit this problem installing Zabbix Server 2.0.1 on CentOS 6.2 64-bit, but it has also been reported on earlier Zabbix versions.

I'm trying to build a system with redundancy, with a backup Zabbix server that uses MySQL replication, and can be quickly enabled if the primary system fails. This requires the use of 2 network interfaces on each Zabbix server: one of them (eth0) is up all the time, has the system default gateway assigned to it, and is not used for monitoring (the Zabbix agents aren't configured to talk to its IP). The other interface (eth1) is enabled on the active Zabbix server and disabled on the backup, both of which use the same IP when they are enabled - the IP the Zabbix agents are configured for. eth0 and eth1 are on different subnets, and there are hosts on each subnet that require monitoring.

SNMP could probably use either IP (although we do have a few things that are configured to talk only to the IP assigned to eth1, so that is preferred). But SNMP has to work.

The problem is that if we specify a "SourceIP" in zabbix_server.conf, SNMP breaks, and does not send any packets at all. All SNMP items become unsupported. But if we don't specify a "SourceIP", then most of the Zabbix agent traffic is sent via eth0, which doesn't match the IP the agents expect, so the agents don't work.

It's possible to externally force the SNMP and Zabbix agent traffic to a specific source IP. Here's how I did it, using policy routing and iptables. For this example, eth1 is the interface we want for the SourceIP, and it has IP address 192.179.251.52, netmask 255.255.255.192 ("/26"), and gateway of 192.168.251.1.

This method requires the "iproute" package which is included with CentOS 6.2.

1) Add a new iproute routing table for Zabbix. Edit the file "/etc/iproute2/rt_tables", and at the end, add the line: "1 zabbix"

2) Create the file "eth1-up.sh" with the following:

# This script is run to bring up interface eth1

# enable eth1 on IP address 192.168.251.52/26
ifconfig eth1 192.168.251.52 netmask 255.255.255.192 up

# use this interface for this subnet
ip route add 192.168.251.0/26 dev eth1 src 192.168.251.52 table zabbix

# add default gateway for this interface
ip route add default via 192.168.251.1 dev eth1 table zabbix

# all traffic coming in to this IP should go out on this IP 
ip rule add from 192.168.251.52/32 table zabbix
ip rule add to 192.168.251.52/32 table zabbix

# apply this table to all packets marked "1" by iptables
ip rule add fwmark 0x1 table zabbix
ip route flush cache

3) Create the file eth1-boot.sh:

# These things need to be set once at boot time, they don't persist across reboots

# all traffic coming in to this IP should go out on this IP 
ip rule add from 192.168.251.52/32 table zabbix
ip rule add to 192.168.251.52/32 table zabbix

# apply this table to all packets marked "1" by iptables
ip rule add fwmark 0x1 table zabbix

# disable reverse packet filter on eth1 so that return packets don't get blocked
sysctl net.ipv4.conf.eth1.rp_filter=0

4) Now set up the rules in iptables:

# This only needs to be done 1 time - these rules persist across reboots

# backup the old configs
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.bak

# open ports 80 for http, 3306 for MySQL, 10051 for zabbix, 161:162 for SNMP
iptables -I INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10051 -j ACCEPT
iptables -I INPUT -p udp --dport 161:162 -j ACCEPT

# mark any outgoing packets destined for zabbix agent on eth0 
# iproute will send them to eth1, but then they will need new SourceIP
iptables -A OUTPUT -t mangle -o eth0 -p tcp --dport 10050 -j MARK --set-mark 1

# Change source IP to eth1 IP on any packets going out eth1 to zabbix agent
iptables -A POSTROUTING -t nat -o eth1 -p tcp --dport 10050 -j SNAT --to 192.168.251.52

# mark any outgoing packets destined for SNMP on eth0 
# iproute will send them to eth1, but they will then need new SourceIP
iptables -A OUTPUT -t mangle -o eth0 -p udp --dport 161 -j MARK --set-mark 1

# Change source IP to eth1 IP on any packets going out eth1 to SNMP
iptables -A POSTROUTING -t nat -o eth1 -p udp --dport 161 -j SNAT --to 192.168.251.52

# save new rules
service iptables save

In zabbix_server.conf, make sure you have nothing set for "SourceIP". I also have nothing set for "ListenIP". This method is not in production use yet, but seems to work fine in testing for Zabbix 2.0.1 and CentOS 6.2 64-bit.