active agent suddenly stopped working

[itvet]

I have a host in zabbix that uses active agent checks to report data. a few days ago it suddenly stopped reporting. after checking the logs I fount this error message:
Get active checks error: *** Cannot connect to [x.x.x.x]:10051 [A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]
I have re-installed the client, removed and re-added the host and checked the corrisponding ports on my firewalls and I still cannot work out what is stopping this thing from working.
does anyone know what the problem is and how to fix it?

[dima_dm]

Use tcpdump on zabbix_server, and see network traffic.
Example:
/usr/sbin/tcpdump -i eth0 -s 0 -X "host IP_your_server_here and tcp port 10051"

Check up that zabbix_server listens on port TCP 10051.
see
/etc/zabbix/zabbix_server.conf
StartTrappers=1

[itvet]

everything seems fine, 10051 is in /etc/services as zabbix-trapper. I replaced sensitive IPs and domains with XXX

StartTrappers=20

administrator@monitoring:~$ sudo tcpdump -i eth0 -s 0 -X "host 172.16.1.21 and tcp port 10051"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:11:10.109015 IP host-XXXXX.38391 > 172.16.1.21.zabbix-trapper: Flags [S], seq 1255953854, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 7044 4000 7c06 0d98 5c1b 77ab E..0pD@.|...\.w.
0x0010: ac10 0115 95f7 2743 4adc 55be 0000 0000 ......'CJ.U.....
0x0020: 7002 ffff a4ae 0000 0204 0564 0101 0402 p..........d....
16:11:10.531914 IP port-XXXXX.33982 > 172.16.1.21.zabbix-trapper: Flags [S], seq 33855753, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 49cf 4000 7506 07f1 50af b633 [email protected]
0x0010: ac10 0115 84be 2743 0204 9909 0000 0000 ......'C........
0x0020: 7002 ffff 8858 0000 0204 0564 0101 0402 p....X.....d....
16:11:10.711997 IP host-XXX.63078 > 172.16.1.21.zabbix-trapper: Flags [S], seq 2646678452, win 8192, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
0x0000: 4500 0034 544e 4000 7d06 2889 5c1b 77ac E..4TN@.}.(.\.w.
0x0010: ac10 0115 f666 2743 9dc1 1bb4 0000 0000 .....f'C........
0x0020: 8002 2000 f753 0000 0204 0564 0103 0308 .....S.....d....
0x0030: 0101 0402 ....
16:11:11.453473 IP host-XXX.32437 > 172.16.1.21.zabbix-trapper: Flags [S], seq 2145533829, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 3566 4000 7b06 57a8 5c1b 6979 E..05f@.{.W.\.iy
0x0010: ac10 0115 7eb5 2743 7fe2 3f85 0000 0000 ....~.'C..?.....
0x0020: 7002 ffff ab55 0000 0204 0564 0101 0402 p....U.....d....
16:11:11.668337 IP XXX.58251 > 172.16.1.21.zabbix-trapper: Flags [S], seq 4092907819, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 4b35 4000 7506 6d0b 50af 4fb3 [email protected].
0x0010: ac10 0115 e38b 2743 f3f4 d12b 0000 0000 ......'C...+....
0x0020: 7002 ffff 65f8 0000 0204 0564 0101 0402 p...e......d....
16:11:14.312149 IP host-XXX.32437 > 172.16.1.21.zabbix-trapper: Flags [S], seq 2145533829, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 35be 4000 7b06 5750 5c1b 6979 E..05.@.{.WP\.iy
0x0010: ac10 0115 7eb5 2743 7fe2 3f85 0000 0000 ....~.'C..?.....
0x0020: 7002 ffff ab55 0000 0204 0564 0101 0402 p....U.....d....
16:11:14.593501 IP XXX.58251 > 172.16.1.21.zabbix-trapper: Flags [S], seq 4092907819, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 4b50 4000 7506 6cf0 50af 4fb3 [email protected].
0x0010: ac10 0115 e38b 2743 f3f4 d12b 0000 0000 ......'C...+....
0x0020: 7002 ffff 65f8 0000 0204 0564 0101 0402 p...e......d....
16:11:14.859729 IP XXX.36202 > 172.16.1.21.zabbix-trapper: Flags [S], seq 1739663828, win 65535, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 32de 4000 7506 90a2 50af 4473 [email protected]
0x0010: ac10 0115 8d6a 2743 67b1 29d4 0000 0000 .....j'Cg.).....
0x0020: 7002 ffff faf4 0000 0204 0564 0101 0402 p..........d....
16:11:16.716243 IP host-XXX.63078 > 172.16.1.21.zabbix-trapper: Flags [S], seq 2646678452, win 8192, options [mss 1380,nop,nop,sackOK], length 0
0x0000: 4500 0030 5594 4000 7d06 2747 5c1b 77ac E..0U.@.}.'G\.w.
0x0010: ac10 0115 f666 2743 9dc1 1bb4 0000 0000 .....f'C........
0x0020: 7002 2000 0b63 0000 0204 0564 0101 0402 p....c.....d....
^C
9 packets captured
900 packets received by filter
617 packets dropped by kernel
administrator@monitoring:~$

[dima_dm]

I don't see TCP connection, I see SYN packets only. Where you see SYN,ASK packets? You must check firewall settings.
Example:
13:08:10.860920 eth0 < 172.16.36.76.1183 > 172.16.38.1.pop3: S 2382260024:2382260024(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl 128, id 12075)
13:08:10.860952 eth0 > 172.16.38.1.pop3 > 172.16.36.76.1183: S 1197354217:1197354217(0) ack 2382260025 win 32120 <mss 1460,nop,nop,sackOK> (DF) (ttl 64, id 58101)
13:08:10.861291 eth0 < 172.16.36.76.1183 > 172.16.38.1.pop3: . 1:1(0) ack 1 win 65535 (DF) (ttl 128, id 12076)

May be monitoring server have 2 or more interfaces, аnd there is routing problem? Can you ping server XXX from zabbix_server and vice versa?

[itvet]

ping isn't being sent thrugh the firewall, and I tripple checked the firewalls, but it seems to have come back up by its self, must have changed a setting in the remote router that fixed it