Received empty response from Zabbix Agent at [xx.xx.xx.xx].

[sltidc]

Hi,
I have installed and working zabbix server. (4.4.4.4)
Now I feel sam virtual machines gave below error.

************************************************** ********

Received empty response from Zabbix Agent at [xx.xx.xx.xx]. Assuming that agent dropped connection because of access permissions.

************************************************** ********
Strange thing is sove error vms gave one or two graphs,like ping,cpu jumps.
But others are not giving any response.

Some vms are working perfectly and propagating graphs nicely.
Please help me to rectify this issue.
I tried somany things but still failed.

Thanks
Prabhath

[batchen_regev]

Try telnet the vms that doesnt work from the zabbix server host , port 10051 \ 10050 see if you get any response. if some works it seems like maybe a firewall issue.

[sltidc]

Thanks batchen_regev ...
Check and let you know.

[sltidc]

From zabbix server can telnet to vm from 10050 but cant telnet on 10051
Vm to zabbix server, cant telnet both ports.

But zabbix server and that vm in same network and both servers system firewall shutdown.

Please help.

[Markku]

What does your Server directive say in zabbix_agentd.conf in some of the affected agents?

Markku

[sltidc]

This is the tcpdum capture from zabbix server.

tcpdump: listening on enp3s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
172.20.111.149 > 172.20.111.102: ICMP echo request, id 23657, seq 9, length 64
172.20.111.102 > 172.20.111.149: ICMP echo reply, id 23657, seq 9, length 64
172.20.111.102.53311 > 172.20.111.149.10051: Flags [S], seq 1264703645, win 29200, options [mss 1460,sackOK,TS val 2592328800 ecr 0,nop,wscale 7], length 0
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94923710 ecr 2592327798,nop,wscale 7], length 0
172.20.111.149.40894 > 172.20.111.102.21: Flags [FP.], seq 231079930:231079936, ack 643010835, win 229, options [nop,nop,TS val 94923968 ecr 2592316354], length 6: FTP, length: 6
172.20.111.102.21 > 172.20.111.149.56510: Flags [FP.], seq 2212611242:2212611256, ack 3141106956, win 227, options [nop,nop,TS val 2592329248 ecr 94611461], length 14: FTP, length: 14
172.20.111.149.56510 > 172.20.111.102.21: Flags [R], seq 3141106956, win 0, length 0
172.20.111.149.36728 > 172.20.111.102.10050: Flags [FP.], seq 254407239:254407267, ack 3706641813, win 229, options [nop,nop,TS val 94924544 ecr 2592281088], length 28
172.20.111.149 > 172.20.111.102: ICMP echo request, id 23657, seq 14, length 64
172.20.111.102 > 172.20.111.149: ICMP echo reply, id 23657, seq 14, length 64
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94924816 ecr 2592327798,nop,wscale 7], length 0
172.20.111.149.57884 > 172.20.111.102.80: Flags [F.], seq 1390471359, ack 3779430478, win 229, options [nop,nop,TS val 94925216 ecr 2592317417], length 0
172.20.111.149.56520 > 172.20.111.102.443: Flags [FP.], seq 2302659740:2302659913, ack 1608652523, win 229, options [nop,nop,TS val 94925984 ecr 2592318180], length 173
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94926816 ecr 2592327798,nop,wscale 7], length 0

[sltidc]

zabbix server ---> 172.20.111.149

[Markku]

172.20.111.102.53311 > 172.20.111.149.10051: Flags [S], seq 1264703645, win 29200, options [mss 1460,sackOK,TS val 2592328800 ecr 0,nop,wscale 7], length 0
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94923710 ecr 2592327798,nop,wscale 7], length 0
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94924816 ecr 2592327798,nop,wscale 7], length 0
172.20.111.149.10051 > 172.20.111.102.53311: Flags [S.], seq 2469275079, ack 1264703646, win 28960, options [mss 1460,sackOK,TS val 94926816 ecr 2592327798,nop,wscale 7], length 0

No timestamps are shown btw. Anyway, this means that client (active agent) 172.20.111.102 tries to connect to the server 172.20.111.149, but the TCP three-way handshake never completes because the agent-side never completes the handshake. That is very unusual. You could run some tcpdump on the client side as well, and check if there is some misconfigured iptables rules or similar in the client. Are the server and the agents in the same subnet? If not, you should check the firewalls or other middleboxes.

Red "ZBX" in the host list means that passive agent connections are failing. From the short tcpdump output you can tell that there are both active and passive items for 172.20.111.102 (which is not a problem as such, just an observation).

Also, check zabbix_server.log (on the server) and zabbix_agentd.log (on the agent) for anything interesting. Raise the logging level if needed.

To limit the outputs when troubleshooting, you could disable all the hosts but just one (wait like three minutes for the Zabbix server and agents to update the configs before proceeding after that).

Markku