Zabbix agent on {HOST.NAME} is unreachable for 5 minutes

[lpossamai]

Hi,

My current setup is:

1 Zabbix server 3.4
2 Zabbix proxies 3.4

Monitoring a total of 40 hosts.

Once in a while (at least twice a week) I get notifications about: Problem: Zabbix agent on {HOST.NAME} is unreachable for 5 minutes
But for all of my hosts! I get that for every host I am monitoring.

I checked my Zabbix servers and they were all ok. No downtime there.

The only log entry I have during the time that happens is:

failed to send email: wrong answer on RCPT TO "550 #5.1.0 Address rejected."

... Which is not related at all.

What I have checked so far:

1. Zabbix server did not restart by itself
2. Zabbix proxies did not restart by themselves
3. Network was up during that time.

What could cause such an error like that?
Cheers!


EDIT 1:

I've noticed that the hosts that are monitored directly from the Zabbix Server did not get that problem. Only the hosts monitored by a Zabbix Proxy host.
I don't see any error message for those Zabbix Proxies. I will increase the log_level to a debug mode and leave like that.. so next time it happens I'll have more data.

EDIT 2:

I was able to find this only Log error:

1833:20180722:112927.573 cannot send proxy data to server at "172.30.1.118": ZBX_TCP_READ() timed out

That server is located in another country and it communicates with "172.30.1.118" (Zabbix Server) via a Site-to-Site VPN.

Maybe the VPN went down for a little bit?
Is there a way to increase the "timeout" option? so I don't get spammed with lots of "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes" errors?

[cybiblow]

Well i don't know how exactly to solve your issue.
however i'll show you how i set up to monitoring "agent.ping" with the trigger

[lpossamai]

I may have found the issue.

The problem happened again during working hours so that made easier for debugging.

I got several messages from my Zabbix Proxy server as follow:

  1050:20180724:115424.987 TCP expect network [B]error[/B]: cannot connect to [[192.168.188.110]:80]: [111] Connection refused

  1053:20180724:115442.942 End of zbx_snmp_get_values():NETWORK_[B]ERROR[/B]

  1053:20180724:115442.942 End of zbx_snmp_process_standard():NETWORK_[B]ERROR[/B]

Basically, all the hosts were getting "connection refused". I checked if the Zabbix Agent was running on that particular server, and it was.
So, that means that my Cisco ASA Router was blocking Zabbix Proxy to receive and send data due to the high volume being transmitted. Our routers block automatically if they think it is an intrusion... an attack.

I've whitelisted the IPs and will monitor if that happens again.

[lpossamai]

FYI: I've increased "StartTrappers=" which fixed the problem.

[asadleo94]

In my i had shifted my host zabbix2 to zabbix4 and then i recieived unreachable alert even mointoring and other trigers are working fine. i forgot to disable on old zabbix when i disable it. It started worked perfectly