Hello Everyone,
We started receiving and still receiving many "Zabbix agent unreachable" alert followed by resolved notification mostly from proxy servers
Hosts monitored under proxies also experience agent unreachable issue.
Upon checking, proxy & agent services are up and running fine. It is just we are receiving frequent unreachable and OK notification of agent for application, proxy and db servers.
Using: Zabbix 4.0.5 version. There has been no recent changes in config file and production environment.
proxy log
cat zabbix_proxy_01.log
13237:20220306:044321.057 SNMP agent item "net.if.speed[ifHighSpeed.10023]" on host "sxxxxxxxxxxx" failed: first network error, wait for 15 seconds
13236:20220306:044324.172 SNMP agent item "vm.memory.used[ciscoMemoryPoolUsed.1]" on host "xxxxxxxxxxx" failed: first network error, wait for 15 seconds
13249:20220306:044336.980 SNMP agent item "net.if.out[ifHCOutOctets.10023]" on host "xxxxxxxxxxx" failed: first network error, wait for 15 seconds
agent log:
13393:20220307:124213.664 active check data upload to [1xx.xx.xxx.xxx:xxxxx] started to fail ([recv] ZBX_TCP_READ() timed out)
13393:20220307:124214.666 active check data upload to [1xx.xx.xxx.xxx:xxxxx] is working again....
What we have done so far
1.predicting it to be network issue, Engaged network team, confirmed no issue related to network.
2. tried shifting virtual proxy servers to different esx hosts. Not much luck.
3. Tried putting physical IP of Zabbix application server in proxy config file instead of vip ip .
4. Restarted application, proxy, agent services ,still receiving these errors.
5. Able to telnet and ping to application server from Proxy servers.
Pardon for such a long post but eagerly awaiting for some inputs on this. Thank you in advance.
We started receiving and still receiving many "Zabbix agent unreachable" alert followed by resolved notification mostly from proxy servers
Hosts monitored under proxies also experience agent unreachable issue.
Upon checking, proxy & agent services are up and running fine. It is just we are receiving frequent unreachable and OK notification of agent for application, proxy and db servers.
Using: Zabbix 4.0.5 version. There has been no recent changes in config file and production environment.
proxy log
cat zabbix_proxy_01.log
13237:20220306:044321.057 SNMP agent item "net.if.speed[ifHighSpeed.10023]" on host "sxxxxxxxxxxx" failed: first network error, wait for 15 seconds
13236:20220306:044324.172 SNMP agent item "vm.memory.used[ciscoMemoryPoolUsed.1]" on host "xxxxxxxxxxx" failed: first network error, wait for 15 seconds
13249:20220306:044336.980 SNMP agent item "net.if.out[ifHCOutOctets.10023]" on host "xxxxxxxxxxx" failed: first network error, wait for 15 seconds
agent log:
13393:20220307:124213.664 active check data upload to [1xx.xx.xxx.xxx:xxxxx] started to fail ([recv] ZBX_TCP_READ() timed out)
13393:20220307:124214.666 active check data upload to [1xx.xx.xxx.xxx:xxxxx] is working again....
What we have done so far
1.predicting it to be network issue, Engaged network team, confirmed no issue related to network.
2. tried shifting virtual proxy servers to different esx hosts. Not much luck.
3. Tried putting physical IP of Zabbix application server in proxy config file instead of vip ip .
4. Restarted application, proxy, agent services ,still receiving these errors.
5. Able to telnet and ping to application server from Proxy servers.
Pardon for such a long post but eagerly awaiting for some inputs on this. Thank you in advance.
Comment