Zabbix 4.2.2 - Active Checks Stopped Working

[maddog2050]

Hi,

I've upgraded our zabbix servers to 4.2.2 from 4.2.1 today and now quiet a lot of my Zabbix Agent Active Checks have stopped working.

How do i go about troubleshooting this issue? As i'm not seeing any errors in the log files.

Many thanks

Adam

[andris]

Choose some agent for which active checks are not working.
You can set DebugLevel=4 in zabbix_agentd_conf and restart agent (or increase logging level at runtime for "active checks" process(es)).
Then look for line "sending [{"request":"active checks"...." - that is agent requesting a list of active checks from server.
What happens after this request ? If it works normally there will be lines like "got [{"response":"success","data":[{"key":.... ", and "In add_check() key:'..." - that is agent has successfully received the list of active checks from server.
If the list is successfully received, later the agent will be sending data to server (or proxy) like "JSON before sending [{"request":"agent data","session":"868bdd64f72ece2dc59c40a2d0761dab" ,"data":[{"host":"...." and server reply like " JSON back [{"response":"success","info":"processed:...".

[maddog2050]

This is the output from the agent log file with the DebugLevel=5, it doesn't really give any indication as to why it failed.

17333:20190605:121150.852 In refresh_active_checks() host:'127.0.0.1' port:10051
17329:20190605:121150.852 zbx_setproctitle() title:'collector [processing data]'
17329:20190605:121150.852 In update_cpustats()
17333:20190605:121150.852 sending [{"request":"active checks","host":"p-l-zabbix01-a.cromwell-tools.co.uk"}]
17333:20190605:121150.852 before read
17329:20190605:121150.852 End of update_cpustats()
17329:20190605:121150.852 zbx_setproctitle() title:'collector [idle 1 sec]'
17329:20190605:121151.852 zbx_setproctitle() title:'collector [processing data]'
17329:20190605:121151.852 In update_cpustats()
17329:20190605:121151.853 End of update_cpustats()
17329:20190605:121151.853 zbx_setproctitle() title:'collector [idle 1 sec]'
17329:20190605:121152.853 zbx_setproctitle() title:'collector [processing data]'
17329:20190605:121152.853 In update_cpustats()
17329:20190605:121152.853 End of update_cpustats()
17329:20190605:121152.853 zbx_setproctitle() title:'collector [idle 1 sec]'
17333:20190605:121153.852 End of refresh_active_checks():FAIL

[andris]

So, the agent successfully establishes TCP connection to server, sends request but gets nothing.
You can check state of the item and its host in frontend - are there no error messages.
Also you can increase DebugLevel for server "trapper" processes at runtime to see how it looks from server side (what happens after "trapper got '{"request":"active checks","host":"..." lines).