Every ZABBIX agentd that establishes a direct connection to the ZABBIX server, repeatedly timed out:
The ZABBIX agentd which is locally installed on the ZABBIX server is also affected. Network problems can thus be excluded.
If I connect those Agents to an ZABBIX proxy, the error messages do not appear.
Affected Versions: ZABBIX Server/Proxy 5.2.3, ZABBIX Agentd 4.4.10, 5.0.7 & 5.2.3
Error messages with LogLevel=4:
Code:
... 556:20210113:190753.025 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:190755.366 active check data upload to [127.0.0.1:10051] is working again 556:20210113:190902.485 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:190906.486 active check data upload to [127.0.0.1:10051] is working again 556:20210113:191656.436 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:191700.437 active check data upload to [127.0.0.1:10051] is working again 556:20210113:191805.561 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) ...
If I connect those Agents to an ZABBIX proxy, the error messages do not appear.
Affected Versions: ZABBIX Server/Proxy 5.2.3, ZABBIX Agentd 4.4.10, 5.0.7 & 5.2.3
Error messages with LogLevel=4:
Code:
30552:20210113:231238.210 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
30548:20210113:231238.979 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231238.979 In update_cpustats()
30548:20210113:231238.979 End of update_cpustats()
30548:20210113:231238.979 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231239.210 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
30552:20210113:231239.210 End of send_buffer():SUCCEED
30552:20210113:231239.211 zbx_setproctitle() title:'active checks #1 [processing active checks]'
30552:20210113:231239.211 In process_active_checks() server:'127.0.0.1' port:10051
30552:20210113:231239.211 for key [agent.ping] received value [1]
30552:20210113:231239.211 In process_value() key:'zabbix.example.org:agent.ping' lastlogsize:null value:'1'
30552:20210113:231239.211 buffer: new element 0
30552:20210113:231239.211 End of process_value():SUCCEED
30552:20210113:231239.211 In need_meta_update() key:agent.ping
30552:20210113:231239.211 End of need_meta_update():FAIL
30552:20210113:231239.211 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100
30552:20210113:231239.211 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[{"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130}],"clock":1610575959,"ns":211418113}]
30548:20210113:231239.980 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231239.980 In update_cpustats()
30548:20210113:231239.980 End of update_cpustats()
30548:20210113:231239.980 zbx_setproctitle() title:'collector [idle 1 sec]'
30548:20210113:231240.980 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231240.980 In update_cpustats()
30548:20210113:231240.980 End of update_cpustats()
30548:20210113:231240.980 zbx_setproctitle() title:'collector [idle 1 sec]'
30548:20210113:231241.980 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231241.980 In update_cpustats()
30548:20210113:231241.980 End of update_cpustats()
30548:20210113:231241.980 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231242.211 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out)
30552:20210113:231242.211 send value error: [recv] ZBX_TCP_READ() timed out
30552:20210113:231242.211 End of send_buffer():FAIL
30552:20210113:231242.211 End of process_active_checks()
30552:20210113:231242.211 In get_min_nextcheck()
30552:20210113:231242.211 End of get_min_nextcheck():1610575969
30552:20210113:231242.211 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100
30552:20210113:231242.211 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[{"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130}],"clock":1610575962,"ns":211738479}]
30548:20210113:231242.981 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231242.981 In update_cpustats()
30548:20210113:231242.981 End of update_cpustats()
30548:20210113:231242.981 zbx_setproctitle() title:'collector [idle 1 sec]'
30548:20210113:231243.981 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231243.981 In update_cpustats()
30548:20210113:231243.981 End of update_cpustats()
30548:20210113:231243.981 zbx_setproctitle() title:'collector [idle 1 sec]'
30548:20210113:231244.981 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231244.981 In update_cpustats()
30548:20210113:231244.981 End of update_cpustats()
30548:20210113:231244.981 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231245.211 send value error: [recv] ZBX_TCP_READ() timed out
30552:20210113:231245.211 End of send_buffer():FAIL
30552:20210113:231245.211 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
30548:20210113:231245.982 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231245.982 In update_cpustats()
30548:20210113:231245.982 End of update_cpustats()
30548:20210113:231245.982 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231246.212 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100
30552:20210113:231246.212 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[{"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130}],"clock":1610575966,"ns":212279051}]
30552:20210113:231246.325 JSON back [{"response":"success","info":"processed: 0; failed: 1; total: 1; seconds spent: 0.000015"}]
30552:20210113:231246.325 In check_response() response:'{"response":"success","info":"processed: 0; failed: 1; total: 1; seconds spent: 0.000015"}'
30552:20210113:231246.325 info from server: 'processed: 0; failed: 1; total: 1; seconds spent: 0.000015'
30552:20210113:231246.325 End of check_response():SUCCEED
30552:20210113:231246.325 OK
30552:20210113:231246.325 active check data upload to [127.0.0.1:10051] is working again
30552:20210113:231246.325 End of send_buffer():SUCCEED
30552:20210113:231246.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
30548:20210113:231246.982 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231246.982 In update_cpustats()
30548:20210113:231246.982 End of update_cpustats()
30548:20210113:231246.982 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231247.325 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
30552:20210113:231247.325 End of send_buffer():SUCCEED
30552:20210113:231247.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
30548:20210113:231247.982 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231247.982 In update_cpustats()
30548:20210113:231247.982 End of update_cpustats()
30548:20210113:231247.983 zbx_setproctitle() title:'collector [idle 1 sec]'
30552:20210113:231248.325 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100
30552:20210113:231248.325 End of send_buffer():SUCCEED
30552:20210113:231248.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
30548:20210113:231248.983 zbx_setproctitle() title:'collector [processing data]'
30548:20210113:231248.983 In update_cpustats()