No Zabbix agents can communicate with server, suddenly

[rhvitfeldt]

Hi,

Since upgrading to Zabbix 6.4.14 we've had issue where no agents couldn't communicate with the server. We saw this error earlier and it somehow got resolved (can't remember if we restarted Zabbix server of it recovered automatically). Then and now we can see hundreds of theese errors in the zabbix_server.log:

3054:20240508:150620.965 failed to accept an incoming connection: from x.x.x.x: unspecified certificate verification error: TLS handshake set result code to 1: file ../ssl/record/rec_layer_s3.c line 308 func ssl3_read_n: error:0A000126:SSL routines::unexpected eof while reading: TLS write fatal alert "decode error"

This time we've restarted the server, the service and agents, and nothing helps. None of our 400 agents can communicate. Not sure if related to 6.4.14 but can't find any reason behind this error.

Agent logs look like this:

2024/05/08 15:37:24.711406 [101] cannot connect to ["hostname":10051]: read tcp xx.xx.xx.xx:61693->xx.xx.xx.xx:10051: i/o timeout
2024/05/08 15:37:24.711406 [101] active check configuration update from host ["hostname"] started to fail zz0.4dzbjay5np6zz