Zabbix Queue backlog after radius failure
Zabbix Infrastructure
OS: Ubuntu 24.04
Zabbix: 7.4.10
Zabbix servers: 2 in HA
Zabbix Proxies: 5 in active mode, one is stand alone the other four are in proxy groups grouped in 2's
PostgreSQL: 18.3, w/timescaleDB 2.23.1
PGbouncer: 1.24.1
Compression: on after 7d
Issue experienced: Around 4am yesterday two of our Zabbix proxies stopped using their value cache and they started backlogging items. This all seems to coincide with our Radius server failing to authenticate requests during this time, the radius server has failed to auth at other times but not necessarily at this scale.
After Radius resumed working after a few moments, the two Zabbix proxies never caught up on the backlog of items you can seen in the queue of the screenshot I've included (both proxy 3 and 4 were about the same).
To resolve the backlog I Restarted the primary Zabbix server at 9:18 AM which failed over to the second Zabbix server, at this point certain metrics started recording again, but the queue stayed backlogged. At 1:12 PM I restarted both proxy 3 and 4 and that resolved their backlog of queued items.
Any suggestions for a bug this could be? or perhaps what I should see in the logs to help narrow down what the problem could be?
OS: Ubuntu 24.04
Zabbix: 7.4.10
Zabbix servers: 2 in HA
Zabbix Proxies: 5 in active mode, one is stand alone the other four are in proxy groups grouped in 2's
PostgreSQL: 18.3, w/timescaleDB 2.23.1
PGbouncer: 1.24.1
Compression: on after 7d
Issue experienced: Around 4am yesterday two of our Zabbix proxies stopped using their value cache and they started backlogging items. This all seems to coincide with our Radius server failing to authenticate requests during this time, the radius server has failed to auth at other times but not necessarily at this scale.
After Radius resumed working after a few moments, the two Zabbix proxies never caught up on the backlog of items you can seen in the queue of the screenshot I've included (both proxy 3 and 4 were about the same).
To resolve the backlog I Restarted the primary Zabbix server at 9:18 AM which failed over to the second Zabbix server, at this point certain metrics started recording again, but the queue stayed backlogged. At 1:12 PM I restarted both proxy 3 and 4 and that resolved their backlog of queued items.
Any suggestions for a bug this could be? or perhaps what I should see in the logs to help narrow down what the problem could be?
Attached Files