Zabbix queue over 10 mins large, large amounts of missing item data

[_nahallac]

Hi all,

Apologies in advance if I don't have enough information, but I am trying to troubleshoot an issue that I think could have any number of reasons so at the moment I am just trying to narrow it down.

We have a Zabbix instance running version 3.2.2, originally installed using the appliance I believe. The queue numbers are consistently very high (currently 500+ delayed more than 10 minutes) and we are missing large chunks of item data from each of our hosts across pretty much all monitored servers. I have attached a couple of screenshots showing the missing data on one of our Windows servers, as well as the Zabbix Queue - Overview by proxy view. Please note that the top proxy monitors 80 of our servers, the second one down 11 and the bottom one none which is why the queue is much larger on the first proxy in the view.

I have read a few forum posts as well as the documentation and I am a bit stuck at where to start on my troubleshooting. Some other things I have noted:

• The Zabbix central server monitors itself and doesn't seem to miss any data at all, nor are any of its items in the queue
• There are constant errors in the Zabbix server log (/var/log/zabbix/zabbix_server.log) relating to ESXi monitoring timeouts. I have found that the credentials for the hosts are wrong so I will be getting that fixed. I have temporarily disabled them to see if the logs clear and if there is an impact on the queue numbers
• There are some errors relating to active checks (Cannot send list of active checks) on five hosts in the Zabbix proxy logs so I will clear them by fixing the agent hostnames
• All queued items are of type 'Zabbix Agent'
• The MySQL DB on both proxies looks like it hasn't had any tuning applied (InnoDB settings in my.cnf are default)

If anyone could provide input I'd be really grateful. I will update this thread with relevant information as I find it. Thanks in advance!

[LenR]

As you have discovered, queue items can be those that can't be collected due to unreachable hosts as well as collection delays. You don't care as much about the first, but they mask the second. Your method of correcting the obvious problems is sound, if you're lucky, it will result in most or all of the queue being resolved. If you have few items under the 10 minute level, you are probably in pretty good shape.

I've never had to do much if any tuning on proxy mysql either.

How is your history syncer and write cache stats?