Performance Issues

[operationcenter]

Hello,
I am having performance issues on a Zabbix deployment; the deployment consists on a Zabbix Server and two Zabbix Proxy, each of which monitors ~20000 SNMPv2 items (~100 vps).
I currently have for each proxy more than 1000 items in queue for more than 10 minutes, some items have been in the queue for hours.
Form the monitors the VM that host the proxies don't have a high resource usage.
I've already changed the StartPollers to 1000.
How can I deal with this situation? I am completely lost in finding where the bottleneck is and why so many objects are still in the queue.
Thanks in advance.

[ZaBeast]

We are not wizards , show us please your Zabbix environment:

1. Attach performance graphs for Zabbix Server for the last 30 days.
2. Attach Zabbix Server and Zabbix Proxy configuration file
3. Attach MySql configuration file
4. Server hardware specification

[AnthonyS_CP]

What version of Zabbix are you on?
Database type? On system or offloaded?
Are you using any custom templates? If so, go under your "Configuration" Tab and then "Hosts", do you see anything showing a error?

Could you Post the "Zabbix internal process busy %", "Zabbix data gathering process busy %", and under screenshot of your "Queue of items to be updated". This is under the "Administration" Tab, go into "Queue" and make sure "Overview" is selected. Then also attach, "Overview by Proxy".

[operationcenter]

Hi,
here are the files. Zabbix performance graphs are pointless since the Zabbix server is not doing any checks.
I'm on Zabbix 4.2, and the database is on-system. Nothing shows an error under hosts.
The server is a Nutanix cluster with SCSI internal storage.

[AnthonyS_CP]

Understandable and thank you for your information. The Performance graphs on your Server would be an indicator of issue with the ingesting of the processed information from the proxy. The Zabbix server then does all the processing, calculation, correlation, etc. But if the performance limitation would be on the Zabbix Server side, it cannot keep up with the data sent from the proxy. (Not a hardware limitation... but a setting bottle-necking the system).

Please provide three more graphs (just the avg % of the Last, min, avg, and max. These should provide an overall view of the processes. For the server and your proxies, provide the “Zabbix Internal process busy %”. In your settings the proxy conf that was provided, it does show Stats being enabled and should be pulled from the server.

I'm thinking the issue could be with your Zabbix Trapper Process on the Server. I lied about only three more graphs. You should be able to see if this is the case by looking at "Monitoring" > "Graphs" > "Group: Zabbix servers" > " Go into your host and select the "Zabbix data gathering process busy %". Look for the "busy trapper processes" and "busy proxy poller processes" <- only for passive proxies. But please check. This could be filtered by just the past hour. If the "avg"s of these are tapped out. You would need to alter your following settings:

zabbix_server.txt:
# StartTrappers=5
# Since you are using two active proxies you will need multiple trappers for pulling / receiving the data from the proxies. I'd start with 50. After a service restart, monitor your gathering process busy % graphs.
StartTrappers=50

The higher the StartTrappers number, you could start hitting the DBSyncer being the next bottleneck.

[Mike2K]

Hi,
here are the files. Zabbix performance graphs are pointless since the Zabbix server is not doing any checks.
I'm on Zabbix 4.2, and the database is on-system. Nothing shows an error under hosts.
The server is a Nutanix cluster with SCSI internal storage.

Are you really using SCSI storage? not SATA or SAS ?

Could you show us the Zabbix performance graphs ?

[operationcenter]

My Bad, our datastore is an NFS 3.
I'm attaching the performance graphs.

I will increase the trappers as soon as possible.

Thanks

[operationcenter]

Hello,
I tried to increase the trappers with no result.

Thanks