Huge Zabbix Queue on Agent Active

[khansari]

Hello,

I have Zabbix 4.2.2 on my server (Centos 7) with about 400 hosts and 9000 items that work in Active mode.
My Server has 8 Core with 12G Ram and 50Gb HDD.

Zabbix queue on admin panel show many items.
I was changed many config in zabbix_server.conf and my.cnf files, But this issue not solved.

All of my graph and configs is on attachment.

Can anybody help me ?

Regards.

[khansari]

Another Graphs :

[Atsushi]

Does the busy state of Zabbix server process occur? Please show us the graph "Zabbix data gathering processes busy%" in your environment. If there are many active checks, you need to increase the trapper process.

[khansari]

Does the busy state of Zabbix server process occur? Please show us the graph "Zabbix data gathering processes busy%" in your environment. If there are many active checks, you need to increase the trapper process.

For now, I changed "StartTrappers" to 600و But nothing changed in queue item counts.
Data Gathering graph is in attach

[ingus.vilnis]

Timestamps between Zabbix server and the monitored servers differ, therefore you might see the queue and "delays"while in the reality there are none.

Tuning your Zabbix server and database with randomly maxing unknown parameters will only make matters worse.

What are the agents? Windows? What version of Zabbix agents is used?

And that More than 10 minutes queue is generated just by monitored hosts being offline.

[khansari]

Timestamps between Zabbix server and the monitored servers differ, therefore you might see the queue and "delays"while in the reality there are none.

Tuning your Zabbix server and database with randomly maxing unknown parameters will only make matters worse.

What are the agents? Windows? What version of Zabbix agents is used?

And that More than 10 minutes queue is generated just by monitored hosts being offline.

All of agents are Windows OS (8 , 10, 2003, 2008, 2016)
Zabbix Agent that installed on them is 4.0.1, But Zabbix server is 4.2.2

[ingus.vilnis]

Try updating the agents to the latest 4.0.8 or 4.2.2. There is a fix since 4.0.5 regarding queue and timestamps on Windows.

[Atsushi]

Are there items that can obtain values for items in the active check?
If the Trapper process is not in the busy state, is it not possible to connect to the Zabbix server side from the Zabbix agent side? Please check the Zabbix agent log and Firewall settings for the route to the Zabbix server.

[khansari]

Try updating the agents to the latest 4.0.8 or 4.2.2. There is a fix since 4.0.5 regarding queue and timestamps on Windows.

After one month, all of agents updated to 4.2.1 (one by one - remotely or physically).
But my problem not solved and zabbix server have many items in queue yet.

For now, I have new problem.
For example one of my hosts have a partition with name F and free space on this partition is 4%.
I have a item for check free space on all partition of this host with 1h interval, and a trigger that will fire when free space is less than 5%.

So we must have a trigger problem in my dashboard, But it is not.
A trigger fired at a time and resolved at another time, without any change in host's partition. (see attachment).

[ingus.vilnis]

Two hosts report to Zabbix server with identical Host names defined in agent configuration files. Find both those hosts and change to appropriate values.

[khansari]

Two hosts report to Zabbix server with identical Host names defined in agent configuration files. Find both those hosts and change to appropriate values.

Thanks, There is no duplicate host name, But i have many hosts with same ip address.
Because my customers have multiple server with only one internet provider. So all of them have same ip for send data to zabbix server.

[ingus.vilnis]

For active agent items IP does not matter as it is only the Hostname parameter in zabbix_agentd.conf file that distinguishes the hosts to Zabbix server.

You have multiple servers with the same Hostname value in their agent configs.

In Latest data for this host look at the history of values returned for the system.hostname item. There is a chance you will see different names returned with every new value. That will give you a hint which servers to fix.

[khansari]

For active agent items IP does not matter as it is only the Hostname parameter in zabbix_agentd.conf file that distinguishes the hosts to Zabbix server.

You have multiple servers with the same Hostname value in their agent configs.

In Latest data for this host look at the history of values returned for the system.hostname item. There is a chance you will see different names returned with every new value. That will give you a hint which servers to fix.

Very nice trick,
I found duplicated hostname for two host.
Thanks

But my problem about huge items in queue not solved yet. Do you have any trick for this ?

[ingus.vilnis]

633 active items in queue over 10 minutes can be generated by servers that are offline or otherwise unreachable. Check queue details for exact hosts and check if they are online in the first place.

Queue under 10 can't be considered an issue at all as it can be caused by some network delays or clock offsets. I am looking now at a queue of ~15000 items at 2000 nvps and I am not bothered at all, treating it as something "acceptable".

[khansari]

633 active items in queue over 10 minutes can be generated by servers that are offline or otherwise unreachable. Check queue details for exact hosts and check if they are online in the first place.

Queue under 10 can't be considered an issue at all as it can be caused by some network delays or clock offsets. I am looking now at a queue of ~15000 items at 2000 nvps and I am not bothered at all, treating it as something "acceptable".

Yeah, Some of host are unreachable. maybe other host that are online, have a network delay.
If you think it's ok. I haven't any problem with this.

Thank You