Zabbix Server high queue on Proxies

[MisterX]

Dear all,


we are using two Zabbix 7.04 Servers in HA mode with 7 Proxies in total (each has mariadb installed) and a seperate mariabdb (tables are partioned) residing on a different box. All running on Redhat 8 VMs with each having 32 CPUs with 96G of RAM.
On 2 of the 7 Proxies we often have noticed high Queue with 300 - 800 Items longer than 10 Minutes.
We have ~1.6k VPS in total and and the 2 Proxies with the higher Queue have ~300 - 500 VPS each.
We have one Proxy with the highest ~600 VPS who is not having high queue problems which is a bit strange,

Number of hosts (enabled/disabled) 260 232 / 28
Number of templates 433
Number of items (enabled/disabled/not supported) 176413 171264 / 4789 / 360
Number of triggers (enabled/disabled [problem/ok]) 61585 59554 / 2031 [62 / 59492]

Mostly network devices are being monitored via SNMP and the delayed items are primarly delayed interface informations from Switches.

We have attached the proxy Template in Zabbix frontend to the Proxies (and made sure they are monitored by themselves) but there is nothing unusual to be seen
or high utilization noticed.


Does somebody have an idea how to bring the queue down?

###########################################

Zabbix Server Config:

StartDiscoverers=15
StartHTTPPollers=5
StartTimers=2
StartAlerters=5
StartEscalators=2
StartPreprocessors=250
StartTrappers=50
StartPingers=50
StartPollers=50
StartPreprocessors=250
ValueCacheSize=2G
TrendCacheSize=512M
HistoryIndexCacheSize=1024M
CacheSize=4G
VMwareCacheSize=128M
StartVMwareCollectors=25


Zabbix Proxies:

ProxyOfflineBuffer=24
ConfigFrequency=300
StartDBSyncers: 8
StartPingers: 5
StartPollers: 80
StartPreprocessors=10
StartPollersUnreachable: 5
StartPreprocessors: 10
StartSNMPTrapper: 1
StartVMwareCollectors: 10
Timeout: 4
CacheSize=4G
LogSlowQueries=3000
CacheSize: 4G
HistoryCacheSize: 1024M
HistoryIndexCacheSize: 1024M
VMwareCacheSize: 512M

ProxyBufferMode=hybrid
ProxyMemoryBufferSize=1G



I was thinking about using asynchronus Pollers for the items discovered via LLD because I have read that this may boost performance.
But this would mean adapting all the Templates. And also if I understood correctly this would mean that the Item prototypes will have the same
intervall than the master item.

BR

[markfree]

I was just about to suggest that you to use the asynchronous pollers for the SNMP items.
Using the SNMP Walk option would be ideal if applicable, but you can also change the legacy format to the new asynchronous Get format, that should help too.

Of course, all of this probably means some invisible work.

[MisterX]

Hi,

Thanks for your answer.
Is the process to change it in the templates the same for Walk and Get?

For the Walk item to use the asynch Pollers:

I have read that you have to create Walk item first, add discovery rule with with Master item as depentend and "SNMP walk to Json" as preprocessing rule.Finally in the item Prototype use "snmp walk value" with oid and Index.
The downside would be if I understood correctly that the intervall is bound to the Master item and can not be changed in the item Prototypes. Is that assumption correct?

[omid1979]

I have silimar problem with my zabbix server due zabbix proxy for windows 's server queue's full and make unreachable untile I restart Zabbix SERVER ,
I aks it here

https://www.zabbix.com/forum/zabbix-help/493726-zabbix-proxy-s-high-queue-due-hanged-until-reset-zabbix-server

any idea about it ?
our windows 's Server use zabbix agent to send Items to proxy server

[MisterX]

I was just about to suggest that you to use the asynchronous pollers for the SNMP items.
Using the SNMP Walk option would be ideal if applicable, but you can also change the legacy format to the new asynchronous Get format, that should help too.

Of course, all of this probably means some invisible work.

I have changed one Template to make use of SNMP WALK and asynchronous pollers . The items were successfully discovered but for some unknown reason the filter I have specified in the discovery rule`s "Filter" tab seems to be ignored (it worked before )

[markfree]

Nice that it is works.
As for the filter, try testing the master item and seeing if the values returned match the filter you added to the discovery rule.

[MisterX]

Maybe I did not express myself correctly.
The Items are getting discovered but each status is not supported. Therefore I have changed it back to legacy.
In need to have only the values matched which specified in the Filters Tab.
So you mean applying the filter on the Master item?