High Preprocessing Manager Queue in Zabbix Proxies and Data collection issue

[Toolsadmin]

Hi All,

We are having Zabbix 6.0.2 environment with

• Two HA Zabbix servers
• Primary and Secondary MySQL 8.0.26 (1TB storage)
• 10 Zabbix proxies ( Active proxies)
• 1957 Hosts (1796 enabled/ 161 disabled) onboarded and 5000 hosts yet to be onboarded
• Number of items - 93387 enabled/ 147975 disabled/ 2575 not supported
• Number of triggers - 40542
• Required server performance, new values per second - 416.5

Two of our Zabbix proxies are having High queue, and data collection is stopped because of that. As a part of troubleshooting we have found that Zabbix proxy was unable to preprocess the item values which is resulting in high preprocessing manager queue as attached in the screenshots.

Zabbix proxy configuration is same for all the proxy servers still only two proxy servers are having the issue, We have followed every possible ways of troubleshooting still no luck in resolving the issue. MySQL DB configuration and fine tuning has also been done according to suggestions made.






Zabbix Proxy Configuration:


ProxyLocalBuffer=1
ProxyOfflineBuffer=4
HeartbeatFrequency=120
ConfigFrequency=3600
DataSenderFrequency=120
StartPollers=5
StartIPMIPollers=1
StartPreprocessors=5
StartPollersUnreachable=10
StartHistoryPollers=2
StartTrappers=10
StartPingers=1
StartDiscoverers=1
StartHTTPPollers=1
HousekeepingFrequency=6
CacheSize=2G
StartDBSyncers=4
HistoryCacheSize=512M
HistoryIndexCacheSize=512M
Timeout=15
TrapperTimeout=300
UnreachablePeriod=300
UnavailableDelay=300
UnreachableDelay=120
LogSlowQueries=10000
StatsAllowedIP=127.0.0.1
ListenBacklog=300


MySQL Proxy DB Configuration:

#GLOBAL BUFFERS

innodb_buffer_pool_size =16G
innodb_log_file_size =1G
innodb_log_buffer_size =256M
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_thread_concurrency=12
innodb_file_per_table=1
innodb_log_files_in_group=2

# Session Buffers

sort_buffer_size =16M
join_buffer_size=16M
net_buffer_length = 1M
read_buffer_size =16M
read_rnd_buffer_size =64M
myisam_sort_buffer_size = 128M

# General Variables

back_log = 200
long_query_time=1
innodb_flush_method=O_DIRECT
thread_cache_size =256
delayed_queue_size = 1500
max_connections = 6500
max_connect_errors = 99999
max_allowed_packet=1024M
tmp_table_size=64M
concurrent_insert=1
log_bin_trust_function_creators=1
group_concat_max_len=4294967295
open_files_limit=10000

Zabbix Server Configuration:

AllowUnsupportedDBVersions=1
HistoryStorageTypes=uint,dbl,str,log,text
HistoryStorageDateIndex=1
ExportDir=/h
ExportFileSize=1G
ExportType=events,history,trends

############ ADVANCED PARAMETERS ################
StartPollers=150
StartIPMIPollers=1
StartPreprocessors=75
StartPollersUnreachable=50
StartHistoryPollers=100
StartTrappers=200
StartPingers=25
StartDiscoverers=2
StartHTTPPollers=1
StartTimers=10
StartEscalators=25
StartAlerters=35
HousekeepingFrequency=4
MaxHousekeeperDelete=50000
CacheSize=12G
CacheUpdateFrequency=300
StartDBSyncers=10
HistoryCacheSize=1G
HistoryIndexCacheSize=1G
TrendCacheSize=1G
TrendFunctionCacheSize=1G
ValueCacheSize=8G
Timeout=30
TrapperTimeout=120
UnreachablePeriod=120
UnavailableDelay=60
UnreachableDelay=60
TmpDir=/tmp
StartLLDProcessors=25
StatsAllowedIP=::/0
ServiceManagerSyncFrequency=300
ProblemHousekeepingFrequency=900
ListenBacklog=3000

Because of high queue on proxies, data collection has stopped completely, Can anyone please help us in resolving this issue?

blogs referred:
https://blog.zabbix.com/how-to-troub...h-queue/12244

[Glencoe]

Upgrade to the latest 6.0.x version. The fixes that deal with slow preprocessing were introduced around 6.0.11/6.0.12.

[Hamardaban]

Avoid the 6.0.15