Utilization of preprocessing manager processes over 75%

[fsaguilieri]

Hello,


My zabbix process manager always has processing at 100% and the queue always remains high.
Thanks in advance

Zabbix version: 6.0.33
Linux: Alma Linux 9.4 Kernel 5.14.0-427.33.1.el9_4.x86_64




Return of zabbix_server -R diaginfo

== history cache diagnostic information ==
Items:0 values:0 time:0.000348
Memory.data:
size: free:67108480 used:0
chunks: free:1 used:0 min:67108480 max:67108480
buckets:
256+:1
Memory.index:
size: free:14928304 used:1848400
chunks: free:3 used:5 min:8072 max:10686664
buckets:
256+:3
Top.values:
==
== preprocessing diagnostic information ==
Values:106746 done:100268 queued:6 processing:2180 pending:4292 time:0.783250
Top.values:
itemid:291148 values:11 steps:0
itemid:712557 values:11 steps:0
itemid:700746 values:11 steps:0
itemid:712551 values:11 steps:0
itemid:394325 values:11 steps:0
itemid:247801 values:11 steps:0
itemid:712555 values:11 steps:0
itemid:600485 values:11 steps:0
itemid:712559 values:11 steps:0
itemid:600486 values:11 steps:0
itemid:750228 values:11 steps:0
itemid:84052 values:11 steps:0
itemid:84051 values:11 steps:0
itemid:307232 values:8 steps:0
itemid:407152 values:8 steps:0
itemid:381703 values:8 steps:0
itemid:601386 values:8 steps:0
itemid:552449 values:7 steps:0
itemid:313545 values:6 steps:0
itemid:348868 values:6 steps:0
itemid:351326 values:6 steps:0
itemid:89760 values:6 steps:0
itemid:630750 values:6 steps:0
itemid:83340 values:6 steps:0
itemid:591780 values:6 steps:0
Top.oldest.preproc.values:
==
== locks diagnostic information ==
Locks:
ZBX_MUTEX_LOG:0x7fd74ac10000
ZBX_MUTEX_CACHE:0x7fd74ac10028
ZBX_MUTEX_TRENDS:0x7fd74ac10050
ZBX_MUTEX_CACHE_IDS:0x7fd74ac10078
ZBX_MUTEX_SELFMON:0x7fd74ac100a0
ZBX_MUTEX_CPUSTATS:0x7fd74ac100c8
ZBX_MUTEX_DISKSTATS:0x7fd74ac100f0
ZBX_MUTEX_VALUECACHE:0x7fd74ac10118
ZBX_MUTEX_VMWARE:0x7fd74ac10140
ZBX_MUTEX_SQLITE3:0x7fd74ac10168
ZBX_MUTEX_PROCSTAT:0x7fd74ac10190
ZBX_MUTEX_PROXY_HISTORY:0x7fd74ac101b8
ZBX_MUTEX_MODBUS:0x7fd74ac101e0
ZBX_MUTEX_TREND_FUNC:0x7fd74ac10208
ZBX_RWLOCK_CONFIG:0x7fd74ac10230
ZBX_RWLOCK_CONFIG_HISTORY:0x7fd74ac10268
ZBX_RWLOCK_VALUECACHE:0x7fd74ac102a0
==
== value cache diagnostic information ==
Items:37664 values:225227 mode:0 time:0.019659
Memory:
size: free:522669376 used:11940176
chunks: free:194 used:141118 min:24 max:522656736
buckets:
24:9
32:103
40:13
48:56
56:2
64:5
256+:6
Top.values:
itemid:744521 values:52 request.values:45
itemid:758407 values:52 request.values:46
itemid:724889 values:51 request.values:46
itemid:717512 values:51 request.values:44
itemid:724888 values:51 request.values:46
itemid:717509 values:51 request.values:44
itemid:655705 values:50 request.values:1
itemid:779910 values:50 request.values:0
itemid:758406 values:50 request.values:46
itemid:717584 values:50 request.values:44
itemid:753135 values:49 request.values:0
itemid:544277 values:49 request.values:0
itemid:753241 values:49 request.values:0
itemid:544195 values:49 request.values:42
itemid:746456 values:49 request.values:42
itemid:746822 values:49 request.values:0
itemid:547948 values:49 request.values:0
itemid:770421 values:49 request.values:0
itemid:546538 values:49 request.values:42
itemid:548702 values:49 request.values:0
itemid:589767 values:49 request.values:0
itemid:526500 values:49 request.values:0
itemid:758756 values:49 request.values:42
itemid:792264 values:49 request.values:0
itemid:740099 values:49 request.values:42
Top.request.values:
itemid:758407 values:52 request.values:46
itemid:724889 values:51 request.values:46
itemid:724888 values:51 request.values:46
itemid:758406 values:50 request.values:46
itemid:744521 values:52 request.values:45
itemid:717512 values:51 request.values:44
itemid:717509 values:51 request.values:44
itemid:717584 values:50 request.values:44
itemid:744448 values:49 request.values:44
itemid:773730 values:46 request.values:44
itemid:544195 values:49 request.values:42
itemid:746456 values:49 request.values:42
itemid:546538 values:49 request.values:42
itemid:758756 values:49 request.values:42
itemid:740099 values:49 request.values:42
itemid:668098 values:49 request.values:42
itemid:542456 values:49 request.values:42
itemid:547795 values:49 request.values:42
itemid:650159 values:49 request.values:42
itemid:650516 values:49 request.values:42
itemid:741596 values:49 request.values:42
itemid:628796 values:49 request.values:42
itemid:544259 values:49 request.values:42
itemid:741597 values:49 request.values:42
itemid:528299 values:49 request.values:42
==
== LLD diagnostic information ==
Rules:0 values:0 time:0.000877
Top.values:
==
== alerting diagnostic information ==
Alerts:0 time:0.000416
Media.alerts:
Source.alerts:
==


Output from log:

66692:20240909:131527.106 In preprocessor_enqueue() itemid: 784646
66692:20240909:131527.106 In preprocessor_enqueue_dependent() itemid: 784646
66692:20240909:131527.106 End of preprocessor_enqueue_dependent()
66692:20240909:131527.106 End of preprocessor_enqueue()
66692:20240909:131527.106 In preprocessor_assign_tasks()
66692:20240909:131527.106 End of preprocessor_assign_tasks()
66692:20240909:131527.106 End of preprocessor_add_request()
66692:20240909:131527.106 In zbx_ipc_service_recv() timeout:1.000
66692:20240909:131527.106 End of zbx_ipc_service_recv():0
66692:20240909:131527.106 In preprocessor_add_request()
66692:20240909:131527.106 In preprocessor_sync_configuration()
66692:20240909:131527.106 In DCconfig_get_preprocessable_items()
66692:20240909:131527.106 End of DCconfig_get_preprocessable_items() items:35390
66692:20240909:131527.106 End of preprocessor_sync_configuration() item config size: 35390, history cache size: 20918
66692:20240909:131527.106 In preprocessor_enqueue() itemid: 737846
66692:20240909:131527.106 In preprocessor_enqueue_dependent() itemid: 737846
66692:20240909:131527.106 End of preprocessor_enqueue_dependent()
66692:20240909:131527.106 End of preprocessor_enqueue()
66692:20240909:131527.107 In preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_add_request()
66692:20240909:131527.107 In zbx_ipc_service_recv() timeout:1.000
66692:20240909:131527.107 End of zbx_ipc_service_recv():0
66692:20240909:131527.107 In preprocessor_add_request()
66692:20240909:131527.107 In preprocessor_sync_configuration()
66692:20240909:131527.107 In DCconfig_get_preprocessable_items()
66692:20240909:131527.107 End of DCconfig_get_preprocessable_items() items:35390
66692:20240909:131527.107 End of preprocessor_sync_configuration() item config size: 35390, history cache size: 20918
66692:20240909:131527.107 In preprocessor_enqueue() itemid: 770096
66692:20240909:131527.107 In preprocessor_enqueue_dependent() itemid: 770096
66692:20240909:131527.107 End of preprocessor_enqueue_dependent()
66692:20240909:131527.107 End of preprocessor_enqueue()
66692:20240909:131527.107 In preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_add_request()
66692:20240909:131527.107 In zbx_ipc_service_recv() timeout:1.000
66692:20240909:131527.107 End of zbx_ipc_service_recv():0
66692:20240909:131527.107 In preprocessor_add_request()
66692:20240909:131527.107 In preprocessor_sync_configuration()
66692:20240909:131527.107 In DCconfig_get_preprocessable_items()
66692:20240909:131527.107 End of DCconfig_get_preprocessable_items() items:35390
66692:20240909:131527.107 End of preprocessor_sync_configuration() item config size: 35390, history cache size: 20918
66692:20240909:131527.107 In preprocessor_enqueue() itemid: 644726
66692:20240909:131527.107 In preprocessor_enqueue_dependent() itemid: 644726
66692:20240909:131527.107 End of preprocessor_enqueue_dependent()
66692:20240909:131527.107 End of preprocessor_enqueue()
66692:20240909:131527.107 In preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_add_request()
66692:20240909:131527.107 In zbx_ipc_service_recv() timeout:1.000
66692:20240909:131527.107 End of zbx_ipc_service_recv():0
66692:20240909:131527.107 In preprocessor_add_request()
66692:20240909:131527.107 In preprocessor_sync_configuration()
66692:20240909:131527.107 In DCconfig_get_preprocessable_items()
66692:20240909:131527.107 End of DCconfig_get_preprocessable_items() items:35390
66692:20240909:131527.107 End of preprocessor_sync_configuration() item config size: 35390, history cache size: 20918
66692:20240909:131527.107 In preprocessor_enqueue() itemid: 525986
66692:20240909:131527.107 In preprocessor_enqueue_dependent() itemid: 525986
66692:20240909:131527.107 End of preprocessor_enqueue_dependent()
66692:20240909:131527.107 End of preprocessor_enqueue()
66692:20240909:131527.107 In preprocessor_assign_tasks()
66692:20240909:131527.107 End of preprocessor_assign_tasks()

[kamil1]

Hi,
Here are two points that you can try:

• Increase preprocessing workers by adjusting StartPreprocessors in zabbix_server.conf
• Consider adjusting HistoryCacheSize

But before that, please provide more information on your current settings (zabbix_server.conf).

[okoli]

Did you see this post:

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/463124-high-preprocessing-manager-queue-in-zabbix-proxies-and-data-collection-issue