Utilization of preprocessing manager processes over 75% / Delay in items

[bvivi57]

Hi

We are migrating from zabbix 4.0.34 to Zabbix 6.2.3

We have installed a new infrastructure and the zabbix server is mounted in docker but the database is on a separate server (maria DB 10.7.7)

At first it was fine, but now we have the "Utilization of preprocessing manager processes over 75%" alert that appears regularly and Preprocessing queue size keeps increasing. We are often forced to restart zabbix. We also notice that some items are not refreshed as they should be (delay in each check).

At the moment we have 283 hosts and 57287 items. Performance index is 434.59 new values per second.

The server is a VM with 20GB of ram and 12vCPU.

The database is on a server with 20GB of RAM and 6 vCPU

ulimits:
   nproc: 65535
   nofile:
    soft: 20000
    hard: 40000
  deploy:
   resources:
    limits:
      cpus: '8'
      memory: 12G
    reservations:
      cpus: '2'
      memory: 1G
​

We don't have enough resources. We tried to play with the server configuration, but nothing happens, the alert reappears.

Here is the current configuration.

# ZBX_LISTENIP=
# ZBX_LISTENBACKLOG=
# ZBX_HISTORYSTORAGEURL=http://elasticsearch:9200/ # Available since 3.4.5
# ZBX_HISTORYSTORAGETYPES=uint,dbl,str,log,text # Available since 3.4.5
# ZBX_ALLOWUNSUPPORTEDDBVERSIONS=0 # Available since 6.0.0
# ZBX_DBTLSCONNECT=required # Available since 5.0.0
# ZBX_DBTLSCAFILE=/run/secrets/root-ca.pem # Available since 5.0.0
# ZBX_DBTLSCERTFILE=/run/secrets/client-cert.pem # Available since 5.0.0
# ZBX_DBTLSKEYFILE=/run/secrets/client-key.pem # Available since 5.0.0
# ZBX_DBTLSCIPHER= # Available since 5.0.0
# ZBX_DBTLSCIPHER13= # Available since 5.0.0
# ZBX_AUTOHANODENAME=fqdn # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_HANODENAME= # Available since 6.0.0
# ZBX_AUTONODEADDRESS=fqdn # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_NODEADDRESSPORT=10051 # Allowed to use with ZBX_AUTONODEADDRESS variable only. Available since 6.0.0
# ZBX_NODEADDRESS=localhost:10051 # Available since 6.0.0
# ZBX_DEBUGLEVEL=3
ZBX_STARTPOLLERS=50
# ZBX_IPMIPOLLERS=0
#ZBX_STARTPREPROCESSORS=100
ZBX_STARTPOLLERSUNREACHABLE=20
# ZBX_STARTTRAPPERS=5
# ZBX_STARTPINGERS=5
# ZBX_STARTDISCOVERERS=1
ZBX_STARTHTTPPOLLERS=5
# ZBX_STARTHISTORYPOLLERS=5 # Available since 5.4.0
# ZBX_STARTTIMERS=1
# ZBX_STARTESCALATORS=1
# ZBX_STARTALERTERS=3 # Available since 3.4.0
# ZBX_STARTLLDPROCESSORS=2
#ZBX_JAVAGATEWAY_ENABLE=true
# ZBX_JAVAGATEWAY=zabbix-java-gateway
# ZBX_JAVAGATEWAYPORT=10052
#ZBX_STARTJAVAPOLLERS=5
ZBX_STARTVMWARECOLLECTORS=3
#ZBX_VMWAREFREQUENCY=60
#ZBX_VMWAREPERFFREQUENCY=60
ZBX_VMWARECACHESIZE=64M
#ZBX_VMWARETIMEOUT=10
ZBX_ENABLE_SNMP_TRAPS=true
# ZBX_SOURCEIP=
# ZBX_HOUSEKEEPINGFREQUENCY=1
# ZBX_MAXHOUSEKEEPERDELETE=5000
# ZBX_PROBLEMHOUSEKEEPINGFREQUENCY=60
# ZBX_SENDERFREQUENCY=30
ZBX_CACHESIZE=512M
# ZBX_CACHEUPDATEFREQUENCY=60
# ZBX_STARTDBSYNCERS=4
# ZBX_HISTORYCACHESIZE=16M
# ZBX_HISTORYINDEXCACHESIZE=4M
# ZBX_HISTORYSTORAGEDATEINDEX=0
ZBX_TRENDCACHESIZE=16M
ZBX_TRENDFUNCTIONCACHESIZE=32M
ZBX_VALUECACHESIZE=32M
ZBX_TIMEOUT=30
# ZBX_TRAPPERTIMEOUT=300
# ZBX_UNREACHABLEPERIOD=45
# ZBX_UNAVAILABLEDELAY=60
# ZBX_UNREACHABLEDELAY=15
​

We also configured the database engine because we had IOwait. Here is the configuration of mariadb

innodb_autoextend_increment        = 256
innodb_buffer_pool_size            = 6G
innodb_flush_log_at_trx_commit     = 2
innodb_flush_method                = O_DIRECT
innodb_old_blocks_time             = 1000
innodb_open_files                  = 2048
innodb_stats_on_metadata           = OFF
join_buffer_size                   = 262144
max_allowed_packet                 = 32M
max_connect_errors                 = 10000
max_connections                    = 1000
max_heap_table_size                = 134217728
query_cache_type                   = 0
query_cache_size                   = 0
slow-query-log                     = ON
table_open_cache                   = 2048
thread_cache_size                  = 4
tmp_table_size                     = 134217728
wait_timeout                       = 86400​

We don't know what parameters to play with and why we can't stabilize the platform. Zabbix server: The preprocessing queue never goes down...we are currently at "Preprocessing queue: 753205"

If someone can help us?