Zabbix 6.4.12 - frontend slowing down and proxy disconnecting every 20 minutes or so

[intense]

I have zabbix version 6.4.12 and I have a problem that about every 20 minutes "something" causes the zabbix frontend to slow down for a while and freeze for 1-2 minutes.
Then disk usage increases significantly and disconnects all proxies for a while.
Has anyone had this problem?
Maybe it's a matter of some configuration setting?
I'm asking for help because I've been struggling with this for several days. There is nothing special in the zabbix server logs at this point. The same in the database logs.
I have MariaDB version 10.5.
I have InnoDB optimizations configured.

During this trim Queue temporarily increases significantly... and during these 20 minutes there are no problems with delayed queries...

[mariadb-10.5]

max_connections = 250

innodb_buffer_pool_size=22G
innodb_flush_method=O_DIRECT
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=1
innodb_log_file_size=5G
innodb_log_buffer_size=256M
innodb_stats_on_metadata=off
tmp_table_size=96M
max_heap_table_size=96M
open_files_limit=65535
max_connect_errors=1000000
connect_timeout=60
wait_timeout=28800
skip-name-resolve=ON
performance_schema=ON
join_buffer_size=256K

My virtual machine has 4 CPus (ESXi) and 32GB RAM.









This is what it looks like on the chart and in the proxy log.
What can you do every 20 minutes or so?

[LenR]

What is the config cache reload time for the server and proxy? Mine was 20 min, but I don't remember if that is the default or if I changed it.

[intense]

What is the config cache reload time for the server and proxy? Mine was 20 min, but I don't remember if that is the default or if I changed it.

I disconnected all proxies for a test and the consumption still increases every 20 minutes and it's the same...

On server I have # this options:
Is it wrong?

Are you talking about another parameter?

[intense]

I did ProxyConfigFrequency=120 but it didn't help

Is this parameter CacheUpdateFrequency?

[intense]

It's not the configuration's fault because I changed it to the default one and the problem still exists. Could this be the reason that my Zabbix Server is based on MariaDB 10.5 database, and all proxies are on version 5.5?
Queue Size and others were jumping like that overnight... and then the entire frontend blocked for a moment and the proxies cut off the connections

[intense]

Zabbix approximately every 20 minutes seems to lag for a moment and then Queue Size and Utilization temporarily increases and the frontend does not work. After a minute or two, everything returns to normal and runs smoothly. Something seems to be turning on every 20 minutes... maybe someone has an idea? I have no problems with Queue and they all pass nicely.

[intense]

I reinstalled everything, the server and all proxies. I configured Zabbix from scratch and ZABBIX still cuts out every 20 minutes for 1 minute... the interface doesn't work and utilization increases. Disconnects all proxies and reconnects them after a while. I have no strength left. What could be the reason? I have Zabbix and proxy version 6.4.13. MariaDB 10.5.22 database everywhere. InnoDB optimization done. I use everything in passive mode. Proxies are ACTIVE. I also have Zabbix agents installed on proxy machines and I have # Server Active in the agent... could this be some reason?

utilization does not increase significantly and it still jams the interface

[intense]

What could be the problem?
I'm out of ideas ...

[intense]

My configuration in attachment file.

I have VM machine 4 CPUs and 16GB RAM.

My optimize for MariaDB 10.5.22 InnoDB:

[mariadb-10.5]

max_connections=250

innodb_buffer_pool_size=8G
innodb_flush_method=O_DIRECT
innodb_log_file_size=2G
innodb_log_buffer_size=128M
performance_schema=ON
join_buffer_size=4M
skip-name-resolve=ON
tmp-table-size = 128M
max-heap-table-size = 96M
open_files_limit = 65535
max_connect_errors = 1000000
connect_timeout = 60

Help me because I have no ideas what could cause these jumps and Zabbix freezes. I turned off Housekeeper for a test and still the same thing.
Jamming occurs almost every 20 minutes.

[LenR]

Does the monitoring.rvn.com.pl host collect data at other times? What is the UnreachablePeriod setting? I'm wondering if it could be unreachable items retrying at 20 minutes.

What are your housekeeping intervals? Is housekeeping active when the problem happens?

[intense]

Does the monitoring.rvn.com.pl host collect data at other times? What is the UnreachablePeriod setting? I'm wondering if it could be unreachable items retrying at 20 minutes.

What are your housekeeping intervals? Is housekeeping active when the problem happens?

But UnreachablePeriod I have #

[tim.mooney]

I have VM machine 4 CPUs and 16GB RAM.

Your Zabbix server, your Zabbix web front end, and your Zabbix database are all in a VM environment. That typically means multiple VMs accessing the same group of disks (or flash storage if you're lucky), so if some other VM is running an I/O intensive job every 20 minutes, it could be drastically impacting the I/O on your VM. You need to check with your vSphere administrators and have them check the VMWare built-in monitors for the storage your system is using. Make certain there is plenty of I/O availability during the period when Zabbix is not functioning correctly. It could easily be that some other VM has a scheduled task or cron job that runs every 20 minutes that is very I/O intensive. Even if it only runs for a minute, it may be impacting your VM.

Next, your initial VM install was 4 CPUs and 32 GB RAM, but your reinstall is 4 CPUs and 16 GB RAM. Giving the VM less RAM is not going to improve performance. At least for the database, the more RAM you can allocate to it, the better.

I have thoughts about several of your MariaDB tuning settings, but I think you should first verify (or rule out) whether the performance problem you're seeing is being caused by high I/O on some storage your VM shares.