Zabbix issues related to history syncer and housekeeper processes

[bclaeys]

Hi,

I have a performance issue with Zabbix.
Every 4 weeks/28 days, the history syncer internal proccesses go up to +80% for a few hours, and during this period, when housekeeping happens, it also goes to 100%, see graph below. Light blue is history syncer processes, orange is housekeeping.
You can see housekeeping being executed every hour, but normally it only runs for a minute.


During the high load, Zabbix performance is very bad, and a lot of triggers fire due to slow response of the checks.

I'm not sure what causes this.

We run zabbix in a docker environment. Mysql is used as database.

Zabbix server env file (non default values):
ZBX_STARTPOLLERSUNREACHABLE=50
ZBX_STARTTRAPPERS=20
ZBX_CACHESIZE=192M
ZBX_STARTDBSYNCERS=12
ZBX_HISTORYCACHESIZE=192M
ZBX_HISTORYINDEXCACHESIZE=64M
ZBX_TRENDCACHESIZE=64M
ZBX_VALUECACHESIZE=96M
ZBX_TIMEOUT=20
ZBX_LOGSLOWQUERIES=1000

Zabbix proxy config (non-default values):
ConfigFrequency=60
StartPollers=200
StartPollersUnreachable=100
CacheSize=64M
StartDBSyncers=4
HistoryCacheSize=128M
HistoryIndexCacheSize=32M
Timeout=20

This is a small environment, 31 hosts and <5000 items (3750 enabled).

Mysql data files sizes:
11.1 GiB - history_uint.ibd
3.0 GiB - history.ibd
2.5 GiB - history_text.ibd
1.1 GiB - trends_uint.ibd

Anyone knows what could've caused this, or how to fix this?
Thanks in advance!

[tim.mooney]

You don't say what version of Zabbix server you're using. It's often useful to provide that info when you're asking questions on the forums. If you're using a long-term support (LTS) release, like 5.0 LTS, making sure you're running the latest minor version in that series is often a good place to start. There are query performance improvements sometimes even late into the lifetime of an LTS.

For a small environment, your Zabbix settings seem too big. I would try reducing your ZBX_STARTTRAPPERS and your ZBX_STARTDBSYNCERS to half of your current value, and I would also reduce ZBX_STARTPOLLERSUNREACHABLE to 1/10th of your current value. With 31 hosts you're monitoring, I can pretty much guarantee that you don't need 50 unreachable pollers. I'm not sure what to say about your various cachesize settings, but I suspect they may also be contributing to your issue. If it were my environment, I would probably try reducing all of the XXXCacheSize settings to half their current values (or maybe 1/4 for the two History-related cache settings) and see if that improves things.

I know it seems counterintuitive that performance could increase if you dedicate less memory for caching and run fewer "worker" processes, but it's definitely possible it will.

So your Zabbix settings have been adjusted considerably from the defaults, but have you done any database tuning? Have you done any tuning related to the InnoDB buffer pool?

How much RAM do you have in your Zabbix server? When these performance issues are happening, how much free memory does your server have?

[bclaeys]

Hi Tim,

Thanks for the info.

Sorry, completely forgot to mention that I was running Zabbix 5.4.1 at the moment.

To answer your question, server memory is 32GB, but shared over different tools running in docker. At all times, at least 12GB was free. We haven't performed any database tuning yet. At first sight, the mysql load wasn't very heavy, but I'll look up some more info related to this.
The configuration files were copied from another (much bigger) zabbix environment. That explains the oversized parameters.

Actions done for now:
Upgraded zabbix to 5.4.10
Changed all parameters to what you advised.

I'll keep an eye on performance the coming days, but will probably only be able to have a good view of it in 4 weeks.

Thanks for the recommendations!

[tim.mooney]

I'll keep an eye on performance the coming days, but will probably only be able to have a good view of it in 4 weeks.

Understood about the 4 week delay before you know if this helps. I think it will, I just don't know if it will help enough, or if additional tuning will be needed.

The thread will still be here, just update once you have more info on the status.

Thanks for the explanation about the configuration files. I was wondering why those values had been tuned so high for a small environment.

[bclaeys]

Hi Tim,

A small update on the issue, your advised changes helped a lot!
The issue isn't completely gone yet, but as you can see, the 100% process use time was much shorter, and we didn't get flooded with alerts anymore.



We'll go on and tweak the parameters some more and eventually try to reach an optimal solution.

I also doubled the size of the mysql InnoDB buffer pool from 125mb (default value from mysql docker image) to 250mb. As we have more than enough memory on the server, this can be further increased if necessary.

Thanks a lot for the help!

[tim.mooney]

Excellent! I'm glad things improved a bunch, and you have future room to tinker and improve things further.

The InnoDB buffer pool change you made is a good one too. As I'm sure you saw in the MySQL or MariaDB docs, increasing the buffer pool is one of the single biggest performance "wins" you can get on a MariaDB/MySQL database using InnoDB/XtraDB tables. Because you've got a system that's shared not just with Zabbix server but also with other tools, you probably can't give InnoDB a huge chunk of memory, but that's one of the few database settings I tune even when there aren't any problems on a system.