[Help]Zabbix housekeeper processes more than 75% busy

[midnight_man]

I have same problem...housekeeper is going to 100% every 1 hour.

Lot off topics here but no solution any help guys?

I have small database mysql...what can be a problem? slow PC?

[kloczek]

Not a PC but storage of this PC.
Housekeeping history and trends data above some nvps does not scale.
Solution is apply history and trends tables partitioning, disable housekeeping and delete oldest data by dropping dally partition.

[midnight_man]

Thanks but why zabbix autors are making housekeeper process when its not usable?

I dont want to use table partitioning because its quite complicated...

I widened inno db buffer pool size from 132mb to 512mb. Now housekeeper is eating 70% CPU not 90% CPU but still it is a problem I think.

[kloczek]

Thanks but why zabbix autors are making housekeeper process when its not usable?

I dont want to use table partitioning because its quite complicated...

I widened inno db buffer pool size from 132mb to 512mb. Now housekeeper is eating 70% CPU not 90% CPU but still it is a problem I think.

Wrong impression.
What housekeeper still is working on small scale monitoring.
Simple more and more people are using zabbix on large and very large scale and things needs to be rearchitected inside Zabbix
I know that for example native support for using partitioned tables is under development. What was provided for biggest Zabbix (as company) customers with paid support as temporary solution now needs to be refined and properly implemented in official source code. Problem only is that as long as switching to partitioned history and trends tables is not a big problem, from developers perspective in zabbix are many other much more important things which will consume limited developers time resources in nearest future.
Zabbix has very steady growth and with growing users base it is quite easy to trash this growth. In such conditions spending limited dev resources on what what demands and needs customers able to pay biggest support quotes is only rational strategy.
Again: at the moment native OOTB partitioning support is not on top of priorities list and switching to partitioned tables is easy process.
Zabbix in many cases is only available monitoring software able to handle quite wide range of monitoring aspects which does not solve any other monitoring software.

Even with biggest possible zabbix database all what you need to switch to partioned layout is slave DB. On slave you can stop syncing data -> partition tables -> sync with master and promote slave as new master with only few seconds downtime. I'm constantly repeating that every => mid size zabbix should have not only master but slave DB as well. For example with slave DB is possible to make full DB backup with zero impact on whole zabbix stack. On every major upgrade slave can be used to test upgrade process, proove that such upgrade will not produce some unexpected errors or measure how long needs to bu actual downtime on prod upgrade. Time to time in some zabbix envs with high volume of constant changes it is good to compact (optimise) tables. Again slave DB instance is perfect place to do this and after promote slave as new master is possible to do such optimisation with few seconds downtime.

At the movement typical zabbix admin must have some level of typical DB skills and IMO it will be like this in next year if not longer. Really in many monitored envs passing barrier of those skills in not a big deal and trust me majority of zabbix users will patiently wait on native partioning even year or two and in meantime instead they will have much more demanded functionalities.

Current zabbix architecture proves that whole zabbix skaffold works up to speed where in some databases people are writing up to few GB of new data every minute. It is really hard to find other monitoring software which may work up tu such scale. It means that this scaffold is enough strong and well architected to try add more features.

[victorhugopa]

Great discussion!!
Thanks!

Wrong impression.
What housekeeper still is working on small scale monitoring.
Simple more and more people are using zabbix on large and very large scale and things needs to be rearchitected inside Zabbix
I know that for example native support for using partitioned tables is under development. What was provided for biggest Zabbix (as company) customers with paid support as temporary solution now needs to be refined and properly implemented in official source code. Problem only is that as long as switching to partitioned history and trends tables is not a big problem, from developers perspective in zabbix are many other much more important things which will consume limited developers time resources in nearest future.
Zabbix has very steady growth and with growing users base it is quite easy to trash this growth. In such conditions spending limited dev resources on what what demands and needs customers able to pay biggest support quotes is only rational strategy.
Again: at the moment native OOTB partitioning support is not on top of priorities list and switching to partitioned tables is easy process.
Zabbix in many cases is only available monitoring software able to handle quite wide range of monitoring aspects which does not solve any other monitoring software.

Even with biggest possible zabbix database all what you need to switch to partioned layout is slave DB. On slave you can stop syncing data -> partition tables -> sync with master and promote slave as new master with only few seconds downtime. I'm constantly repeating that every => mid size zabbix should have not only master but slave DB as well. For example with slave DB is possible to make full DB backup with zero impact on whole zabbix stack. On every major upgrade slave can be used to test upgrade process, proove that such upgrade will not produce some unexpected errors or measure how long needs to bu actual downtime on prod upgrade. Time to time in some zabbix envs with high volume of constant changes it is good to compact (optimise) tables. Again slave DB instance is perfect place to do this and after promote slave as new master is possible to do such optimisation with few seconds downtime.

At the movement typical zabbix admin must have some level of typical DB skills and IMO it will be like this in next year if not longer. Really in many monitored envs passing barrier of those skills in not a big deal and trust me majority of zabbix users will patiently wait on native partioning even year or two and in meantime instead they will have much more demanded functionalities.

Current zabbix architecture proves that whole zabbix skaffold works up to speed where in some databases people are writing up to few GB of new data every minute. It is really hard to find other monitoring software which may work up tu such scale. It means that this scaffold is enough strong and well architected to try add more features.

[kloczek]

Hi all,

I have this issue zabbix housekeeper process every 1,5 hour is loaded heavily to 100%. It doesn't impact overall performance but bothers me. I've got no idea how to debug this issue. Could please someone help?

Should i adjust values bellow?

in zabbix_server.conf:
HousekeepingFrequency=1
MaxHousekeeperDelete=500

This trigger about housekeeper should be removed from standard templates because it is completely normal that if HK is working it shows that it is 100% busy.

[nepto]

This trigger about housekeeper should be removed from standard templates because it is completely normal that if HK is working it shows that it is 100% busy.

I absolutely agree with this.

Such trigger does not make any sense, as:

• there is only one housekeeper job;
• when this housekeeper job runs, it is desirable;
• triggering an alert when 75% of processes are busy is very confusing; it gives an impression there are more housekeeping jobs and some limit should be increased, while this is not actually possible.

[exkg]

My two cents ... now a days we have another (and a lot of) ways to control Housekeeper behavior:

https://www.zabbix.com/documentation/current/manual/web_interface/frontend_sections/administration/general

[]s,
Luciano