Help with Housekeeper Processes (100% busy for 3 hours every hour)

[msantangelo]

Hello all,

I've been trying to get some tweaking done on the newly implemented Zabbix Server and I'm hitting a wall pretty regularly now. I am running Zabbix 2.4.4 which I installed on an Ubuntu 14.04.2 LTS VM with 8 GB of RAM (6.25 GB consumed) and 8 vCPUs. Frontend, MySQL, and Zabbix Server are all running on this box currently.

The long and short of it is that roughly every hour the Zabbix housekeeper process jumps to 100% (this I understand is normal) but for on average 3 hours (this is not normal as I understand it). I have attached a picture of the internal process graphs.

The box has 177 hosts with 27115 items (a lot of network monitoring for the core switches; i'm working on pairing this down since I don't need things like operational status on a lot of the ports).

Looking at the Zabbix Server log I am seeing a LOT of "Z3005 query failed 2006 MySQL server has gone away" messages. There is nothing in the mysql logs though.

StartPollers=80
StartIPMIPollers=10
StartPollersUnreachable=80
StartTrappers=20
StartPingers=10
HousekeepingFrequency=1
MaxHousekeeperDelete=100
CacheSize=1G
StartDBSyncers=8
HistoryCacheSize=512M
TrendCacheSize=512M
HistoryTextCacheSize=512M
ValueCacheSize=512M
Timeout=15

Is it just that MySQL can't keep up with the housekeeper process? Is there a tweak to MySQL that I am missing?

[msantangelo]

Aside: I've knocked the number of items down to 7346 enabled changing the NVPS from >350 to 119. The SNMP Devices template with SNMP Interfaces generated a bunch of data that we don't need (errors, description, admin status, operational status). Disabling them is giving me some flex, but housekeeper is still getting busy.

[ingus.vilnis]

Hi,

Looks like your MySQL is having hard times dealing with all the data Zabbix is sending in.

There is not one small tweak that you might be missing. Lots of different configuration parameters might affect your system.

If you really want to tune your Zabbix properly, please check these resources:
Zabbix tuning https://www.zabbix.com/forum/showthread.php?t=47781
MySQL tuning https://www.zabbix.com/forum/showthread.php?t=9925
Database partitioning (which will disable housekeeper completely) http://zabbix.org/wiki/Docs/howto/mysql_partitioning

I know this is a lot or reading but hope you will be able to tune your system properly.

Best Regards,
Ingus

[msantangelo]

I don't mind some reading. I'll start digging through. I'm hoping that pruning the items down from 27.1k to 7.3k will help. I'll still want to get this really well tweaked out though.

Thanks, I'll report back if I see some improvements.

Hi,

Looks like your MySQL is having hard times dealing with all the data Zabbix is sending in.

There is not one small tweak that you might be missing. Lots of different configuration parameters might affect your system.

If you really want to tune your Zabbix properly, please check these resources:
Zabbix tuning https://www.zabbix.com/forum/showthread.php?t=47781
MySQL tuning https://www.zabbix.com/forum/showthread.php?t=9925
Database partitioning (which will disable housekeeper completely) http://zabbix.org/wiki/Docs/howto/mysql_partitioning

I know this is a lot or reading but hope you will be able to tune your system properly.

Best Regards,
Ingus

[ingus.vilnis]

Cool, glad you are not afraid to dig into this issue properly.

First of all check Zabbix server configuration.
Tune your settings according to performance graphs.

Check the amount of pollers you have. I don't think you need that much. Keep them max. ~40% busy.
Check all cache sizes. Have them ~90% free. See the graphs and tune accordingly.

StartDBSyncers=4 Default 4 is totally enough for your instance. Probably it is actually causing you some trouble.

LogSlowQueries=3000 Add this to your server config. You will see slow queries logged in Zabbix log file that indicates bad DB performance.

Check MySQL config file. This is also a very good article. http://www.percona.com/blog/2014/11/...-mysql-zabbix/

innodb_buffer_pool_size=4G - set it to be 50% of your RAM.

Also check that you have innodb_file_per_table enabled. If not then you should re-create your Zabbix DB from backup with it enabled. (optional step, don't focus on it if you don't have it yet.)

Check other params but be careful, don't break working stuff. Make sure you have your config backups in place so you can revert if anything.

When done, go for partitioning and disable housekeeper.

Good luck!

Best Regards,
Ingus

[msantangelo]

Alright, I followed the guide at percona and made a lot of tweaks to my MySQL conf. I also tuned the zabbix conf for starting various inputs as well as changed the cache sizes around a little bit. I also modified poll intervals, history, and trend times. I'm gonna let it run overnight while tail'ing the zabbix_server log and see how it goes. I'll check in tomorrow.

Thanks a ton!

[msantangelo]

A lot of this reading is leading me towards the conclusion that I should just start from scratch and make the VM right from the get go. I'm fine with doing that and all the re-config that it requires. That being said:

If I go to Config -> Hosts -> select all the hosts -> export all the hosts

That would be sufficient to re-import them when I get the new VM setup, right?

Cool, glad you are not afraid to dig into this issue properly.

First of all check Zabbix server configuration.
Tune your settings according to performance graphs.

Check the amount of pollers you have. I don't think you need that much. Keep them max. ~40% busy.
Check all cache sizes. Have them ~90% free. See the graphs and tune accordingly.

StartDBSyncers=4 Default 4 is totally enough for your instance. Probably it is actually causing you some trouble.

LogSlowQueries=3000 Add this to your server config. You will see slow queries logged in Zabbix log file that indicates bad DB performance.

Check MySQL config file. This is also a very good article. http://www.percona.com/blog/2014/11/...-mysql-zabbix/

innodb_buffer_pool_size=4G - set it to be 50% of your RAM.

Also check that you have innodb_file_per_table enabled. If not then you should re-create your Zabbix DB from backup with it enabled. (optional step, don't focus on it if you don't have it yet.)

Check other params but be careful, don't break working stuff. Make sure you have your config backups in place so you can revert if anything.

When done, go for partitioning and disable housekeeper.

Good luck!

Best Regards,
Ingus

[ingus.vilnis]

Hi,

Why do you think your instance is that bad that you need to start all over? Of course, sometimes that helps a lot so it is up to you to decide.

Yes, you have to export all the hosts and you described the steps correctly. Additionally you should also export all the templates that you have added or modified.

One important note if you start over from scratch.
MySQL innodb_file_per_table parameter is the only which you have to specify before you create your Zabbix database. All other MySQL tuning parameters can be added later.

Best Regards,
Ingus

[msantangelo]

It seemed like partitioning the SQL db was definitely the right way to go and since this is devel not prod I'd much rather start from scratch, rebuild, take meticulous notes and get it right from day 0.

Guess I know what I'm doing today

Hi,

Why do you think your instance is that bad that you need to start all over? Of course, sometimes that helps a lot so it is up to you to decide.

Yes, you have to export all the hosts and you described the steps correctly. Additionally you should also export all the templates that you have added or modified.

One important note if you start over from scratch.
MySQL innodb_file_per_table parameter is the only which you have to specify before you create your Zabbix database. All other MySQL tuning parameters can be added later.

Best Regards,
Ingus

[msantangelo]

Alright, finally got partitioning and partition scheduling done, I re-added all my hosts via the backed up config, and now I'm going to sit here and wait a day while watching the graphs to see what happens. The box itself already seems a little more responsive. It hasn't done the discovery for the switches yet, and that will be the real test.

[lpossamai]

Alright, finally got partitioning and partition scheduling done, I re-added all my hosts via the backed up config, and now I'm going to sit here and wait a day while watching the graphs to see what happens. The box itself already seems a little more responsive. It hasn't done the discovery for the switches yet, and that will be the real test.

hey msantangelo ... I know it's been a while but I am having pretty much the same issue here.... were you able to fix it after getting the partitioning done? Do you have any other tips? Cheers!

[vso]

Which version do you use? Does upgrading to latest one help ?

[scuba]

I've been using the latest version 4.0 Alpha 8,everday igetthe message "Zabbix housekeeper processes more than 75% busy" . Think it's an embedded problem of Zabbix and not solved through the years

[kloczek]

I've been using the latest version 4.0 Alpha 8,everday igetthe message "Zabbix housekeeper processes more than 75% busy" . Think it's an embedded problem of Zabbix and not solved through the years

No. There is no any problem here.
Housekeeper it is single process. 100% means that this process is active and 0% means that it is not running.

[scuba]

If so , why it's listed in the problems list. At least its severity should be 'Information'