Housekeeper & alerter more than 75% busy

[akarim]

Hi.

I'm lost. On latest days, get a several problems. The symptoms are Housekeeper & alerter more than 75% busy, but result a lot of

host unavailable, first network error, wait for 15 seconds, connection restored, host became available, and false positives.

Zabbix 2.0.6

conf on agents-->

LogFile=/tmp/zabbix_agentd.log
LogFileSize=10
EnableRemoteCommands=0
LogRemoteCommands=0
Server=144.76.XXX.XXX
ListenPort=10050
StartAgents=3
ServerActive=144.76.XXX.XXX
Hostname=XXXX.XXXX.XXXX
RefreshActiveChecks=120
BufferSend=30
BufferSize=500
MaxLinesPerSecond=100
Timeout=30

On server.conf

NodeID=0
ListenPort=10051
LogFile=/tmp/zabbix_server.log
LogFileSize=10
DBHost=localhost
DBName=XXXX
DBUser=XXX
DBPassword=XXXXXXXX
StartPollers=35
StartIPMIPollers=0
StartPollersUnreachable=30
StartTrappers=30
StartPingers=10
StartDiscoverers=3
StartHTTPPollers=1
ListenIP=0.0.0.0
HousekeepingFrequency=24
MaxHousekeeperDelete=100000
DisableHousekeeping=0
CacheSize=128M
StartDBSyncers=8

my,cnf

[client]
port		= 3306
socket		= /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0
[mysqld]
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
log-error = /var/log/mysqld.error.log
max_connections = 300
key_buffer = 384M
max_allowed_packet = 64M
thread_cache_size = 8
myisam-recover         = BACKUP
query_cache_limit	= 1M
query_cache_size = 64M
join_buffer_size = 1M
expire_logs_days	= 10
max_binlog_size         = 100M
innodb_file_per_table = 1
thread_concurrency = 8
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 8M
read_buffer_size = 2M
sort_buffer_size = 2M
table_cache = 512
[mysqldump]
quick
quote-names
max_allowed_packet	= 16M
[mysql]
[isamchk]
key_buffer = 256M
write_buffer = 2M
read_buffer = 2M
sort_buffer_size = 256M
!includedir /etc/mysql/conf.d/

https://docs.google.com/file/d/0BwTAn45qcvZIc2tDSndRR1VtT28/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIV2NUSGQ2RkpHVmc/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIQ2JneHhLaHplaTA/edit?usp=drivesdk

Apreciate help.

[akarim]

https://docs.google.com/file/d/0BwTAn45qcvZIZThnaG1Yak9QTVU/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIRFlSWXZuM1BZQ0k/edit?usp=drivesdk

[tchjts1]

How many hosts do you have reporting in? Your screenshots are showing only an hour of time. 6 hours or 12 hours would be better to show a pattern.

In your zabbix_server.conf file, do you have a Timeout= setting? If not, put it in and try setting it at 10. Default is 3.

Something that strikes me as odd is that you have MaxHousekeeperDelete set that high. When housekeeper runs, it does usually hit 100% for awhile. I believe the setting default is 500? I have mine set to 300.

With mine set like that, it runs about 10 minutes every hour. I think maybe you have housekeeper running so much that the Zabbix server can't do anything else. I have Zabbix DB and Zabbix APP on 2 different VM's and still see performance degradation in Zabbix screens and graphs when housekeeper runs.

[tchjts1]

It would also be helpful if you showed a screenshot of your memory and swap usage (total and free) just like this from the server that has your Zabbix DB on it

[akarim]

How many hosts do you have reporting in? Your screenshots are showing only an hour of time. 6 hours or 12 hours would be better to show a pattern.

https://docs.google.com/file/d/0BwTAn45qcvZITW1jLU91NTBzaVE/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZINWMtR0dKS05haFU/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIYVR6dU1neFFHT1k/edit?usp=drivesdk

In your zabbix_server.conf file, do you have a Timeout= setting? If not, put it in and try setting it at 10. Default is 3.

Put 10...

Something that strikes me as odd is that you have MaxHousekeeperDelete set that high. When housekeeper runs, it does usually hit 100% for awhile. I believe the setting default is 500? I have mine set to 300.

10000 move to 300
[/QUOTE]

Thanks..

[akarim]

https://docs.google.com/file/d/0BwTAn45qcvZIdDA5NUNmRWRPM3M/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIdzlkaVk5THJrSWs/edit?usp=drivesdk

https://docs.google.com/file/d/0BwTAn45qcvZIY2kwYWgzMzQ0OWM/edit?usp=drivesdk

[tchjts1]

So... now you're good?

[akarim]

Yes.. only

Zabbix alerter processes more than 75% busy


A lot of thanks.

[pc99096]

any clue what to do when the alerter process is more than 75% busy?
which parameter to increase? unfortunately there is no google help

[mdk]

any clue what to do when the alerter process is more than 75% busy?
which parameter to increase? unfortunately there is no google help

hi,pc99096,you can set Default operation step duration on zabbix web to deal with alerter process is more than 75% busy, web->configuration->Action->Operations->Default operation step duration

[mdk]

how to deal with Zabbix history syncer processes more than 75% busy? I set StartDBSyncers=50

[ingus.vilnis]

Hi,

Please don't mess with StartDBSyncers and set it back to what it was before. (4 by default)

Take a look at other performance issues - poller amounts and cache usages and database performance tuning.

Best Regards,
Ingus

[implode]

I have the same problem. My housekeeper and alerter processes are at 100% and from what I can tell, housekeeper has been over 75% for 1h 11m 19s and the alerter 1h 1m 17s. It seems that when the housekeeper kicks in, a lot of triggers may timeout and this causes an alerts snowstorm on false alerts. Even though housekeeper now is at 100%, the zabbix server CPU is still 85-90% idle. Can housekeeper and alerter processes be parallelized such that there can be more than one of each such processes.

[Slash]

Housekeeping is a typical problem when your database begin to grow.

The common advice is to completely disable it and use database partitioning instead, which does the same but with little (if any) overhead.

Currently what the housekeeper does is that it parse your history/trend and remove items that are exceeding their maximum age (the "History storage period (in days)" and the "Trend storage period (in days)" from items configuration). As you can guest, this process become more and more heavy as your database grow... and since the housekeeper run every hour, this can become a big issue.

To give you an idea, I have seen some housekeeping taking more than 1 hour, at this point you are always in housekeeping and zabbix is ultra slow at all time...

When you use data partitionning, your table are separated in different child table, per period (one per day, one per month), at this point, you just have to drop a child table when you don't need the data any more and that's all.

Personally, I'm using this method with postgresql: https://www.zabbix.com/forum/showthread.php?t=33583 and there are other method on the wiki.