I'm using a template for my zabbix server and this morning, it started alerting me that the processes reached 100%, how do I fix this?
Attached Files
-
-
Please check MaxHouseKeeper delete in the zabbix_server.conf file.
You can adjust it and find a balance point.
I have also ready about partitioning the MySQL zabbix DB so that housekeeper deletes become easier. -
What should I adjust it to? Is it okay to just turn off this trigger?
I did a grep of the zabbix log and saw the following:
16762:20170730:080309.468 housekeeper [deleted 173963 hist/trends, 22410014 items, 0 events, 1669 problems, 0 sessions, 0 alarms, 0 audit items in 101291.599422 sec, idle for 1 hour(s)]
16762:20170730:090310.087 executing housekeeper
16762:20170730:230440.196 housekeeper [deleted 133023 hist/trends, 18230343 items, 0 events, 2526 problems, 0 sessions, 0 alarms, 0 audit items in 50490.097106 sec, idle for 1 hour(s)]
16762:20170731:000440.804 executing housekeeper
16762:20170731:033010.975 housekeeper [deleted 121728 hist/trends, 7253154 items, 0 events, 386 problems, 0 sessions, 0 alarms, 0 audit items in 12330.165797 sec, idle for 1 hour(s)]
16762:20170731:043011.584 executing housekeeper
16762:20170731:060800.996 housekeeper [deleted 122704 hist/trends, 4171233 items, 0 events, 106 problems, 0 sessions, 0 alarms, 0 audit items in 5869.406861 sec, idle for 1 hour(s)]
16762:20170731:070801.612 executing housekeeper
16762:20170731:072944.995 housekeeper [deleted 132195 hist/trends, 1738522 items, 0 events, 139 problems, 0 sessions, 0 alarms, 0 audit items in 1303.380750 sec, idle for 1 hour(s)]Comment
-
The logs are showing items getting deleted, per the housekeeping policy you have set up in Zabbix
Does the housekeeper running at 100% cause monitoring issues?
Please check the /usr/local/etc/zabbix_server.conf MaxHousekeeperdelete settings. You can set this to lower, which will enable housekeeper to complete the run faster.Comment
-
I can't recall the details of the default, but i think it fires if the housekeeper is over 75% busy. The problem is, at least in any settings I've found, either 0% or 100%, nothing in between, and if it is running long enough at 100% you get alerts, even if it is running just fine.
I changed mine from something like 30m or whatever it was to 48 hours. Honestly, with occasionally deleting of hosts or other generation of dangling items to clean up, it is not abnormal for it to take a few hours to run at a time, you mostly need be worried if it never stops running. Admittedly 48 hours may be pretty long, but I think the default is much too short, and leads people to panic and start asking how to fix something not really broken, especially with new installs and lots of churn in the database as a natural side effect of experimentation.
I also set:
HousekeepingFrequency=24
MaxHousekeeperDelete=0
These settings might not be right for others, especially in MySQL which I think is a bit less sophisticated than Postgresql about such things. Basically it says "run frequently" so as to have less to do at any given time, and secondly when you need to delete for a specific set of items, delete them all in one big go, not a few at a time.
I found doing these two things dramatically reduced the overall housekeeping runtime I needed, mostly from the latter. Your mileage may vary especially if your database/server is already taxed just running zabbix.
Partitioning is a better overall solution as it goes to the cause, not just the symptoms. I just didn't need to, so didn't.Comment
Comment