Hi everybody,
this is my first post, so please be kind with me :-) I promise I have searched the forum for similar topics and haven't found one.
I am also experiencing heavy load on my server due to the zabbix housekeeper. I read a lot about: "disable housekeeper at all", "partition your tables" ...., but I first wanted to know if there is any way to improve the algorithm of the housekeeper. I am using Debian squeeze and therefore picked the code of the Debian installation. During inspection of the code I stumbled over the following lines in housekeeper.c:
and
The first lines are clear. Grab me le oldest available clock value from itemid. But what for is the statement in line 283? My only guess is, every item should keep it's values at least for four housekeeper cycles, even keep_history is set to 0.
But what does the statement do?
Case 1:
We create a item with default values (history 90 days). The housekeeper runs every hour. The item values start do drop in (let's say every minute).
Let's say the first housekeeper run is after half an hour. We will get min_clock = "1/2 hour ago" ( at that time the first value arrived). So our statement in line 283 evaluates to:
min_clock = MIN(now - 90 days , "1/2 hour ago" + 4 hours);
=> "now - 90 days" wins: Nothing will be deleted, as we have no values < now - 90 days.
Case 2:
We have monitored our item now quite a while and reach the magic day 90. Let's assume our first value is 90 day and half an hour old when the housekeeper runs. What does the statement in line 283 evaluate to?
min_clock = MIN(now - 90 days , "-90 days - 1/2 hour + 4 hours);
=> "now - 90 days" wins: All values older than 90 days will be deleted.
So far so good!
Case 3:
Let's say: I don't need history of the last 90 days, 7 days is enough. I set history in configuration to 7 days. What happens? Our oldest value is 90 days old but we only want to keep history for the last 7 days from now on. What does our statement evaluate to:
min_clock = MIN(now - 7 days , "-90 days + 4 hours);
=> "-90 days + 4 hours " wins: Only the values older than 90 days + 4 hours will be deleted. Or am I completely wrong?
Case 4:
I want to get rid of history completely and set history to 0. It doesn't matter. It's the same behaviour as in Case 3.
Case 5:
I create an item and set it's history to 0 from the beginning. What does our statement? Once again, we assume the housekeeper runs half an hour after we got our first value and housekeeping cycle is one hour.
min_clock = MIN(now - 0 days , - 1/2 hour + 4 hours);
"now - 0 " is smaller than 3 1/2 hours in the future. now wins: All values older than now will be deleted, which means ALL values will be deleted. Am I wrong again?
Patch:
What about: I don't care how old the values in the database are. All item values older than history should be deleted. No matter, what history is set to, I want to keep at least the values for 4 housekeeper cycles to have my triggers something to evaluate. Replace line 274 - 283 by:
Please enlighten me :-)
this is my first post, so please be kind with me :-) I promise I have searched the forum for similar topics and haven't found one.
I am also experiencing heavy load on my server due to the zabbix housekeeper. I read a lot about: "disable housekeeper at all", "partition your tables" ...., but I first wanted to know if there is any way to improve the algorithm of the housekeeper. I am using Debian squeeze and therefore picked the code of the Debian installation. During inspection of the code I stumbled over the following lines in housekeeper.c:
Code:
274 result = DBselect("select min(clock) from %s where itemid=" ZBX_FS_UI64, table, itemid);
275
276 if (NULL == (row = DBfetch(result)) || SUCCEED == DBis_null(row[0]))
277 {
278 DBfree_result(result);
279 return 0;
280 }
281
282 min_clock = atoi(row[0]);
Code:
283 min_clock = MIN(now - keep_history * SEC_PER_DAY, min_clock + 4 * CONFIG_HOUSEKEEPING_FREQUENCY * SEC_PER_HOUR);
284 DBfree_result(result);
285
286 deleted = DBexecute("delete from %s where itemid=" ZBX_FS_UI64 " and clock<%d", table, itemid, min_clock);
But what does the statement do?
Case 1:
We create a item with default values (history 90 days). The housekeeper runs every hour. The item values start do drop in (let's say every minute).
Let's say the first housekeeper run is after half an hour. We will get min_clock = "1/2 hour ago" ( at that time the first value arrived). So our statement in line 283 evaluates to:
min_clock = MIN(now - 90 days , "1/2 hour ago" + 4 hours);
=> "now - 90 days" wins: Nothing will be deleted, as we have no values < now - 90 days.
Case 2:
We have monitored our item now quite a while and reach the magic day 90. Let's assume our first value is 90 day and half an hour old when the housekeeper runs. What does the statement in line 283 evaluate to?
min_clock = MIN(now - 90 days , "-90 days - 1/2 hour + 4 hours);
=> "now - 90 days" wins: All values older than 90 days will be deleted.
So far so good!
Case 3:
Let's say: I don't need history of the last 90 days, 7 days is enough. I set history in configuration to 7 days. What happens? Our oldest value is 90 days old but we only want to keep history for the last 7 days from now on. What does our statement evaluate to:
min_clock = MIN(now - 7 days , "-90 days + 4 hours);
=> "-90 days + 4 hours " wins: Only the values older than 90 days + 4 hours will be deleted. Or am I completely wrong?
Case 4:
I want to get rid of history completely and set history to 0. It doesn't matter. It's the same behaviour as in Case 3.
Case 5:
I create an item and set it's history to 0 from the beginning. What does our statement? Once again, we assume the housekeeper runs half an hour after we got our first value and housekeeping cycle is one hour.
min_clock = MIN(now - 0 days , - 1/2 hour + 4 hours);
"now - 0 " is smaller than 3 1/2 hours in the future. now wins: All values older than now will be deleted, which means ALL values will be deleted. Am I wrong again?
Patch:
What about: I don't care how old the values in the database are. All item values older than history should be deleted. No matter, what history is set to, I want to keep at least the values for 4 housekeeper cycles to have my triggers something to evaluate. Replace line 274 - 283 by:
Code:
if( keep_history * SEC_PER_DAY < 4 * CONFIG_HOUSEKEEPING_FREQUENCY * SEC_PER_HOUR ) {
min_clock = now - 4 * CONFIG_HOUSEKEEPING_FREQUENCY * SEC_PER_HOUR;
} else {
min_clock = now - keep_history * SEC_PER_DAY;
}
Comment