Zabbix Housekeeper not functioning.

[akhil48]

Hi all,
I've been using Zabbix 5 to monitor about 3000 devices for a while now. But I initially forgot to consider the space utilization and set the housekeeper to handle more data clearing.
So now, I have almost 1.5 TB of data in the DB. Which is quite concerning. I have set the trends and history to 90d in the Housekeeper page. But when I check the Latest data, there is data from over 1 year.

Hence, I came to the conclusion that the housekeeper isn't doing its task. So I tried manually running it using "zabbix_server -R housekeeper". But literally nothing happens.
The server log shows "forced execution of housekeeper" and "executing housekeeper" nothing else.

Could someone please help me fix this urgently. The housekeeper service is running fine and starts properly when server service is restarted. I'm attaching the screenshots of the logs.

The housekeeper frequency is set to "1" and MaxHousekeeperDelete=5000 .

Thanks in advance.

[Markku]

Based on your outputs, the housekeeper starts its job but never completes (it should log the run summary after the run). Most probable reason for that is that it has so much to do. The database seems to be under heavy load ("slow query" when inserting data).

You can always go to the various history and trends tables in the database and delete old rows manually. I'd suggest stopping Zabbix server service before that to give the database more time to execute the delete statements.

Markku

[akhil48]

Thanks for the advice Markku, But I had doubts regarding manual deletion of data. We have to "OPTIMIZE" the DB after deleting the rows, but the drive I have is almost full.
So I read that we need almost twice the space to OPTIMIZE a table in MySQL. Hence, I'm sort of in a pickle now.

Could you suggest some other way to get the housekeeper started? I tried reducing the MaxHousekeeperDelete to small values like 100 to see if it would work, but still the same result.

[Markku]

Disclaimer: I'm not a professional database administrator, just self-learned, so YMMV.

My understanding is that deleting the rows as such does not require more disk space. If you delete so much data that huge majority of that 1.5 TB is just slack, I'd guess OPTIMIZE would be happy with much smaller amount of extra space (I'd guess it only needs temporary space for the actual data, not for the deleted data). Also, you could (after deleting the rows) do a database dump to an external host, then delete the old database, and restore the dump again, without using extra disk space on the same server.

I don't think you are going to be able to handle the mess with just the housekeeper anymore, but that's just my opinion. And you are going to need OPTIMIZE (or dump+import) at some point anyway.

Markku

[Markku]

I'd also recommend implementing database partitioning (after cleaning up the mess), assuming you are using MySQL/MariaDB and are able to make the setup: https://blog.zabbix.com/partitioning...cedures/13531/

Markku

[akhil48]

Thanks a lot, will try this as last resort.

[cyber]

Based on your outputs, the housekeeper starts its job but never completes (it should log the run summary after the run).

I think that output picture just does not have summary line...
Just an example...

8974:20240730:141337.597 executing housekeeper
...
...
...
8974:20240730:141418.509 housekeeper [deleted 10913 hist/trends, 2056 items/triggers, 0 events, 13 problems, 0 sessions, 0 alarms, 0 audit, 0 autoreg_host, 0 records in 40.836741 sec, idle for 1 hour(s)]

There's whole 40 seconds between those lines and log can contain a lot of lines for that time..

But if there is really no such summary lines in log, then something weird is going on...

[akhil48]

I think it's the latter, we've been observing the logs for a few days and there are no summary line after forced execution. Weird indeed.