How is trend data calculated?

[mallyn]

Hello,

I have the problem that history data is not deleted any more. That for the DB is getting incredibly huge. I wanted to check if the trend data is calculated correctly and just the history does not get deleted or if the trend data is not getting calculated at all.

I just cannot figure out where or how this is done. Please can someone point me in the right direction?

Best regards
Norbert

[Zdenek_OMNISENSUIT]

Hello

How did you recognize that history date is not deleted anymore?
What about you housekeeper - working well?
What means incredibly huge db?
What about db growing speed?
And more. ...


-------------------------
omnisensuit.com

[mallyn]

Hello

How did you recognize that history date is not deleted anymore?

Because it increases in size like mad

What about you housekeeper - working well?

Yes, it runs every hour and deletes quite some items.

What means incredibly huge db?

from always ~450GB in 2020 it exploded to ~1TB within some weeks

What about db growing speed?

from the 1st of April to now ~300GB, which is 13GB/day

Our DB Admin "chunked" the tables so I can see the disk size of every month, and the sizes are neary the same although retention for most items should be 1 month.

[gofree]

Hi

when it comes to trends this is how they're being calculated:

Zabbix keeps a 1-hour max/min/avg/count set of values for each item in the table trends. The data is used for trending and long period graphs. The one hour period can not be customized.

https://www.zabbix.com/documentation/current/manual/installation/requirements#database_size

when it comes to housekeeper - it tries to remove 4 times the time of history etc. as the interval its being run - so in your case it will be 4hrs

In my opinion your DB is too big for standard housekeeper and there are few recommendations that always apply for DB :

- did you change anything related to history setting recently, number of hosts, metrics ?
- is it a sudden increase in DB growth - something happened ..sometimes you have metrics of type "text" that are storing whole blocks of text instead of simple string, you can use preprocessing validation
- reconfigure your metrics to strong use of preprocessing - this can save a lot of data ( all the things that are not changing frequently can be dropped , throttling ) https://www.zabbix.com/documentation.../preprocessing
- to frequent samples ( especially without preprocessing, example 1m interval ) is a road to hell - DB will be huge - optimize
- reasonable history, how long you store data ? the longer than bigger the DB is - optimize this paramater
- with DB that big its time to consider partitioning instead of housekeeper ( will work fine for MySQL or Timescale - out of the box )

[mallyn]

Thank you very much, I will have a look at these points - especially preprocessing sounds promising