TimescaleDB - partitioning of event and problem tables

[gofree]

Just curious wuold it be beneficial to partition event and problem tables as its done with history and trends in TimescaleDB.

I can see a benefit in case of problem storm ( can happen ) - so you can easily drop chunks from problem storm - definitelly loosing some problem but making zabbix working again.

What you think?

[gospodin.horoshiy]

If you have lots of events - such configuration is beneficial for sure to offload Zabbix housekeeper. Be careful with changing the DB schema though - you may have problems upgrading zabbix in the future.

[gofree]

I'm just wondering if that not worth to be part of zabbix default setup - as prevention in case of event storms. If zabbix already partitions history and trens in timescale why not problems and events doing it on my own without official support is risk not worth to take

[gofree]

Thanks for explanation , putting in that way it makes sense. Kinda the suggestion was not to enable automatic deletion of the problem ( as houseekeper is doing with history and trends ), the idea was to have the possibility to easily drop problems in case of event storm ( in theory ) where you dont care that much about the problems in some time frame ( zabbix is not usable at that point anyway ). Its a hypotetical ( but it hapenned ) situation and sometimes cant be prevented in big environment with dynamically generated triggers.

[09Richard]

Thanks for explanation , putting in that way it makes sense. Kinda the suggestion was not to enable automatic deletion of the problem ( as houseekeper is doing with history and trends ), the idea was to have the possibility to easily drop problems in case of event storm ( in theory ) where you dont care telldunkin that much about the problems in some time frame ( zabbix is not usable at that point anyway ). Its a hypotetical ( but it hapenned ) situation and sometimes cant be prevented in big environment with dynamically generated triggers.

I think partitioning of event and problem tables is different thing that partitioning tables with data. The data we keep for some time, but event and problems must be kept as long as problem persist. So one event can block partition from deletion. Before droping partition you must be sure that events and problems are closed.

[lakeland]

I don't use TimescaleDB，but
I think whether to partition those tables depends on the number of data in them.
While zabbix work on thousands of hosts, the quantities of events or problems may archive hundreds of millions. If we don't partition them, house keeper may become slower and slower , and maybe the problem pages will takes you 1 minute to open (these problem really happend before).
But I agree with you on that: this situation is special, you should not partition your table because other people do so.

[lakeland]

By the way, we partition table to different interval.
We follow a principle that no more than 5 million rows in one partition.

[gofree]

I guess youre missing the point guys...what Im saying is that if you have an event storm ( thousands of problems ) you're done - zabbix will simply not work. This can happen in rare case scenario with dynamic trigger especially in large environments. This is not about speed , its more about ressurect zabbix in case of event storm.

[astrinrow]

what problems can occur after changing the DB schema?

[Miama_34]

what problems can occur after changing the DB schema?

Changing the database schema after an initial implementation is common during the development stages and may be necessary after application deployment. The technique for altering the database schema and programming considerations vary depending on the APIs used to access the database. It is important to understand the difference between static and dynamic schemas as they are totally different entities and can not be used together, though they can be compatible.


During early development, schema changes may be frequent and the process of regenerating the database APIs and application consists of repeating the normal steps of application development for the API of choice. But once an application is in use in a production environment, changes to the database schema may require more planning and careful deployment. To address this need, eXtremeDB incorporates Binary Schema Evolution (BSE) capability.
Mybkexperience