Zabbix PSQL Database Scaling Issues

[doctorbal82]

Hello,

We are running into continuous issues of the Zabbix database getting filled up too quickly. Within 1 month the Zabbix sucked up ~50GB. We expanded the monitoring system to other users and see it being slightly abused. When confronted they did mention that they need to retain most of the data for the time period.

I have already increased the Database twice to a larger disk but don't want to keep performing this. I would like a fully self-contained, scalable environment regarding space of the database. I would like to build the environment to be able to handle the abuse from other users.

Thus my questions are as follows:

1. Are there any recommend postgresql scaling articles for handling hundreds and thousands of metrics and potentially 1000s of hosts?

2. How do I check which database entries (psql commands) take up the most space from which host? Is there any easy way to check this via an API command to the frontend instead of sifting through 100s of hosts?

3. Is a HA database scenario recommended to help with performance?

4. Is there a blueprint for the recommended database size to handle a certain amount of metrics, hosts, items?

5. For large scale environments what shall be modified on the /etc/zabbix/zabbix_server.conf file to address Housekeeping? Shall it be maxed out?

Any recommended information is highly appreciated.

[syndeysider]

Hi doctorbal82

I think you should probably start here :

https://www.zabbix.com/documentation/3.2/manual/installation/requirements

Scroll down to "Database size" and spend some time understanding the concepts behind History vs Trend data and Housekeeping.

Once you've got a better understanding on how Zabbix stores data, in which tables and for how long, you will be in a better position to understand how table partitioning comes into the picture.

https://www.zabbix.org/wiki/Docs/howto/zabbix2_postgresql_autopartitioning

With these two key pieces of information you should be able to assess how large your database is going to grow according to your requirements around historical/trend retention settings and also how large a disk system to provide to accommodate this database. With partitioning enable you can further automate the "cleaning" of old data, adding some form of normalisation to the running instance of zabbix.

[doctorbal82]

Thanks syndeysider

Hello syndeysider,

Thank you for the references; very helpful! I will look through those docs and see what I can find.

If anyone else has any perspective on database optimization and architectural layout it would be appreciated. The documentation provided online is great, but it would be interesting to see other deployment perspective of scaling to hundreds of thousands of metrics.