Best practice Zabix architecture

[RustyP]

Hi Guys

I am in the process of building a POC for Zabbix to replace our current solution and I would like your experience to advise best practice for our architecture.

We have up to 5k devises which have a total for 20k individual monitoring checks. These devices are spread across 12 different datacentres globally, all these figures are likely to increase as we use zabbix to monitor more of our infrastructure, so scalability is important.

What I would like to know is:

• Is it best to separate the Zabbix server and DB? I understand that the DB IO can be considered a bottle neck at times?
• Would you recommend having a zabbix proxy at each datacentre? considering the number of datacentres could increase would this become unmanageable? What would the advantages/disadvantages be to having a proxy at each datacentre?

Thanks for your time

[1berto]

The performance always depend on how many items are you monitoring. But for the 'average' scenario it's worth to separate the DB and use a connection pool (i use pgbouncer).
Also it's probable that you need to use proxies (again it depends on how many items are you monitoring), IN MY SCENARIO, the proxies databases are local (to each proxy).
If your network is spreaded globally could be a good idea to have a proxy in each continet/country/region, etc.

[LenR]

A popular setup a few years back was zabbix-server & db on the same server, data collection by proxies and server(s) for web and api.

[ITOMDave]

Hi Rusty, I'm doing a similar exercise myself and after reading articles from a lot of very clever people I've made a few relatively straightforward design decisions to give me hopefully a highly scalable architecture :

1. Separate the Zabbix Server and DB.
2. Partition the DB
3. Have multiple web front ends in place and put a load balancer in front of them
4. Setup an active-passive zabbix cluster
5. Setup database HA, clustered DB.
6. Don't use SQLite on zabbix proxies

I'm also playing around with other architectural principles but I still need to work on the potential impact of them on overall scalability and availability. E.G.

A. Should I encrypt all Agent<->Proxy<->Server comms ?
B. Should I encrypt the Database when at rest ?
C. Should I enforce encrypted comms between the Zabbix Server and the database ?

Finally, I've not yet found a definitive "best practice" architecture that would give me the finer details such as database tuning, proxy tuning, data retention, memory and cache sizes etc etc.

Happy to learn along with you on this.

[elemarmb]

Why don't use SqLite?

[ITOMDave]

Hi,
My reasons for not using SQLite were mainly due to wanting a consistent DB platform and best flexibility and scalability. As far as I can see, there is nothing wrong with SQLite per se, but if I'm using MySQL as my Zabbix database, then to me it makes sense to keep things consistent. I also wanted to avoid future scalability issues when managing nodes via a proxy. So if I had a site with 50 devices being handled via proxy, sure - I could use SQLite I'm sure. However with the dynamic nature of IT infrastructures and the ever growing use of Docker as a production platform, then I wouldn't want to be struggling to monitor 250 nodes with hundreds of items using SQLite. It's just a personal preference - As always, YMMV.

Regards,
Dave

[kloczek]

Hi,
My reasons for not using SQLite were mainly due to wanting a consistent DB platform and best flexibility and scalability. As far as I can see, there is nothing wrong with SQLite per se, but if I'm using MySQL as my Zabbix database, then to me it makes sense to keep things consistent. I also wanted to avoid future scalability issues when managing nodes via a proxy.

Using sqlite DB backend meand rewriting whole db file on every transaction.
That means scaleing almost linearly number of IOs and database insertc/updates latency with size of the DB.
MySQL with even 128MB innodb pool takes less memory and IO bandwidth in case such smallest DBs.
MySQL is not quite well modularised so it is possible to drop loading many modules loaded by default.
And using sqlite is orthogonal with scaleability.