Best infrastructure design for monitoring 1600+ devices

[guille.rodriguez]

Hi all, a this moment our Zabbix infrastructure has this design




Zabbix Version is 6.2.7 on all nodes (server & proxies)


- Virtual MachineZabbix Server + Frontend + Database on RedHat 8 with 4cores, 12gb RAM (with MySQL Partitioning)


- Virtual Machine Zabbix Proxy Building num.1 + Proxy Database on Debian 11 with 6 cores 12gb RAM, monitors all vlans on this building (firewalls + switches + routers using snmp & snmptraps, windows + linux using zabbix passive). Monitors about 800 devices




- Dell Optiplex 3020 Zabbix Proxy Building num.2 + Proxy Database on Debian 11 with 4 cores 16gb RAM, monitors all vlans on this building (firewalls + switches + routers using snmp & snmptraps, windows + linux using zabbix passive). Monitors about 700 devices




- Dell Optiplex 3020 Zabbix Proxy Building num.3 + Proxy Database on Debian 11 with 4 cores 8gb RAM, monitors all vlans on this building (firewalls + switches + routers using snmp & snmptraps, linux using zabbix passive). Monitors about 100 devices



- Virtual Machine Zabbix Proxy Vehicles num.4 + Proxy Database on Debian 11 with 2 cores 4gb RAM, monitors bus vehicles who had routers 4G and switches that bus has inside and cctv server (windows). Every bus has 5 router/switches and 1 cctv with 6 cameras. Now we are testing with a few vehicles, but there are 50 vehicles, also the polling checks are time relaxed (compared with other proxies)




Proxies are running ok, only the Proxy Building num.1 sometimes is busy, all the others are ok, this proxy now is monitoring some vehicles... so in the future some hosts monitored by this host will be monitored by the num.4 proxy


I want to improve the system health and I want to ask you , how do you think is the best infrastructure design? For server improvement I'm thinking about run separately server , database and frontend, so what are the hardware requirements (cpu, ram, disk) for


VM running zabbix-server service
VM running mysql database for server
VM running zabbix-frontend

Also, I'm planning to change the passive agent windows and linux checks to active agent

[LenR]

We were about 4x your number of hosts (I've retired, don't have access to the specs) with a very similar design. We kept the Zabbix server and mysql server on the same host, that seemed to be what was recommended 5+ years ago when we designed our environment. We had 4 busy SNMP proxies, 1 busy proxy for hosts and a couple of other proxies that were for net access, not load. We ran our web frontend on a different host, but I don't think it mattered much. I think we were 8000 NVS.

Mysql tuning on the Zabbix server can make a huge difference. We used half our 32G ram for mysql innodb buffers (used hugepages) and adjust the redo logs larger. Use some of the tuning advisors, phpmyadmin can offer suggestions. I did notice increased write rates the older the partitions got so I partitioned the largest history tables more frequently than daily. Watch the Zabbix server and proxy stats, some adjust number of processes as needed, don't want them too busy, don't want too many unnecessary processes.

[tim.mooney]

Congratulations on the retirement LenR! I hope you'll continue to contribute here in your spare time, as you find questions you can field. You've shared lots of great info over the years.

[supervos]

Well we have about 350 hosts in 2 location VMware Cloud director and we do it like this


The JAVA gateways are missing in this picture they reside on the proxy's. SRV en PRX have 4 CPU 16gb MEM. Web had 2 and 8GB,

[Mrbibi]

Hey,
Question that has nothing to do with it. What tool did you use to make your diagram? I'd be interested in making my diagram with the same icons!
Thank's.

[lmonasterio]

Hi! Very interesting topic. Do you use containers (with Docker Swarm or Kubernetes) to deploy those infrastructures? Or just use VMs?

[guille.rodriguez]

At this moment most of them are VMs,

1 Redhat (Server + Database + Frontend)
2 Debians (Proxies)

And two Dells Optiplex MFF 3020 (4cores, 16gb ram, 250gb ssd)

[guille.rodriguez]

Just for update, I changed all my Windows to active monitoring and now proxy is more relaxed. No more "unreachable pollers" trigger