Very busy process poller

[Amiram]

Hey all,

I'm running a Zabbix-Server version 6.2 with a MySQL (8.0.28) database.

• The MySQL has been tune to partition the tables ~50Gb per partition.
• The data disk is running an XFS file system

  https://serverfault.com/questions/29193/what-is-the-best-linux-filesystem-for-mysql-innodb

• Additional MySQL performance tuning has been added to improve its performance.

The single server (hardware - 36Cores, 512Mb RAM and 20T HD`) is monitoring around 4000 server and the magic number are:

• Number of items ~1.5M
• Number of processes values per second ~10K
• Nvps ~10K

I'm currently running 811 Pollers (normal poller for passive checks) and it busy more then 75% of the time.

I expect the environment to grow in all directions (more server, additional items - per server, PDUs and maybe switches) in the near future.


Any ideas how to handle it?

1. It will handle it?
2. Additional pollers (more then 1000 - maximum)
3. Stronger hardware?
4. Use Proxies?

P.S
I'm using some external scripts that take a bit of time to finish (~25seconds)

Best regards,
Amiram

[Hamardaban]

First of all, transfer some of the load to the proxy (sqlite)
https://www.zabbix.com/documentation...concepts/proxy

Secondly, a fast file system for the database.

Thirdly, optimization of the database server (I prefer postgresql with timescaledb)

[Amiram]

I know that a Proxy is an option but it will require me to do some architecture changes in addition to another component to monitor and maintain.

FileSystem information has been added to the original post in addition to the Database information

[Markku]

I'm using some external scripts that take a bit of time to finish (~25seconds)

I believe this contributes a lot to the poller busyness, depending on the amount of those items: the poller sends the request and keeps waiting for the results to arrive, cannot do anything else meanwhile.

But anyway, yes, implementing proxies is one of the most important things when scaling Zabbix, especially with passive items.

Markku

[Amiram]

depending on the amount of those items

There are two items per servers that are running this external script, but they are running every once 15 minutes.

There are additional solution besides proxies?

[Markku]

There are two items per servers that are running this external script, but they are running every once 15 minutes.

15 minutes = 900 seconds, and you have ~4000 hosts = ~8000 items. It means that every second (in average) there are ~10 of those items being newly queried, and each of them holds a poller for ~25 seconds. And during that 25 seconds there are ~250 other pollers entering the same hold. So no wonder if pollers are busy. (Someone let me know if I calculated something incorrectly.)

[Amiram]

Additional information:

* All the monitored servers are located at the same DC as the `Zabbix Server`.


So, how to distribute servers to a proxy? What is the correct way to do it?

[Hamardaban]

The proxy does not have to be on another "network".
You will have to manually distribute which servers will access which proxies - by editing agent configs and configuring host via the web interface.

[Markku]

So, how to distribute servers to a proxy? What is the correct way to do it?

There is no single correct way, it all depends.

If your hardware server (I take it that it is hardware dedicated to the Zabbix server) has plenty of CPU resources available, you can even install the proxy on the same server. In that case you need to reconfigure the Zabbix proxy to use a different ListenPort than the default 10051 (because that's already used by the Zabbix server), you can use for example 10151. Then you can change your hosts to be monitored by the proxy. If your hosts have also active items or trapper items, you need to change the agents to use the new port 10151.

The usual option is to install the proxy/proxies on new server(s) (or VMs). Then you don't need to change the ListenPort on the proxy, but you need to change the agent Server/ServerActive configurations to accept/use the proxy address instead (or in addition to) the Zabbix server address.

Markku

[Amiram]

This is what I feared from.

Thank you everyone.