Poll Distribution for 1M Items

[mnarubin]

Good day,
We have ~12000 network devices that Zabbix poll with SNMP to get configuration information for cards, ports, entity and so on (Serial Number, Type, Name e.t.c). It is a huge number of items, more than 1M.
All the information for the items must be collected twice per day, but not in one moment. We try to configure poll intervals for 12hours, but it is not good idea. When Zabbix and proxies start they generate a high spike of traffic, without any distribution during the day. How can we configure the custom random poll for all items in that case?

[Markku]

What is the Zabbix version you are using?

Markku

[mnarubin]

Hello, Markku
We used Zabbix 6.0.0
Right now we try to use Java Sript for calculate random value for custom inteval. But may be more simple way exist.

[Markku]

Right. Now, what is the actual problem you are trying to solve? What is not currently working or where is a problem?

FWIW, anyway, I have thought Zabbix poller automatically distributes the polling (time-wise) somehow.

But you say that you have verified that there is a short busy time for the pollers and then they idle all the other time. Is that the correct interpretation?

Markku

[mnarubin]

Hello Markku,
It is not a problem with Zabbix, itself. We have a problem with external system that we use for post processing. Metrics collected and Real-time exported with Zabbix. We desired that this flow was distributed on 24 hours in equal. But metric flow have burst spikes so huge that overloading our external Postgress DB.

[Markku]

Ok, so you have some external code that reads the real-time export files and inserts data in your other PostgreSQL server. In that external code you could leverage some kind of queueing that throttles or bulk-optimizes the database inserts. I mean, if you know your target is not very performant, you can't escape it in the long run anyway.

Again, did you really verify that your pollers polled all the items right after start and then stay idle after that?

Markku

[medl]

I'd agree to Markku that your Problem seems to be outside of zabbix, if you need help with that then you should describe what your external processing looks like.
E.g. are you using a Zabbix Server Plugin or are these DB Triggers that kick of some processing.

Anyway, one possibility would be to reduce the number of pollers or deploy a special Proxy for these devices (with a limited amount of pollers).
Request will queue up at first and depending on the latency in between the proxy and your devices zabbix can/will process only "so much".
For example 50ms latency and one Poller will result in 20 Requests per Second. Bulk requests would still return more values...
Long term (based on my observation) zabbix may distribute those requests based on what the proxy can process.

Hope that helps