Update interval is fluctuating a lot

[aib]

Hello!
Our Zabbix infrastructure have ~10 proxies to monitor >4,000 hosts / >1 million Items

We are using "SNMP agent" to pull information from CISCO devices every minute.
Quite often, Zabbix decreases or increases polling interval dramatically (lowest update interval that I noticed was 6 seconds, highest was 1 minute 59 seconds).
Unfortunately, some CISCO devices have an SNMP cache that updates only once per 30 seconds.
Therefore, if Zabbix request information earlier than 30 seconds from previous try, device returning the same number and after pre-processing (multiply by 8 and save change-per-second) the wrong speed saving in database (zero bps)

I would like to know how to decrease the limits for update interval changes, or even disable that feature and always use pre-configured Update interval (1 minute) without increasing/decreasing it.

[aib]

There are a couple of screenshots that support my initial post.
1. Update interval configured for interface HundredGigE0/0/0/30

2. Small piece of history showing two items together - "Change-per-second" and "RAW" data from SNMP pull.

[Markku]

How is your StartPollers configured?

How does the poller utilization look like?

My understanding is that if the pollers are too busy, they need to flex the polling intervals for these passive items.

Markku

[aib]

Utilization graph for last two days for proxy is attached.
Average load = 82%,
Min load = 4%
Max load = 100%

[Markku]

I would add some more pollers. How is your StartPollers configured at the moment?

Markku

[aib]

StartPollers=400
StartPollersUnreachable=20

[LenR]

What version of Zabbix? What database? Are you partitioned? Get the graphs of zabbix processes too. What is the zabbix NVPS?

Are those load graphs from the zabbix server? How is the load on the proxies?

[aib]

Zabbix version is 5.0.17
We are using Postgres DB
Yes, we are using partitions
Zabbix NVPS ~13,000
Those graphs from Zabbix proxy configured for the host having polling interval issue

[Markku]

So, my understanding is that the pollers (as configured with StartPollers) are responsible for doing the SNMP polling and assigning the timestamps to the results. The poller utilization graph from the proxy showed that that the pollers are very heavily utilized. Thus, the number of pollers should be increased, provided that the CPU capacity is sufficient, too.

Anyway, in this large system all the Zabbix system metrics should be fully monitored and understood to be able to keep the system running properly.

Markku

[LenR]

What are the server specs for the problem proxy? Can you add more processors? See what processes are using the CPU on that proxy.