Delayed SNMP items (over 30 minutes instead of 1 minute)

[staerkl]

Summarized: I have set up SNMP monitoring for the firewalls and Netapps.
The firewall items are queried every minute and are also visible in the dashboard after a short time.
The items of the Netapps are probably not queried. A query is set up every minute. However, these are often only queried a few times an hour.
snmpget on the console returns the data immediately.

About my environment:
I operate a server with 5 proxies and 250 hosts.
All servers run on Debian 12. I installed Zabbix 7.0.3.

What I have checked or tested:
- Queue is almost always empty
- VPS is at 37
- 10 pollers and 50 snmp pollers are started with 4007 values
- via snmpget in the console I get the values
- “Use combined requests” deactivated for testing
- SNMP items from other hosts of the same proxy are received

In the debug log I see the affected item keys every minute:
sudo tail -f /var/log/zabbix/zabbix_proxy.log | grep CannotTakeover
4317:20240820:185701.692 In substitute_key_macros_impl() data:'fas3220.ha[haCannotTakeoverCause, "Hostname"]'
4317:20240820:185701.692 End of substitute_key_macros_impl():SUCCEED data:'fas3220.ha[haCannotTakeoverCause, "Hostname"]'
The values do not appear in the dashboard.

What else can I check?
Have I forgotten any important details?

I appreciate any tips.
Thank you very much in advance.

[Markku]

Are those items configured to use preprocessing rule that discards unchanged data?

Markku

[muelli]

Interestingly I have the same or a similar problem, mentioned here:

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/490035-some-snmp-item-not-collecting-data-regularly-for-ilo4

In the debug log I can see the collected data every minute (in this example 1.3.6.1.4.1.232.3.2.2.2.1.6.0 is "system.hw.diskarray.cache.battery.status[cpqDaAccelBattery.0]"):

data prepare seems OK:
364918:20240822:081540.086 In substitute_key_macros_impl() data:'1.3.6.1.4.1.232.3.2.2.2.1.6.0'
364918:20240822:081540.086 End of substitute_key_macros_impl():SUCCEED data:'1.3.6.1.4.1.232.3.2.2.2.1.6.0'
364918:20240822:081540.153 In zbx_snmp_translate() OID:'1.3.6.1.4.1.232.3.2.2.2.1.6.0'
364918:20240822:081540.153 End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.3.2.2.2.1.6.0'

Then it collects the value (in this case "2", which is battery status OK):
364924:20240822:081840.128 In get_values_snmp() host:'X.X.X.X' addr:'X.X.X.X' num:1
364924:20240822:081840.128 In zbx_snmp_open_session()
364924:20240822:081840.128 In zbx_is_ip4() ip:'X.X.X.X'
364924:20240822:081840.128 End of zbx_is_ip4():SUCCEED
364882:20240822:081840.130 End of zbx_ipc_service_recv():2

However the data is then not displayed in the frontend.
Either I am misinterpreting the debug log or this is some sort of a bug.....

[Markku]

Are those items configured to use preprocessing rule that discards unchanged data?

Markku

[muelli]

hmmm indeed, preprocessing says "Discard unchanged with heartbeat" and "6h" as value.
I was not aware of this setting at all, thanks for bringing it up. So basically I will habe to remove this preprocessing to get recent data to be stored in the DB and used for graphs etc....

[staerkl]

Markku This tip was very helpful.Thank you very much.
After removing the preprocessing, the data is displayed correctly.

Unfortunately, the recorded values do not match the 6h:

Is this a bug?

[Markku]

I haven't seen that kind of behavior. I'd recommend using the Values list to see the distinct data values, not the graphical view, to be sure the values are really identical.

Also, I'd guess restarting the relevant server/proxy components may cause data to be saved more frequently than the heartbeat interval.

Markku

[staerkl]

I haven't restarted the servers and proxies today.
Unfortunately I can't find any patterns.

[Markku]

When having an SNMP item that does not seem to be updating according to the configured interval, the most common reason is along the lines that the Zabbix server/proxy poller is not able to do all the tasks it is supposed to do.

Two factors affect that:

1. The number of items the poller needs to take care of
2. The number of configured pollers in the Zabbix server/proxy configuration.

Now, with Zabbix 7.0, the SNMP item configurations can also use walk[] or get[] syntax, instead of the legacy textual or numeric OID. The difference is:

- The legacy OID configurations will use the synchronous pollers (configured with StartPollers directive)
- The new-style walk[] and get[] items will use the new asynchronous SNMP pollers (configured with StartSNMPPollers directive)

The major difference between "sync" and "async" is that the "sync" pollers (or items) work serially, meaning that a slow or unresponsive SNMP agent can cause problems in all SNMP items. Async items are executed in a multithreading poller so that the execution is much more effective and slow SNMP responders don't affect the overall process that much.

If your SNMP items are using the legacy OID syntax, check your server/proxy utilization graphs and increase the number of pollers if needed. Or, rewrite the items/templates to use the more efficient async pollers.

Markku