Problems with multiple SNMP host items being unsupported

[mkmb]

Hello all,

we are currently having huge issues with most of our SNMP hosts.
For example, we have a couple of HPE iLos and network switches we want to monitor via SNMP, but they are giving us a headache.

We are currently using Zabbix Server 5.4.3 on a Ubuntu Server 20.04.2 LTS with all current package updates.
Some SNMP hosts switch back an forth from "No data collection", some only show many items as "Unsupported" - with the error message "SNMP error: noCreation (That table does not support row creation or that object can not ever be created)"

Weird thing is, although the items are all unsupported, they still collect data - only with gaps, but at least that's something.
An example:


This shows the PSU temperature being collected randomly over time. As you can see, data collection is spotty.
I couldn't confirm that 100%, but to me it looks like the Items get unsupported and stop collecting data until it becomes supported again.
This also reflects in the Zabbix server logs, which shows a block of items becoming unsupported, then becoming supported again. The error message in those cases is always "SNMP error: noCreation (That table does not support row creation or that object can not ever be created)"

Also, this happens on hosts monitored with and without a proxy.

Since this happens on many hosts and not just iLos, but also Switches, I suspect, we possibly have an error in our configuration or need to adjust something in the SNMP data collection.

Does anyone have an idea on how to fix this and get monitoring working again for those hosts?

[Milord]

Same problem. Have you solved it?

[cyber]

Verify all credentials. Try to switch off bulk queries. Add additional pollers and unreachable pollers...

[Tpe]

Did you find fix for this? I'm running into same issue.
It is working fine with SNMP from pfsense but from ILO4 (latest firmware 2.80 Jan 25 2022), have updating issues even receives correct values from the server.
Discovers all the items fine, but will update the values only once. Some few values still get updated. After restarting zabbix-server, all values get updated one time.

What I have done/tried:
-Installed Zabbix 6.0.3 LTS Apr 04, 2022 appliance
-Updated Centos 8 with dnf update

-Tested with SNMPv2 and SNMPv3, tried to turn off bulk requests. Same issue.
-Tested with 3 different templates HPE ProLiant DL360 SNMP, HPE ProLiant DL380 SNMP and HP iLO SNMP. Same issue with all. Even they all manage to discover items and the pull values once.
-tcpdump shows that values are respond correct and fast from the iLO, also zabbix debug log shows that respond is ok without error.
-Tried to increase pollers/unreachable pollers/timeout value. Does not effect. At the same time, data from pfsense works just fine.
-Disabled all other items and left only one item from iLO (which did not got updated). Item has 1minute interval. Still wont update even it's only active item in zabbix. If restarting the zabbix, again, updates one time.
-Tested using zabbix "Test item" feature with different values. Still wont show up at "Latest data".

ZABBIX DEBUG LOG WHILE VALUES DO NOT UPDATE INTO "LATEST DATA":
End of zbx_snmp_open_session()
In zbx_snmp_process_standard()
In zbx_snmp_translate() OID:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
In zbx_snmp_get_values() num:1 level:0
zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
In zbx_snmp_set_result() type:2
End of zbx_snmp_set_result():SUCCEED
End of zbx_snmp_get_values():SUCCEED
End of zbx_snmp_process_standard():SUCCEED
In zbx_snmp_close_session()
End of zbx_snmp_close_session()
End of get_values_snmp()

TCPDUMP WHILE NOT UPDATING:
GetRequest(35) E:232.6.2.6.8.1.4.0.32
GetResponse(36) E:232.6.2.6.8.1.4.0.32=33

After restarting zabbix-server, values will get updated only once:

ZABBIX DEBUG LOG WHILE VALUES DO UPDATE CORRECTLY:
End of zbx_snmp_open_session()
In zbx_snmp_process_standard()
In zbx_snmp_translate() OID:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.232.6.2.6.8.1.4.0.32'
In zbx_snmp_get_values() num:1 level:0
zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
In zbx_snmp_set_result() type:2
End of zbx_snmp_set_result():SUCCEED
End of zbx_snmp_get_values():SUCCEED
End of zbx_snmp_process_standard():SUCCEED
In zbx_snmp_close_session()
End of zbx_snmp_close_session()
End of get_values_snmp()

TCPDUMP WHILE UPDATING WORKS:
GetRequest(35) E:232.6.2.6.8.1.4.0.32
GetResponse(36) E:232.6.2.6.8.1.4.0.32=33

Updating seems to be totally random. If I enable all items, most of the items do not update.
e.g.
232.6.2.6.8.1.4.0.32 System-0.32: Temperature - Do not update
232.6.2.6.8.1.4.0.38 System-0.38: Temperature - Does update every time in 1min interval.
When I disable all other items and leave only these two, now 232.6.2.6.8.1.4.0.38 stops updating as well.

[supportmi]

hi, same issue. have you solved this?

From what i see problem fully persists on iLO4, while on iLO5 all items are discovered without error but graphs have gaps...

[Markku]

This? https://www.zabbix.com/forum/zabbix-...342#post444342

Markku

[supportmi]

i think problem here is different, items randomly become unsupported and then supported again...

[Tpe]

i think problem here is different, items randomly become unsupported and then supported again...

I did see this behavior while testing when I had bulk requests enabled. So make sure you disable em.
If they are already disabled, make sure you have latest firmware on ILO. Some earlier version (can't recall which one), had this issue when ILO side started to throttle communications.

[supportmi]

Currently I am testing Zabbix 6.0.4 while my previous Zabbix 5.4.1 is up and running on different machine (the idea is to switch to the latest versions later). So on my old production Zabbix "bulk requests" are enabled and all works fine. iLOs are running on the latest version (2.80 Jan 25 2022). What I am currently testing is, I have actually removed preprocessing as suggested Markku if this won't help I will definitely try bulk requests. Thanks for the advise! Will update you later.

[Tpe]

Currently I am testing Zabbix 6.0.4 while my previous Zabbix 5.4.1 is up and running on different machine (the idea is to switch to the latest versions later). So on my old production Zabbix "bulk requests" are enabled and all works fine. iLOs are running on the latest version (2.80 Jan 25 2022). What I am currently testing is, I have actually removed preprocessing as suggested Markku if this won't help I will definitely try bulk requests. Thanks for the advise! Will let u know the results in few days.

Yes, Markku did help me on other thread and advice to turn off the preprocessing. That did indeed fixed the issue where request value didn't save to the db if it was same as previous one. There was no errors on requests.
This "SNMP error: noCreation" issue I've seen only when bulk requests were enabled, so it is different.
I think on some point I tested with SNMPv3 & bulk requests enabled and I didn't notice that problem, so could be only SNMPv2 related.

[supportmi]

Yes, Markku did help me on other thread and advice to turn off the preprocessing. That did indeed fixed the issue where request value didn't save to the db if it was same as previous one. There was no errors on requests.
This "SNMP error: noCreation" issue I've seen only when bulk requests were enabled, so it is different.
I think on some point I tested with SNMPv3 with bulk requests and I didn't notice that problem, so could be only SNMPv2 related.

i saw your thread, did you remove preprocessing from all discovery items in the template or? seems like have to try SNMPv3 as well...

[Tpe]

i saw your thread, did you remove preprocessing from all discovery items in the template or? seems like have to try SNMPv3 as well...

I did remove preprocessing under prototype's from those items which I want to monitor continuously, like temperature sensors.
Had to do it on the template as it wont let you remove em from the discovered items. So I removed all discovered items, removed preprocessing from the template and then re-run discovery.

[supportmi]

I did remove preprocessing under prototype's from those items which I want to monitor continuously, like temperature sensors.
Had to do it on the template as it wont let you remove em from the discovered items. So I removed all discovered items, removed preprocessing from the template and then re-run discovery.

I have removed preprocessing under prototype's on the template from the discovered items and also have disabled Bulk Requests on every host using snmpv2. I confirm that it works as intended now. Thanks!

[czechsys]

Tests with 2 servers concurrently (main 5.4 and testing 6.2):
Zabbix 5.4 - ILO4 2.79 & 2.80 works with bulk requests without hiccup
Zabbix 6.2rc - ILO4 2.79 & 2.80 fails with: This "SNMP error: noCreation" issue when bulk. Without bulk works, but its costly - tens of connections vs 1-2 per minute per server.
Both snmpv2.
Tests with snmpv3:
Bulks works, but snmp starts timeout after some short time. And...when using macros for security name/authpass/privpass, items starts to fail with "Cannot connect to "host.example.com:161": Unknown user name.", but host snmp state is ok. And host snmp timeout after some short time (probably ilo restrictions or so.).