SNMPv3 intermittently failing

[choppy]

I'm still new to SNMPv3.

Zabbix sends a blank get-request with a list of fields but no values (as if it wants the other end to populate them). The router I'm configuring then replies with an incremented usmStatsUnknownEngineIds.

Zabbix then sends the get-request for the OID I wanted, and the device replies.

The item is alternating being supported and then erroring with

Authentication failure (incorrect password, community or key) (Sub-id not found: (top) -> ospfNbmaNbrStatus)

I'm not even querying for ospfNbmaNbrStatus, nor am I seeing that OID in the reply packet from the device.

I'm using AES/SHA.

I think this is something to do with Zabbix sending the blank get-request, and the router responding with the UnknownEngineIds value.

The OID I'm actually querying 1.3.6.1.6.3.10.2.1.1.0 works fine if queried via snmpget and looks fine in the pcap I take.

I'm running Zabbix 4.0. Direction welcome at this point as I'm utterly stuck. All the advice online is about duplicate devices, but I don't have any other SNMPv3 devices running.

[cyber]

No, it is ok to send blank request for starting a conversation. It will trigger sending correct engineID etc.

https://www.ietf.org/rfc/rfc3414.txt?number=3414

3.2. Processing an Incoming SNMP Message

[choppy]

No, it is ok to send blank request for starting a conversation. It will trigger sending correct engineID etc.

https://www.ietf.org/rfc/rfc3414.txt?number=3414

3.2. Processing an Incoming SNMP Message

Thanks, that helps me eliminate what's going on there.

Unfortunately I'm no closer to understanding what Zabbix is complaining about.

[cyber]

Anything related to snmp is probably related to magic and sorcery....

EDIT: Jokes aside... I have had a feeling sometimes, that devices are not able to authorize queries quickly enough and then things get enabled/disabled at their own pace...
Only thing here is, that it does this only with Zabbix..:P other tools seem to work fine... Never seem to have any such issues with NNMi...

[choppy]

I have had a feeling sometimes, that devices are not able to authorize queries quickly enough and then things get enabled/disabled at their own pace...

Responses are coming back in within 0.001 seconds, so I don't think it's that.

[cyber]

Yes they come back as "not authorized" or something..
I have not paid too much attention to this lately, but maybe a year ago or so I had to deal with this for a while... Same box, same queries... sometimes you get values, sometimes you get "not authorized", I never found the reason, but as it was not so hurting issue, it remained somewhere on a back burner..

[choppy]

Okay, thanks. If I don't get anywhere by the end of the day I'm going to raise it as a Zabbix bug

[ISiroshtan]

Well I do use SNMPv3 Agent items on 4.4 and had no issues with it. Could you show the item configuration? And maybe packets capture where it has failed and OK packets?

And lastly: there is no chance you monitor multiple devices and some of them have duplicate EngineID? (pretty common issue with SNMPv3)

And to make sure, you don't have any other item/discovery that might try to grab 'ospfNbmaNbrStatus' OID do you?(on this host or on another host that is also monitored via SNMPv3)

[choppy]

Well I do use SNMPv3 Agent items on 4.4 and had no issues with it. Could you show the item configuration? And maybe packets capture where it has failed and OK packets?

I have a capture, but it just shows the same thing over and over:

Occasionally I see a packet going from Zabbix to the client that Wireshark is malformed, but because I can't see anything useful in them I can't tell you what they're doing.

And lastly: there is no chance you monitor multiple devices and some of them have duplicate EngineID? (pretty common issue with SNMPv3)

I tried to set up a few devices (all same manufacturer) and they all showed the same behaviour. When the EngineID actually made it to Zabbix then they all looked difference.

I disabled all the other hosts on Friday, but I'm still seeing this.

And to make sure, you don't have any other item/discovery that might try to grab 'ospfNbmaNbrStatus' OID do you?(on this host or on another host that is also monitored via SNMPv3)

Nope. This is the only host with SNMPv3 items that is enabled.

[ISiroshtan]

Looking at one of my SNMPv3 Agent captures it goes as follows:

Message from Zabbix:

snmp.msgAuthoritativeEngineID: <MISSING>
snmp.msgAuthoritativeEngineBoots: 0
snmp.msgAuthoritativeEngineTime: 0
Encryption fields : empty
msgData: empty

Reply from monitored device:

snmp.msgAuthoritativeEngineID: <EngineID>
snmp.msgAuthoritativeEngineBoots: X
snmp.msgAuthoritativeEngineTime: Y
Encryption fields : empty
msgData: unencrypted varbind usmStatsUnknownEngineIDs

2nd Message from Zabbix:

snmp.msgAuthoritativeEngineID: <EngineID> (matches what was received previously)
snmp.msgAuthoritativeEngineBoots: X (matches what was received previously)
snmp.msgAuthoritativeEngineTime: Y (matches what was received previously)
Encryption fields : present
msgData: EncryptedPDU

2nd Reply from monitored device:

snmp.msgAuthoritativeEngineID: <EngineID> (matches what was received previously)
snmp.msgAuthoritativeEngineBoots: X (matches what was sent previously)
snmp.msgAuthoritativeEngineTime: Y
Encryption fields : present
msgData: encrypteedPDU

Based on you saying that item changes from supported to not supported I'd expect that you should see some SNMP flows going like i described above, but some go in some different pattern. I'd be curious to see those that go differently.

P.S. you should be able to decrypt PDU of SNMP messages to see what is going in there if desired by supplying Wireshark with needed auth data in Edit -> Preference -> Protocols -> SNMP -> Edit User Table

[choppy]

Thanks. That helps explain a few things.

What I'm seeing is

msgAuthoritativeEngineID: <MISSING>
msgAuthoritativeEngineBoots: 0
msgAuthoritativeEngineTime: 0
msgUserName:
msgAuthenticationParameters: <MISSING>
msgPrivacyParameters: <MISSING>
msgData: plaintext (0)
plaintext

followed by

msgAuthoritativeEngineID: 8000a12f0452db1e6a34784df1acc61ce255ab0e42
msgAuthoritativeEngineBoots: 2
msgAuthoritativeEngineTime: 342532
msgUserName:
msgAuthenticationParameters: <MISSING>
msgPrivacyParameters: <MISSING>
msgData: plaintext (0)
plaintext

then Zabbix transmits a packet that Wireshark can't make sense of. I think it's an attempt at authentication. I can see the username and some authentication/privacy parameters.

The error Wireshark throws when parsing that packet is

Expert Info (Warning/Malformed): BER Error: Sequence expected but class:UNIVERSAL(0) Constructed tag:335278 was unexpected

The host responds with a usmStatsWrongDigests packet

msgAuthoritativeEngineID: 8000a12f0452db1e6a34784df1acc61ce255ab0e42
msgAuthoritativeEngineBoots: 2
msgAuthoritativeEngineTime: 342532

We then get a couple of rounds of useful packets - a blank get-request, a report response with the msg info in it, and then the actual OID being queried.

......

What's weird is that Zabbix then sends two authentication attempts, but with different msgAuthoritativeEngineTimes (significantly far apart). The remote host seems to ignore these.

I wonder if some rogue process within Zabbix has got stuck and keeps trying to authenticate on its own.

[ISiroshtan]

The issue seems similar to this. It references this old article and this old bug. I did not dig too deep into those, but skimming it diagonally seems to suggest 3 options to try:

1. Change EngineID on monitored device
2. Updating NET-SNMP to 5.4.1 or newer
3. Finding devices with duplicate EngineID.

As you said 3 is not applicable to your case, you can try/check if 1 and 2 can be used in your case.

[choppy]

The issue seems similar to this. It references this old article and this old bug. I did not dig too deep into those, but skimming it diagonally seems to suggest 3 options to try:

1. Change EngineID on monitored device
2. Updating NET-SNMP to 5.4.1 or newer
3. Finding devices with duplicate EngineID.

As you said 3 is not applicable to your case, you can try/check if 1 and 2 can be used in your case.

1 is out of my control, as it's hard-coded on the device.
3 is, as you say, not a factor.

I'll have a read of the bug post, and I'll look in to 2.

[elementalwindx]

The issue seems similar to this. It references this old article and this old bug. I did not dig too deep into those, but skimming it diagonally seems to suggest 3 options to try:

1. Change EngineID on monitored device
2. Updating NET-SNMP to 5.4.1 or newer
3. Finding devices with duplicate EngineID.

As you said 3 is not applicable to your case, you can try/check if 1 and 2 can be used in your case.

Well here we are in 2/18/24 and I wanted to say I do believe this bug STILL exists. Trying to connect SNMPv3 Zabbix v7.0beta1 to a v20 Sophos XGS. If I change any of the encryption cyphers in Sophos and Zabbix, but use the same security name, it will not make a good connection -_- . You have to choose a new security name in zabbix and make it in Sophos and then set your encryption cyphers and don't touch them again. This wasted 2+ hours of my life. Thanks.

[cyber]

reloading of snmp cache in zabbix after change in any of credentials is a good practice...