Zabbix Docker -SNMPTrapd - Debugging USM SNMPV3 - additional debug option?

[Wolfsbane2k]

Hi.

I'm currently struggling with getting an SNMPv3 Trap to be handled at all in Zabbix running inside a containerised instance. It works perfectly well with an SNMPv2 instance.
I'm therefore pointing my finger at the USM, but can't find a way to enable debugging within the container itself to assist with this - it could be something as simple as a malformed engine id (caps or lower case...) or something else.

Has anyone managed to do this? I'm tempted to fork the official zabbix snmptraps container build file to add the -Dusm (or even -Dall) option to the snmptrapd command line, but that currently looks like the only way to do it.

Any guidance gratefully received!

Ta

[Wolfsbane2k]

Ah.. simplier just to change the existing docker compose file to add the -Dusm option at the end of the command line.

No to work out how to "just" restart that container from the compose file, without the rest.

(edited to make clear that no spaces should exist between the -D and usm (or other debug options)

[Wolfsbane2k]

Ok, finally getting somewhere with debugging the issue.. but not solving it

I have:
1) on an entirely seperate machine, used the snmptrapd setup and test guidance from https://net-snmp.sourceforge.io/wiki..._SNMPv3_notifi cations confirmed that I can successfully send SNMPv3 Traps to a snmptrapd over the network, and have it decoded:

In a /tmp/snmptrapd.conf file put:
createUser -e 0x8000000001020304 traptest SHA mypassword AES
authuser log traptest


Then start snmptrapd pointing to that file (runs in the foreground, uses only that config file and logs to stderr):
snmptrapd -f -C -c /tmp/snmptrapd.conf -Le


Then run snmptrap (in another window) to send a linkup trap:
snmptrap -v 3 -n "" -a SHA -A mypassword -x AES -X mypassword -l authPriv -u traptest -e 0x8000000001020304 localhost 0 linkUp.0

You should see this in the output of the window running snmptrapd:
2007-10-10 10:19:11 localhost [UDP: [127.0.0.1]:46380]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (0) 0:00:00.00 SNMPv2-MIB::snmpTrapOID.0 = OID: IF-MIB::linkUp.0

(although i didn't have the mibs loaded, so i just used 1.3.6.1.0 instead of linkUp.0

This proved to me that an instance using net-snmp 5.9.1 was able to decode the trap.

I then bumped this to SHA-256 and AES-256, and got the same result - Positive.

I then added a "proper" engine, username and credentials, and that used the snmptrap to create the SNMPTrap - Positive: Yay!
I then used the real equipment to generate a real trap, and that was also properly decoded. Smashing.

copying the snmptrapd.config for "traptest" and real usernames from the test system to the operational system and restarting the container failed, with no messages reported in the container log.files.
Sadly, the "traptest" also failed, with no messages.

Using wireshark to capture the message, and the same credentials as the live system to decode the message being sent confirmed that it was an valid message on the bus.

Modifying the "compose-zabbix-components.yaml" file to add additonal debug flags in the snmptrap's docker command "-Dusm,snmpUsm,usmUser,recv", assuming we were failing the USM aspects, lead to the logs reporting

"usm:USM Processsing Begun..
usm: Unknown User: (traptest)

Clearly, as expected, failing USM.

To test if the snmptrapd.conf is being loaded at all, with differences between "defined" users and "undefined" users, i've just spammed a compltely trash engine ID and username to the docker instance, which has exactly the same result

inspections have shown that my test system has net-snmp-5.9.1, the current alpine docker instances have net-snmp-5.9.4 (according to snmptrapd --version from within the docker)

Enough for today, will keep all readers (cough, all 0 of you) informed...
(plan tomorrow - check the snmptrapd.conf file is actually being loaded by the snmptrapd docker instance, and what, if any changes exist btween net-snmp versions)

[Wolfsbane2k]

Ahhhh,.... Do'h: persistent snmptrapd.conf file isn't being loaded.

container boot logs with the additional debug options show a

read_config_store open failure on /var/lib/zabbix/snmptrapd_config/snmptrapd.conf

Which will, of course, result in all the above behaviours:

Now to work out why.

Nooo... that's for tomorrow.

[Wolfsbane2k]

ARGH

Went and changed permissions so that the snmptrapd container could write to it, and it completely blanked the file - meaning it's not persistent at all.

Tried a few different routes to try and make that persistent, and every time the container is stopped, it blanks it out.

[Wolfsbane2k]

Formal issue raised on the Zabbix Docker github.

https://github.com/zabbix/zabbix-docker/issues/1795

not sure if this is an upstream (net-snmp) problem or zabbix docker build problem.

[Wolfsbane2k]

Solved. issue with the snmptrapd command due to use of "-C", in the default Dockerfile which prevents use of additional snmp config files other than those defined by the "-c" command line option.

That's been, over all, about 6 months of "must solve that so we can fully migrate to SNMPv3": a great relief to get to the bottom of!

formal fix will be added to the github issue later.