Evaluate data from snmpTrap in trigger expression

[tflolo]

Hello,

I'm quite new to Zabbix, and I have run into som issues monitoring systems via snmp traps.

I have this trap:

20220207.085428 UDP: [127.0.0.1]:47048->[172.16.238.2]:1162
DISMAN-EVENT-MIB::sysUpTimeInstance = 315569
SNMPv2-MIB::snmpTrapOID.0 = SNMPv2-SMI::enterprises.35642.1.2.0.2
SNMPv2-MIB::sysName.0 = New system
SNMPv2-SMI::enterprises.35642.1.2.2.1.1.6 = 146
SNMPv2-SMI::enterprises.35642.1.2.2.1.1.5 = 3700
SNMPv2-SMI::enterprises.35642.1.2.2.1.1.2 = "Device 1"
SNMPv2-SMI::enterprises.35642.1.2.2.1.1.4 = -200
SNMPv2-SMI::enterprises.35642.1.2.0.0.1.0 = 10
SNMPv2-SMI::enterprises.35642.1.2.2.1.1.3 = 1260
SNMPv2-MIB::snmpTrapEnterprise.0 = SNMPv2-SMI::enterprises.35642

This trap tels me that there is a problem a device called Device 1, and there are a few OID's that tell me about temperature, power etc.. All important stuff.

I have created an SNMPTrap item

snmptrap[SNMPv2-SMI::enterprises.35642.1.2.0.2]

And this item logs the trap in zabbix.

Next I am trying to make a trigger to alert me when the trap indicates a problem with the device.

SNMPv2-SMI::enterprises.35642.1.2.0.0.1.0 = 10

The OID above tells me what the problem with the device is based on bitwise flags in the value. 0 = OK !0 = some bit flags are set

How can I create a trigger expression that pick the OID I want and evaluate it to raise a problem, here I need to do some bitwise operations to find out if a certain bit is set in the Int? Is this possible or am I handling the problem completely wrong. I can't preprocess the data and only pick the SNMPv2-SMI::enterprises.35642.1.2.0.0.1.0 = 10, because I need to "save" the information in the other OID's aswell, like the device name for instance.

I have been working on this for a few days now. I'm on 5.4 so the new expression syntax in triggers has limited the examples I can find on the topic.

I'm not sure how to continue forward.

[ISiroshtan]

Hey Mate.

Create a dependent item that will extract only the value of SNMPv2-SMI::enterprises.35642.1.2.0.0.1.0. As result you have one items that saves full trap for possible investigations(or for use in {ITEM.VALUE} macro in trigger) and one item that holds just the number.

In trigger use bitwise function against dependent item to check for state of specific bits you need.

I think this direction should allow you to solve your problem.

[tflolo]

Hey ISiroshtan,

I actually found the dependant items, and can extract the values via preprocessing it works great. Now I need to make some tags I guess to use the SNMPv2-SMI::enterprises.35642.1.2.2.1.1.2 = "Device 1" OID to tell me what device failed. I have been able to tag triggers before with {{ITEM.VALUE}.iregsub()}

Thanks for the help

[tflolo]

Now I'm 95% of the way, I now know there is a problem in my system, but not which device has the problem.

I made dependant items for all the different OID's, so now I can pick out and do my bitAnd() command on my value and trigger an alarm. Works like a charm.

My problem is that this same trap with same OID's can trigger for N amount of devices in the system so Device 1 - N triggering should not resovle my error if it's not the same device, and I should get a new error triggered if a different device has a problem.

I guess the design issue is that my host has multiple devices that can report problems

[ISiroshtan]

So to make sure I understand correct:
You have a system that sends SNMP traps and this system can report issue about different devices behind it.
To understand which device is actually reporting the issue should we should look at ''SNMPv2-SMI::enterprises.35642.1.2.2.1.1.2" OID inside the trap.

Is it correct?

[ISiroshtan]

The way I'd go is modify Zabbix traps receiver (seen it's done when perl script receiver is used) - add extra check to see if trap matches needed criteria and if it does - use value of OID "SNMPv2-SMI::enterprises.35642.1.2.2.1.1.2" as source, instead of actual source IP. Then create all the devices that are behind your SNMP sender as separate hosts using their names as DNS names.

This way you will have all the devices defined as separate hosts, and despite traps coming all from same source they will be properly mapped to corresponding hosts in Zabbix.



There is also possibility of using tags + trigger event correlation. But for it you would need to have master item(to extract data for tag) + dependent item in trigger expression. Which means Zabbix will do double check for trigger and create duplicate alarms (first evaluation when master item gets new value and few milliseconds later it re-evaluates again when dependent item gets processed). To fight that you'd also need global event correlation to close the duplicate alarms.
Pros: all control through UI
Cons: in my opinion it's more complicated.

[tflolo]

Hey,

Yeah I did the dependant item way and pulled in the "name" oid with a length check and referenced it in a tag with ITEM.LASTVALUE2 macro. But as you say i get duplicate alarms, for everything, and sometimes the wrong tag.

I will have a look at the trap receiver maybe i can mangle it enought so it does what I want it to do.

The use-case for snmp is usually 1 device per interface, but in my case i have 1 interface for multiple devices, and it's all dynamic as well, so 1 host can have 10 devices 1 can have just 1.

Thank your for your help

[ISiroshtan]

Obviously what I proposed is workaround to make it work. But as you have device name in one of OIDs, you can make it work.

So if you using zabbix_trap_receiver.pl as SNMP handler, adding following code can work for you:

foreach my $x (@varbinds)
{
if ( $x->[0] =~ /SNMPv2-SMI::enterprises.35642.1.2.2.1.1.2/)
{
$hostname = $x->[1];
$hostname =~ /.*\"(.*)\".*/;
$hostname = $1 || 'unknown';
}
}

It should be added at line 79-80. After the default hostname selection was done.

Now please, don't ask how it works, I have no clue about perl, it was written not by me. But it works (tho with different system and different OID in my case).

Post this change, original sender address should be replaced by value in that OID. So traps will no longer be registered with current host in Zabbix. Instead, you'd need to create a new host for each 'device'. In interface you need to set device name that is passed in that varbind as DNS name and set interface to using DNS instead of IP. This way, no matter from where the trap was recieved, it will arrive to actual Host that represents said device.



If you using SNMPTT instead - sorry, have no clue how to make it work. Never used it actually.