zabbix 6.4 configuring trigger for snmp interface down/ up traps

[Marijana]

hi!
I ve tried x times to configure these trigger but without success.
i am recieving traps down/up. see them in logs
se in gui as problem when traps arrived .but when it has to be cleared it is failing.sometimes is working somtimes not .i am really lost ideas what to try anymore.i am pretty new to zabbix .
i put in attach my last trigger conf.
i will be grateful for your help.

[ISiroshtan]

Hey Marijana
So let me try to explain what is the issue and why it not working as you expect.

Main point to understand here - recovery expression. Recovery expressions is evaluated only AFTER trigger expressions switches to false. So if you have two different items (A and B) and want a trigger to open based on value of A but be resolved based on value of B, you need to make sure that by time B arrives the trigger expression is already resolved. So if you use find(A, ...)=1 and item A only gets values that match this condition, the trigger can not be automatically resolved. Ever. Because latest value of value A that find(A) takes for evaluation is always matches the alert condition. Simply put, if you got trap "power lost" at 15:39, the last value of item A will still be "power lost" at 16:00, 20:00, 24:00, etc. It will never change. No matter how many traps arrive to item B, value of find(A) will remain same and will satisfy the trigger expression.

In my initial message in this thread I offered the easiest way to solve it - you combine traps that trigger alert and those that resolve alert into same item. This way find() function will be fetching the latest trap. If latest trap received indicates alert fire condition - trigger condition will match and alert will fire. If latest trap received indicates alert resolution condition (or NOT alert fire condition) trigger expression will be false (and recovery condition will be true, if set) resulting in alert resolving.


Alternative approach (which I think is working, tho it was long time since i had to use it so not sure if it still would work same way in current version of Zabbix) is to limit time for how far back in the past we looking with find() function.
This way you keep your items A and B separately. Then you define trigger like find(A,1m,"regexp","some_matching_string")=1. Recovery like find(B,1m,"regexp","some_other_matching_string")=1 .
Now how it work:
Trap arrives to item A with matching string - Zabbix will take all values A over last minute and check if any of them match trigger conditions -> last one matches -> alert fired.
Few minutes later (it's important, we set 1m in find() function, so there should be at least 1 minute between alert trap and recovery trap) a recovery trap arriver -> as new value that is used for trigger arrive Zabbix needs to reevaluate the trigger -> zabbix takes all values over last minutes for item A -> there should be no values, as alert open trap arrived more than a minute ago -> trigger expression is not satisfied(false) so zabbix can proceed to check recovery expression -> it takes all values of item B now over last minute -> it checks if find function matches -> it's matched -> alert resolved

Big problem with second approach is the fact you are setting static lookback (1m in my example) to function. If time between alert trap and recovery traps will be less then set time (1m) - recovery will not work. That is why I recommend first approach whenever possible.
(And writing regexp with oid name instead of oid number is easier (IMO), that is why I usually recommend to have mibs installed)

Either way, pretty lengthy reply and I never saw myself as good at explaining, so feel free to follow up with questions

[cyber]

Problem is, that you have not learned the concept of "recovery expressions"

Recovery expression

Logical expression (optional) defining additional conditions that have to be met before the problem is resolved, after the original problem expression has already been evaluated as FALSE. Recovery expression is useful for trigger hysteresis. It is not possible to resolve a problem by recovery expression alone if the problem expression is still TRUE. This field is only available if 'Recovery expression' is selected for OK event generation.

If your initial expression is still true, it will never look into recovery one... and using different items in main and recovery does not work out always... and nodata(, 10s) does not work either.. it shoudl not be less thatn 30 sec as this is the period when history syncer calculates nodata...

sec period should not be less than 30 seconds because the history syncer process calculates this function only every 30 seconds.

You could use tags in triggers instead and configure event orrelation rule for this...

[Marijana]

i am quite familiar with concept of recovery expression. I read that def 10x times .and have expirience of 20y in monitoring itself . But I am new into Zabbix and something strange is happening here . for some interfaces is working for some doesnt.i did for sure 10 diff trigger configuration and nothing works consistently. .In pic in attach you could see i have tag for correlation .and it works fine. something else is problem.
you wrote not to use diff item for problem expression and recovery expression? why? it is naturaly for snmptrap down to be closed with snmptrap up. but i saw when both traps has same timestamp problem stays. zabbix couldnt successed to close.
If your initial expression is still true, it will never look into recovery one... and using different items in main and recovery does not work out always... and nodata(, 10s) does not work either.. it shoudl not be less thatn 30 sec as this is the period when history syncer calculates nodata... -thx for this info for nodata function. nodata i saw a lot of example and think is necessery .I am not sure when it says trap evaluation ,is zabbix itself evaluate its snmptrapd.log periodically or per received trap?
addittionaly to this I have request from my network guys to couple interface down trap with alert which is coming from polling operational state of interfaces?

Could You be kind to write me concrete example for this particular case:received snmptrap interface down item ,how trigger should look like to work properly? I lost ideas.
thx a lot!
br,
Marijana

[Marijana]

and what would snmptrap interface down put in state false?
this is my new conf of trigger :
find(/SNMP Interface Traps/snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.3"],,"regexp","down")=1
it works even like this
find(/SNMP Interface Traps/snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.3"])=1

it generate problem
i put for OK event option "expression"
I create tag for matchin exact interfcace .it works. {{ITEM.VALUE}.regsub("\s+"(.*?)"(.*)", "\1")}

But RESOLVED never happend. in trap logs there is no new interface down traps but PROBLEM persist.
so what should I do?
br,
Marijana

[Marijana]

hi! here is my new trigger configuration for snmptrap interface down. and recovery is not working at all. in attachment is configuration of trigger and tag
i ll appreciate much your help.
add request is to couple snmptrap interface down with snmp polling operative status of interface alert in terms if trap interface down is received than snmp polling will not generate alert fro the same interface. that configuration i couldnt figure it out how to do it. in theory i know what i have to do ,but dont know how to do in zabbix.
br,
Marijana

[cyber]

Please, if you answer, just quote a post or qrite another... its quite impossible to quote you back if you use "comment"...

But.. to the topic... you use one trap to trigger it. so you "find" the value of "down" from data and then trigger the event... but your item never receives any other value than those containing "down"... so your expression never gets false again. Thus whatever you have in recovery expression, will never be considered.
Having tags for closure only works after your trigger is evaluated to false, be it with or without recovery. Only after that matching event can be closed.

I would instead create one trigger that has ok generation "none" and event generation "multiple". Extract tag for device name similarily as you already do... and add additional tag "down" (no value)
Then I would create another trigger, that would work based on that other trap that receives only OK traps with similar tags except instedad down use "up".
And then create event correlation rule, where you do the matching -> from host group X, with Tag for device pair matching and new event having tag UP, close both events...

[ISiroshtan]

Why overcomplicate things? Just feed both linkDown and linkUp traps into same zabbix item. I think the key should be snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.[34]"] or snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.(3|4)"]
Then have trigger "find(/SNMP Interface Traps/snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.(3|4)"],,"regexp","down")=1" trigger and it will work. When last message is ifDown - alert will fire. When last message is ifUp - alert will resolve.
(Unrelated: why using numeric OIDs? it's horrible! Why not instal proper MIBs and deal with OID names instead?)

After you test the above, you can proceed to specific interface detection setup.
Assuming your regex to extract interface index works:
Add recovery expression "find(/SNMP Interface Traps/snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.(3|4)"],,"regexp","up")=1"
Problem event generation mode - multiple
OK even closes - if tag matches
Tag for match - RADI

[Marijana]

Hi!
Thx Y ou so much for Your help. I tried this before but didnt work at first. Dont know why. so i was thinking is wrong. Now, listening Your advice, I tried again and it works !!! .Once more big thank YOU !
regarding mibs ,i will set that too. Didnt get time to sort it out .I am new to this and inherited this zabbix installation and configuration.
p.s.
Add to this I have request to pair link down alert (problem) which is triggering from snmp polling operstatus every 5min with snmp trap "interface down" in manner that if snmp trap for "interface down" comes that link down is not triggered. And PROBLEM to be closed through snmp trap "interface up" or polling operstatus is "up".I know that triggers will connect through depending mechanism but my issue is how to match attributes between 2 triggers ? .Trigger in link down from snmp polling is using discovery mechanism for getting ifname and ifalias and ifndex, in snmptrap i get those vaules in trap and could get them out in tag through item.value but how to reffer correctly with each other in configuration because link down trigger is not aware of item .value in trap trigger ,and vice versa, trap trigger is not aware of ifalias ,ifname ,ifindex variables. do I have to extract this variables as macros in template which holds both triggers ? or preprocess snmp trap somehow and then pair with link down trigger ?
Really dont know how.Maybe is not possible at all.
thx again in advance for Your answer.

br,
Marijana

[ISiroshtan]

Marijana glad it worked for you.

About MIBs: I would highly advise to do it first step, as when you install them it will stop having numeric OID data in it, so any items you have set with numeric OIDs will stop working and you will need to rework them again. I do think numeric OIDs are horrible to read, but if you already have a lot of work done with them it might make sense to keep it as is until you are ready to do a rework of all traps configuration you have.

Now about what you ask for interface status trigger dependance: it's not impossible but... it's not easy. What you did set up till now is not compatible with what you want. So you essentially have to drop what you have to trash and start from scratch (I would advise to just drop the idea).
The direction you could take is to integrate ifup/ifdown trap processing into interface discovery from snmp polling.
(I did not work with new style SNMP discovery, so follow up is written from knowledge of old snmp discovery process)So you would need to check if final part of OID matches actual interface index (in Cisco i think they are, on other vendors - no so sure). If they are you need to setup item prototype for snmp trap but not just "\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.[34]" regex but in a way where it would match both ifUp/ifDown AND interface index (that you can get as {#SNMPINDEX} LLD macro) via same regexp. Then simple form of trigger prototype, like single problem on "find(/SNMP Interface Traps/snmptrap["\s\.1\.3\.6\.1\.6\.3\.1\.1\.5\.(3|4)\s+{#SNMP INDE X}"],,"regexp","down")=1" (regex is not correct here, would need to study trap to write proper one) and then you can set dependent trigger relationship.
But I honestly don't think the effort is worth it. At best it will be backlog task for somewhere in distant future with lowest possible priority.

[cyber]

Extract index to tag and use it to match closing trap...

And MIB-s are just another thing to maintain..:P Thank you, I will skip.. Numeric will work always..

[ISiroshtan]

And MIB-s are just another thing to maintain..:P ... Numeric will work always..

Maintain as in "install it one time and forget it exists"?
Fair point that it will work without them, but in my experience if you will get a task to rework something in SNMP Trap monitoring after 6-12-24 month (or worse, you are given somebody else solution), it's way easier to make sense of it if you read OID names and not OID numbers
I will concur on not installing them if it's some often-changed MIB file and you need to operate in cross-version environment... or some other quite specific cases. But still think for majority of cases it's easier to deal with MIBs installed

I will skip..

Everybody does it the way they feel more comfortable, right? While I would question a newbies if they sure they want to do it that way, I'm in no position to ask you that, now am I?

[cyber]

You can always question everything.. And you should, that is how good solutions are worked out..
I have never found working with numeric OID-s somewhat difficult... (not that I like that snmp crap .. :P ). But maybe, I am weird... Some mib browser in other tab to check things and all is good..:P

[Marijana]

hi guys!
I have new problem with snmptraps. I set it up as ISiroshtan suggested and it works. but for some nodes and interfaces I see traps in zabbix_trap.log but i dont see them in problem console or in items. traps are correct format as for one which I see in problems.
example of received interface down trap in zabbix_trap.log which I dont see in problems or items of node (its not due to conf of item or trigger because same template is applied on all nodes ,and for some is working and random for some is not, restart of zabbix then helps ,but it is not solution.i dont know what is the root case of this behavior so i can permanently remove the problem):
-----------------------------------------------------------------
10:52:42 2024/12/06 ZBXTRAP xx.xx.xx.xx PDU INFO: requestid 388 errorindex 0 receivedfrom UDP: [xx.xx.xx.xx]:161->[172.xx.xx.xx]:162 transactionid 3732907 notificationtype TRAP version 1 messageid 0 community bla errorstatus 0 VARBINDS .1.3.6.1.2.1.1.3.0 type=67 value=Timeticks: (93890461) 10 days, 20:48:24.61 .1.3.6.1.6.3.1.1.4.1.0 type=6 value=OID: .1.3.6.1.6.3.1.1.5.3 .1.3.6.1.2.1.2.2.1.1.61 type=2 value=INTEGER: 61 .1.3.6.1.2.1.2.2.1.2.61 type=4 value=STRING: "TenGigE0/0/0/2" .1.3.6.1.2.1.2.2.1.3.61 type=2 value=INTEGER: 6 .1.3.6.1.4.1.9.9.276.1.1.2.1.3.61 type=4 value=STRING: "down"
----------------------------------------------------------------------
i am running zabbix 6.4. version on 4.18.0-425.3.1.el8.x86_64 #1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux.

Do you know what could be the problem ?
Did you met this problem ,and what should I do?
thx a lot for your answer!
br,
Marijana

[ISiroshtan]

Hey Marijana

So as you know flow of data is like
(SNMP Sender) -> Zabbix server snmp engine -> zabbix_trap.log file -> latest data -> trigger
So if you see the trap present in zabbix_trap.log the next step is to go check if it's present in latest data of corresponding host/item at expected timestamp.
From my experience with cases with similar symptoms, I would expect it not be in Zabbix latest data. If it is indeed not in there - check if IP address in 'ZBXTRAP xx.xx.xx.xx' properly matches to host ip set in Zabbix. Might be that device sent it from a different interface for whatever reason..

If the trap IS in Zabbix latest data, but trigger did not fire and at the same time properly fires for other interface traps... that would be way more interesting and weird case. In this scenario I would ask a screenshots of trap in Latest data + trigger configuration screenshots + screenshot of problems history of that host at that time...and then some extra time to think about it

[Marijana]

hi ISirothan,
my case is that trap is present in zabbix_trap.log file ,but not in the latest data. and ip address is matching. unfortunately i dont have screenshots at the moment .i reapplied template on node to see if that will help. so i lost previous data .But asap i notice again i ll send you pics. Now i wanted to confirm which case iz mine.
keep in touch.
br,Marijana

[Marijana]

hi!
Sry for late update info.  Anyway I am positive  sometimes traps are recevied and seen in zabbix_traps.log ,but there is nothing in item,and then I do pure restart of zabbix and it starts working. So I didnt touch or change anything. i check configuration. it is ok. anyway it will not work after pure restart if configuration is problem. Does anybody expirience same or similar problem?
For me this is big problem ,because i dont know when it will show problem or not. because it doesnt stop working  for all  traps .it is random.
br,
Marijana

[Marijana]

Can this problem be due to lower memory amount for caches in snmptrapper ? maybe i should increase that?
br,
marijana

[Marijana]

hi!
I have add on question.
I am using snmp polling and snmp traps for registering interface down or up state. I got request when interface is adminstrative shut down than for that interface ,alert interface down coming from snmp trap "interface down" to be automatically closed /resolved .I really dont know how? i try by adding nodata function in recovery expression but it sis not working. what am I missing or is not possible?
to add info my snmp polling interface oper state alert is dependable of alert from snmptrap so not to have too much alerts . so i can not make snmp tap alert dependable of polling admin sate of itnerface or I can? ( to add additional trigger ifadmin down check? ) or it will not work if admin shut down remove from snmp table that interface , ( this I am not sure ,I have to check)

thx in advance for your help1

br,
m.

[cyber]

one item that keeps track of IF admin status and then you consider that value in your trigger... ie "admin status is not down and if status is down"

[Marijana]

I ll try and let you know if it works.thx!

[Marijana]

Hi!
Here am I again. now I have request fo "cold start" trap to clear "dying gasp" trap .
I didnt successed to make expretion which works. All trpas are coming but cold start anyhow cant clear dying gasp. I made to version but both not working .any idea? in future i will have more request liek this and is really becoming painfull every time I have trouble with clear match. ( .so some universal guidance for matching traps for clear would appriciate .thx a lot in advance!
br,
marijana