Multiple event generation trigger and auto OK event upon a given timeout

[20k]

Hi,

I'm using the zabbix_sender tool to send some customs alarms.
I have a very simple Zabbix trapper text item, whose key is "alarm[name][severity]".
From this item, I want a trigger which fires for any new value, and which automatically turns to OK in case no value is received for a given timeout (60s for example).

My trigger configuration is the following, with "PROBLEM event generation mode" set to Multiple :
{MyHost:alarm[name][severity].regexp(.*)}=1 and
{MyHost:alarm[name][severity].nodata(60)}=0 and
({TRIGGER.VALUE}=0 or {MyHost:alarm[name][severity].diff()}=1)

My trigger fires as expected when I send a new value from the zabbix_sender but it then fires again every 30sec until the timeout whereas no new value is received.

Here the history of my trigger : http://img4.hostingpics.net/pics/175...emshistory.jpg
And the history values of my item : http://img4.hostingpics.net/pics/553...latestdata.jpg


Any idea what's wrong ?
Thanks!

[20k]

Some additional informations below.

First, I forgot to indicate that I'm running on Zabbix 3.2.3.
Then, looking into the zabbix_server.log (with DebugLevel=5), I do see my trigger expression is evaluated every 30sec irrespectively of whether new values are received or not. This is why I tried to add the diff expression in my trigger but it still does not work as I would expect.

I also tried with "PROBLEM event generation mode" set to Single but in that case I'm missing the new values coming before the timeout so it is not the behaviour i'm looking for.

[20k]

There is really something that I don't understand.
I changed my trigger expression to :
{MyHost:alarm[name][severity].regexp(.*)}=1 and
{MyHost:alarm[name][severity].count(60)}>0

I do not have multiple triggers for the same item value anymore but my trigger never goes OK after the timeout. Looking at the logs, I see that my trigger expression is not evaluate anymore every 30sec. I guess this explain why it never goes to OK.

I'm starting to think that there might be a bug here.
It should not be that complicated to simply raise a trigger for any new value of a simple text item, this look to be a very basic and common need.

Thanks!

[20k]

I definetely think there is something wrong with Zabbix trapper item and the nodata() function.

I finally tried the following trigger configuration :

Problem: {MyHost:alarm.regexp(.*)}=1
Recovery: {MyHost:alarm.nodata(60)}=1

I don't see any good reason for this simple definition to not clear my alarm after 60sec.

[Atsushi]

nodata () is evaluated every 30 seconds.

If you set the PROBLEM event generation mode to Multiple, event will be generated every 30 seconds.

Manual:

https://www.zabbix.com/documentation/3.2/manual/config/triggers

If time-based functions (nodata(), date(), dayofmonth(), dayofweek(), time(), now()) are used in the expression, the trigger is recalculated every 30 seconds by a Zabbix timer process.

[ovas]

I definetely think there is something wrong with Zabbix trapper item and the nodata() function.

I finally tried the following trigger configuration :

Problem: {MyHost:alarm.regexp(.*)}=1
Recovery: {MyHost:alarm.nodata(60)}=1

I don't see any good reason for this simple definition to not clear my alarm after 60sec.

It does not recover, because problem definition is still meeting trigger condition.

[20k]

Thanks ovas.
Yes, I understand that after looking at ZBX-12067, even if that's not really the behaviour I would expect.

That being said, do you have an idea about how I could get the behaviour I'm looking for, which again seems very basic and common.

[ovas]

I have actually been trying to resolve this for two days now, but with only partial success. Step by step:

Item
Type - Zabbix trapper
Type of information - Text
Key - alarm

Event correlation
Condition - Old event tag Error = new event tag Error
Operation - Close new event

Trigger
Name - Alarm: {{ITEM.VALUE}.regsub("\#([0-9]+)", \1)}
Problem expression - {centos7_zbserver3.2:alarm.regexp(.*)}=1 and {centos7_zbserver3.2:alarm.nodata(60)}=0 and {centos7_zbserver3.2:alarm.diff()}=1
Recovery expression - {centos7_zbserver3.2:alarm.nodata(60)}=1
PROBLEM event generation mode - Multiple
OK event closes - All problems if tag values match
Tag for matching - Error
Tags - Error = {{ITEM.VALUE}.regsub("\#([0-9]+)", \1)}
Allow manual close - Yes

Testing via terminal
zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -s centos7_zbserver3.2 -k alarm -o 'This is test #1'
zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -s centos7_zbserver3.2 -k alarm -o 'This is test #2'
etc, but send the data within 60 seconds.

The conclusion
There are two ways of achieving two different results.
1. If we leave the trigger setup as it is written above and send two values, Zabbix - as it should actually - will create additional 2 events at */30 * * * * for the second value you send because of how timer functions work in Zabbix. Those two additional events will be automatically closed because of event correlation. After a bit of time, the second value you send will be closed, because of the recovery expression condition. The first event value will remain open until manually closed, because event tag will change upon the second value sent to item and our we switch to event to OK status only if tag values match. Screenshot: http://prntscr.com/eyb7hq
2. If we change "OK event closes" setting to "All problems", again, Zabbix will create two additional events because of how timer functions work. But over defined time, if no data is sent to item, all the trigger events will close by timeout. That is, if you have bursts of data sent to trapper, it is OK. But in case trapper is updated with information constantly, more often than withing nodata() defined timeout, the event will never close. Screenshot: http://prntscr.com/eybe20

Hope this helps.

P.S. I wonder if it is convenient to use a single item to store values for multiple data. Depends on the data amount obviously, but I would have made separate items or separate triggers. Items offer more flexibility and triggers can be adjusted by regular expressions.

[20k]

Hi ovas,

Many thanks for your time, I've also spent a lot of time trying many triggers expression.
I did not have time to look at your findings but I'll surely do and I'll let you know. I've never played with the Event correlation feature so I'll especially give a try at this.

Regarding your last remark, I'm actually using a distinct item (automatically created by LLD rule) for each single alarm type. But even for a given alarm type, I might got more than a single alarm within a minute.

[ovas]

So it means you have multiple items. Still different data in all of the items? If yes, then it doesn't resolve the above mentioned issues, but in case of opposite event correlation can actually automatically close the copies of events made by nodata timer function.

[20k]

I've looked at what is possible with the Event corelation but if I'm not mistaking this can only be used to automatically close some events, not to ignore them. This does not really fill my need as the duplicated events would still be count, for example in the "100 busiest triggers" menu.

I think I'm going to give up the idea of automatic trigger closing :-(

Actually, my original concern is that when an event is not closed, the "Last change" of the related trigger is never updated when a new occurence of this same trigger is raised. From the "Last 20 issues" widget in the Dashboard, I would like this new event occurence to be updated and displayed on top of the list. But I guess there is no way to achieve this.

Thanks again for your help anyway.

[Vaku]

Hi, I'm fighting with the same problem. Already 2 days.
Very frustrated that such a common and expected behavior does not work in Zabbix.
Have you considered creating another item, which would change it's state to 1 or 0 every 5 minutes (for example) and use that second item instead of nodata ?

[Vaku]

Please vote the this feature request: https://support.zabbix.com/browse/ZBXNEXT-5400
it is related to this topic and described problem

[Kos]

Please vote the this feature request: https://support.zabbix.com/browse/ZBXNEXT-5400
it is related to this topic and described problem

Probably, more appropriate is the older ZBXNEXT-2452 (opened since version 2.0.x).
It has a detailed description of the problem, examples of use-cases and proposed solution; however, still has not enough votes for implementing.