Unconditional trigger of zabbix trapper item

[extremeeryan]

We have a zabbix trapper item that receives input from a script that runs zabbix_sender. The data stored in the item is a short text blurb about the status of a service. zabbix_sender does not run on a schedule, but rather is event based (so there's no way to tell when the next trapper item will be received). We want to configure a trigger for this item to report a problem every time zabbix_sender is ran and the item receives new data. We were initially using the following expression, with the item being named item.message:

{Template:item.message.change()}=1

However, there are two issues with this method:

• When the item has no previous data, zabbix will fail to trigger on the first zabbix_sender run, as the previous state does not exist. This is a problem, as this item/trigger combo will be used by a large number of hosts.
• We set a relatively low global housekeeper history storage period to avoid database bloat. According to this page: https://www.zabbix.com/documentation...#value_caching , these values will be cached even if the housekeeper cleans them up, but may be deleted after a service restart. In this situation, we may miss important alerts from servers.

The ideal setup would be a trigger configuration that alerts any time zabbix_sender runs and the item is updated, but without the issues present when using change(). We tried a number of workarounds, like setting the expression to a condition that always evaluated to true, which resulted in zabbix immediately spamming problems for all servers with instances of the item (we were hoping the conditional was only evaluated when zabbix_sender was ran, which didn't seem to be the case).

Is there a way to unconditionally trigger a trigger on update of a trapper item without spamming alerts or having previous item data to use as a comparison?

[tim.mooney]

Does it work the way you want if you switch to using

{Template:item.message.nodata(5m)} <> 1

That should trigger if nodata() for the past 5 minutes is not true (in other words, you've received data within the last 5 minutes).

Also, I recommend using distinct integer codes with custom items, rather than textual blurbs, with a custom value mapping so they are clear and obvious in your interface, graphs, etc. This uses less storage (usually), can often be easier to trigger upon, makes it possible to graph values, etc.

[Singularity]

With item type of information as text, use strlen in trigger expression. Whenever item receives data, the trigger condition will become true and it will fire.
strlen > 0

[ncc]

(I work with extremeeryan, and am following up for him)

Does it work the way you want if you switch to using

{Template:item.message.nodata(5m)} <> 1

That should trigger if nodata() for the past 5 minutes is not true (in other words, you've received data within the last 5 minutes).

If that behaves the same as our always-truthy condition described in the OP (which for the record was {Template Database Alert:db.message.now()}>0 - our hope was that the condition was only evaluated when the item was updated, which did not seem to be the case), I would expect alerts to be rapidly sent for the 5 minute period that that expression is true. Should that not be the case?

Also, I recommend using distinct integer codes with custom items, rather than textual blurbs, with a custom value mapping so they are clear and obvious in your interface, graphs, etc. This uses less storage (usually), can often be easier to trigger upon, makes it possible to graph values, etc.

I definitely understand the value in doing that, however we don't have control of the types of alerts being sent, so we'd have to have the script calling zabbix_sender map textual blurbs to an integer, which is then mapped back to an actual message in zabbix, which wouldn't be ideal.

[ncc]

With item type of information as text, use strlen in trigger expression. Whenever item receives data, the trigger condition will become true and it will fire.
strlen > 0

Will we encounter the same issue detailed above, where the trigger continually fires, because the expression always evaluates as true?

[Singularity]

Will we encounter the same issue detailed above, where the trigger continually fires, because the expression always evaluates as true?

Sorry my bad, I interpreted a little differently. Will get back on this

[ncc]

Any updates on this? We can try testing {Template:item.message.nodata(5m)} <> 1 if you are confident it shouldn't fire off a multitude of alerts for a single value increase, but we want to avoid spamming our alert system like before when we used the always-truthy value.

[tim.mooney]

For the action that matches this problem and is generating the alerts, what "Default operation step duration" and/or Operation Details "Operation step duration" are you using?

[ncc]

For the action that matches this problem and is generating the alerts, what "Default operation step duration" and/or Operation Details "Operation step duration" are you using?

The "Default operation step duration" is set to 60, and the operation's step duration is set to 0 (which sounds like should use the default). The "Start in" value for the operation is "Immediately," as we need the alerts to go out ASAP, which they do.

[tim.mooney]

Your triggers and your actions work together. The trigger that I suggested should detect the issue that you want to detect (I haven't tested it, so I'm not certain, but I think it matches your stated goals). Because of the use of (5m) in the trigger, it will evaluate to true for 5 minutes. If you only want it to be true for a shorter period, use some other time period.

Once the issue has been detected by the trigger (trigger expression evaluates to true), how soon and how frequently you get alerted will depend on which action(s) match the conditions and what their operation configuration is. If your action configuration allows it to match and execute many times during the period where the trigger is true, then you will be alerted multiple times. You can be alerted quickly but then have a back-off for subsequent alerts by using escalations or using some other operation step detail than your default of 60 (seconds).

[ncc]

Your triggers and your actions work together. The trigger that I suggested should detect the issue that you want to detect (I haven't tested it, so I'm not certain, but I think it matches your stated goals). Because of the use of (5m) in the trigger, it will evaluate to true for 5 minutes. If you only want it to be true for a shorter period, use some other time period.

Once the issue has been detected by the trigger (trigger expression evaluates to true), how soon and how frequently you get alerted will depend on which action(s) match the conditions and what their operation configuration is. If your action configuration allows it to match and execute many times during the period where the trigger is true, then you will be alerted multiple times. You can be alerted quickly but then have a back-off for subsequent alerts by using escalations or using some other operation step detail than your default of 60 (seconds).

Is there an escalation rule or another step detail that would allow us to only get new alerts sent once, but still have newer alerts after it come through immediately? I imagine we could have a step wait for 5 minutes or so, which would effectively solve the issue of using 5m in the trigger, but we'd be missing important alerts that came through while the action was waiting. Or am I incorrect in the the assumption that they don't function asynchronously?

[tim.mooney]

Is there an escalation rule or another step detail that would allow us to only get new alerts sent once, but still have newer alerts after it come through immediately?

Sorry, I'm not understanding what you're asking here. If two unrelated events cause your trigger to fire (twice), then your action that matches will perform its initial step within whatever default operation step duration you have defined, once for each new problem event. After that, additional alerts for the existing event(s) follow your action's escalation schedule. If you just want to be notified once per individual problem event, that can be done by modifying your step schedule.

I think you should read up on escalations and spend some time experimenting. If you want to avoid potentially spamming alerts during your tests, use some kind of block (perhaps a firewall rule) to prevent Zabbix from being able to talk to your external systems that actually send the alerts. Just remember to remove the block when you're done testing.

[ncc]

>Sorry, I'm not understanding what you're asking here. If two unrelated events cause your trigger to fire (twice), then your action that matches will perform its initial step within whatever default operation step duration you have defined, once for each new problem event.

I was concerned about how actions would handle the "Multiple" PROBLEM event generation mode, but in testing, it didn't seem to be an issue. However, after a good amount of testing, a coworker and I were still unable to get a system in-place where each update to an item would trigger an action only once. Using `nodata` with the "Multiple" PROBLEM event generation mode resulted in multiple actions being triggered regardless of the step configuration (or resulted in multiple subsequent alert messages not being processed), and the "Single" PROBLEM event generation mode doesn't trigger multiple actions by design. If you have any insight into the exact configuration needed to achieve what we're looking for, that would be appreciated, otherwise we're going to pursue a system outside of Zabbix to process these alerts.

It seems this feature request and this forum post are related to what we're trying to implement, and similarly mention some of the same problems we ran into.