How to avoid OK event actions for a new node

[mbrackeva]

When a new node is registered, every related trigger fires an OK event once. Since we integrate with another event management framework, this causes issues and unnecessary notifications.

How can I avoid this, either in the trigger or in the action?

This is Zabbix 3.0

[Linwood]

Are you speaking of an OK trigger, or an alert that goes out?

Every trigger (which is the state of a condition) is always either OK or PROBLEM.

An Alert (action based on the trigger) is sent based on how you set up the conditions in the alert message. Normally you include "is a problem" and also check the recovery box. The recovery message is sent as an OK when a trigger that was a problem (and only if it was a problem) changes to OK, and it is only sent to those who already received the problem message (more relevant to escalations).

So if you are getting (for example) email for "OK" without a problem, you probably forgot to include the condition for "is a problem".

If you are just looking at monitor-triggers and seeing OK triggers, that's expected.

[mbrackeva]

I'm speaking of the fact that when a new node is registered, every trigger on one of the new node's items will create an OK event. I can understand that is to be expected since the trigger's status changes from noting to OK.

My question is: how can I suppress those first OK events after a new node was created? I don't want to suppress any other events after that.

[mbrackeva]

The documentation speaks of the macro {TRIGGER.STATE}. I guess I need something like that.
However, it is only available in internal actions.

[Linwood]

I'm speaking of the fact that when a new node is registered, every trigger on one of the new node's items will create an OK event. I can understand that is to be expected since the trigger's status changes from noting to OK.

My question is: how can I suppress those first OK events after a new node was created? I don't want to suppress any other events after that.

I do not think I can help, perhaps someone else can jump in, but...

I really think you are chasing a ghost. The OK event seems a normal part of the trigger evaluation process. The trigger state, on initial evaluation, has to be something (internally 0 or 1), or unsupported (if the included items are not present). I'm not sure what it would take to separate the trigger value from the event.

I really do think you are either looking for a solution to a non-problem, or if it is causing you a problem, something else may not be set up correctly.

But maybe someone more familiar with the internals will show up who can comment.

[mbrackeva]

No, this is a real problem, but I can't seem to make myself clear.

Our Zabbix environment sends events (through actions) to an event management system. From within that system people receive notifications from those events, problem as well as ok. That works just fine under pretty much all conditions!

The only issue I have is that, when a new node is added, all triggers for items on that new node will fire an OK event. That OK event is, like all other OK events, sent to the event management system. The problem there is that, under those isolated conditions, people receive an OK notification for something that has never been in a PROBLEM state! If you have worked in a large operations environment for a while, you know that something like that is unacceptable.

So, again: the issue only applies right after adding a new node. I want to be able to avoid sending those "first-time" OK events to the event management system, but I can't find a way to separate them from the other events.

[Linwood]

OK, let me speculate.

You are using an Action, and the actions do not include a "trigger value = problem"?

You are doing that so you get OK events so the management system can know the events cleared?

My suggestion is set the action for "trigger value = problem" and check the recovery box. It will send the event only if there is a problem, and it will then also send when the event "OK" occurs but only if it previously sent the problem for the same event.

There are issues of course with triggers that are allowed to generate multiple events.

I suspect your problem will also occur when adding a new trigger (e.g. a new template associated with an old host)? That would be a similar circumstance.

If that's not on track, I apologize for being dense.

[mbrackeva]

Your assumptions are correct. I will test your suggestion. I will have to see how it behaves in more complex situations.

I'll give feedback. I'm on holiday for the next week though... But thanks in advance for the feedback!

[mbrackeva]

As I suspected from the documentation the "Recovery message" only applies to the operation type "Send message" and not to "Remote command". So I'm afraid I'm still stuck...

[Linwood]

As I suspected from the documentation the "Recovery message" only applies to the operation type "Send message" and not to "Remote command". So I'm afraid I'm still stuck...

Any chance of switching to the Send Message to communicate with the other system? It can call an external script and pass data (I don't let zabbix format mail at all, I just send all the macros in a text file to a perl script and let it do all the work).

Just thinking out loud.

[mbrackeva]

I haven't really played with media types yet. I'll setup something in our test environment and see how that works out. It is a substantial change from using the remote command facility, so it'll take me some time.
But I'll report back.

Tx again for the tips!

[Linwood]

Perhaps; I've never done anything with the remote commands, but substantially the external form of the media types lets you call a script with parameters. You can use the 3.x additional parameters, or what I found easier was just to create the "message" as a list of item values, e.g.

item.value@@@{ITEM.VALUE}~~~
host.name@@@{HOST.NAME}~~~

I have a list of basically every macro available, and a few functions. I pass exactly the same list out every time, the script loads all this into a perl hash, then the script decides what data it needs and ignores the rest. Essentially it just exposes the event context (via the macros available) to the external routine. The odd syntax is to provide a delimiter that won't occur in macro replacement text (remember some of that is multi-line so end of line delimiters are not reliable).

The only real interaction with zabbix is that the message conditions, escalation, and recovery (or not) control when the external routine is invoked, not what it does. You may need to set up a dummy user to divorce these from real messaging to real users since media needs a user.

[mbrackeva]

I see now that I should have started a while ago with media types instead of remote commands to send events to our framework. I can now simplify the setup and, most important, no more unwanted events!

Thanks for the valuable input!

[Linwood]

I see now that I should have started a while ago with media types instead of remote commands to send events to our framework. I can now simplify the setup and, most important, no more unwanted events!

Sometimes the hardest thing is to know what aspect of a system to mis-use to get what you want.