Stop Escalation emails!!!!

[JSylvia007]

I'm tearing my hair out trying to get these emails to stop.

My "Action" is:



All of my "Events" are acknowledged. WHY dont the damn email stop. I'm getting 9 emails every 5 minutes. All of this because I added a template to monitor Exchange. We Have 2 Exchange Servers, but we don't utilize POP3, IMAP, etc. Every one of those things throws a trigger. I went through and Acknowledged all of the events by hand, disabled the triggers AND items, yet i am STILL getting emails.

HELP! Where are they coming from???

[JSylvia007]

I've now disabled the action, and I'm STILL getting emails...

Someone PLEASE tell me how i can fix this!

[JSylvia007]

OK... Further explanation on my setup.

This is an upgrade from 1.4

Everything worked fine until I added an item and trigger to the Template_Windows that I wanted on ALL of my windows servers except one.

I figured that all I'd need to do was just disable the "Item" on the specified host (Machine X).

I added the Item, no issues.

I added the Trigger, no issues (except on the one server which i planned on disabling the item)

I have one action (in the OP above) to cover all trigger flips.

Right off the bat, of cource, the trigger flipped on Machine X, and the action caused an email to go out.

I figured, "alright, disable the item on Machine X"... No such luck here. I disabled the item, and i keep getting emails. I acknowledged EVERY event that I can find, and I am STILL getting emails.

I went one step further, disabled the action, yet the EMails are STILL coming.

I've had to stop zabbix-server for the night because the emails are unbearable.

Any help on this to save me from having to rebuild from scratch would be greatly appreciated.

[NOB]

Hi

after reading your post I hesitated to check it. But, finally,
the curiosity was bigger than the fear of an E-Mail flood.
Especially if I configured to send E-Mail to a colleague

If you configure it like me (see attachments) it works perfectly.
We would like to be able to set the delay (1 hour per step is, maybe,
to long).
I think, setting this a, say, 15 min. per step is fine.
It is not necessary to specify it in a more fine grain way.

The comparison of your screenshot with mine, show three potential
problems in yours:

1. Step From 1 To 0 (0 means infinity)
2. Period: 0 (= default) - does that mean immediately ?
3. The condition with Event acknowleged = "Not Ack" is at the wrong place

I hope that the mail flood did stop after a while. I am not sure, how long the backlog of
your mail-server is
You can watch the actions, i.e. sending the mails, under Administration/Audit The select (on the top right) Audit logs.

HTH,

Norbert.

[JSylvia007]

Thank you greatly for the insight, i'd like to state a few things...

First, is that my actions / configuration is correct for how i need the system setup.

Basically, "Keep emailing the admin every 5 minutes until the problem is either resolved or acknowledged."

My issue is really in the post before yours about HOW i came about the problem.

Luckily, there is hope. Its a simple fix.

Basically, edit your actions so that "Escalation" is not enabled, and save the changes. Handle all the Ack's that need to be remedied. Disable all triggers that should have been disabled anyway. Re-enable escalation.

Hope I'm being clear, and I hope that this helps out others as well.

~Jake

[NOB]

Hi

we did test some more parts and found out, that every From step
will add 1 hour of delay.
So:

1. Step 1 has zero delay of doing an action
2. Step 2 has 1 hour delay prior to the first action
3. Step 3 has 2 hours delay ...
4. and so on.

In our setup (the images attached to my previous post) the Step 2-3
works like this:

The first mail is sent after 1 hour and from then on the mail is repeated
every three minutes (180 seconds) until the event is either acknowledged or solved.

In case of a solution, the "Recovery" mail is send to both users.
So, if you want to send repeated mails immediately:

• choose From step 1 to avoid the delay
• Set the Period to 300 seconds
• Specify the Operation type, etc.

and then I expect that every 5 minutes a mail is send to the user.

HTH,

Norbert.

[Alexei]

I am lost after reading this thread. Is there anything we should fix?

[NOB]

I am lost after reading this thread. Is there anything we should fix?

I really don't know, either

IMHO the result of this thread shows the following:

You can configure the escalation in a way which may result in a mail flood.
How to fix it was mentioned in this thread.

I simply tried to show what we tested and how it works if the
configuration is done like we did. If you configure it that way,
the request of JSilvia IMO can be fulfilled without any patch.

I am not sure whether you can check for a potential mail flood
in the frontend to avoid such problems.
The cause of this mail flood is not clear and I don't want to
test it
If a configuration like Step to 0 (infinity) combined with a Period of 0, etc. really results in
a mail flood a patch is required, I think.

We experienced a case where one problem message arrived and 22 resolved
messages !
The case is still open but we need more testing. I am not sure how
we configured the actions, but I am sure we didn't use the escalation part.

What is needed most is a documentation and training of the users.
If they try a feature which is not well documented, some misconfiguration
based mostly on misunderstandings will happen and result in posts
like these.

Regards

Norbert.

[JSylvia007]

In response to NOB and Alexi - First - Thanks for creating an amazing product, and being here for the support of those who use it.

NOB, you are correct, there IS a SOLUTION, which I attempted to (poorly) explain, which solves this issue.

Here is a deeper description of the problem, again for further understanding, hopefully to help other users.

With an action like is setup in the first post, if the following happens, you WILL receive an email flood.

1. Create a Template (X)
2. Add an item to the template (Xi)
3. Add a trigger (Xit)
4. Apply the Template (X) to a host (H)
----Side Note on Host (H): The item on Host H will NEVER be able to collect data. The specified service will NEVER be active on this host. This is because you have 100 Windows Servers, 80 run this service and 20 do not. Its just easier to add it to a template that affects all windows servers, you can always disable the item and trigger on Host H later...

Here's what happens:

Because the item can never collect data, the trigger is thrown instantly, causing the action to be fired immediately. This is the start of the email flood.

One would think that all they need to do is disable either the item, or trigger causing this issue, and the action would cease firing. This is NOT the case.

If you disable the item and/or trigger, you still NEVER receive the conditions that cause the action NOT to be fired.

Steps to stop the email flood:

After further research, it seems easier than my previous post.

1. Edit the action causing the mail flood and temporarily click off the "escalation" checkbox - save the action
2. Disable the offending trigger and item
3. re-enable the escalation on the Action


Again, I hope this helps, and I hope I'm being clear to help further the flow of knowledge.

I don't believe this requires a patch, i believe that its actually behaving as its supposed to. But, as NOB said, a little knowledge and the proper configuration goes a long way.

The only thing I would say that MAY need a patch, or some further investigation, is that if an item / host / trigger is disabled, any currently occurring actions / escalations for that object should be stopped.

[cbidwell]

I've got the same issue. When I upgraded from 1.4 to 1.6, zabbix seemed to reenable some of the triggers that I initially had disabled on various servers. So when I went to my actions and disabled them, it was too late! Mail kept continuously being sent even with the action disabled. The only way I was able to fix it was to run that service on that server so that the trigger would disable and the email would be sent out that it came back up.

Is there something that can be done with this? I found that editing an action after emails are already sent out doesn't make a difference, it seems any editing of actions are applied to future triggered alarms.