Limit the number of notification mails

[Alex___]

Hi there,

I've configured triggers in Zabbix to send notification mails and it works fine.
But is there a way to limit the number of outgoing notification mails within a given time frame (e.g. 50 mails/day)?

Today, due to some sort of "oscillating issue", I've received over 7000 mails so far: "Server XXX is unreachable: PROBLEM, Server XXX is unreachable: OK, Server XXX is unreachable: PROBLEM, Server XXX is unreachable: OK, ...". I had already reduced the validation interval but the problem is the number of servers I'm monitoring. Zabbix "thinks" that they are available/unavailable all the time. I don't know why but this is an other issue.

Best regards, Alex

[efrain02]

I'm also looking for a solution for this. Right now i disable the notification while i find some solution. And i'm making proofs with the values inside the actions. If i found something i'll let you know.

[Alex___]

Added a feature request: https://support.zabbix.com/browse/ZBXNEXT-736

Another option might be to limit the maximum number of outgoing mail within the MTA.

[qix]

Maybe the escalation system could provide you with the tools to solve this?

[Alex___]

Maybe the escalation system could provide you with the tools to solve this?

Mh, maybe. Do you know how?

[qix]

First, you should try to minimize affected triggers by using trigger dependencies. Once that is tuned, you can use the escalation framework to minimize the number of sent messages by using a cooling down period. (delayed notifications)

Also, try to use timebased triggers (min/max/nodata) to ensure that a specified item has been in an error state for a while.

Most of this will be in the manual and wiki.

Good luck!

[Alex___]

Thanks for the answer.

First, you should try to minimize affected triggers

The issue here was that the node availability information was oscillating and therefore Zabbix sent n (=number of monitored servers) messages within each interval. And since I was not able to login via SSH (file system error) the mail account got flooded.

Most of this will be in the manual and wiki.

I'll have a closer look at it to optimize the configuration.

[fmrapid]

The system should include automatic flapping detection as part of a standard option. Without having to go through hoops.

I am sure there is a ZBXNEXT somewhere, need to find it and vote for it.

Zabbix has great functionality but sometimes there is basic stuff that does not get covered off very well.

Cheers

fmrapid

[qix]

The system should include automatic flapping detection as part of a standard option. Without having to go through hoops.

No please, I like to know when something is bouncing before the customer calls. It builds credits

[Alex___]

The system should include automatic flapping detection as part of a standard option. Without having to go through hoops.

I am sure there is a ZBXNEXT somewhere, need to find it and vote for it.

There seems to be a flapping detection (see https://support.zabbix.com/browse/ZBX-1148) but this is another issue.

[jszaszvari]

No please, I like to know when something is bouncing before the customer calls. It builds credits

Not when you have a Wireless Access Point that is rebooting every 30 seconds at the office leading to 5000-6000 emails over the weekend.

Sure you could disable the host for the weekend, but that shouldn't have to be done.

You should be able to set flap detection on a Template or Trigger level.

I wouldn't mind so much if i could limit those emails to lets say, 1 per hour.

[qix]

I'm sure that it is possible to construct a trigger in such a matter that you can limit the number of events that occur within a specific timeframe.
However, in the thread it was suggested that Zabbix would need to detect this automatically and that is not something I personally would want.

IMHO the power in Zabbix is that you have the flexiblity to create very complex triggers but that Zabbix behaviour is very predictable none the less.
I feel that introducing these kinds of automatic 'improvements' would actually lower Zabbix flexibility and power.
It's like being a Linux CLI god and having to work with the Windows explorer to sort out 100's of files.

Anyway, If you need help setting up 'trigger flap detection' for a specific trigger, you could post the trigger here and we'll see if we can come up with a solution.

[qix]

Btw, did you know you can use time periods in actions and media as well as in triggers?
You can disable getting events or messages over the weekend in theses cases.

[jszaszvari]

I was thinking about custom , but i still need to at least be notified once.

It also wouldn't be lowering the flexibility, flap detection is a feature in almost all enterprise monitoring systems that i have used in the past.

Most open source ones have it too. Just not zabbix.

If you have a situation where a host on your network has a hardware issue causing it to drop offline ever 15 seconds, it should not be a hard task to detect that happening.

I just had a idea about creating a new trigger looking for X events in X amount of time. Will report back how i go....

Here is it explained much better: http://nagios.sourceforge.net/docs/3_0/flapping.html

Thanks for your advice. Much appreciated.