Zabbix Sending Lots of Mail when host is down

[Sara]

Hi,

I am using Zabbix Monitoring Tool for my servers.

All goes well but when server gets rebooted we are getting all the configured alerts instead of the single alert informing the reboot.

Is there any way we can limit the number of alerts?

Hope to hear from you soon.

[Sara]

Hi,

Kindly get me some solution for that.

[Linwood]

The only way I've found is to build dependencies into the service related items (as opposed to performance related which should not be alerting, e.g. disk space too high should not be alerting, right?).

So for example, on things like FTP, NTP, SMTP, etc. I build a dependency on the ICMP trigger for unreachable, so that if a node is unreachable, those triggers do not fire.

You also need to ensure that the timing of the trigger on which you are dependent is such that it will go first, e.g. if on 3 ping failures you trigger ICMP unreachable and these are every 60 seconds, then nothing dependent on it should fire in less than 3+ minutes (more like 4 to be safe).

[Sara]

Reason for lots of alert

Kindly tell me the reason why i am getting lots of alerts. I donot want to use Escalation feature for alert issue.

Please advise me some other option.

[Linwood]

Well, I tried to help, but apparently did not address your issue, I will try once more.

Alerts generally come from triggers, and in the trigger you can control when it goes off. Consider a server that works fine, is shut down for a while, then rebooted. All services go down, then all come back up.

For triggers that look at something like disk space used, these should not trigger alerts as the disk space does not change (indeed, it just stops receiving new data).

For triggers that look at a service, say whether the HTTP port is open, or the SSH port is open, these ports implicitly close when the server goes down. So every trigger looking for an open port, or an explicit response, goes off.

The simplest way to prevent them from all going off at once is to write your triggers so they do what you want (those in the templates are EXAMPLES and you are responsible for writing your own -- welcome to open source free software!). So for example, if you have 10 such triggers including one that checks if the host is up (usually via ICMP Ping), then 9 of them should be set to alert only if the ICMP Ping is NOT alerting.

Then (+/- a bit of careful timing) what happens is this:

- Server goes down
- Ping trigger alerts
- HTTP (or other) trigger TRIES to alert but cannot as it sees PING down
- Server comes up
- Ping alert clears
- Other triggers no longer need to fire as they come up also.

This is done by the trigger dependency setting (a tab on the trigger definition) and can be added to the template.

It is worth noting that out of the box this might not work completely, as there are race conditions -- zabbix might notice the HTTP service down before it checks the PING response. The normal solution for that is put some delay in the other checks, more delay than PING, e.g. wait for at least 3 polls of HTTP before the trigger alerts, and make sure the PING is faster to alert (compare both how many samples and rate at which each is sampling).