Make the Alerts stop!

[garumph]

We have an issue where we get flooded with alerts.

host is a netscaler with snmp items and triggers to monitor down hosts in load balancer queues.

set to alert on a trigger (host down >0)
escalations enabled
no recovery message
operation is to send email steps 1 - 0 every 300 seconds

it will work fine, then last night (and a couple other times) we had a release where we shutdown several hosts then brought them back up generating a couple dozen problem/ok alert pairs. We re-enabled the email for the alert.

At this point every event (problem or ok) for the last about 24 hours started alerting every 5 minutes. This continued until I went through and ack'd each event. Even after the problem events were ack'd it continued to alert on the ok events until they were ack'd also.

What can we do to stop this? We don't want Zabbix alerting us if a trigger is no longer in a problem state.

[untergeek]

I just had this happen this morning, but without any downtime on the server.

My web tests paged down for a bit, and the service was unavailable for about 10 minutes. This was expected. But when Zabbix kept on alerting after things were back up I became concerned. The "Latest Data" page for the host in question showed that ALL of the tests were running perfectly. Even the Dashboard showed that 12 of 12 web tests came back perfectly okay. I still got alerts saying that it was down.

Is there a way to clean Zabbix when it's gotten in this state? Even just for one host?

I have disabled the alert, the items, the triggers and even the web tests. I am waiting for things to clear out a bit. Then I'm going to re-enable the stuff and wait to see what happens next.

This isn't the first time something like this has happened.

[MrKen]

Is there a way to clean Zabbix when it's gotten in this state?

I had a similar problem recently. Once the alerts start coming there's no stopping them! Even stopping zabbix_server for a couple of days didn't stop them. In my case we're talking about hundreds of thousands of alerts.

This solution worked for me http://www.zabbix.com/forum/showthread.php?t=13326

Hope it can be useful for you too.

MrKen

[untergeek]

Actually, I was able to get mine under control differently.

First, I disabled the items, the triggers and the action that were in question. Then I waited a few minutes. Then I re-enabled them. That seems to have corrected it.

I obviously can't say whether that will work for everyone, but if I hit this bug again I will be trying the same fix first.