Zabbix alerting even though ACK'd

[mattsims]

We had an issue over the weekend. We were performing maintenance in our production environment. We set our hosts in to Maintenance Mode.

During the work, we Acknowledged the alerts and let the maintenance run out. Once it did, we started receiving page alerts for the hosts that were alerting but acknowledged.

Even setting maintenance back on the hosts, Zabbix continued to send out emails to our paging system (I was tailing the maillog and could see Zabbix sending out messages).

After some zabbix-server restarts, we ended up disabling the hosts which sent a final "Escalation cancelled: host disabled" page.

After some period of quiet, we re-enabled the hosts.

All our Actions have the Condition of Event acknowledged = "Not Ack" in order to qualify to page us.

This was very frustrating for us. Any idea of what may have happened?

[Pada]

When you put hosts/host groups in maintenance, make sure that you add a maintenance period too. When I started using Zabbix, I often forgot to add the maintenance period, although I did specify the "Active since" & "Active till" values.

Could you please tell us what version of Zabbix you're using and screenshots of how your initial Action looks like, as well as how your Escalations look like?

In our environment, our initial Actions are ALWAYS configured with:
Type of Calculation: (A) and (B) and (C) and ...
Conditions:

(A) Trigger value = "PROBLEM"
(B) Maintenance status not in "maintenance"
(C) Trigger severity >= "Warning"

Then our 2nd step (escalation) is configured with delay between 10-15 minutes, with the condition: Event acknowledged = "Not Ack"

Lastly, also take note that when there was an alert sent our for a particular trigger and the host goes into maintenance after that, that the OK (and probably escalation [if not acknowledged) alerts will be sent out as well.

[mattsims]

Thank you for replying.

Yes, we had a Period for the initial duration of the maintenance. The hosts did alert on the dashboard but didn't page us. We Acknowledged the alerts and purposely let the maintenance run out. As soon as the maintenance ran out, we started getting paged even though the hosts were Acknowledged.

We're using version 2.0.6.

Attached are some screen shots

[Pada]

I can't see anything wrong with your configuration.

It may be due to a bug (ZBX-6681) that is present in versions older than 2.0.7:

https://www.zabbix.com/rn2.0.7.php

The description of that bug doesn't sound like the issue that you had, because you said that you didn't get the first set of notifications, due to the host(s) being in maintenance...

I think you'll need an answer from more experienced users / developers / Zabbix support staff regarding this matter.

[mattsims]

Thanks Pada, appreciate your input.