Trigger for a % of servers down

[jfasselin]

I have a need for varied alerts to monitor a Citrix server farm. Essentially, I have 14 servers, and I want different levels of alerts if I have 1-3, 4-13 or 14 servers down (Average, High, Disaster); because the servers are autonomously redundant and I can still meet my SLA with 3 servers down.

For 1-3, I just need a simple ping alert for the specific server, it's easy. For the disaster condition, it's also pretty easy to do a trigger that will alert me if ALL servers are down.

But how do I make a trigger that will generate a high alert if 4+ servers are down? I thought about doing conditional logic, as I've done with a similar 3 server setup before, but with 14 it's pretty much undoable.

[aib]

Did you ever try to use Aggregate Checks, for example grpsum?

Something (from top of my head) like that

grpsum["Citrix Servers","icmpping",last,0]

And then create three triggers:
- grpsum = 0 (all servers down)
- grpsum < 11 (more than 3 servers down)
- grpsum < 14 (at least 1 server down)

Also you can control not the last value but average for period of time (for example, average value for last 5 minutes (300 sec))

grpavg["Citrix Servers","icmpping",avg,300]