Our current solution is simple enough. We run a script on the target to see if any filesystem is too full, and report WARNING or ERROR based on a local config file. The filesystem name and usage is propagated to the action, and our Operations staff uses the notes from the action email to determine the escalation.
This works okay, but there are a few things that worry me. If a second filesystem should fill up when we have an active alert, we can't notify the second team. (it's the same trigger/action that already went off) We also have room for human error when handling the alert that I would like to remove.
We have enough systems with non-standard configurations that I can't soften the blow with templates. Besides, each one may need different trigger levels and/or escalations.
The only way that I can currently see to have this work is to configure each filesystem on each host with it's own item, trigger, and action. That way we can adjust thresholds (on the trigger) and escalation information (on the action).
Of course, to maintain my sanity, I will need to use the API to create all of the entries, and I'm going to need to make a script to cross-check between the systems and the alerts. Otherwise eventually someone will miss creating an alert and we won't notice until something bad happens.
I would love to be able to specify the thresholds and the escalation group (or short note) in a config file on the client. Something like:
FS:WARN:CRITICAL:GROUP:notes
/apps/app1:90:95:"APP Owner 1":
/apps/app2:90:105:"APP Owner 2": no one is hurt if this fills up, so don't ERROR just WARN
/apps/app3:50:60:"APP Owner 3":Must keep lots of space free for this app.
Is there some special option I missed that would let me do this the easy way (client script) instead of making an avalanche full of individual snowflakes?
This works okay, but there are a few things that worry me. If a second filesystem should fill up when we have an active alert, we can't notify the second team. (it's the same trigger/action that already went off) We also have room for human error when handling the alert that I would like to remove.
We have enough systems with non-standard configurations that I can't soften the blow with templates. Besides, each one may need different trigger levels and/or escalations.
The only way that I can currently see to have this work is to configure each filesystem on each host with it's own item, trigger, and action. That way we can adjust thresholds (on the trigger) and escalation information (on the action).
Of course, to maintain my sanity, I will need to use the API to create all of the entries, and I'm going to need to make a script to cross-check between the systems and the alerts. Otherwise eventually someone will miss creating an alert and we won't notice until something bad happens.
I would love to be able to specify the thresholds and the escalation group (or short note) in a config file on the client. Something like:
FS:WARN:CRITICAL:GROUP:notes
/apps/app1:90:95:"APP Owner 1":
/apps/app2:90:105:"APP Owner 2": no one is hurt if this fills up, so don't ERROR just WARN
/apps/app3:50:60:"APP Owner 3":Must keep lots of space free for this app.
Is there some special option I missed that would let me do this the easy way (client script) instead of making an avalanche full of individual snowflakes?