We use a script here to push out code releases to our servers. During the push to a server, several outages occur which triggers actions in Zabbix as services are stopped and restarted and CPU load spikes, etc.
In order to avoid these false alarms I have extended our release script to create a maintenance period, then add/remove hosts as the script performs the upgrades on each host in turn.
I'm hoping to come up with an elegant way to shorten the latency between updating the maintenance period via the frontend (API) and when the back end server recognizes the change. Right now I just have the script sleep for a minute to allow for the back-end to (I presume) poll the database and learn of the changes. Without this delay a few alarms slip out at the beginning.
Is there an approved, supported, or simply effective way that I can reduce the delay in this process? Adding a delay of one minute per host to our software releases really adds up to significant time as the number of hosts grows.
If there's not a way to do it in Zabbix, my next best plan is to just start manipulating the maintenance window for a host at some point during the previous host's processing, but I don't really look forward to that complexity and confusion in what is otherwise a pretty straightforward piece of code.
Anyone have any clever ideas?
In order to avoid these false alarms I have extended our release script to create a maintenance period, then add/remove hosts as the script performs the upgrades on each host in turn.
I'm hoping to come up with an elegant way to shorten the latency between updating the maintenance period via the frontend (API) and when the back end server recognizes the change. Right now I just have the script sleep for a minute to allow for the back-end to (I presume) poll the database and learn of the changes. Without this delay a few alarms slip out at the beginning.
Is there an approved, supported, or simply effective way that I can reduce the delay in this process? Adding a delay of one minute per host to our software releases really adds up to significant time as the number of hosts grows.
If there's not a way to do it in Zabbix, my next best plan is to just start manipulating the maintenance window for a host at some point during the previous host's processing, but I don't really look forward to that complexity and confusion in what is otherwise a pretty straightforward piece of code.
Anyone have any clever ideas?