Zabbix Trigger Logic

[SamuelD]

I would like to implement this structure to the way my triggers occur. I have tried many options but it never seems to work.

Trigger name:
FMS Master is Down

Expression:
last(/LINUX FMS/system.run[pgrep -x fmshelper | wc -l])=0
and
last(/LINUX FMS/system.uptime)>60m

Trigger name:
Scripting Engine is Down

Expression:
last(/LINUX FMS/system.run[pgrep -x fmsased | wc -l])=0
and
last(/LINUX FMS/system.uptime)>60m
and
last(/LINUX FMS/system.run[pgrep -x fmshelper | wc -l])=1


I tried to make it that it gives it an hour before it says something but it still sends all the messages to the users when the server is restarted. We don't want it to send us these triggers unless the system hasn't been restarted, because its already implied these processes are down. If FMS Helper Service goes down, all FMS sub-processes go down with it. Nginx does not, but we do not want Nginx to only alert us if it down, not if the system restarted, causing it to be down with it. If anyone could please help, that would be very lovely!


Here is my current infrastructure.

[rozols]

Hi,
The issue is a race condition: during a restart, the service actually stops a few seconds before the OS finishes rebooting. At that exact moment, your process check correctly reads 0, but system.uptime hasn not reset yet — so last(pgrep)=0 AND last(uptime)>60m is still true right then, and the alert fires before uptime ever drops. The >60m condition can't catch this because the two facts do not happen at the same instant.
To fix this you need to use a dependency trigger as a suppression gate, and require sustained downtime instead of a single bad poll.

1. A dedicated just rebooted trigger - no action/notification attached, it only exists to know if server rebooted recently and help suppress other triggers.
Name: Recently rebooted
Expression: last(/LINUX FMS/system.uptime)<60m
If uptime is less than 60 min this will fire and provide dependency for next triggers not to fire. As for actions dont use this trigger for notifications and as for monitoring you can set this to be not classified and and in monitoring dont use this severity.

2. FMS Master, using min() instead of last() so a one bad poll cant trip it.
Name: FMS Master is Down
Expression: min(/LINUX FMS/system.run[pgrep -x fmshelper | wc -l],30m)=0
Dependency: Recently rebooted
If minimal value for fmshelper over last 30 min is equal to 0 and server is not recently rebooted the alarm will fire for FMS Master is Down, this will be a real issue and can be used for actions and notifications.

3. Scripting Engine matches your point about sub-processes going down with FMS Helper.
Name: Scripting Engine is Down
Expression: min(/LINUX FMS/system.run[pgrep -x fmsased | wc -l],30m)=0
Dependencies: Recently rebooted
FMS Master is Down
Do the same for any other FMS sub-process. Since they depend on both the reboot trigger and the master trigger, you get one alert (Master) instead of a storm when it takes everything else down with it.

4. Nginx stays independent of FMS Master, but still respects the reboot grace:
Name: Nginx is Down
Expression: last(/LINUX FMS/net.tcp.service[http,"{$NGINX.STUB_STATUS.HOST}","{$NGINX.STUB_STA TUS.PORT}"])=0
Dependency: Recently rebooted

After tests on your end you can drop the AND last(uptime)>60m clause from every individual trigger now — the "Recently rebooted" dependency does that job centrally, so if you ever want to change the grace period you only edit one trigger instead of hunting through all of them.

I tested this exact structure in a lab (fake processes + systemd, real reboot cycle) no false alert fired during restart, and a real outage still alerts correctly after the sustained-downtime window. Recommend testing this in a disposable VM before deploying: make sure your system.run[...] items are set to Numeric (unsigned) type, not Text, or min() will reject them; and if you are on Zabbix agent 2, system.run may be blocked by default unless you add an AllowKey rule for it.