Recovery expression not working as expected

[comboloid]

Hello!!

I am trying to monitor our machines and their downtimes in a more effective way, so i created the following trigger which checks if the agent is not responding or load is over 300. The problem with this, is that recovery expression i am using very often takes 1-2 hours to be recognized from zabbix as resolved.


For example check the following machine. It has a CPU spike for some minutes and the load dropped down within 10 minutes



But the trigger actually recovered on it's own after 2 hours, which is totally misleading for our stats.



Do you have any idea that might be causing this or how we can debug this further?

[cyber]

ok .. lets debug your expressions ... How long is your {$AGENT.TIMEOUT}`?
"agent is not available for X amount of time" and " second to last value of cpu load is over or equal to 300"
recovery ... "3rd value from "now" should be less or equal to 200"

First .. you probably missed the "#N" usage of last() is different from other functions... it is not N values but "the Nth most recent value"
Second, I think you host availability just keeps it open... even if your load values are already back to normal and even 2 or 3 values back should show ok... then maybe availability is still not OK? Look for both items values when trigger is active....

[comboloid]

hey cyber, {$AGENT.TIMEOUT} is set to 1m.

The weird part of this is that this setup when used to check only load it works as expected and the recovery expression works as expected.



then maybe availability is still not OK? Look for both items values when trigger is active....

Zabbix agent is available in all these cases. I am only thinking if Zabbix gets somehow confused or buggy when combining two triggers and using recovery expression only for one of them?

[markfree]

To me, it doesn't make much sense to mix these two items in the same trigger.
As Cyber already mentioned, when you use the "#num" parameter, you are actually evaluating the "#Nth" most recent value, not the last one.
This means that your trigger will always evaluate values from previous interval cycles, not the most recent item cycle.

Anyway, how long is your item interval?

[comboloid]

hey Mark!!

The reasoning behind these two items in a single trigger, is i want to calculate the real downtime of our machines, saying when the agent is not available and/or load is >300 seems quite accurate. So the idea is to have a single trigger about it, which will autoresolve when load is less than 200 and monitor the results of the trigger. If you have any other better idea on how to achieve this i am still searching for alternatives

The interval for system.cpu.load[all,avg1] is 1min. The reason for using the "#num" parameter is to let 1-2 minutes pass before checking that there real is a problem and avoid false positives.

thanks!

[cyber]

Zabbix agent is available in all these cases. I am only thinking if Zabbix gets somehow confused or buggy when combining two triggers and using recovery expression only for one of them?

Recovery expression is additional expression that has to be TRUE after your initial trigger expression is already turned FALSE. I guess if your expression works OK without availability item, then that one is the culprit here, which does not allow it to turn false in time..

[comboloid]

hey cyber thanks for the help. I managed to sort this out with your help.

The problem was that despite in the recovery expression only load was ok to turn true, Zabbix was anyway looking for data regarding agent availability to proceed. And i have been using the "discard unchanged with heartbeat 3 hours" option for the agent, meaning that when unchanged, Zabbix had to wait at max 3 hours to get a new update about it to proceed with the recovery or not. I totally removed the option of heartbeat and since then it started working as expected.