Trigger Dependencies - Zabbix Proxy Down

[pxm_cr]

Hi all,

We would like to adjust our environment to support the following scenario, when the Zabbix Proxy is down, we don't want to receive notifications that all the monitored hosts are being down or inaccessible.

We have managed to address this first part with the use of trigger dependencies, so when the Zabbix Proxy is down, we only receive a single email stating that the proxy is indeed down, so far so good. However, when the Zabbix Proxy becomes accessible again, all the notifications from the monitored hosts are being sent in a spammed fashion. It's like all those notifications are being generated the first time when the Zabbix Proxy is down, stored somewhere temporarily and then when the Zabbix Proxy is back up online, the notifications are being released.

Is anyone experiencing the same issue? Any thoughts on how it can be resolved?

Thank you in advance.

[batchenr]

i have 2 advice's :

1. check zabbix logs at :
Reports -> Action log

2.are they recovery messages ? if so you can delete them.

[pxm_cr]

i have 2 advice's :

1. check zabbix logs at :
Reports -> Action log

2.are they recovery messages ? if so you can delete them.

Hi batchenr,

Thank you for your prompt reply.

Sure, I can delete them, but wouldn't those recovery messages be generated again the next time the Zabbix Proxy is down? I am mostly interested in a permanent solution where the Zabbix Proxy loses connectivity, a single email notification informs you so and when the it comes back again, nothing else to be sent apart from a recovery message from the Zabbix Proxy itself.

[Pada]

I'm not sure if there is a better way of doing it with Zabbix versions newer than 1.8, but here is our elaborate approach we started using in v1.8:

We have a base template called "Template_Host_via_Zabbix_Proxy", where the template includes as single "Zabbix internal" item:

Description: {$PROXY_HOSTNAME}: last access time
Key: zabbix[proxy,{$PROXY_HOSTNAME},lastaccess]
Update Interval: 30s

And a trigger:

Name: {$PROXY_NAME} not connected for 30 seconds
Expression:

({Template_Host_via_Zabbix_Proxy:zabbix[proxy,{$PROXY_HOSTNAME},lastaccess].fuzzytime(30)}=0)|(({TRIGGER.VALUE}=1)&(    (({Template_Host_via_Zabbix_Proxy:zabbix[proxy,{$PROXY_HOSTNAME},lastaccess].avg(#11)}-{Template_Host_via_Zabbix_Proxy:zabbix[proxy,{$PROXY_HOSTNAME},lastaccess].last(#11)}-150)>2)|(({Template_Host_via_Zabbix_Proxy:zabbix[proxy,{$PROXY_HOSTNAME},lastaccess].avg(#11)}-{Template_Host_via_Zabbix_Proxy:zabbix[proxy,{$PROXY_HOSTNAME},lastaccess].last(#11)}-150)<-2)))

Severity: Information
Comments: This trigger would only recover 10min after the proxy came back alive

Then also fill in the user macro for:

{$PROXY_HOSTNAME}=dummyhostname

The reason for having the trigger on "Informational" is because we never send notifications out on "Informational" severity level events.

We then create a new template for each proxy that we have, linked with our Template_Host_via_Zabbix_Proxy template, where we then override the user macro with the actual proxy's hostname.
eg. A template with the name: "Template_Host_via_Zabbix_Proxy_Africa" and macro {$PROXY_HOSTNAME}=zabbixproxy-africa.example.com

Then for the templates that we apply to our hosts, we create a new template of them for each proxy that we have, and then link both the Template_Host_via_Zabbix_Proxy_<associated proxy> and the <original template>
For example if our original template was named "Template_Apache_httpd", we'd now create 2 called say:
Template_Apache_httpd_via_Zabbix_Proxy_Africa & Template_Apache_httpd_via_Zabbix_Proxy_EU
The Africa one has 2 templates linked: Template_Host_via_Zabbix_Proxy_Africa & Template_Apache_httpd
The EU one has 2 templates linked:Template_Apache_httpd_via_Zabbix_Proxy_EU & Template_Apache_httpd
It is important to then go and edit the triggers on each of these new templates, so that the triggers that would've previously caused alerts when the proxy went down, to now depend on the "{$PROXY_NAME} not connected for 30 seconds" trigger.
In our setup, its basically just the triggers that had ".nodata" or ".fuzzytime" in them that we had to change, but yours may include others as well.

We also had to increase the duration of our .nodata and .fuzzytime expressions, so that its slower than the time it takes to detect that the proxy is down, which is detected by our "{$PROXY_NAME} not connected for 30 seconds" trigger.
If you don't do this, you end up having your triggers sending out alerts before you detected that the proxy is down.

I really hope there is a better way of doing this in Zabbix 3.2, since we're now in a process of upgrading to it.

[Pada]

I see with Zabbix 3.2 (perhaps earlier versions too) it is possible to add a Trigger dependency from a Trigger in a template to depend on a Trigger on a host, which resulted in the following error in Zabbix 1.8:

Error adding dependencies. Following dependencies are not allowed: 1) from a host trigger to a template trigger; 2) from a template trigger to a host trigger; 3) between template triggers, where templates have links to other templates.

It looks like since 2.0 they have improved on how triggers may depend between hosts and templates, and it is pretty well defined in terms of the limitations. See for example the Trigger dependencies docs for 3.2: https://www.zabbix.com/documentation...s/dependencies

So now its not necessary to add an Internal Item or Trigger to every host that is monitored by a Zabbix proxy any longer, but you may still want to have a custom templates per Zabbix proxy where their connectivity triggers then depend on the Zabbix Internal item that tracks the specific Zabbix proxy's lastaccess, which you can add to your Zabbix server host.

[pxm_cr]

I see with Zabbix 3.2 (perhaps earlier versions too) it is possible to add a Trigger dependency from a Trigger in a template to depend on a Trigger on a host, which resulted in the following error in Zabbix 1.8:


It looks like since 2.0 they have improved on how triggers may depend between hosts and templates, and it is pretty well defined in terms of the limitations. See for example the Trigger dependencies docs for 3.2: https://www.zabbix.com/documentation...s/dependencies

So now its not necessary to add an Internal Item or Trigger to every host that is monitored by a Zabbix proxy any longer, but you may still want to have a custom templates per Zabbix proxy where their connectivity triggers then depend on the Zabbix Internal item that tracks the specific Zabbix proxy's lastaccess, which you can add to your Zabbix server host.

Indeed, we have updated to 3.2 and testing the new implementation. Thank you all for your input, much appreciated.

[Pada]

I'm definitely looking for something better than my proposed solution, because we had an issue yesterday where our link between our Zabbix proxies and Zabbix server was too slow, which resulted in:
1) our "{$PROXY_NAME} not connected for 30 seconds" trigger not going into a problem state, since the link was still up
2) like 150 alerts being sent out for like "Missing Zabbix Agent data for past 5 minutes"

I'll see if I can perhaps combine the the checks with the Zabbix Internal item "zabbix[proxy_history]" when the Proxy backlog grew too much.
Unfortunately I cannot seem to get that "zabbix[proxy_history]" item to work!

[Trumpfz]

Hi all,

We would like to adjust our environment to support the following scenario, when the Zabbix Proxy is down, we don't want to receive notifications that all the monitored hosts are being down or inaccessible.

Hello,

I would like to get this notification when the Proxy is down, but unfortunately I don't get them and all the Hosts from this proxy are "online" even if the proxy is definately offline.

Scenario is like this:
Zabbix Server is at a remote location. The Zabbix-Proxy is working in active mode on a hypervisor and is monitoring through zabbix-agent itself and some virtual machines running on the hypervisor.

When I turn on the Hypervisor, all Hosts come up and "Availability" switches from red to green, just as it should be. But when I turn off the Hypervisor, all virtual machines stay green in availabilty. The Zabbix server just seems to be waiting forever to get a signal from the proxy that the hosts are down, never gets one and they stay up.

I bet there is a simple solution to this, but it seems I just can not find it since days.

[[email protected]]

Hello,
Sorry for unearthing a topic that old. But did you get a solution to your issue ? I'm actually very interrested in the exact same scenario.