Zabbix server shows "Host unreachable" alerts when zabbix proxy is down.

[JustinFossy]

Hello,

In our organization , we have a Zabbix Proxy server that monitors several hosts ( arround 1000 hosts ). This Zabbix Proxy reports to a central Zabbix server.

The problem is that , when the Zabbix Proxy goes down, we receive several false alerts stating "host unreachable" on the Central Zabbix server console.

In actual scenario, the hosts are up , but it is the Zabbix proxy that is down. Due to this, the users monitoring Zabbix, get a false impression that all the 1000 hosts are down.

Is there a way we can configure so that in the Zabbix monitoring console, we only get one alert stating "Zabbix proxy down " and rest of the alerts don't appear or get suppressed.

Kindly suggest.

[JustinFossy]

Hello ,

Please help me in finding a solution for this.

Thanks in advance.

[LenR]

I've not done this, but would a trigger dependency on the proxy fix this? See http://www.zabbix.com/documentation/...s/dependencies

I think the case is zabbix - proxy - host, so per the doc, if proxy is down, don't alert on host.

[JustinFossy]

Thank you LenR, for the reply. I was totally blank , but after going through
the link you provided me , i got an idea as to how to go about it. Still have some confusion though.

1.) If i am setting the trigger dependency on Zabbix proxy, then how will i relate this trigger to all the 1000 hosts which is getting monitored by Zabbix proxy ?

Can you please provide the detailed steps as to setting the trigger dependency on Zabbix proxy to achieve this so that i can understand it better.

Initially , i thought of setting the trigger dependency on each of the hosts, but doing this on 1000 hosts, would be way to tedious.


2.) What if the hosts are really down ? Would the "Host Unreachable" trigger get activated , or is it that this trigger would get suppressed since it depends on another trigger.

Thanks in advance.

[LenR]

1: I think if you setup the trigger in a template, the dependency would be to the specific proxy, the other tests would apply to the attached host. I would make a template with the trigger, then attach it to each host defendant on the proxy. The guy down the hall would use the api, zabcon and magic I don't do :-)

2: If the proxy is down, you won't know if it's hosts are up or down, the proxy being down is a "Disaster" level alert. Now you are getting 1000 down host alerts with the down proxy alert hiding in the confusion. It the proxy is up, it will work as it does now.

I have dependencies for the (crappy) MS SCOM service we have to run on Linux. If SCOM is installed, then a listener must be active. First contact from the listener should generate a ssl cert, so if the listener is active, Z checks for that cert being present. When the cert is present, then another process should be active to gather and send to SCOM. (We use puppet and zabbix to tag-team this process)

[heaje]

What does your trigger look like for the hosts being unreachable? Does it use the nodata function? If so, that explains why all 1000 go off if the proxy goes down. Without the proxy up, no data goes into the item and that causes the alert to go off after whatever time period is given in the trigger definition.

As for how to resolve the issue (besides the method already described in the thread), I'm not aware of another way. It would be nice if the zabbix[host,<type>,available] internal check worked properly on proxies, then this whole issue could be avoided.

[JustinFossy]

Hi Heaje,

We have defined the trigger as {hostname of server :agent.ping.nodata(300)}=1} . Any suggestions ,as to what needs to be changed here .

Thanks in advance.

[heaje]

Yup, you're using the nodata() function. That's why all your hosts show as offline when the proxy goes down. The Zabbix server doesn't see any data show up for however long your Proxy is down.

In any case, probably the only thing you can do would be something along the lines of what LenR said. You could create a template with the agent.ping item (and the same trigger definition as what you already have). In that template, make the trigger dependent on the proxy up/down check that already exists for the proxy host. Then link that template to each host behind that proxy. In effect, this would prevent all the extra alerts from going off.

[JustinFossy]

Thank you LenR and Heaje for your help on this topic. I had this problem going on for quite some time in our organization .

The solution which you have provided has given me a clear picture as to how this problem can be resolved.

[JustinFossy]

Hello LenR/Heaje


If i am creating a new Template with an Item agent.ping ,it's trigger and creating the dependency trigger, do i need to remove the agent.ping trigger which is already present in each of these hosts. ?

Would the ping trigger which is already present in each of these hosts interfere with the trigger which i will create in the template.

[LenR]

Is the present agent.ping item in a template? If so, just disable it in the template, it will replicate down to each host. Test the new trigger, then you can probably delete the old one.

[JustinFossy]

Hello LenR,

I will disable it, create a new template with the "agent.ping" and its corresponding trigger ( along with the trigger deendency ) .

But , since this template is attached to each of the 1000 hosts, we will again get 1000 alerts on the console, but this time , the alert will be "Zabbix proxy is down" instead of "Hosts unreachable " .

Please correct me if i am wrong ..

Thanks in advance,

[LenR]

Try one with a trigger dependency. Trigger a depends on b. Fail A, you should get an alert. Fail B, no alert fom this trigger at least, then fail A, firsl no alert from this trigger.

Always develop in a small test environment and and test until you understand everything. Only after that migrate to production.

[PirateZebedee]

Some Ideas

Can you cluster a zabbix proxy?, if one box is down then the monitoring still happens via the other box!

If you put the proxy into maintance mode does this put all the montored systems into maintance as well?

[heaje]

1. I could be wrong, but I don't believe Zabbix has any built-in functionality for fail-over. If it does, I would love to learn about it. Failover can be done with third-party implementations though.
2. No. Putting a proxy in maintenance does not put anything else in maintenance.