I have found that false alerts are somewhat common when you have servers scattered all over the place. Sometimes certain routing goes down and you get an alert but it's only for routing from where zabbix is being hosted to where the server is you are monitoring. Sometimes it's because the zabbix server cannot receive the data for whatever reason. Those are the worst because if you are monitoring hundreds of servers you will get hundreds of false alerts.
How do people avoid this? I have gone to extraordinary extremes by having 2 zabbix servers in separate datacenters monitoring all my servers. So I know it's a false alert if only one zabbix server alerts me. The other thing I have done, which should be a built in macro or something, is only alert if the server cannot contact some fixed reference in addition to the server. In my case I am pinging google DNS server 8.8.8.8. Zabbix will only alert if the Google DNS server is still pingable but the server I am monitoring is not.
Seems to me there should be a better way where I don't need to maintain 2 zabbix servers and don't need every trigger having to check if it can contact external reference points in addition to the server.
How do people avoid this? I have gone to extraordinary extremes by having 2 zabbix servers in separate datacenters monitoring all my servers. So I know it's a false alert if only one zabbix server alerts me. The other thing I have done, which should be a built in macro or something, is only alert if the server cannot contact some fixed reference in addition to the server. In my case I am pinging google DNS server 8.8.8.8. Zabbix will only alert if the Google DNS server is still pingable but the server I am monitoring is not.
Seems to me there should be a better way where I don't need to maintain 2 zabbix servers and don't need every trigger having to check if it can contact external reference points in addition to the server.