High Availability for Proxy

[r.riva]

Good Morning to all !

I have the following infrastructure:
- 2 MySQL Database server in sync for all MySQL database
- 2 Zabbix instances in HA for High Availability

I need to create "n" (from 5 to 10) Zabbix proxy to distribute the load for checking a couple of thousands of devices with some check on each device.

Which will be the best way to create an high availability for Zabbix Proxies ?
What I would like to obtain is that one proxy became unreachable it will be possible to have the host configured on it to be checked from another proxy.

I know I'll could do it manually by a "Massive Move" but I wuold like to have an automatic procedure.

Thank you all in advance
Regards
Riccardo

[onallion]

If you configure all proxy server addresses in your agents, and make your checks Active, maybe that would be okay for your scenario?

[kloczek]

Which will be the best way to create an high availability for Zabbix Proxies ?
What I would like to obtain is that one proxy became unreachable it will be possible to have the host configured on it to be checked from another proxy.

Base filaments of zabbix stack HA are standing on two things:
- monitoring all possible metrics over proxies (only metrics which you cannot monitor over the proxies are zabbix server internal metrics)
- securing main zabbix server database

With above you can:
- schedule zabbix server maintenance window as long as it will be not longer than proxies ProxyOfflineBuffer
- if you are doing regular main zabbix server database backups (for example by switch DB backend to read only mode -> flush all not written data to storage -> make for example btrfs or zfs snapshot -> switch back DB in R/W mode) on the proxy you need still to preserve ProxyLocalBuffer at least equal to your DB backup cycle
- On agents side you need to preserve small buffer which could allow hold monitoring data in case when proxy will be lost/temporarily not available to hold unsynced data enough time allowing to switch to new physical location proxy process with DB backend with proxy database with only imported database scheme.
- As long as all monitored metrics are monitored over proxies temporary switching off zabbix server has only one impact: you have no alarming layer. All monitoring data will be still collected on the proxies and when srv<>prx connectivity will be restored all pending data will be pushed to srv.

All above will be working with minimal losses of the data if all agents will be active agents and proxies as well. It can work as well with passive proxies as well as long as it will be relatively short period of pushing cfg data to proxies (like <=60s). Firs what does active proxy i request cfg data from server. After receive those data proxy is able to process all agents requests without additional losses.

It is really not possible to reduce loses of the data in whole stack to zero ..
All JMX, SNMP, VmWare metrics allways will have some loses because all those metrics arr probed/initiated from proxy. This is why using for example ODBC interface is not so useful as it should be in real case scenarios because better is organize DB monitoring on top of agent without zabbix ODBC keys (using template which is using loadable module or not by simple query DB metrics using standalone DB cli client).

You can raise significantly HA of whole zabbix stack if you your zabbix server DB backend will have slave DB. In case loosing main zabbix DB all what is necessary to do is promote slave as new master and restart zabbix server process and recreate ASAP (even from scratch) new slave. In such case nothing will be lost because server will resync all missing data from proxies very/relatively quickly.
Additionally slave will have yet another propose. It can be used as a point in which you are doing regular DB backups. Whatever backup procedure you will be using it can even saturate 100% bytes/IOs bandwidth to storage and it will be null performance impact on zabbix server<>DB backend interaction.

I'll try to rephrase your question: "do I need strong HA on proxies layer?"
Answer is: no you don't need all this.

[r.riva]

If you configure all proxy server addresses in your agents, and make your checks Active, maybe that would be okay for your scenario?

Thank you for your answer but,
If I'll configure all proxy in my agents, wich will be the scenario ?
All proxy makes all the check to all my devices ?
Or there will be a sort of balancing between proxies because when they have to make the check "A" to the Device "1" the command will be executed only from one proxy at a time ?

Excuse my ignorance ;-)

[jjahn]

what is the recommended configuration when one uses multiple proxies?

So is there a recommended configuration for both hosts and proxy when one uses multiple proxies?

If for example one configures each host running a proxy process with the same "Hostname" value (e.g. Proxy1) and then have one proxy of the same name in the ZBX server...is this even valid or will this cause data conflicts?

If for example one configures each host running a proxy process with the different "Hostname" values (e.g. Proxy1, Proxy2, ... ProxyN) and then have multiple proxies using these names in the ZBX server...how will a host (which can only be configured with one proxy to use in the ZBX server) fail-over to some other proxy of the proxy it is using fails?

[banzayats]

...
I'll try to rephrase your question: "do I need strong HA on proxies layer?"
Answer is: no you don't need all this.

I can not agree with you.
First, I do not like lots of junk in logs:

53867:20170727:164109.534 cannot send list of active checks to ...
53867:20170727:164109.975 cannot send list of active checks to ...

They appear if zabbix agent tries to send active checks to all proxies. But since only one is specified in the settings, all the others will output such messages.
Therefore, I prefer to specify only one proxy for Server/ServerActive option. It also allows the zabbix_sender to send data to the desired proxy, since it uses the first entry from the ServerActive parameter.

But this solution has a drawback - if we lose any of the proxy, then we lose all data about the hosts monitored by it.
To solve this isuue we can built HA active-standby cluster. But this is a little bit complicated and we will depend on one more component.

Another soluton is to use Zabbix internal tools: alerting and API. If one proxy is down - Zabbix will activate an action which changes the "Monitored by proxy" option. Within two minutes, the proxy updates the parameters and starts polling the hosts. At this time, a small script runs on hosts (every minute), which receives a new proxy address, updates the config and restarts zabbix agent. The maximum downtime is about 2 minutes. This is not optimal solution, but it works.