Zabbix proxy not switching to the active HA node

[nioh]

Hi,

Im trying the HA feature for Zabbix server. Right now I have a cluster setup.

Node 1: zabbic-server-c1
Node 2: zabbic-server-c2

I have a proxy installed using helm in an Azure Kubernetes Cluster.
It reports successfully to the active server and i can collect data about my K8S cluster.

However when i shut down Node 1, the second node takes over as active but the proxy is lost and start being reported as offline.
When i take Node one online again and change it to the active node, proxy goes online.

I have this configured in the proxy configuration file:
- name: ZBX_SERVER_HOST
value: "zabbix-server-c1.example.com:10051 ;zabbix-server-c2.example.com:10051"

If i change the value and put the node 2 first, then the proxy will only report to node 2.
Seems it only reports to the first node in the value.

Is there any configuration i have missed or is this an expected behavior?

[markosa]

You have space before semicolon,is that copy-paste problem or is it in configuration? It would help if you would show all active lines(not starting with #) within this(remember sanitize). For example using grep -v "#" /etc/zabbix/zabbix_proxy.conf, in our system it returns like this:
Server=1.1.1.1;2.2.2.2;3.3.3.3

HostnameItem=system.hostname


LogFile=/var/log/zabbix/zabbix_proxy.log

LogFileSize=0

EnableRemoteCommands=1

PidFile=/run/zabbix/zabbix_proxy.pid

SocketDir=/run/zabbix


DBName=zabbix_proxy

[nioh]

Hi!
Hmm could be it, there is seem to be a space in the configuration.
Here is the config from within the zabbix proxy pod:
ProxyMode=0
Server=zabbix-server-c1.bravedesert-7cf30aab.swedencentral.azurecontainerapps.io:10051 ;zabbix-server-c2.reddesert-b01b9526.norwayeast.azurecontainerapps.io:10051
Hostname=zabbix-aks03-proxy
LogType=console
DebugLevel=3
SocketDir=/tmp/
DBName=/var/lib/zabbix/db_data/zabbix-aks03-proxy.sqlite
ProxyConfigFrequency=10
CacheSize=128M
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/sbin/fping
SSHKeyLocation=/var/lib/zabbix/ssh_keys
User=zabbix
SSLCertLocation=/var/lib/zabbix/ssl/certs/
SSLKeyLocation=/var/lib/zabbix/ssl/keys/
SSLCALocation=/var/lib/zabbix/ssl/ssl_ca/
LoadModulePath=/var/lib/zabbix/modules/

Here is the helm variables (no space):

[nioh]

Just upgraded to 7.0.0 in hopes this would get resolved but it seems to be the same. Zabbix Proxy logs says the following:


Seems the proxy does not even try to send data to the second node.

[nioh]

As soon as I enable the C1 node and make it primary it starts working again and the proxy goes online.

[shaunmaher]

Ah, good! If this issue is happening to someone other than just me, it's (maybe) not something I'm doing wrong.

My "Server=" line doesn't have any spaces:

Server=portainer-az-enwst1.internal.cloudapp.net;portainer-az-enwst2.internal.cloudapp.net

I have tried with specifying the ports and without (we're using the default port anyway).

If I change the order of the servers on the configuration line, so that the "active" server is first, it works.

If I force the server that is mentioned first in the configuration to become active, the proxy starts working.

If, once the proxy is successfully connected to a server, the active server changes, the proxy follows. This is an issue only when the proxy is starting and making it's first call out to a server.

Cheers.
Shaun.

[nioh]

Ah, good! If this issue is happening to someone other than just me, it's (maybe) not something I'm doing wrong.

My "Server=" line doesn't have any spaces:

Server=portainer-az-enwst1.internal.cloudapp.net;portainer-az-enwst2.internal.cloudapp.net

I have tried with specifying the ports and without (we're using the default port anyway).

If I change the order of the servers on the configuration line, so that the "active" server is first, it works.

If I force the server that is mentioned first in the configuration to become active, the proxy starts working.

If, once the proxy is successfully connected to a server, the active server changes, the proxy follows. This is an issue only when the proxy is starting and making it's first call out to a server.

Cheers.
Shaun.

I have still not been able to make this work. Could you explain more what you mean with this?

If, once the proxy is successfully connected to a server, the active server changes, the proxy follows. This is an issue only when the proxy is starting and making it's first call out to a server.

My proxy is connected to the server mentioned first in the config. If i stop that server, it does not switch to the secondary node.

[Markku]

Works here with 7.0.2

Server=192.168.7.102;192.168.7.103

Either server can be active (the other is standby), and the proxy can connect to the available server.

What kind of firewall or some other middlebox do you have between the proxy and the servers? The log texts in the screenshots above look like the proxy was able to connect to something that is not a Zabbix server. That could mess with the HA logic.

Markku

[vivetech]

Greetings of the day..

"Server=192.168.7.102;192.168.7.103

Either server can be active (the other is standby), and the proxy can connect to the available server."

This way can we have redundancy of servers also?

[Markku]

Looks like this: https://support.zabbix.com/browse/ZBX-25029

Markku

[shaunmaher]

Hi.

A quick update.

With a little more experimentation (not with 7.0.2 yet) I did notice that the issue only seems to effect proxies that are connected to the Zabbix server without TLS. I discovered the above issue while working on a proof-of-concept environment where I was not using TLS. When I switched to TLS for the production implementation, the issue did not appear.

> If, once the proxy is successfully connected to a server, the active server changes, the proxy follows. This is an issue only when the proxy is starting and making it's first call out to a server.

Sorry. Maybe I was mistaken.

Cheers.
Shaun.

[Markku]

"Server=192.168.7.102;192.168.7.103 (in the proxy configuration)"

This way can we have redundancy of servers also?

vivetech That is called Zabbix server clustering or HA (high availability): https://www.zabbix.com/documentation...epts/server/ha

Requires a shared database so that two independent Zabbix servers (and frontends) can use the same database.

Markku

[nioh]

Looks like this: https://support.zabbix.com/browse/ZBX-25029

Markku

Yep exactly this

[nioh]

Hi.

A quick update.

With a little more experimentation (not with 7.0.2 yet) I did notice that the issue only seems to effect proxies that are connected to the Zabbix server without TLS. I discovered the above issue while working on a proof-of-concept environment where I was not using TLS. When I switched to TLS for the production implementation, the issue did not appear.

> If, once the proxy is successfully connected to a server, the active server changes, the proxy follows. This is an issue only when the proxy is starting and making it's first call out to a server.

Sorry. Maybe I was mistaken.

Cheers.
Shaun.

Interesting, mine is also a test environment, had not yet setup TLS. I will try this as well.

[nioh]

Can confirm this works when enabling TLS, we are using PSK.