Zabbix HA cluster, inactive server UI says Connection to Zabbix server refused

[c.h.]

We've set up two hosts running Zabbix Server 6.2.6 in HA mode, using nginx, patroni, pgbouncer, haproxy, and postgres 15.

On the active server, port 10051 is open; on the standby, it's closed. This is as expected, I think.

On the active server, the application looks correct; on the standby, the top-right pane (system information) on the default dashboard says

Connection to Zabbix server "127.0.0.1:10051" refused. Possible reasons:
1. Incorrect "NodeAddress" or "ListenPort" in the "zabbix_server.conf" or server IP/DNS override in the "zabbix.conf.php";
2. Security environment (for example, SELinux) is blocking the connection;
3. Zabbix server daemon not running;
4. Firewall is blocking TCP connection.
Connection refused

I'm looking at https://www.zabbix.com/documentation...epts/server/ha, and it says

• NodeAddress parameter must be specified for each node.

The NodeAddress parameter (address/port) will be used by Zabbix frontend to connect to the active server node. NodeAddress must match the IP or FQDN name of the respective Zabbix server.

Currently it's set to localhost:10051, but that appears to be the wrong answer.

What should it be? It certainly shouldn't be the IP or FQDN of the active Zabbix server, since the active Zabbix server will change as dictated by the HA process.

Thanks!

[Markku]

NodeAddress should be set to the IP address of the server, that's the address that the frontend will use to connect to the server. As mentioned in the page you linked:

Zabbix frontend will autodetect the active node by reading settings from the nodes table in Zabbix database. Node address of the active node will be used as the Zabbix server address.

That's why the ZBX_SERVER settings in the frontend are set to empty.

Markku

[c.h.]

I'm truly confused.
When you say "NodeAddress should be set to the IP address of the server" -- which server? The active one, or the standby one? The localhost one? The remote one? Do I have to edit /etc/zabbix/zabbix_server.conf, update NodeAddress and restart the server every time the active node changes?

When the documentation says "Zabbix frontend will autodetect the active node by reading settings from the nodes table in the Zabbix database", it makes me wonder "why is NodeAddress in the config file?"

The /etc/zabbix/zabbix_server.conf file says this:

## Option: NodeAddress
#       IP or hostname with optional port to specify how frontend should connect to the server.
#       Format: <address>[:<port>]
#
#       If IP or hostname is not set, then ListenIP value will be used. In case ListenIP is not set, localhost will be used.
#       If port is not set, then ListenPort value will be used. In case ListenPort is not set, 10051 will be used.
#       This option can be overridden by address specified in frontend configuration.
#
# Mandatory: no
# Default:
# NodeAddress=localhost:10051
​

The /etc/zabbix/web/zabbix.conf.php file says this:

// Uncomment and set to desired values to override Zabbix hostname/IP and port.
// $ZBX_SERVER                  = '';
// $ZBX_SERVER_PORT             = '';

I feel that I'm doing all the right things, but it isn't working. What am I overlooking/misunderstanding?

[Markku]

When you say "NodeAddress should be set to the IP address of the server" -- which server?

That server whose zabbix_server.conf you are editing.

Markku

[c.h.]

So, 127.0.0.1 is good or bad?

[Markku]

When the documentation says "Zabbix frontend will autodetect the active node by reading settings from the nodes table in the Zabbix database", it makes me wonder "why is NodeAddress in the config file?"

Because the server process will write the NodeAddress value in the nodes tables in the database. (Yes, in theory the server could try to autodetect its IP address without the NodeAddress configuration but that would be highly unreliable in many cases, that's why the NodeAddress must be manually set by the administrator.)

And it's the database from where the frontend finds out who is the currently active server and is then able to connect to the correct server.

Markku

[Markku]

So, 127.0.0.1 is good or bad?

Bad because the frontend cannot use that to connect to the remote server.

Set it to the real reachable IP address of the server. (Edit: Actually you can use FQDN instead if the server has one, as mentioned by the documentation.)

Markku

[c.h.]

Thank you, now it's working!

The missing piece in my understanding was filled in when I changed the NodeAddress value to 10.1.2.3 on the standby server and restarted it; surprisingly, the error message still said "cannot connect to 127.0.0.1:10051".

The NodeAddress value isn't the ip address to connect to now, it's the ip address that's being written into the Nodes table for this node, which will be used by the frontends on all of the Zabbix servers when this node eventually becomes active.

I then edited /etc/zabbix/zabbix_server.conf file on the other Zabbix server in this HA cluster, setting the NodeAddress value to 10.5.2.3, and restarted it.

Once I'd done that, the error messages went away.

Please, change the documentation a little, just to say that all of the frontends in the HA cluster will try connecting to this NodeAddress when this node switches from standby to active.

Also, make it mandatory, and make it an error to set it to 127.0.0.1 or localhost, because they will never work. Having them default to localhost the way they are now is not a good idea.

Also, HANodeName seems to be unnecessary, since NodeAddress is what frontends will connect to. The HANodeName and NodeAddress together sort-of make up an entry in a simple /etc/hosts file, but the server doesn't need the name, it's cosmetic. Or, the HANodeName could be shown in the 'system information' pane ("Zabbix server is running: Yes 10.1.2.3:10051 (HANodeName)").

Thank you for helping me solve my problem!

[somesh_k]

Hi c.h. , I have a same situation and switchover from standby to active is working properly but not able to open the GUI using standbyIP....Can you please provide the solution.
Just a thought do we need the file /etc/zabbix/web/zabbix.conf.php on standby node as well

[cyber]

Why you should be able to connect to STANDBY node? Connect to active one...

This HA setup where people run 2 all-in-one hosts side by side ... please understand HA works only for Server component.. not for DB or GUI... Those should be separated...

[somesh_k]

Hi cyber ,Yes I understood that HA works for server component only, please see below query.


scenario 1: as shown in above snap ,I am able to open the GUI using servre-01 IP i.e. the Active one.
scenario 2: Now lets consider switchover happened and server02 becomes active which was previously standby. In this case the URL should shift to server02 IP or it is remain same?

[cyber]

How is your setup actually done? Too little info... Only assumptions... and we all know that assuming will make an ass of u and me...

[somesh_k]

Ref LiNK:

https://www.youtube.com/watch?v=n7rp9OujJRY

https://www.zabbix.com/documentation/current/en/manual/concepts/server/ha

Server-01 config:
1. /etc/zabbix/zabbix_server.conf
DBHost=<databaseip>
DBName=zabbix
DBUser=zabbix
DBPassword=passwd
HANodeName=server-01
NodeAddress==XX.XX.XX.68:10051


2./etc/zabbix/zabbix_agentd.conf

Server=XX.XX.XX.68,XX.XX.XX.69
ServerActive=XX.XX.XX.68;172.XX.XX.69

3./etc/zabbix/web/zabbix.conf.php

// $ZBX_SERVER = '';
// $ZBX_SERVER_PORT = '';


Server-02 config
1. /etc/zabbix/zabbix_server.conf
DBHost=<databaseip>
DBName=zabbix
DBUser=zabbix
DBPassword=passwd
HANodeName=server-02
NodeAddress==XX.XX.XX.69:10051



2./etc/zabbix/zabbix_agentd.conf

Server=XX.XX.XX.68,XX.XX.XX.69
ServerActive=XX.XX.XX.68;XX.XX.XX.69


2./etc/zabbix/zabbix_agentd.conf

Server=XX.XX.XX.68,XX.XX.XX.69
ServerActive=XX.XX.XX.68;XX.XX.XX.69

Now when failover happen and server-02[XX.XX.XX.69] becomes active as shown below in snap1 then I can open the frontend using xx.xx.xx.68 ip and not by using XX.XX.XX.69.




Snap for frontend.(snap-2)



Now in above pic we can see the zabbix server is running on XX.XX.XX.69 but we can access the frontend using XX.XX.XX.68 only so is this the correct as per architecture or not just,want to confirm only.

[cyber]

ok .. at least you have separate DB there..
And GUI-s are installed on both hosts? What kind of error you get when connecting to that .69?
Check if httpd (or nginx) is running on both... From my point of view GUI should work on both hosts all the time, as they both use same DB. Only communication towards real server is for checking if it is up and running... and that is also selected from DB at first...

But you should just add separate host for GUI (you can build a HA version also..) no point in checking different IP-s etc... just one entry point. You have come already so far... just the last step..

[somesh_k]

yes,as per my understanding if we use seperate DB then we won't be able to achieve HA.[Ref zabbix official docs],httpd is running on 69 server. In my setup communication towards real/active is happening without any issue.

Please see below error but I can connect to db server from 69 using CLI.



Frontend error


GUI's packages are installed on Host-2