Ad Widget

**kernbug** · 18-06-2018, 07:26

Originally posted by eset

Found some subjects here but there aren't lot of discuss about that so maybe I will raised the subject : Zabbix Servers behind HAproxy in Active/Passive mode
If I may I will just leave the link to Stackoverflow where I described my problem: https://stackoverflow.com/questions/...behind-haproxy

Hello

Zabbix Server/Zabbix Proxy are stateless, you can run another instance in case of crash immediately.

Example of the more reliable schema:

How to configure a High Availability Zabbix Server using Pacemaker on CentOS 7 / RHEL 7 - YallaLabs

http://yallalabs.com/linux/how-to-configure-a-high-availability-zabbi-server-using-pacemaker-on-centos-7-rhel-7/

In this tutorial we will cover how to set up high availability Zabbix server. We are going to create a HA Active/Passive Cluster that consists of two CentOS7 / RHEL7

**kernbug** · 18-06-2018, 12:04

Hi,

By default host interface of the Zabbix Server is 127.0.0.1 and name "Zabbix server". You can choose any name and set it up in the Web Frontend and Zabbix Agent "hostname=". IP address also could be from main network interface, not only from loopback.

**eset** · 18-06-2018, 12:06

Originally posted by kernbug

Hi,

By default host interface of the Zabbix Server is 127.0.0.1 and name "Zabbix server". You can choose any name and set it up in the Web Frontend and Zabbix Agent "hostname=". IP address also could be from main network interface, not only from loopback.

could be ? So I can add Server= and ServerActive as 127.0.0.1,10.x.x.x ?

**kernbug** · 18-06-2018, 12:10

Originally posted by eset

could be ? So I can add Server= and ServerActive as 127.0.0.1,10.x.x.x ?

Why not? Local to the server Zabbix Agent will work.

**eset** · 18-06-2018, 13:07

Originally posted by kernbug

Why not? Local to the server Zabbix Agent will work.

Ok I'm asking because in zabbix_server.log I've got a lots of entries like this:

Code:

sudo tail -f /var/log/zabbix/zabbix_server.log
 11990:20180618:130038.934 [Z3005] query failed: [1062] Duplicate entry '28538-1529316000' for key 'PRIMARY' [insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (28538,1529316000,60,1529316038,1529317808,1529319578),(28658,1529316000,60,0,0,1),(28718,1529316000,60,3776,17368,28616);
]
 11990:20180618:130040.045 [Z3005] query failed: [1062] Duplicate entry '28599-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28599,1529316000,60,0.000000,0.000000,0.000000);
]
 11988:20180618:130041.121 [Z3005] query failed: [1062] Duplicate entry '28600-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28600,1529316000,60,0.000000,0.000565,0.016946),(28540,1529316000,60,100.000000,100.000000,100.000000),(28720,1529316000,60,89.032898,89.032972,89.033203);
]
 11990:20180618:130058.874 [Z3005] query failed: [1062] Duplicate entry '28618-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28618,1529316000,60,99.593544,99.594612,99.619675);
]
 11990:20180618:130059.970 [Z3005] query failed: [1062] Duplicate entry '28619-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28619,1529316000,60,2.294561,2.328483,2.542593);
]
 11987:20180618:130523.996 [Z3005] query failed: [1062] Duplicate entry '28523-1529316000' for key 'PRIMARY' [insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (28523,1529316000,6,1529005764,1529005764,1529005764);

and on second zabbix-server

Code:

sudo tail -f /var/log/zabbix/zabbix_server.log
  2648:20180618:130055.568 [Z3005] query failed: [1062] Duplicate entry '28614-1529316000' for key 'PRIMARY' [insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (28614,1529316000,60,0,0,0);
]
  2648:20180618:130055.623 [Z3005] query failed: [1062] Duplicate entry '28555-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28555,1529316000,60,0.000000,0.015979,0.958740);
]
  2648:20180618:130056.684 [Z3005] query failed: [1062] Duplicate entry '28616-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28616,1529316000,60,99.999094,99.999974,100.000000),(28556,1529316000,60,0.287211,0.349100,0.497427);
]
  2648:20180618:130057.775 [Z3005] query failed: [1062] Duplicate entry '28617-1529316000' for key 'PRIMARY' [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (28617,1529316000,60,99.775293,99.779178,99.779489),(28557,1529316000,60,0.000000,0.000565,0.016946);
]
  2643:20180618:130222.173 executing housekeeper
  2643:20180618:130222.733 housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 0.085167 sec, idle for 1 hour(s)]

It looks like the same zabbix-server entries have been duplicated.

Ok I cleared configuration on front , probably those entries above where because I add those zabbix application server wrong.
So on both zabbix applications servers (zabbix server) I added on zabbix_agent.conf:

monit-app-01 (10.156.0.10)

Code:

Server=127.0.0.1,10.156.0.10
ServerActive=127.0.0.1,10.156.0.10

monit-app-02 (10.164.0.10)

Code:

Server=127.0.0.1,10.164.0.10
ServerActive=127.0.0.1,10.164.0.10

How about when adding a node to zabbix-server so in zabbix_agentd.conf I should add it like this?

Code:

Server=172.31.255.254,10.156.0.10,10.164.0.10
ServerActive=172.31.255.254,10.156.0.10,10.164.0.10

from the left are in sequence: Floating IP, monit-app-01, monit-app-02

So it's not working properly:

On monit-app-01 I get:

Code:

 15490:20180618:134120.265 failed to accept an incoming connection: connection from "10.164.0.10" rejected, allowed hosts: "127.0.0.1,10.156.0.10"

where 10.164.0.10 is monit-app-02 strange.

On monit-app-02 I get:

Code:

  8958:20180618:134233.790 failed to accept an incoming connection: connection from "10.156.0.10" rejected, allowed hosts: "127.0.0.1,10.164.0.10"

where 10.156.0.10 is 10.156.0.10

**eset** · 18-06-2018, 13:39

When adding on front two zabbix servers into Configuration / Hosts with their local IP addresses I get

Code:

Zabbix agent on monit-app-01 is unreachable for 5 minutes

**kernbug** · 18-06-2018, 14:08

Originally posted by eset

When adding on front two zabbix servers into Configuration / Hosts with their local IP addresses I get

Code:

Zabbix agent on monit-app-01 is unreachable for 5 minutes

Zabbix Agent configuration parameters:

"ServerActive" -> outgoing connection to server -> Zabbix Server
"Server" <- incoming connection from server <- Zabbix Server

Code:

15490:20180618:134120.265 failed to accept an incoming connection: connection from "10.164.0.10" rejected, allowed hosts: "127.0.0.1,10.156.0.10"

you need

Code:

Server=10.164.0.10,10.156.0.10 #(incoming passive checks)
ServerActive=10.164.0.10,10.156.0.10 #(outgoing active checks)

in frontend:

Code:

monit-app-01 (10.156.0.10) monit-app-02 (10.164.0.10)

**eset** · 18-06-2018, 14:11

Yeah ok I've added it like that and it works on. I mean the zabbix-agents on zabbix-servers. But other nodes added directly to Zabbix Servers (without proxy) should have this:

Code:

 Server=172.31.255.254,10.156.0.10,10.164.0.10 ServerActive=172.31.255.254,10.156.0.10,10.164.0.10

172.31.255.254 is floating IP so basically for all agents (directly connected) and proxy's it should be like that?

**eset** · 18-06-2018, 22:57

Originally posted by kernbug

Code:

Server=10.164.0.10,10.156.0.10 #(incoming passive checks)
ServerActive=10.164.0.10,10.156.0.10 #(outgoing active checks)

Ok but Are you sure this is correct? I added it so and now, Zabbix Bot send notifications twice

Code:

Problem: PROBLEM:monit-app-02:Zabbix agent on monit-app-02 is unreachable for 5 minutes
Host: monit-app-02
Trigger: PROBLEM: Zabbix agent on monit-app-02 is unreachable for 5 minutes: High
Timestamp: 2018.06.18 22:42:00


Eventid: 367

this is from monit-app-01

Code:

root@monit-app-01:/usr/lib/zabbix/alertscripts# tail -f /var/log/zabbix/zabbix_server.log |grep -i --color slack
 29140:20180618:224202.295 In substitute_simple_macros() data:'--slack'
Eventid: 367' '--api_token=xxxx' '--slack''
Eventid: 367' '--api_token=xxxx' '--slack' output:

on slack I see something like this

but there is also a second event I receive is with ID 369 from monit-app-02

and on frontend I see this

It's duplicated

kernburg

I think you are wrong about ServerActive. If Zabbix node with Zabbix-agent needs to retrieve information it only needs to retrieve from one (active - behind HAProxy or active on Floating IP) zabbix-server not both like you proposed. So in ServerActive=<FLOATING_IP_or_HAProxy_IP> where those two zabbix-server stands behind. And Server= I agree it has incoming connections so both zabbix-server IP addresses needs to be there because their aren't SRC-NATed through HAProxy or FLOATING_IP that's why incoming connection comes from specific (active at the time it comes) server so It needs to be listed there for access.

**kernbug** · 22-06-2018, 09:26

Originally posted by eset

I think you are wrong about ServerActive. If Zabbix node with Zabbix-agent needs to retrieve information it only needs to retrieve from one (active - behind HAProxy or active on Floating IP) zabbix-server not both like you proposed. So in ServerActive=<FLOATING_IP_or_HAProxy_IP> where those two zabbix-server stands behind. And Server= I agree it has incoming connections so both zabbix-server IP addresses needs to be there because their aren't SRC-NATed through HAProxy or FLOATING_IP that's why incoming connection comes from specific (active at the time it comes) server so It needs to be listed there for access.

Hi, sorry, but I don't know about your exact setup, so ServerActive= of course should contain Virtual IP. It was my example suggestion.

**eset** · 26-06-2018, 23:16

Originally posted by kernbug

Hi, sorry, but I don't know about your exact setup, so ServerActive= of course should contain Virtual IP. It was my example suggestion.

I gave you my exact setup. Server and ServerActive these are the most important things.

Ok I managed to set keepalive and floating ip. Two zabbix-server works, one is active second is deactivated (active only if the primary server goes down).

I set also zabbix-proxy server, added in zabbix_proxy.conf

Server=10.156.0.10,10.164.0.10

those IP addresses are the zabbix servers addresses (first is active, second deactivated).

On zabbix-agents nodes I've set IP address of zabbix-proxy node. Example of zabbix_agentd.conf:

Server=10.5.0.149
ServerActive=10.5.0.149

but that doesn't work. Example host: samariumhas the following error:

Code:

 17922:20180626:230607.241 no active checks on server [10.5.0.149:10051]: host [samarium] not found
 17922:20180626:230807.291 no active checks on server [10.5.0.149:10051]: host [samarium] not found
 17922:20180626:231007.346 no active checks on server [10.5.0.149:10051]: host [samarium] not found
 17922:20180626:231207.397 no active checks on server [10.5.0.149:10051]: host [samarium] not found

and in zabbix-proxy

Code:

  8072:20180626:230341.062 trapper got '{"request":"active checks","host":"samarium","host_metadata":"Linux UpCloud"}'
  8072:20180626:230341.062 In get_hostid_by_host() host:'samarium'
  8072:20180626:230341.062 query [txnlev:0] [select hostid,status,tls_accept,tls_issuer,tls_subject,tls_psk_identity from hosts where host='samarium' and status in (0,1) and flags<>2 and proxy_hostid is null]
  8072:20180626:230341.063 query [txnlev:1] [insert into proxy_autoreg_host (clock,host,listen_ip,listen_dns,listen_port,host_metadata) values (1530047021,'samarium','10.2.2.116','samarium',10050,'Linux UpCloud')]
  8072:20180626:230341.064 cannot send list of active checks to "10.2.2.116": host [samarium] not found
  8072:20180626:230341.064 send_list_of_active_checks_json() sending [{"response":"failed","info":"host [samarium] not found"}]

I changed in zabbix_proxy.conf
Server=10.156.0.10,10.164.0.10 => Server=172.31.255.254

Now server, those behind zabbix-proxy are correctly added using auto-registration but on Configuration / Hosts the ZBX isn't green on those hosts which indicates that zabbix-agent isn't reachable.

Logs on zabbix-proxy say:

Code:

 11091:20180627:000643.912 In connect_to_server() [172.31.255.254]:10051 [timeout:600]
 11091:20180627:000645.031 In connect_to_server() [172.31.255.254]:10051 [timeout:600]
 11091:20180627:000646.137 In connect_to_server() [172.31.255.254]:10051 [timeout:600]

**kernbug** · 27-06-2018, 07:12

Hi

Zabbix Servers behind HAproxy in Active/Passive mode

Two zabbix-server works, one is active second is deactivated (active only if the primary server goes down)

If one server is deactivated, you primary IP is 'floating IP', for 'Server' and 'ServerActive' of the Agents and Proxies configuration file.

I changed in zabbix_proxy.conf
Server=10.156.0.10,10.164.0.10 => Server=172.31.255.254

And this is correct in this case.

Now server, those behind zabbix-proxy are correctly added using auto-registration but on Configuration / Hosts the ZBX isn't green on those hosts which indicates that zabbix-agent isn't reachable.

ZBX icon stay green if host has passive checks.

11091:20180627:000643.912 In connect_to_server() [172.31.255.254]:10051 [timeout:600]

From Zabbix Proxy check up port 10051, it should be opened on the Zabbix Server for your Zabbix Proxy:

Code:

telnet 172.31.255.254 10051
or
##netcat utility
nc -vz 172.31.255.254 10051

**eset** · 27-06-2018, 11:37

Hi

If one server is deactivated, you primary IP is 'floating IP', for 'Server' and 'ServerActive' of the Agents and Proxies configuration file.

True but only if zabbix-server is on that IP
SourceIP & ListenIP

ZBX icon stay green if host has passive checks.

I nedd that also to work and are you sure ? I can change item type for zabbix agent template to Zabbix Agent (Active) then this is checked by zabbix-agent not by zabbix-server

From Zabbix Proxy check up port 10051, it should be opened on the Zabbix Server for your Zabbix Proxy:

Code:

telnet 172.31.255.254 10051
or
##netcat utility
nc -vz 172.31.255.254 10051

Checking connection from proxy -> zabbx

Code:

nmap 172.31.255.254 -Pn -p 10051

Starting Nmap 7.40 ( https://nmap.org ) at 2018-06-27 11:32 CEST
Nmap scan report for 172.31.255.254
Host is up (0.025s latency).
PORT      STATE SERVICE
10051/tcp open  zabbix-trapper

Code:

telnet 172.31.255.254 10051
Trying 172.31.255.254...
Connected to 172.31.255.254.
Escape character is '^]'.
Connection closed by foreign host

Code:

 nc -vz 172.31.255.254 10051
172.31.255.254: inverse host lookup failed: Unknown host
(UNKNOWN) [172.31.255.254] 10051 (zabbix-trapper) open

**eset** · 30-06-2018, 10:36

I changed Type in Items from Zabbix Agent to -> Zabbix Agent (active) and it worked almost for all templates.

Ad Widget

Zabbix Servers behind HAProxy

Zabbix Servers behind HAProxy

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment