Zabbix Agent registering with wrong IP Address

[Maximepj]

Hi there,

We're upgrading our Zabbix 3.2 to Zabbix 4.4. Our new Zabbix server is running in AWS ( EC2 ) behind an ELB , what is happening here is if we use the elb dns name as the zabbix server in our agents config it actually registers the agent with the ELB IP address. It means that the Zabbix server will drop communication because it can't reach the Zabbix agent. Is there a way to bypass the ELB and get the IP address of the real agent instead. Looks like we have an usual setup, havent seen that kind of Zabbix setup on the internet so I can't seem to find a decent answer. Any tips for such setup?


Thank you

[Markku]

Hi, please describe you environment a bit better to clear/prevent confusions.

• Is there a specific reason for using ELB?
• Are you using agent autoregistration?
• Are you using active or passive agents? For specific reason?
• What type of ELB are you using? https://aws.amazon.com/premiumsuppor...d-balancer-IP/

Markku

[Maximepj]

• No specific reason we wanted an easy and fast way to enable HTTPS on our Zabbix server and we're also using AutoScaling groups / Launch configurations. I know we can use the nginx config file to do the same thing but we chose not to use it ( for now at least )
• Yes, we're using agent autoregistration
• Active checks are used
• We're using Classic ELB

I tried to setup a proxy policy on our ELB on port 10051 but did not seem to work see AWS doc here: https://docs.aws.amazon.com/elasticl...-protocol.html

As soon as I did that the "Zabbix server is running" started to fail ( "No" ) and I also started to see some "Message from 10.X.X.X.X is missing header. Message ignored." which is the ELB's Ip address. The only way I found to "bypass" my issue was to set the server active directly to the Zabbix server itself but then there is no point of using the ELB then.


Additional information:
Zabbix Server version is 4.4
Zabbix agents are still using 3.2 since it's supposed to be fully compatible with 4.4 we will update them later
Zabbix running in dockers
Also deploying Zabbix as code

[Markku]

Ok, thanks for the information.

The described proxy protocol is not usable with Zabbix server/agent traffic, it requires specific support on the application level.

With active agents the visible IP address of the agent is not a significant thing, that should work with the ELB's source address translation. But, the agents need to be able to contact the Zabbix server in the specified port (10051/tcp by default). If they cannot, then the Zabbix server items checking for the agent reachability will obviously trigger.

Have you tried configuring a 10051/tcp listener in your ELB (in addition to the https listener), forwarding the connections to port 10051/tcp on the only Zabbix server?

(You might already know this but do note that it is not possible to use load balancer to distribute load between several Zabbix servers.)

Markku

[Maximepj]

Markku Yes 10051 is opened in my ELB forwarding to 10051 instance port. I have ServerActive=%ELB-DNS% in our agents configuration. This is what I have in one of my auto registration if that changes anything. Also please note that i'm running Zabbix in dockers ( 1 docker for server 1 docker for web ). Let me know if you have any tips that would help me out figure it out.




Also, probably that the server is not receiving an IP address and uses the one used for the incoming connection AKA the ELB (like you kind of said in your previous post ) :

Server, when adding the new auto-registered host, uses the received IP address and port to configure the agent. If no IP address value is received, the one used for the incoming connection is used. If no port value is received, 10050 is used.

[Markku]

Unfortunately I don't have any Zabbix in containers, too complicated for me at the moment

Do note that the IP address entered in the Agent interfaces field (either manually or by autoregistration) is not significant when the agent is active. This is because the agent initiates the TCP connection (see for example https://majornetwork.net/2017/10/zab...nts-and-ports/ for an overview), and Zabbix only cares about the host name that is inside the message payload (well TLS settings as well but you didn't say you are running any TLS with Zabbix agent connections so I assume that you are not using it). That said, can you please clarify this:

It means that the Zabbix server will drop communication because it can't reach the Zabbix agent.

What are the actual error messages that you get (and where) that you say Zabbix drops the communication?

Are you sure your host items are "Zabbix agent (active)" and not "Zabbix agent"?

If you are unsure if agent traffic reaches Zabbix server you can always run tcpdump on your EC2 instance to see the incoming connections. Here are some Wireshark dissectors if you want to play with them: https://majornetwork.net/2019/06/zab...for-wireshark/

Markku

[Maximepj]

Ahhh!!!! you are right it looks like some of our Zabbix items are Zabbix agent ( active ) and some of them are Zabbix agent. Most of them are Zabbix agent though ( like 99% of them ),



I'm not sure why it's like that it's probably some legacy stuff that we've had since the day we started using Zabbix. I assumed it was active since we had a "serverActive" set in the config file and that we had auto registration but I was wrong. What I understand from your last statement is that the 1% of active checks would work and the rest will not work even thought when you click on the host in Zabbix the "ZBX" icon is red and that Zabbix can't communicate with the agent.

[Markku]

Right, the red ZBX icon only reflects the status of the passive items, active items do not change the color of the icon.

In the Latest data page you can of course verify which items have received values within last 24 hours (that's the limit AFAIK).

Markku

[Maximepj]

good to know! That being said since my checks are mostly passive in the end. Do you know how i could prevent the ELB IP to be set on all hosts to stop it from disabling my hosts? If not I might just remove the ELB of the equation and point directly to the server and that would be it.

[Markku]

I don't see any benefit for an Internet-accessible Zabbix frontend+server being behind an ELB. Just attach an Elastic IP in the Zabbix server and use Let's Encrypt to manage the https certificate for the frontend. Then you don't have any source address translation in front of Zabbix server.

(Whether the agents are actually reachable by the Zabbix server for passive checks, that depends of course now on the agent-side of the network.)

You can mass-update the items from passive to active to fix the templates you are using for the hosts. That's the recommended way because active checks are more efficient from the Zabbix server point of view. And then you don't have any problems if the agent-side is protected with an ingress firewall.

Markku