Difficulty getting Active Checks working

[SkilledAlpaca]

• Zabbix Appliance Version: 6.4.15
  • IP: 192.168.168.14
  • Hostname: ZABBIX01
  • OS: AlmaLinux 8.10
• Windows Server: 2022 Standard v21H2
  • IP: 192.168.168.8
  • Hostname: WEBSERVER03
• Debian Version: 12 (bookworm)
  • IP: 192.168.168.23
  • Hostname: deb-intproxy-prod-01
• Deployed Agent Version: 6.0.17
• Deployed Agent2 Version: 6.4.15

I currently have the normal passive agent deployed to multiple Linux and window servers without issue. I am attempting to setup Active Checks and running into a strange snag.
The firewalls on both Linux and windows allow ports 10050 and 10051 TCP.

Windows Agent Logs

12224:20240603:090636.850 Unable to connect to [192.168.168.14]:10051 [cannot connect to [[192.168.168.14]:10051]: (null)]
 12224:20240603:090636.851 Active check configuration update started to fail​

Windows Agent2 Logs

2024/06/03 10:09:26.834148 Zabbix Agent2 hostname: [WEBSERVER03]
2024/06/03 10:09:30.839333 [101] cannot connect to [192.168.168.14:10051]: dial tcp :0->192.168.168.14:10051: i/o timeout
2024/06/03 10:09:30.839333 [101] active check configuration update from host [WEBSERVER03] started to fail
2024/06/03 10:09:33.851383 [101] cannot connect to [192.168.168.14:10051]: dial tcp :0->192.168.168.14:10051: i/o timeout
2024/06/03 10:09:33.851383 [101] sending of heartbeat message for [WEBSERVER03] started to fail​

Linux Agent Logs

242987:20240603:092919.979 Unable to connect to [192.168.168.14]:10051 [cannot connect to [[192.168.168.14]:10051]: [4] Interrupted system call]
242987:20240603:092919.979 Unable to send heartbeat message to [192.168.168.14]:10051 [cannot connect to [[192.168.168.14]:10051]: [4] Interrupted system call]
242987:20240603:092922.979 Unable to connect to [192.168.168.14]:10051 [cannot connect to [[192.168.168.14]:10051]: [4] Interrupted system call]
242987:20240603:092922.979 Active check configuration update started to fail​

The zabbix_agent.conf and zabbix_agent2.conf are configured as follows:

Server=192.168.168.14
ServerActive=192.168.168.14
Hostname=WEBSERVER03

The frontend 'Host name' is configured to 'WEBSERVER03' and the passive checks are working. I have configured an Active Check and it shows as 'Unknown' in the front end. The Linux hostname is configured properly too.


Attempting to telnet from WEBSERVER03 to ZABBIX01 on either 10050 or 10051 both fail.
Attempting to telnet from ZABBIX01 to WEBSERVER03 on port 10050, success, while 10051 fails
These are the same results on the win-intproxy-prod-01 Linux server.

PS C:\Logs> Test-NetConnection -Computer 192.168.168.14 -Port 10051 -InformationLevel Detailed
WARNING: TCP connect to (192.168.168.14 : 10051) failed

ComputerName            : 192.168.168.14
RemoteAddress           : 192.168.168.14
RemotePort              : 10051
NameResolutionResults   : 192.168.168.14
                          zabbix01.ad.domain.com
MatchingIPsecRules      :
NetworkIsolationContext : Private Network
InterfaceAlias          : Ethernet
SourceAddress           : 192.168.168.8
NetRoute (NextHop)      : 0.0.0.0
PingSucceeded           : True
PingReplyDetails (RTT)  : 0 ms
TcpTestSucceeded        : False​

deb-intproxy-prod-01:~$ telnet 192.168.168.14 10051
Trying 192.168.168.14...
telnet: Unable to connect to remote host: Connection timed out​

[root@zabbix01 ~]# telnet 192.168.168.8 10050
Trying 192.168.168.8...
Connected to 192.168.168.8.
Escape character is '^]'.
Connection closed by foreign host.​

We have Unifi managed switches and Meraki MX Firewall, but they are not configured for any kind of LAN port blocking. I'm at my wits end trying to get the active checks working. I was able to get "Active Checks" by running an instance of each agent and sending the checks to different hosts in the frontend, which I would like to get away from. Any help would be appreciated! Apologies if my initial post is all over the place, been scratching my head on this for a few hours.

[Markku]

-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 10050,10051,10053 -j ZBX

For me it looks like it drops all incoming TCP SYN packets, before accepting inbound Zabbix protocol packets.

Markku

[Markku]

deb-intproxy-prod-01:~$ telnet 192.168.168.14 10051
Trying 192.168.168.14...
telnet: Unable to connect to remote host: Connection timed out

This looks pretty clear: there is a firewall on 192.168.168.14 that blocks the incoming TCP port 10051, or, the appliance is not in that IP address.

The appliance manual (https://www.zabbix.com/documentation...nual/appliance) says that the firewall allows that by default. Maybe you could check that?

Markku

[SkilledAlpaca]

Performing a iptable lookup, 10050,10051,10053 are open, as far as I can tell. The only thing I haven't looked into is the nginx side of things on the appliance, but for IP address, that shouldn't be an issue.

[root@zabbix01 ~]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N ICMP
-N NTP
-N SSH
-N ZBX
-N WEB
-N SNMPTRAP
-N SQL
-A INPUT -i lo -j ACCEPT
-A INPUT -s 8.8.8.8/32 -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -p icmp -j ICMP
-A INPUT -p udp -m udp --dport 123 -j NTP
-A INPUT -p tcp -m tcp --dport 20:22 -j SSH
-A INPUT -p tcp -m multiport --dports 80,443 -j WEB
-A INPUT -p udp -m udp --dport 162 -j SNMPTRAP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 10050,10051,10053 -j ZBX
-A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A ICMP -p icmp -m limit --limit 5/sec -j ACCEPT
-A SSH -p tcp -m tcp --dport 22 -j ACCEPT
-A ZBX -p tcp -m tcp --dport 10050 -j ACCEPT
-A ZBX -p tcp -m tcp --dport 10051 -j ACCEPT
-A WEB -p tcp -m multiport --dports 80,443 -j ACCEPT
-A WEB -p udp -m udp --dport 162 -j ACCEPT

[root@zabbix01 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     udp  --  dns.google           anywhere             udp spt:domain
ACCEPT     udp  --  dns.google           anywhere             udp spt:domain
ICMP       icmp --  anywhere             anywhere
NTP        udp  --  anywhere             anywhere             udp dpt:ntp
SSH        tcp  --  anywhere             anywhere             tcp dpts:ftp-data:ssh
WEB        tcp  --  anywhere             anywhere             multiport dports http,https
SNMPTRAP   udp  --  anywhere             anywhere             udp dpt:snmptrap
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ZBX        tcp  --  anywhere             anywhere             multiport dports zabbix-agent,zabbix-trapper,10053

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply

Chain ICMP (1 references)
target     prot opt source               destination
ACCEPT     icmp --  anywhere             anywhere             limit: avg 5/sec burst 5

Chain NTP (1 references)
target     prot opt source               destination

Chain SSH (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

Chain ZBX (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-agent
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-trapper

Chain WEB (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             multiport dports http,https
ACCEPT     udp  --  anywhere             anywhere             udp dpt:snmptrap

Chain SNMPTRAP (1 references)
target     prot opt source               destination

Chain SQL (0 references)
target     prot opt source               destination

Appears telnet on the localhost works.

[root@zabbix01 ~]# telnet localhost 10051
Trying ::1...
Connected to localhost.
Escape character is '^]'.
^CConnection closed by foreign host.​

[Markku]

-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 10050,10051,10053 -j ZBX

For me it looks like it drops all incoming TCP SYN packets, before accepting inbound Zabbix protocol packets.

Markku

[SkilledAlpaca]

For me it looks like it drops all incoming TCP SYN packets, before accepting inbound Zabbix protocol packets.

Markku

I've made no changes to the iptables and unfortunately I am also unfamiliar with this specific tool. If I'm reading it correctly, it places the zbx groups above the deny rule, which should allow incoming traffic? I guess it's time to learn iptables. What would I need to do to test if that rule is the issue?

[Markku]

The INPUT chain rules are evaluated in the order of shown in your "iptables -L" command, thus the drop happens before even checking the Zabbix ports.

Additionally, port 10053 is not even mentioned in the firewall rules in the documentation, so either the documentation is incorrect or someone has been reconfiguring your iptables rules.

Markku

[Markku]

You imply that you are behind a Meraki MX firewall anyway, so just flush all the rules with sudo iptables -F if you feel like it.

Markku

[SkilledAlpaca]

The INPUT chain rules are evaluated in the order of shown in your "iptables -L" command, thus the drop happens before even checking the Zabbix ports.

Additionally, port 10053 is not even mentioned in the firewall rules in the documentation, so either the documentation is incorrect or someone has been reconfiguring your iptables rules.

Markku

I just disabled the iptables rules, and the active checks are working. I'm in the process of spooling up a "new" appliance so I can get the fresh iptable rules to compare. It does seem I made some changes to the rules at one point and would like to get them reverted. I will post any differences I find.

[SkilledAlpaca]

For me it looks like it drops all incoming TCP SYN packets, before accepting inbound Zabbix protocol packets.

Markku

This was the issue. I guess when I was making changes a year or so ago, I made some kind of change to the iptables and moved the zbx group after the DENY. Thank you for all your help! I guess it's time for me to do a deep dive into iptables so I don't run into this again.

Edit: A small note, I did have to upgrade my installed agents from 6.0.4 to 6.4.15 for the active checks to begin reporting properly. They were showing Unavailable and the "zabbix[host,active_agent,available]" key was returning "2 - not available" until I upgrade the agents. After that things began working perfectly.

Export of 'iptables-save > /root/iptables_backup.fresh' from a brand new net install of the appliance (version 6.4.15). I was able to take this export and restore it to the production machine and active checks started working instantly. I did initially compare this backup to the production back and as Markku said, it was dropping the ZBX group due to the rule ordering.

# Generated by iptables-save v1.8.5 on Mon Jun  3 12:06:16 2024
*raw
:PREROUTING ACCEPT [2814732:209811427]
:OUTPUT ACCEPT [3324527:260574895]
-A PREROUTING -p udp -m udp --sport 53 -j NOTRACK
-A OUTPUT -p udp -m udp --dport 53 -j NOTRACK
COMMIT
# Completed on Mon Jun  3 12:06:16 2024
# Generated by iptables-save v1.8.5 on Mon Jun  3 12:06:16 2024
*filter
:INPUT ACCEPT [83094:10477346]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3317609:259993999]
:ICMP - [0:0]
:NTP - [0:0]
:SSH - [0:0]
:ZBX - [0:0]
:WEB - [0:0]
:SNMPTRAP - [0:0]
:SQL - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -s 8.8.8.8/32 -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -s 8.8.4.4/32 -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -p icmp -j ICMP
-A INPUT -p udp -m udp --dport 123 -j NTP
-A INPUT -p tcp -m tcp --dport 20:22 -j SSH
-A INPUT -p tcp -m multiport --dports 10050,10051 -j ZBX
-A INPUT -p tcp -m multiport --dports 80,443 -j WEB
-A INPUT -p udp -m udp --dport 162 -j SNMPTRAP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A ICMP -p icmp -m limit --limit 5/sec -j ACCEPT
-A SSH -p tcp -m tcp --dport 22 -j ACCEPT
-A ZBX -p tcp -m tcp --dport 10050 -j ACCEPT
-A ZBX -p tcp -m tcp --dport 10051 -j ACCEPT
-A WEB -p tcp -m multiport --dports 80,443 -j ACCEPT
-A WEB -p udp -m udp --dport 162 -j ACCEPT
COMMIT
# Completed on Mon Jun  3 12:06:16 2024 zz0.5btlwjxjp2nzz