Active Agent wont connet to proxy bind says port in use when it's not.
Im having a weird issue with active agent showing as inactive in our server. The initial startup says it can't bind to address already in use, however a netstat of what's running on our RHEL8 server shows no active ports open to 10051 we are connecting to proxy.
Log from agent
2024/04/17 19:06:25.872346 [0] starting listener for '0.0.0.0:10050'
2024/04/17 19:06:25.873102 listening for control connections on /tmp/agent.sock
2024/04/17 19:06:26.872663 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:06:26.878740 [101] cannot connect to [10.62.52.101:10051]: dial tcp :0->10.62.52.101:10051: bind: address already in use <--Shows in use but it's not
2024/04/17 19:06:26.878764 [101] active check configuration update from host [catl0dlas28244] started to fail
2024/04/17 19:06:26.878772 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:06:26.878778 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:06:26.884317 [101] cannot connect to [10.62.52.101:10051]: dial tcp :0->10.62.52.101:10051: bind: address already in use
2024/04/17 19:06:26.884341 [101] sending of heartbeat message for [catl0dlas28244] started to fail
2024/04/17 19:06:26.884348 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
I have tcpdumps open on the server and if I do a nc -p 10051 to the server port I see it pop in tcpdump
but I never see anything come in with only the active agent running.
Leaving it running it just keeps scrolling these:
024/04/17 19:06:26.884348 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:07:27.872425 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:07:27.878804 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:08:27.872810 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:08:27.879003 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:08:28.873101 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:08:28.879164 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:09:29.873110 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:09:29.879187 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:10:28.873268 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:10:28.879306 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:10:30.873193 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:10:30.879211 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
I don't believe this to be firewall related as every other host in our subnets which this is a /22 can connect to the proxy fine.
Nmap shows it as open as well
Starting Nmap 7.70 ( https://nmap.org ) at 2024-04-17 19:12 EDT
Nmap scan report for 10.62.52.101
Host is up (0.00085s latency).
PORT STATE SERVICE
10051/tcp open zabbix-trapper
Nmap done: 1 IP address (1 host up) scanned in 0.31 seconds
OS is RedHat version 8.8
agent version is 6.2.7
Agent installs are automated and config file generation is also automated using ansible to our environment so it's odd there would be a misconfiguration. I have yet to find anything that looks to be incorrect.
I just never see anything going out over port 10051 from that heartbeat message I've done tcpdump of the servers running the agent and it doesn't show any outbound traffic to the proxy other than ICMP. The only time I can generate traffic over the port is using nc.
TCPDUMP Output on proxy....
19:09:35.474814 IP (tos 0x0, ttl 59, id 2780, offset 0, flags [DF], proto TCP (6), length 60)
(AGENT HOST)10.64.52.224.zabbix-trapper >(PROXY)10.62.52.101: Flags [S], cksum 0xf5df (correct), seq 3460400561, win 29200, options [mss 1460,sackOK,TS val 218455976 ecr 0,nop,wscale 7], length 0
Log from agent
2024/04/17 19:06:25.872346 [0] starting listener for '0.0.0.0:10050'
2024/04/17 19:06:25.873102 listening for control connections on /tmp/agent.sock
2024/04/17 19:06:26.872663 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:06:26.878740 [101] cannot connect to [10.62.52.101:10051]: dial tcp :0->10.62.52.101:10051: bind: address already in use <--Shows in use but it's not
2024/04/17 19:06:26.878764 [101] active check configuration update from host [catl0dlas28244] started to fail
2024/04/17 19:06:26.878772 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:06:26.878778 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:06:26.884317 [101] cannot connect to [10.62.52.101:10051]: dial tcp :0->10.62.52.101:10051: bind: address already in use
2024/04/17 19:06:26.884341 [101] sending of heartbeat message for [catl0dlas28244] started to fail
2024/04/17 19:06:26.884348 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
I have tcpdumps open on the server and if I do a nc -p 10051 to the server port I see it pop in tcpdump
but I never see anything come in with only the active agent running.
Leaving it running it just keeps scrolling these:
024/04/17 19:06:26.884348 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:07:27.872425 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:07:27.878804 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:08:27.872810 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:08:27.879003 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:08:28.873101 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:08:28.879164 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:09:29.873110 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:09:29.879187 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
2024/04/17 19:10:28.873268 [101] In refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:10:28.879306 [101] End of refreshActiveChecks() from [10.62.52.101:10051]
2024/04/17 19:10:30.873193 [101] In sendHeartbeatMsg() from [10.62.52.101:10051]
2024/04/17 19:10:30.879211 [101] End of sendHeartBeatMsg() from [10.62.52.101:10051]
I don't believe this to be firewall related as every other host in our subnets which this is a /22 can connect to the proxy fine.
Nmap shows it as open as well
Starting Nmap 7.70 ( https://nmap.org ) at 2024-04-17 19:12 EDT
Nmap scan report for 10.62.52.101
Host is up (0.00085s latency).
PORT STATE SERVICE
10051/tcp open zabbix-trapper
Nmap done: 1 IP address (1 host up) scanned in 0.31 seconds
OS is RedHat version 8.8
agent version is 6.2.7
Agent installs are automated and config file generation is also automated using ansible to our environment so it's odd there would be a misconfiguration. I have yet to find anything that looks to be incorrect.
I just never see anything going out over port 10051 from that heartbeat message I've done tcpdump of the servers running the agent and it doesn't show any outbound traffic to the proxy other than ICMP. The only time I can generate traffic over the port is using nc.
TCPDUMP Output on proxy....
19:09:35.474814 IP (tos 0x0, ttl 59, id 2780, offset 0, flags [DF], proto TCP (6), length 60)
(AGENT HOST)10.64.52.224.zabbix-trapper >(PROXY)10.62.52.101: Flags [S], cksum 0xf5df (correct), seq 3460400561, win 29200, options [mss 1460,sackOK,TS val 218455976 ecr 0,nop,wscale 7], length 0
Comment