Zabbix agent v3.4.6 Frequent ZBX_TCP_READ() failed : [104] on Windows Server 2003

[soonhin]

Hi,

I'm having an issue with zabbix agent where the agent always have issue or 'down". Every time it happen, I will just restart the agent from services.msc then after few hours or a day, it goes down again.
When the zabbix server failed to read data from the agent, I tried to telnet:
From agent to server port 10051 --> OK
From server to agent port 10050 --> OK
Ping agent to server --> OK
Ping server to agent --> OK
I even tried turn off both side firewall then restart the agent, still same thing happened after few hours or day.

From zabbix monitoring, the problem reported are "get value from agent failed:ZBX_TCP_READ() failed: [104] Connection reset by peer"

Through some searching online, comes upon "tcp port exhaustion" problem, so I tried to tweak the Windows Server registryby editing HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\Tcpip\Parameters following the recommendation at https://www.filehold.com/help/techni...ort-Exhaustion.
Still, the issue persist.

I tried to tweak the agent conf as below:
StartAgents=10
BufferSend=10
BufferSize=150
MaxLinesPerSecond=100
Timeout=20
Still, the issue persist.

Below are my environment architecture:

server1 ----------------------
server2 ---------------------- 10.16.145.0/24
server3 ----------------------

server4 ----------------------
server5 ---------------------- 10.16.144.0/24
server6 ----------------------
zabbix ----------------------

server1,2,and 3 are the servers that having the problems. server 4,5 and 6 are fine. All server running Windows Server 2003 SP2.

Zabbix Server version = 3.4.12
Zabbix Agent version = 3.4.6
Zabbix Server OS = Centos 7.5

I attached together the agent log and agent conf file from server2.

The error are regarding TCP, so I guess is something to do with network or port. But there were no issue with connectivity, I really have no idea on how to further check the problem anymore.

I really appreciate some help.
Thanks
soonhin

[plpagador]

Hi, soonhin.

I always find it helpful when I'm getting this kind of errors to troubleshoot with zabbix_get utility from the server's command line. Whenever you are getting this error again, try moving to the directory where zabbix_get is located in your Zabbix server and executing something like:

./zabbix_get -s XXX.XXX.XXX.XXX -p 10051 -k agent.ping

Where XXX.XXX.XXX.XXX is the agent's IP. If everything is OK, you should be getting a 1 from Zabbix agent. If not, you may get an error. I always perform this request to the agent while I monitor (tail -f) the agent's log in a different window, so that I can see what the agent logs exactly when I request the value from server.

Maybe you could post an extraction of Zabbix agent's log just after executing this request with zabbix_get.

Hope this helps you to find the root of your problem too!

[soonhin]

Hi plpagador,

Thank you on your reply.
I've done the troubleshooting using the zabbix_get, below are my result:

[root@spbkzabbix bin]# ./zabbix_get -s 10.16.145.21 -p 10050 -k "agent.ping"
zabbix_get [8598]: Get value error: ZBX_TCP_READ() failed: [104] Connection reset by peer
zabbix_get [8598]: Check access restrictions in Zabbix agent configuration

Then I try do ping test:
[root@spbkzabbix bin]# ping 10.16.145.21
PING 10.16.145.21 (10.16.145.21) 56(84) bytes of data.
64 bytes from 10.16.145.21: icmp_seq=1 ttl=127 time=1.49 ms
64 bytes from 10.16.145.21: icmp_seq=2 ttl=127 time=0.416 ms
^C
--- 10.16.145.21 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.416/0.954/1.493/0.539 ms

After that I try do telnet to port 10050:
[root@spbkzabbix bin]# telnet 10.16.145.21 10050
Trying 10.16.145.21...
Connected to 10.16.145.21.
Escape character is '^]'.
Connection closed by foreign host.


Ping and Telnet port 10051 from the agent host to the zabbix server also successful.
Why it still says "Check access restrictions in Zabbix agent configuration"


There's no log in the agent when I execute the zabbix_get. Below are the fraction of the agent log:
8440:20180816:125518.305 Requested [vm.vmemory.size[pavailable]]
8440:20180816:125518.305 Sending back [27.954112]
5032:20180816:125518.477 In collect_perfstat()
5032:20180816:125518.477 End of collect_perfstat()
9764:20180816:125519.305 Requested [vfs.fs.size[C:,free]]
9764:20180816:125519.321 In zbx_execute_threaded_metric() key:'vfs.fs.size'
4016:20180816:125519.321 executing in data thread for key:'vfs.fs.size'
9764:20180816:125519.321 End of zbx_execute_threaded_metric():SYSINFO_SUCCEED ''
9764:20180816:125519.321 Sending back [44571402240]
4444:20180816:125519.321 Requested [net.if.in[HP NC7761 Gigabit Server Adapter]]
4444:20180816:125519.321 Sending back [1082066782]
5032:20180816:125519.477 In collect_perfstat()
5032:20180816:125519.477 End of collect_perfstat()
5976:20180816:125520.321 Requested [net.if.in[MS TCP Loopback interface]]
5976:20180816:125520.321 Sending back [2297826777]
4960:20180816:125520.337 Requested [vfs.fs.size[D:,free]]
4960:20180816:125520.337 In zbx_execute_threaded_metric() key:'vfs.fs.size'
4652:20180816:125520.337 executing in data thread for key:'vfs.fs.size' <----------------------the latest log up until here only

The zabbix server detected the agent down at 13:00:30, I execute the zabbix_get at around 21:00:00