net.tcp.service is not working on most ports

[Jesse,Lewis]

Setup Zabbix 5.4 on Red Hat Linux Enterprise, love it btw, and seem unable to check most ports.
HTTP, HTTPS, and FTP checks seem to work but all other ports I've checked are not working.


For example, I have another RHEL server on the same subnet as my Zabbix server that is open on ports 22, 80, 111, & 443
When I perform a simple check net.tcp.service[ssh] or net.tcp.service[tcp,,22] I get a 0 returned.
If I login to the Zabbix server OS and simply SSH to the other RHEL server in question is works just fine meaning the Zabbix server's OS can communicate over port 22.

Is there something in the Zabbix-server.conf or other configuration setting I need to enable for net.tcp.service to work?

More details
net.tcp.service[tcp,,80] - works
net.tcp.service[tcp,,21] - works
net.tcp.service[tcp,,22] - does not work
net.tcp.service[tcp,,111] - does not work
net.tcp.service[tcp,,443] - works
net.tcp.service[tcp,,8080] - does not work

[Hamardaban]

Read this and try to understand:

https://www.zabbix.com/documentation/current/en/manual/appendix/items/service_check_details

net.tcp.service[service,<ip>,<port>]
	Checks if service is running and accepting TCP connections.	0 - service is down 1 - service is running	service - either of: ssh, ldap, smtp, ftp, http, pop, nntp, imap, tcp, https, telnet (see details) ip - IP address (default is 127.0.0.1) port - port number (by default standard service port number is used)	Example: => net.tcp.service[ftp,,45] → can be used to test the availability of FTP server on TCP port 45. Note that these checks may result in additional messages in system daemon logfiles (SMTP and SSH sessions being logged usually). Checking of encrypted protocols (like IMAP on port 993 or POP on port 995) is currently not supported. As a workaround, please use net.tcp.port for checks like these. Checking of LDAP and HTTPS on Windows is only supported by Zabbix agent 2. Note that the telnet check looks for a login prompt (':' at the end). See also known issues of checking HTTPS service. https and telnet services are supported since Zabbix 2.0.

[Jesse,Lewis]

Read this and try to understand:

https://www.zabbix.com/documentation..._check_details

net.tcp.service[service,<ip>,<port>]
	Checks if service is running and accepting TCP connections.	0 - service is down 1 - service is running	service - either of: ssh, ldap, smtp, ftp, http, pop, nntp, imap, tcp, https, telnet (see details) ip - IP address (default is 127.0.0.1) port - port number (by default standard service port number is used)	Example: => net.tcp.service[ftp,,45] → can be used to test the availability of FTP server on TCP port 45. Note that these checks may result in additional messages in system daemon logfiles (SMTP and SSH sessions being logged usually). Checking of encrypted protocols (like IMAP on port 993 or POP on port 995) is currently not supported. As a workaround, please use net.tcp.port for checks like these. Checking of LDAP and HTTPS on Windows is only supported by Zabbix agent 2. Note that the telnet check looks for a login prompt (':' at the end). See also known issues of checking HTTPS service. https and telnet services are supported since Zabbix 2.0.

Thank you. I read that when building and troubleshooting my simple check and reread it again just now. I believe my syntax is correct and my understanding that a TCP check on port 22 should return a value of 1 when running that check against a host that has an open SSH service (listening). Am I misunderstanding, or is my syntax wrong? net.tcp.service[tcp,,22]

[Hamardaban]

Does SSH use keys? or just login \ password?
Try tcpdump's you connection attempt on port 22 in case "net.tcp.service[tcp,,22]"

[Jesse,Lewis]

Interesting, there is just no activity, When I run the test it does not seem to even try on port 22. I changed the check to port 80 and could see that in tcpdump

[root@n5sZabbix ~]# tcpdump -i any -n host 10.1.5.110 and port 22
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
^C
0 packets captured
1 packet received by filter
0 packets dropped by kernel
[root@n5sZabbix ~]# tcpdump -i any -n host 10.1.5.110 and port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
09:10:31.245928 IP 10.1.5.6.44228 > 10.1.5.110.http: Flags [S], seq 3939532490, win 29200, options [mss 1460,sackOK,TS val 2016716484 ecr 0,nop,wscale 7], length 0
09:10:31.246675 IP 10.1.5.110.http > 10.1.5.6.44228: Flags [S.], seq 4191191324, ack 3939532491, win 28960, options [mss 1460,sackOK,TS val 3040834668 ecr 2016716484,nop,wscale 7], length 0
09:10:31.246707 IP 10.1.5.6.44228 > 10.1.5.110.http: Flags [.], ack 1, win 229, options [nop,nop,TS val 2016716484 ecr 3040834668], length 0
09:10:31.246735 IP 10.1.5.6.44228 > 10.1.5.110.http: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 2016716484 ecr 3040834668], length 0
09:10:31.246947 IP 10.1.5.110.http > 10.1.5.6.44228: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 3040834668 ecr 2016716484], length 0
09:10:31.246959 IP 10.1.5.6.44228 > 10.1.5.110.http: Flags [.], ack 2, win 229, options [nop,nop,TS val 2016716485 ecr 3040834668], length 0
^C
6 packets captured
8 packets received by filter
0 packets dropped by kernel
[root@n5sZabbix ~]# tcpdump -i any -n host 10.1.5.110 and port 22
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
^C
0 packets captured
1 packet received by filter
0 packets dropped by kernel
[root@n5sZabbix ~]#

[Hamardaban]

First of all - check firewals on both side!

[Jesse,Lewis]

First of all - check firewals on both side!

Great point. I ensured the firewall on the Zabbix server is completely disabled. Also, if it were firewall related then SSH from terminal would not work, but I can SSH from the Zabbix server terminal into the other servers just fine which should indicate the traffic is allowed just for some reason Zabbox is not trying. This felt similar to when I had to enable ping in the Zabbix config but found nothing in the config file related to SSH or TCP ports in general.

[Hamardaban]

If I remember correctly, Zabix uses curl to connect when checking the availability of services. Try to establish a connection from the terminal using curl.
And try net.tcp.port item for check 22 port open.

[Jesse,Lewis]

If I remember correctly, Zabix uses curl to connect when checking the availability of services. Try to establish a connection from the terminal using curl.
And try net.tcp.port item for check 22 port open.

I don't think 5.4 supports net.tcp.port any longer. The syntax is being rejected.

Curl appears to work:
[root@n5sZabbix ~]# curl 10.1.5.110:22
SSH-2.0-OpenSSH_7.4
Protocol mismatch.
curl: (56) Recv failure: Connection reset by peer


TCPDump while performing the curl
[root@n5sZabbix ~]# tcpdump -i any -n host 10.1.5.110 and port 22
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:30:46.405974 IP 10.1.5.6.38708 > 10.1.5.110.ssh: Flags [S], seq 3371036425, win 29200, options [mss 1460,sackOK,TS val 2483318163 ecr 0,nop,wscale 7], length 0
13:30:46.406586 IP 10.1.5.110.ssh > 10.1.5.6.38708: Flags [S.], seq 525557599, ack 3371036426, win 28960, options [mss 1460,sackOK,TS val 3056449826 ecr 2483318163,nop,wscale 7], length 0
13:30:46.406630 IP 10.1.5.6.38708 > 10.1.5.110.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 2483318164 ecr 3056449826], length 0
13:30:46.406669 IP 10.1.5.6.38708 > 10.1.5.110.ssh: Flags [P.], seq 1:78, ack 1, win 229, options [nop,nop,TS val 2483318164 ecr 3056449826], length 77
13:30:46.406738 IP 10.1.5.110.ssh > 10.1.5.6.38708: Flags [.], ack 78, win 227, options [nop,nop,TS val 3056449827 ecr 2483318164], length 0
13:30:46.421791 IP 10.1.5.110.ssh > 10.1.5.6.38708: Flags [P.], seq 1:22, ack 78, win 227, options [nop,nop,TS val 3056449842 ecr 2483318164], length 21
13:30:46.421810 IP 10.1.5.6.38708 > 10.1.5.110.ssh: Flags [.], ack 22, win 229, options [nop,nop,TS val 2483318179 ecr 3056449842], length 0
13:30:46.421890 IP 10.1.5.110.ssh > 10.1.5.6.38708: Flags [P.], seq 22:41, ack 78, win 227, options [nop,nop,TS val 3056449842 ecr 2483318179], length 19
13:30:46.421894 IP 10.1.5.6.38708 > 10.1.5.110.ssh: Flags [.], ack 41, win 229, options [nop,nop,TS val 2483318179 ecr 3056449842], length 0
13:30:46.422402 IP 10.1.5.110.ssh > 10.1.5.6.38708: Flags [R.], seq 41, ack 78, win 227, options [nop,nop,TS val 3056449842 ecr 2483318179], length 0



So it appears to just be Zabbix itself.

[Hamardaban]

we use the 5.4.6 version and the net.tcp.port works perfectly. see the docs.
try to increase the logging level of the server and see what happens
if you are sure that this is a zabix problem - write to technical support.