Many TIME_WAIT connection

[bee]

Hi all,

I saw many (over 50 in total) time_wait connection from zabbix client to zabbix server, as below

TCP zabbixclient:10050 zabbixserver:53944 TIME_WAIT
TCP zabbixclient:10050 zabbixserver:53947 TIME_WAIT
TCP zabbixclient:10050 zabbixserver:53960 TIME_WAIT
....
....
....

is it normal? is one connection above represent one monitored item?

Thanks,
BEE

[trikke]

I'm having the same "Problem".
Have my zabbix server (zobel) on Solaris, here is what i get with "netstat -a":

zobel.zabbix_agent zobel.52877 49152 0 49152 0 TIME_WAIT
zobel.zabbix_trap SRV-VM-WSUS-01.bvch.ch.1520 65476 0 49640 0 TIME_WAIT
zobel.zabbix_agent zobel.52875 49152 0 49178 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2262 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2330 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2329 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2328 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2327 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2326 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2325 64510 0 49680 0 TIME_WAIT
zobel.zabbix_agent zobel.52829 49152 0 49173 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2324 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2323 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2322 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2321 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2320 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2319 64510 0 49680 0 TIME_WAIT
zobel.52885 ora1LD2P.bvch.ch.12051 49680 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2318 64510 0 49680 0 TIME_WAIT
zobel.zabbix_agent zobel.52884 49152 0 49181 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2317 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2316 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2315 64510 0 49680 0 TIME_WAIT
zobel.52882 ora1LD1P.bvch.ch.12051 49680 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2314 64510 0 49680 0 TIME_WAIT
zobel.zabbix_agent zobel.52881 49152 0 49181 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2313 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2312 64510 0 49680 0 TIME_WAIT
zobel.zabbix_trap SRV-HS-REM-07.bvch.ch.2311 64510 0 49680 0 TIME_WAIT

Some connections were made by zabbix agents ( windows) and i guess it are items of type "zabbix Agent (active)" ( monitoring windows eventlog) and definetly items of type "Zabbix Agent" and key "net.tcp.port"

It looks like the socket opened has not been closed correctly.
Can some developer get a look at it ?

greets
Patrick

[Palmertree]

Had the same problem but it was an OS issue. Fixed it by modifing the sysctl.conf file as follows:

net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

[trikke]

Hi P.,

tahnx for the suggestion, I can't seem to find the file on Solaris (on solaris one should use the NDD Command).
Anyway as far as I know it is a application problem. If the application opens a tcp-socket, the application should close the socket as well and not wait for the OS to close the port!!! (See http://rfc.sunsite.dk/rfc/rfc793.html)

Till this issue will be fixed, I think changing the TIME-WAIT interval is a good work-around,

thanx again,
Patrick

[Alexei]

Anyway as far as I know it is a application problem.

You are wrong There is no problem, really... Just run a netstat for a busy TCP based server (Apache) to see bunch of sockets in a TIME_WAIT state.

[ziaansari]

Ok, But why zabbix-server restart after every 10 to 15min.

[trikke]

Which maybe prooves that some people are not able to write "decent" Applications ( reusing/opening Ports, binding ports, closeing ports ... )

[bbrendon]

My zabbix server has between 250 to 500 connections in TIME_WAIT.

Clients have about 0 to 15 connections in TIME_WAIT.

...though I'm not sure this is a bad thing. I have started monitoring this TIME_WAIT issue though because I'm trying to solve a problem where agents seem to go to sleep for about 15 minutes causing false positives and large amounts of panic

[emmanux]

We do have a problem here, in our Zabbix Proxy.

Oracle Linux 6 on top of Hyper-V.

39624 TIME_WAIT connections. That's a lot of TIME_WAIT connections, and a lot of problems with the agents as you can imagine.

EDIT: after applying Palmertree's suggestion, the TIME_WAIT connections dropped to 835. Thanks!

[bbrendon]

Looking at this, there might be something wrong with zabbix. I checked a very busy apache server and it ranges from 200-1000 time_wait at once.

very busy as in 1.5 million hits per day.

Though my zabbix server shows time_wait between 6-20 but I wouldn't say its very busy.

[emmanux]

Reading this: https://vincent.bernat.im/en/blog/20...ate-linux.html I'm thinking in switching to active agents for this customer. But it's a mess with the templates, changing everything to active.
EDIT: read my next posts.

[emmanux]

We do have a problem here, in our Zabbix Proxy.

Oracle Linux 6 on top of Hyper-V.

39624 TIME_WAIT connections. That's a lot of TIME_WAIT connections, and a lot of problems with the agents as you can imagine.

EDIT: after applying Palmertree's suggestion, the TIME_WAIT connections dropped to 835. Thanks!

OK, doing this didn't solve my problem. I only have a cleaner netstat output, but agents still hanging.

I completely stopped the proxy, restarted the network two times, and all connections disappeared from the proxy's OS.

I'm now thinking that it is agents' fault, not proxy's. The hanging agents have a lot of CLOSE_WAIT connections, even with the proxy completely stopped and no TIME_WAIT connections on the proxy's side.

[kloczek]

Till this issue will be fixed, I think changing the TIME-WAIT interval is a good work-around,

No it is not.
Creating and deallocating in kernel space socket structures is quite heavy operation.
To avoid this only way is apply suggested settings.

PS. It is really shame that after many years still Linux does not come with default TCP stack settings allowing reuse sockets. Solaris, *BSD systems and even Windows solved such issues many years ago ad typically such kernel tuning is not necessary.

[emmanux]

Great, I used this program:

https://github.com/rghose/kill-close-wait-connections/blob/master/kill_close_wait_connections.pl

in order to kill all those CLOSE_WAIT connections on an Ubuntu system with a hung up zabbix agent. No avail. Still hang.

[emmanux]

UPDATE:

I used that same program to kill all the ESTABLISHED connections (five in total), and guess what... the agent started to respond again.

This is what I got on the log while I was killing the connections:

  4426:20160911:023042.070 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023043.093 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023043.093 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023044.101 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023044.101 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023045.109 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023045.109 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023046.117 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023046.117 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023047.125 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023047.125 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023048.133 Cannot get socket IP address: [107] Transport endpoint is not connected
  4426:20160911:023048.133 failed to accept an incoming connection: connection rejected, getpeername() failed: [107] Transport endpoint is not connected
  4426:20160911:023049.141 Cannot get socket IP address: [107] Transport endpoint is not connected

However, it's interesting to note, that those connections were completely bogus. The Proxy's OS wasn't aware of those connections (they didn't appear on netstat). Even more, I can shut down the Zabbix Proxy service completely, and the affected agents still hang.

What are those ESTABLISHED connections? Why isn't the agent killing them after a while? No timeout for them? Bizarre.

[emmanux]

This is my lsof, for an affected agent:

zabbix_ag  6180     zabbix  cwd       DIR        8,1      4096          2 /
zabbix_ag  6180     zabbix  rtd       DIR        8,1      4096          2 /
zabbix_ag  6180     zabbix  txt       REG        8,1    291488    5248453 /opt/zabbix-agent/sbin/zabbix_agentd
zabbix_ag  6180     zabbix  DEL       REG        0,4               393216 /SYSV6c0115c3
zabbix_ag  6180     zabbix  mem       REG        8,1    124663    3673965 /lib/i386-linux-gnu/libpthread-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1   1734120    3673976 /lib/i386-linux-gnu/libc-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1     83776    3673962 /lib/i386-linux-gnu/libresolv-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1     30684    3673967 /lib/i386-linux-gnu/librt-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1     13940    3673975 /lib/i386-linux-gnu/libdl-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1    173576    3673971 /lib/i386-linux-gnu/libm-2.15.so
zabbix_ag  6180     zabbix  mem       REG        8,1    134344    3673966 /lib/i386-linux-gnu/ld-2.15.so
zabbix_ag  6180     zabbix    0r      CHR        1,3       0t0       1029 /dev/null
zabbix_ag  6180     zabbix    1w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6180     zabbix    2w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6180     zabbix    3wW     REG        8,1         4    5248088 /opt/zabbix-agent/tmp/zabbix_agentd.pid
zabbix_ag  6180     zabbix    4u     IPv4   28607462       0t0        TCP *:zabbix-agent (LISTEN)
zabbix_ag  6181     zabbix  cwd       DIR        8,1      4096          2 /
zabbix_ag  6181     zabbix  rtd       DIR        8,1      4096          2 /
zabbix_ag  6181     zabbix  txt       REG        8,1    291488    5248453 /opt/zabbix-agent/sbin/zabbix_agentd
zabbix_ag  6181     zabbix  DEL       REG        0,4               393216 /SYSV6c0115c3
zabbix_ag  6181     zabbix  mem       REG        8,1    124663    3673965 /lib/i386-linux-gnu/libpthread-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1   1734120    3673976 /lib/i386-linux-gnu/libc-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1     83776    3673962 /lib/i386-linux-gnu/libresolv-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1     30684    3673967 /lib/i386-linux-gnu/librt-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1     13940    3673975 /lib/i386-linux-gnu/libdl-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1    173576    3673971 /lib/i386-linux-gnu/libm-2.15.so
zabbix_ag  6181     zabbix  mem       REG        8,1    134344    3673966 /lib/i386-linux-gnu/ld-2.15.so
zabbix_ag  6181     zabbix    0r      CHR        1,3       0t0       1029 /dev/null
zabbix_ag  6181     zabbix    1w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6181     zabbix    2w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6181     zabbix    3w      REG        8,1         4    5248088 /opt/zabbix-agent/tmp/zabbix_agentd.pid
zabbix_ag  6181     zabbix    4u     IPv4   28607462       0t0        TCP *:zabbix-agent (LISTEN)
zabbix_ag  6182     zabbix  cwd       DIR        8,1      4096          2 /
zabbix_ag  6182     zabbix  rtd       DIR        8,1      4096          2 /
zabbix_ag  6182     zabbix  txt       REG        8,1    291488    5248453 /opt/zabbix-agent/sbin/zabbix_agentd
zabbix_ag  6182     zabbix  DEL       REG        0,4               393216 /SYSV6c0115c3
zabbix_ag  6182     zabbix  mem       REG        8,1    124663    3673965 /lib/i386-linux-gnu/libpthread-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1   1734120    3673976 /lib/i386-linux-gnu/libc-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1     83776    3673962 /lib/i386-linux-gnu/libresolv-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1     30684    3673967 /lib/i386-linux-gnu/librt-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1     13940    3673975 /lib/i386-linux-gnu/libdl-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1    173576    3673971 /lib/i386-linux-gnu/libm-2.15.so
zabbix_ag  6182     zabbix  mem       REG        8,1    134344    3673966 /lib/i386-linux-gnu/ld-2.15.so
zabbix_ag  6182     zabbix    0r      CHR        1,3       0t0       1029 /dev/null
zabbix_ag  6182     zabbix    1w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6182     zabbix    2w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6182     zabbix    3w      REG        8,1         4    5248088 /opt/zabbix-agent/tmp/zabbix_agentd.pid
zabbix_ag  6182     zabbix    4u     IPv4   28607462       0t0        TCP *:zabbix-agent (LISTEN)
zabbix_ag  6182     zabbix    5u     IPv4   29160531       0t0        TCP srvurh111.customer.local:zabbix-agent->10.100.0.94:41474 (ESTABLISHED)
zabbix_ag  6183     zabbix  cwd       DIR        8,1      4096          2 /
zabbix_ag  6183     zabbix  rtd       DIR        8,1      4096          2 /
zabbix_ag  6183     zabbix  txt       REG        8,1    291488    5248453 /opt/zabbix-agent/sbin/zabbix_agentd
zabbix_ag  6183     zabbix  DEL       REG        0,4               393216 /SYSV6c0115c3
zabbix_ag  6183     zabbix  mem       REG        8,1    124663    3673965 /lib/i386-linux-gnu/libpthread-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1   1734120    3673976 /lib/i386-linux-gnu/libc-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1     83776    3673962 /lib/i386-linux-gnu/libresolv-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1     30684    3673967 /lib/i386-linux-gnu/librt-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1     13940    3673975 /lib/i386-linux-gnu/libdl-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1    173576    3673971 /lib/i386-linux-gnu/libm-2.15.so
zabbix_ag  6183     zabbix  mem       REG        8,1    134344    3673966 /lib/i386-linux-gnu/ld-2.15.so
zabbix_ag  6183     zabbix    0r      CHR        1,3       0t0       1029 /dev/null
zabbix_ag  6183     zabbix    1w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6183     zabbix    2w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6183     zabbix    3w      REG        8,1         4    5248088 /opt/zabbix-agent/tmp/zabbix_agentd.pid
zabbix_ag  6183     zabbix    4u     IPv4   28607462       0t0        TCP *:zabbix-agent (LISTEN)
zabbix_ag  6183     zabbix    5u     IPv4   29157699       0t0        TCP srvurh111.customer.local:zabbix-agent->10.100.0.94:45347 (ESTABLISHED)
zabbix_ag  6184     zabbix  cwd       DIR        8,1      4096          2 /
zabbix_ag  6184     zabbix  rtd       DIR        8,1      4096          2 /
zabbix_ag  6184     zabbix  txt       REG        8,1    291488    5248453 /opt/zabbix-agent/sbin/zabbix_agentd
zabbix_ag  6184     zabbix  DEL       REG        0,4               393216 /SYSV6c0115c3
zabbix_ag  6184     zabbix  mem       REG        8,1    124663    3673965 /lib/i386-linux-gnu/libpthread-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1   1734120    3673976 /lib/i386-linux-gnu/libc-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1     83776    3673962 /lib/i386-linux-gnu/libresolv-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1     30684    3673967 /lib/i386-linux-gnu/librt-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1     13940    3673975 /lib/i386-linux-gnu/libdl-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1    173576    3673971 /lib/i386-linux-gnu/libm-2.15.so
zabbix_ag  6184     zabbix  mem       REG        8,1    134344    3673966 /lib/i386-linux-gnu/ld-2.15.so
zabbix_ag  6184     zabbix    0r      CHR        1,3       0t0       1029 /dev/null
zabbix_ag  6184     zabbix    1w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6184     zabbix    2w      REG        8,1     18292    5248452 /opt/zabbix-agent/tmp/zabbix_agentd.log
zabbix_ag  6184     zabbix    3w      REG        8,1         4    5248088 /opt/zabbix-agent/tmp/zabbix_agentd.pid
zabbix_ag  6184     zabbix    4u     IPv4   28607462       0t0        TCP *:zabbix-agent (LISTEN)
zabbix_ag  6184     zabbix    5u     IPv4   29160562       0t0        TCP srvurh111.customer.local:zabbix-agent->10.100.0.94:40752 (ESTABLISHED)

Any ideas?