For some reason or another, the incoming connection of one of my 30 proxies to my zabbix-server dies half-way in or something. I don't know if a firewall sends a connection-reset to the proxy, or what's going on. There's some problems there to one proxy's specific IP-address.
All other proxy connections are 100% fine. The problem is a local system problem (as I'll describe).
This situation causes all the other proxy-connections to freeze up! All 30 of them. When I check the TCP-stack (command ss -tn) on the server of all the other current proxy connections, I can see that the TCP-buffers for incoming data are full since the zabbix server isn't processing the incoming data: it just lies there on the TCP stack.. the zabbix process(es) aren't fetching it. The consequence is that those other proxie's data/connections aren't processed at all and "last seen" status in the proxy meny in the zabbix WebUI is just counting up virtually forever, until I reboot the whole OS in some cases (restarting server not enough). That's the big picture.
I've inspected the traffic between the other proxies and the server (tcpdumped the endpoints) - all traffic goes through and does so correctly, there is no network problem there whatsoever.
So, faulty firewall rules and what not aside - what's going on with Zabbix here? It seems like there's just one process doing a recv() on the one faulty proxy or something, and gets stuck and blocks forever. Maybe the faulty proxy stacks up connections forever which locks up the recv() system calls on other proxies' connections?
All other proxy connections are 100% fine. The problem is a local system problem (as I'll describe).
This situation causes all the other proxy-connections to freeze up! All 30 of them. When I check the TCP-stack (command ss -tn) on the server of all the other current proxy connections, I can see that the TCP-buffers for incoming data are full since the zabbix server isn't processing the incoming data: it just lies there on the TCP stack.. the zabbix process(es) aren't fetching it. The consequence is that those other proxie's data/connections aren't processed at all and "last seen" status in the proxy meny in the zabbix WebUI is just counting up virtually forever, until I reboot the whole OS in some cases (restarting server not enough). That's the big picture.
I've inspected the traffic between the other proxies and the server (tcpdumped the endpoints) - all traffic goes through and does so correctly, there is no network problem there whatsoever.
So, faulty firewall rules and what not aside - what's going on with Zabbix here? It seems like there's just one process doing a recv() on the one faulty proxy or something, and gets stuck and blocks forever. Maybe the faulty proxy stacks up connections forever which locks up the recv() system calls on other proxies' connections?
Comment