Server and Proxy stop communicating after 6-12 hours?!

[sai.barker]

Hi,

I'm new to Zabbix and I'm have been racking my brain for the past week trying to get a test lab setup with a Zabbix Server and Proxy on a remote network connected over an L2TP/IPSEC VPN.

I can get everything up and running a few hosts monitored through the proxy and it all seems to work fine... Until it doesn't. It seems random but after 6-12 hours the Zabbix server will stop receiving data from the proxy despite showing online in the frontend. I have re-installed both the server and proxy with various combinations of OS and DB but the problem still keeps happening.

Currently I have the latest versions of Zabbix-Server and Zabbix-Proxy (7.0LTS) running on Ubuntu Server 24.04 and using PostgreSQL. Using this installation guide from Zabbix.

Again, the Zabbix server says the proxy is online through the front end.


There are errors in the Zabbix-Server and Zabbix-Proxy.

Extract from /var/log/zabbix_server.log

979:20240928:180217.750 sending configuration data to proxy "customer01.proxy" at "10.120.0.223", datalen 425, bytes 205 with compression ratio 2.1
982:20240928:180219.255 failed to accept an incoming connection: from 10.120.0.223: reading first byte from connection failed: [11] Resource temporarily unavailable
982:20240928:180225.616 failed to accept an incoming connection: from 10.120.0.223: reading first byte from connection failed: [11] Resource temporarily unavailable
981:20240928:180259.931 sending configuration data to proxy "customer01.proxy" at "10.120.0.223", datalen 425, bytes 205 with compression ratio 2.1

Extract from /var/log/zabbix_proxy.log

924:20240928:075946.022 cannot send proxy data to server at "10.240.0.71": empty string received
924:20240928:075952.415 cannot send proxy data to server at "10.240.0.71": empty string received
913:20240928:075955.614 received configuration data from server at "10.240.0.71", datalen 425
924:20240928:075958.814 cannot send proxy data to server at "10.240.0.71": empty string received​

Telnet successfully connects on port 10051 from server > proxy AND proxy > server.
postgresql.service is still running on the server and proxy also.

Any ideas or suggestions of things to try would be greatly appreciated!

[kamil1]

HI,
Seems like connection works "partially" since your telnet testing is successful and also config can be received from server based on the log:
"913:20240928:075955.614 received configuration data from server at "10.240.0.71", datalen 425"

First of all, try to increase DebugLog to at least 4 and see whether the proxy/server logs return more useful information.
Couple of thought to consider:

• Have you tried to see the load metrics? Maybe we are facing resource exhaustion? That can be verified with:
  top -b -d 10 -n 60 > top_output.txt
• Also once problem occurs, you might want to capture the network traffic (on server side) with:
  tcpdump -npi any -s 0 -w /path/to/save/file.pcap host <proxy_ip>
• Additionally, you can check for packet losses once error comes up (with ping utility):
  ping -s 1400 -c 100 -i 0.01 <proxy_ip>
• Also make sure that your VPN connection is stable (maybe it gets disconnected for some time after those 6-12h?).

Regards,
Kamil

[sai.barker]

Hi,

Thanks for your response.

I have found that deleting the hosts and re-discovering them works. The proxy will start sending metrics the the server and the server will collect them again as normal. However the proxy stops working again after several hours.

Monitoring the hosts directly from the zabbix server over the VPN, essentially bypassing the proxy works perfectly.

I have checked the VPN logs and there are no drop outs in VPN connectivity. The VPN session time is 6 days. There is also no packet loss between server and proxy.

Here is a link to the log files and pcap which I have uploaded to OneDrive. The forum won't allow me to attache them.

[sai.barker]

Update:

I have installed zabbix proxy on a completely different bare metal (MSI Cubi 4 with i3-1215u CPU) and haven't had any problems what so ever for approximately a week of testing. Previously I was using an Ali-express mini PC with a J1900 CPU. I'm still unsure of the actual cause of the problem but I'll just stick with the MIS Cubi or Intel NUC for Proxies for now on.