Getting Connection reset by peer by server from proxies

[maxxer]

Hi.
Since some days I'm having troubles getting data from remote proxies to the main server.
Server connectivity is fine, I tried running mtr and I get only few ping loss (less than 2%). Load is fine as well, mostly around 0.5, so I tend to exclude OS issues.
I didn't change load or behaviour in the recent days (or at least nothing relevant I can recall about), but from time to time all of the proxies (many different connection providers) lag data behind because of server connection issues.

Server and proxies are all up to date with the latest LTS 4.0.19. I upgraded from 3.0.x a couple of weeks ago and has been running smooth until now.
I asked the server provider for a checkup but they confirmed everything is fine on their side.

I increased both server and proxy log but couldn't spot aborted or connection errors. Just this

 13588:20200425:203353.349 In connect_to_server() [zabbix.server.com]:10051 [timeout:60]

This is the proxy log without debug

 27602:20200426:151001.724 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27602:20200426:152019.126 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27602:20200426:152315.710 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27602:20200426:153417.892 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27595:20200426:153840.422 received configuration data from server at "zabbix.server.com", datalen 97801
 27602:20200426:154831.131 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27602:20200426:155330.366 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27602:20200426:155924.541 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27603:20200426:160800.850 executing housekeeper
 27603:20200426:160800.891 housekeeper [deleted 11106 records in 0.033240 sec, idle for 1 hour(s)]
 27602:20200426:161740.216 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27602:20200426:162135.359 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27602:20200426:163201.958 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27601:20200426:163542.058 cannot send heartbeat message to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27595:20200426:163841.073 received configuration data from server at "zabbix.server.com", datalen 97801
 27602:20200426:164018.175 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() failed: [104] Connection reset by peer
 27602:20200426:165028.933 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out
 27602:20200426:170324.359 cannot send proxy data to server at "zabbix.server.com": ZBX_TCP_READ() timed out

I don't think this is a strict Zabbix issue but I ran out of ideas so I'm starting from here. Thanks

[maxxer]

The connection issue went away on its own. Possibly it has been a (global) routing issue not related to server or clients

[davidkillingsworth]

I'm getting exactly this problem.

• We are an MSP that has a main Zimbra console at our head office. The primary server is publicly available outside our firewall via ports 10050/10051.
• We have Zabbix proxies setup at our customers sites behind their firewalls and they report back to the primary server. They are using PSK encryption for communication.
• This Zabbix infrastructure has been running without any problem for the past couple of years.
• Just over the last 4 or 5 days, we are getting constant notification alerts with the following errors from 3 or 4 of the proxy servers.
• The problem duration usually lasts for about 6 minutes, then resolves itself. A few minutes later, the problem happens again.
• Any idea how to resolve this, or what might be causing the issue?

Errors:

Zabbix data sender processes more than 75% busy
Zabbix configuration syncer processes more than 75% busy

In our Zabbix Proxy log, I find this:

1129:20220306:174524.978 received configuration data from server at "server.hostname.com", datalen 683928
1187:20220306:174554.227 cannot send proxy data to server at "server.hostname.com": SSL_read() connection closed by peer
1129:20220306:174705.483 received configuration data from server at "server.hostname.com", datalen 683928
1187:20220306:174830.133 Unable to connect to the server [server.hostname.com]:10051 [cannot connect to [[server.hostname.com]:10051]: [111] Connection refused]. Will ret>
1187:20220306:174830.147 Connection restored.
1129:20220306:174845.943 received configuration data from server at "server.hostname.com", datalen 683928
1129:20220306:175026.410 received configuration data from server at "server.hostname.com", datalen 683928
1129:20220306:175206.864 received configuration data from server at "server.hostname.com", datalen 683928
1129:20220306:175347.315 received configuration data from server at "server.hostname.com", datalen 683928

Servers and Proxy's are both Ubuntu 20.04.4 LTS fully patched.
Zabbix version is 5.04 LTS.

Any ideas?

[Hernandes Martins]

The same case, please check you network connections, maybe have issue with vpn or route.

Best regards