Problems when sending configuration to 1 out of 14 passive proxies

[TomPW]

I've got a zabbix 3.2.3 server running on a VPS, connected to 14 passive proxies in equally as many locations.
The zabbix traffic is routed over IPSEC VPN tunnels to all locations using Strongswan on the zabbix server, and various firewalls (Fortinet, Check Point, Cisco) on the 14 different locations.
Communication to 13 of the proxies is fine, but one proxy is giving me issues. The config cant be pushed, which gives me the following error:

cannot send configuration data to proxy <hostname> at <IP-address>: empty string received

I've redacted the hostname and IP address, those numbers are valid.

When running a tcpdump on the zabbix server I can see the connection being setup correctly (syn, syn/ack, ack), then 10 seconds of silence (I've raised the timeout on the proxy from 4 to 10 seconds) and then a fin/ack from the proxy.
Other agent/proxy communication IS working, the Administration -> Proxies has a valid "last seen" time for this proxy, and I can also see valid connections in tcpdump. Also communication with the zabbix agent (I know, different port) is running fine.

Comparing this to a working proxy:
syn, syn/ack, ack, 0,3ms silence, and then the config file is pushed succesfully.

This leads me to believe this isn't an issue in the server/proxy communication, but in generating the proxy config.
I've tried deleting the proxy and re-adding it, but that also didn't solve my issue. Raising the log level to 5 on the server also didn't tell me anything I didn't know.
When googling for "cannot send configuration data to proxy" "empty string received" (including the quotes) I'm not getting any results (I even tried Bing and DuckDuckGo out of desperation).

Any pointers on how to further troubleshoot this issue are highly appreciated.

[allexpetrov]

Can you telnet from Server to that proxy. Issue the command:

telnet <ip/hostname> 10051

Regards,
Alex!

[TomPW]

Telnet to port 10051 works fine, just like all other server/proxy traffic over port 10051. Only pushing the configuration doesn't work.

[allexpetrov]

You can show the performance graphs from proxy, this could also give some clue

Regards,
Alex!

[TomPW]

I've uploaded the performance graph.

[allexpetrov]

Have you checked Proxy Pollers load on your Zabbix Server ?

Regards,
Aleksejs!

[TomPW]

I can't imagine that to be the problem, considering generating and sending the config for the other 13 proxies is working fine.

[brhunt]

TomPW, did you ever have any luck solving this?

We are installing our first proxy, and I am seeing the same behavior.

We are monitoring two SNMP devices, and I see some data coming over, and some data not.

And the log is full of the "empty string received" messages.

Thanks.

Bryan Hunt

[TomPW]

Unfortunately we are still struggling with this issue.

[brhunt]

We were able to resolve our issue, although I am not exactly sure which change that we made was the one to do it.

First, we noticed that we were having memory issues with the zabbix server. We modified the zabbix_server.conf file to increase the: trend write cache; configuration cache; history index cache; history write cache; and value cache.

Second, we made some recommended modifications to the hosts files on both the zabbix server and the zabbix proxy. The proxy server is across the Internet from the zabbix server, so there is a NAT device between the two and the proxy is accessed via a public IP address.

On the zabbix server, we added the proxy server name and its public IP address (our proxy Linux host name and the proxy name in Administration/Proxies are exactly the same).
127.0.0.1 localhost
::1 localhost
8.9.10.11 ZABBIX_PROXY_NAME

On the proxy server, we added the proxy server name to the localhost addresses.
127.0.0.1 ZABBIX_PROXY_NAME locahost
::1 ZABBIX_PROXY_NAME locahost


Since both changes (memory settings and hosts file entries) were made at the same time, I'm not sure which one corrected the issue. But, all are running well now.

Hopefully one of these will help someone in the future.

Thanks.

Bryan Hunt

[kaloyan.s]

Hi all,

I had the same issue a few times.

First time i needed to re-create the database on the proxy and it worked for 30 days.
Second time i needed to delete the proxy from zabbix Web UI and then add it again (after 5 mins)
It seems like a strange issue - probably on the master, but i cannot prove it for now.
We are using passive proxies.

Regards,
Kaloyan S.

[dmarsh]

Saw this exact error today (cannot send proxy data to server at "<hostname>": empty string received). It was caused by a mis-match between the zabbix proxy and server versions. Once the proxy version was changed to match the server it started working.