Random connections errors to remote zabbix_agentd

[Firm]

2551:20091217:132135.687 Item [<hostname>:agent.ping] error: Get value from agent failed: Cannot connect to [<ip_address>:10050] [Interrupted system call]
2551:20091217:132136.011 ZABBIX Host [<hostname>]: first network error, wait for 15 seconds

It happens for different hosts even in local network segment. No connectivity errors observed. No errors found in zabbix_agentd.log files on remote hosts.

What may be the reason of this?

[MrKen]

This happens to me on random hosts from time to time, without explanation.

In fact it happened just yesterday. agent.ping error Get value from agent failed. Opened a terminal session and pinged the offending host, and the ping works fine. Log in to the offending server, everything appears fine; zabbix agentd is running. Restart the agentd, check the agentd log, nothing unusual in the log, but still the errors in the zabbix_server.log

And then 20 to 30 minutes later, the offending host begins monitoring again as if nothing had happened.

Why does it happen? My guess is some sort of bottleneck within the network at that time. Time to put on the Sherlock Holmes outfit to determine the wheres and whys, and then get zabbix onto the job for next time!

MrKen

[elvar]

2551:20091217:132135.687 Item [<hostname>:agent.ping] error: Get value from agent failed: Cannot connect to [<ip_address>:10050] [Interrupted system call]
2551:20091217:132136.011 ZABBIX Host [<hostname>]: first network error, wait for 15 seconds

It happens for different hosts even in local network segment. No connectivity errors observed. No errors found in zabbix_agentd.log files on remote hosts.

What may be the reason of this?

I'm experiencing this exact same problem for one of my hosts, but only one. I've tested using manual constant pings of my own and don't see connection issues so I'm really confused as to why agent.ping keeps failing.

[robertmcox]

I see this too...

Server 1.6.6, Agents 1.6.1, active checks ONLY

What is funny to me is that the issue seems to occur with hosts on the LAN (on a gigE link). Other hosts that we check via the Internet don't show the issue.... :-(

[elvar]

I see this too...

Server 1.6.6, Agents 1.6.1, active checks ONLY

What is funny to me is that the issue seems to occur with hosts on the LAN (on a gigE link). Other hosts that we check via the Internet don't show the issue.... :-(

The one I'm having the problem with is not a LAN host FYI.

[robertmcox]

It took me a while to figure out, but (fingers crossed) I may have discovered my issue.

I have host-based firewalling and I had a chain dropping "invalid" packets. Logging all traffic to/from Zabbix, I saw lots of "invalid" packets being dropped. I adjusted my rules and the issue seems to have cleared up!

[elvar]

It took me a while to figure out, but (fingers crossed) I may have discovered my issue.

I have host-based firewalling and I had a chain dropping "invalid" packets. Logging all traffic to/from Zabbix, I saw lots of "invalid" packets being dropped. I adjusted my rules and the issue seems to have cleared up!

You using iptables?

[robertmcox]

Yes.

Specifically, I was using csf firewall on the host(s) in question. The configuration variable to change is:

packet_filter="0"
drop_pf_logging="0"

Alternatively, you could "whitelist" the Zabbix server IP address.

[elvar]

It took me a while to figure out, but (fingers crossed) I may have discovered my issue.

I have host-based firewalling and I had a chain dropping "invalid" packets. Logging all traffic to/from Zabbix, I saw lots of "invalid" packets being dropped. I adjusted my rules and the issue seems to have cleared up!

The firewall at the site where I'm having this issue with the agent is a Fortigate 111c. I have other agents / hosts behind other Fortigates without issues. It's only this one.

[elvar]

Solved

I fixed my problem by raising Timeout=3 to Timeout=5 in my zabbix_server.conf. The default was fine for all hosts but this one. Such an easy fix I wish I had tried that a long time ago.