Zabbix 2.2.2 Cannot get socket IP address and crash

[marcinkloc]

Hello,

I dont know if it zabbix problem (or php, mysql) but today at 5:20 AM zabbix stopped accepting data, and write in log lot of "Cannot get socket IP address: [107] Transport endpoint is not connected" errors.

35686:20140227:045507.914 housekeeper [deleted 200397 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 42.671649 sec, idle 1 hour(s)]
 35654:20140227:045844.172 sending configuration data to proxy "Z_IBA_proxy", datalen 39883
 35654:20140227:050041.063 sending configuration data to proxy "Z_BCC_proxy", datalen 426842
 35654:20140227:051423.303 sending configuration data to proxy "Z_DWRK_proxy", datalen 109943
 35654:20140227:051734.255 sending configuration data to proxy "Z_PCC_proxy", datalen 332744
 35654:20140227:052458.133 Cannot get socket IP address: [107] Transport endpoint is not connected
 35654:20140227:052458.135 Cannot get socket IP address: [107] Transport endpoint is not connected
 35654:20140227:052458.137 Cannot get socket IP address: [107] Transport endpoint is not connected
 35654:20140227:052458.141 Cannot get socket IP address: [107] Transport endpoint is not connected
 35654:20140227:052458.145 Cannot get socket IP address: [107] Transport endpoint is not connected
 35654:20140227:052458.153 Cannot get socket IP address: [107] Transport endpoint is not connected

Zabbix queue raised up to 4.5K delayed items. At the end of the log was some "mysql has gone away" and "pooler 99 unrechable" entries.

Where should I look for problems? Is it zabbix or mysql/php problem. My zabbix installation is about 100 hosts / 5200 items / 2000 triggers with 5 proxies and zabbix server processes are bored.

Thanks in advance
Marcin

[steveboyson]

Do you run zabbix on a NFS volume?

[MOREA]

Hi,

We got exactly the same issue.
We do not run zabbix on a NFS volume.

Restarting Zabbix server solved it.

Could it be a zabbix 2.2.2 bug, since we never got this issue before ugrade zabbix to this version ?

Regards,

Karl.

[marcinkloc]

Do you run zabbix on a NFS volume?

No, it's local lvm ext4 volume

Yes, restarting zabbix service solved the problem.

[steveboyson]

Hmmm. I'm not too sure what causes this.
What interfaces have you defined for zabbix in the zabbix_server.conf and for the database connect? Is the DB running on the same system or is it a remote DB?

It looks like it loses the connection between proxy and DB. Are they on the same LAN or is the traffic to be routed through a gateway/router?

[marcinkloc]

In my case, zabbix DB running on the same host as zabbix server (it's HyperV vm). Proxies are running on different locations (active proxy) with own databases (mysql or sqlite).

When problem appear I checked mysql server status and it was running but zabbix can not connect to it and write data. Please look at attached graph - it seems that some data was stored.

[safpsr]

I have the same behavior:


With these activities at the same time:


But in my case zabbix server doesn't crash. After 30 minutes, the server works properly ...

Very strange. I didn't see that before. My zabbix server works with 2.2.2 version.

[johansen]

I have the same problem on one of our proxy running now 2.2.4 (before it was 2.2.2). Was anybody able to solve this?

Proxy DB and service running on same host (mysql)
Master DB running on different host

[ZabbixFun]

Cannot get socket IP address: [107] Transport endpoint is not connected

I had the same errors on zabbix 2.2.5 server log:

Cannot get socket IP address: [107] Transport endpoint is not connected

At this moment I got messages from 5 servers that zabbix agent is unavailable.
After 5 minutes everything was OK.

Then I upgraded zabbix server and agents to v.2.2.6 but next morning, at the same time, I got the same errors (only 2 of the servers were different - the other 3 were the same servers).

The server is a VM on CentOS 6.5 monitoring around 30 servers and all updates are installed.

Any idea where to look?
Anybody else had the same issue?

[johansen]

I rebooted my proxy server and since then problems are gone

[ZabbixFun]

Thanks for the reply...

I have rebooted the server also and I'll see tomorrow if this solved the issue.

[ZabbixFun]

The error still persists, every day at the same time - around 8:05 AM:

11465:20141021:080443.776 Cannot get socket IP address: [107] Transport endpoint is not connected
  5807:20141022:080449.108 Cannot get socket IP address: [107] Transport endpoint is not connected
  5808:20141022:080449.109 Cannot get socket IP address: [107] Transport endpoint is not connected
  5806:20141022:080449.109 Cannot get socket IP address: [107] Transport endpoint is not connected
  1818:20141023:080512.636 Cannot get socket IP address: [107] Transport endpoint is not connected
  1819:20141023:080512.637 Cannot get socket IP address: [107] Transport endpoint is not connected

Actions performed so far:
• Upgraded Zabbix server (v.2.2.5 > v.2.2.6) and OS
• Upgraded zabbix agents on some of the above and other servers (v.2.2.5, v.2.2.4 > v.2.2.6)
• Restarted network service on Zabbix server
• Restarted Zabbbix server

Some of the servers are monitored via IPsec, some of them not.

Anyone else having the same issues?

Thanks

EDIT:
At the same time I get the following error on the monitored server - agent log:

active check configuration update from [123.45.67.89:10051] started to fail (ZBX_TCP_READ() failed: [4] Interrupted system call)

[ZabbixFun]

For a few days everything was OK and suddenly today I got a lot af messages that zabbix agent is unavailable on server <hostname>

On zabbix server log I only see this:

Cannot get socket IP address: [107] Transport endpoint is not connected

Any one experiences this issue?

Regards,

[Alex_SYB]

Me, I have started to get these.

2.2.4

any resolution ?

[troffasky]

I get this at the same time every night. All hosts monitored with Zabbix agent show as down for about 20 minutes, then come up again. I get this same error about 'Transport endpoint not connected'. Hosts monitored with SNMP also show a hole in the graph data for the same period. Hosts monitored with ICMP are fine, however. I suspect it's snapshots [Zabbix is running in a VM] or the SAN throughput being maxed out by a backup job on another guest.

[Edit]: OK never mind, I found the cause: a cron job at 2030 that runs a mysqldump --all-databases. I am not sure why it only seems to affect Zabbix agent and SNMP checks, not ICMP. It also doesn't affect other MySQL applications on the same server, eg Cacti.