ZBX_TCP_READ() failed: [4] Interrupted system call

[clahti]

Hello all:

I have researched this and cannot determine the cause of my issue. We relocated 5 virtual machines from one of our datacenters to a new cloud, these were being monitored by zabbix just fine until the move. This seems to be a network problem, however I cannot figure this out. For example, server=zabbix.shotspotter.com and client=fisheye.shotspotter.com

Here is what I see in the agent log:

 25680:20160801:102904.387 active check configuration update from [zabbix.shotspotter.com:10051] started to fail (ZBX_TCP_READ() failed: [4] Interrupted system call)

I see nothing in the server log regarding fisheye client. There does not appear to be any issue connecting to the required ports via a dedicated VPN connection:

[root@zabbix ~]# nc -v -z fisheye.shotspotter.com 10050
Connection to fisheye.shotspotter.com 10050 port [tcp/zabbix-agent] succeeded!
[root@zabbix ~]# 

[root@fisheye ~]# nc -v -z zabbix.shotspotter.com 10051
Connection to zabbix.shotspotter.com 10051 port [tcp/zabbix-trapper] succeeded!
[root@fisheye ~]#

I can reach the client using zabbix_get no problem, this is strictly a problem with Agent Active items:

[root@zabbix ~]# zabbix_get -s fisheye -k agent.version
2.4.8
[root@zabbix ~]#

[clahti]

If I increase the agent log to DebugLevel=4 I get:

 26578:20160801:105559.012 active checks #1 [getting list of active checks]
 26578:20160801:105559.013 In refresh_active_checks() host:'zabbix.shotspotter.com' port:10051
 26578:20160801:105559.055 sending [{"request":"active checks","host":"fisheye"}]
 26578:20160801:105559.055 before read
 26574:20160801:105600.012 collector [processing data]
 26574:20160801:105600.012 In update_cpustats()
 26574:20160801:105600.012 End of update_cpustats()
 26574:20160801:105600.012 collector [idle 1 sec]
 26574:20160801:105601.013 collector [processing data]
 26574:20160801:105601.013 In update_cpustats()
 26574:20160801:105601.014 End of update_cpustats()
 26574:20160801:105601.014 collector [idle 1 sec]
 26574:20160801:105602.014 collector [processing data]
 26574:20160801:105602.014 In update_cpustats()
 26574:20160801:105602.014 End of update_cpustats()
 26574:20160801:105602.014 collector [idle 1 sec]
 26574:20160801:105603.014 collector [processing data]
 26574:20160801:105603.014 In update_cpustats()
 26574:20160801:105603.015 End of update_cpustats()
 26574:20160801:105603.015 collector [idle 1 sec]
 26574:20160801:105604.015 collector [processing data]
 26574:20160801:105604.015 In update_cpustats()
 26574:20160801:105604.015 End of update_cpustats()
 26574:20160801:105604.015 collector [idle 1 sec]
 26574:20160801:105605.015 collector [processing data]
 26574:20160801:105605.015 In update_cpustats()
 26574:20160801:105605.015 End of update_cpustats()
 26574:20160801:105605.015 collector [idle 1 sec]
 26574:20160801:105606.015 collector [processing data]
 26574:20160801:105606.016 In update_cpustats()
 26574:20160801:105606.016 End of update_cpustats()
 26574:20160801:105606.016 collector [idle 1 sec]
 26574:20160801:105607.016 collector [processing data]
 26574:20160801:105607.016 In update_cpustats()
 26574:20160801:105607.016 End of update_cpustats()
 26574:20160801:105607.016 collector [idle 1 sec]
 26574:20160801:105608.016 collector [processing data]
 26574:20160801:105608.016 In update_cpustats()
 26574:20160801:105608.016 End of update_cpustats()
 26574:20160801:105608.016 collector [idle 1 sec]
 26578:20160801:105609.014 active check configuration update from [zabbix.shotspotter.com:10051] started to fail (ZBX_TCP_READ() failed: [4] Interrupted system call)
 26578:20160801:105609.014 End of refresh_active_checks():FAIL
 26578:20160801:105609.014 active checks #1 [processing active checks]
 26578:20160801:105609.014 In process_active_checks() server:'zabbix.shotspotter.com' port:10051)
 26578:20160801:105609.014 End of process_active_checks()
 26578:20160801:105609.014 In get_min_nextcheck()
 26578:20160801:105609.014 In send_buffer() host:'zabbix.shotspotter.com' port:10051 values:0/100
 26578:20160801:105609.014 End of send_buffer():SUCCEED
 26578:20160801:105609.014 active checks #1 [idle 1 sec]
 26574:20160801:105609.017 collector [processing data]
 26574:20160801:105609.017 In update_cpustats()

[clahti]

here is my agentd.conf file

[root@fisheye zabbix]# cat /etc/zabbix/zabbix_agentd.conf
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=10
DebugLevel=4
EnableRemoteCommands=1
Server=zabbix.shotspotter.com
ServerActive=zabbix.shotspotter.com
Hostname=fisheye
RefreshActiveChecks=300
Timeout=10

[root@fisheye zabbix]#

[clahti]

here is my server.conf

[root@zabbix ~]# grep -v -e ^# -e ^$ /etc/zabbix/zabbix_server.conf 
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
DebugLevel=3
PidFile=/var/run/zabbix/zabbix_server.pid
DBName=<private>
DBUser=<private>
DBPassword=<private>
DBSocket=/var/lib/mysql/mysql.sock
StartPollers=15
StartPollersUnreachable=15
StartTrappers=15
StartPingers=10
StartDiscoverers=3
StartHTTPPollers=1
SNMPTrapperFile=/var/log/snmptt/snmptt.log
MaxHousekeeperDelete=10000
CacheSize=512M
StartDBSyncers=10
HistoryCacheSize=256M
TrendCacheSize=128M
HistoryTextCacheSize=256M
Timeout=30
TrapperTimeout=300
UnreachablePeriod=600
UnavailableDelay=120
UnreachableDelay=30
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=3000
StartVMwareCollectors=25
VMwareCacheSize=1G
VMwareFrequency=60
[root@zabbix ~]#

[clahti]

(SOLVED) ZBX_TCP_READ() failed: [4] Interrupted system call

this was not a zabbix issue after all. The MTU on the IPSEC tunnel was not set low enough, too many hops to the cloud.