zabbix_server stops data colletion with 'file log.c lock failed Invalid argument'

[marafado88]

Hello everyone,

I am using Zabbix 3.4.7 for a year, and in the pass few days I have encounter an issue where zabbix_server stops collecting data, but webservice continues to work fine. Also zabbix_server is running in the system, who is debian.

In zabbix_server.log those are the last lines until it stops, seems like zabbix server is looping(dont know if its his normal behaviour but I dont think so):

1099:20190213:031416.980 Starting Zabbix Server. Zabbix 3.4.7 (revision 77720).
1099:20190213:031416.980 ****** Enabled features ******
1099:20190213:031416.980 SNMP monitoring: YES
1099:20190213:031416.980 IPMI monitoring: YES
1099:20190213:031416.980 Web monitoring: YES
1099:20190213:031416.980 VMware monitoring: YES
1099:20190213:031416.980 SMTP authentication: YES
1099:20190213:031416.980 Jabber notifications: YES
1099:20190213:031416.980 Ez Texting notifications: YES
1099:20190213:031416.980 ODBC: YES
1099:20190213:031416.980 SSH2 support: YES
1099:20190213:031416.980 IPv6 support: YES
1099:20190213:031416.980 TLS support: YES
1099:20190213:031416.980 ******************************
1099:20190213:031416.980 using configuration file: /etc/zabbix/zabbix_server.conf
1099:20190213:031417.021 current database version (mandatory/optional): 03040000/03040006
1099:20190213:031417.021 required mandatory version: 03040000
1099:20190213:031417.193 server #0 started [main process]
1103:20190213:031417.194 server #1 started [configuration syncer #1]
1104:20190213:031417.194 server #2 started [alerter #1]
1105:20190213:031417.195 server #3 started [alerter #2]
1106:20190213:031417.195 server #4 started [alerter #3]
1107:20190213:031417.196 server #5 started [housekeeper #1]
1108:20190213:031417.196 server #6 started [timer #1]
1109:20190213:031417.197 server #7 started [http poller #1]
1110:20190213:031417.198 server #8 started [discoverer #1]
1112:20190213:031417.198 server #9 started [history syncer #1]
1115:20190213:031417.199 server #11 started [history syncer #3]
1114:20190213:031417.201 server #10 started [history syncer #2]
1122:20190213:031417.202 server #14 started [proxy poller #1]
1125:20190213:031417.204 server #16 started [task manager #1]
1126:20190213:031417.206 server #17 started [poller #1]
1120:20190213:031417.206 server #13 started [escalator #1]
1130:20190213:031417.210 server #20 started [poller #4]
1135:20190213:031417.212 server #24 started [trapper #2]
1118:20190213:031417.212 server #12 started [history syncer #4]
1129:20190213:031417.213 server #19 started [poller #3]
1132:20190213:031417.214 server #21 started [poller #5]
1136:20190213:031417.219 server #25 started [trapper #3]
1124:20190213:031417.219 server #15 started [self-monitoring #1]
1145:20190213:031417.221 server #28 started [icmp pinger #1]
1138:20190213:031417.222 server #26 started [trapper #4]
1150:20190213:031417.222 server #30 started [preprocessing manager #1]
1134:20190213:031417.222 server #23 started [trapper #1]
1149:20190213:031417.224 server #29 started [alert manager #1]
1128:20190213:031417.225 server #18 started [poller #2]
1143:20190213:031417.235 server #27 started [trapper #5]
1133:20190213:031417.238 server #22 started [unreachable poller #1]
1151:20190213:031418.049 server #31 started [preprocessing worker #1]
1152:20190213:031418.135 server #32 started [preprocessing worker #2]
1153:20190213:031418.135 server #33 started [preprocessing worker #3]
1132:20190213:031615.862 Zabbix agent item "net.if.in[Miniport WAN (monitor de rede)]" on host "xxx-Server" f$
1133:20190213:031634.789 Zabbix agent item "service.info[HealthAlertsSvc,state]" on host "xxx-Server" failed:$
1133:20190213:031653.822 Zabbix agent item "net.if.out[Adaptador de Rede do Microsoft Hyper-V-QoS Packet Scheduler$
1133:20190213:031712.868 temporarily disabling Zabbix agent checks on host "xxx-Server": host unavailable
1126:20190213:031746.906 Zabbix agent item "service.info[SQLAgent$ACRONYM,state]" on host "x-SERVER" failed: fir$
1133:20190213:031801.084 resuming Zabbix agent checks on host "x-SERVER": connection restored
1130:20190213:032003.936 Zabbix agent item "service.info[UxSms,state]" on host "x-SERVER" failed: first network $
1133:20190213:032022.669 Zabbix agent item "net.if.out[WAN Miniport (L2TP)]" on host "x-SERVER" failed: another $
MIB search path: $HOME/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/shar$
Cannot find module (SNMPv2-SMI): At line 1 in (none)
1133:20190213:032038.343 resuming Zabbix agent checks on host "x-SERVER": connection restored
zabbix_server [1115]: [file:'selfmon.c',line:434] lock failed: [22] Invalid argument
zabbix_server [1099]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1103]: zabbix_server [1108]: zabbix_server [1107]: [file:'log.c',line:245] lock failed: [22] Invalid $


zabbix_server [1106]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1105]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1110]: zabbix_server [1109]: [file:'log.c',line:245] lock failed: [22] Invalid argument[file:'log.c',$

zabbix_server [1104]: zabbix_server [1149]: [file:'log.c',line:245] lock failed: [22] Invalid argument[file:'selfmon$

zabbix_server [1118]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1114]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1128]: zabbix_server [1124]: [file:'log.c',line:245] lock failed: [22] Invalid argument[file:'log.c',$

zabbix_server [1736]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1721]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1735]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1758]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1757]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1745]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1725]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1747]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1759]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1743]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1746]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1744]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1756]: [file:'log.c',line:245] lock failed: [22] Invalid argument
1706:20190213:032058.461 Starting Zabbix Server. Zabbix 3.4.7 (revision 77720).
1706:20190213:032058.461 ****** Enabled features ******
1706:20190213:032058.461 SNMP monitoring: YES
1706:20190213:032058.461 IPMI monitoring: YES
1706:20190213:032058.461 Web monitoring: YES
1706:20190213:032058.461 VMware monitoring: YES
1706:20190213:032058.461 SMTP authentication: YES
1706:20190213:032058.461 Jabber notifications: YES
1706:20190213:032058.461 Ez Texting notifications: YES
1706:20190213:032058.461 ODBC: YES
1706:20190213:032058.461 SSH2 support: YES
1706:20190213:032058.461 IPv6 support: YES
1706:20190213:032058.461 TLS support: YES
1706:20190213:032058.461 ******************************
1706:20190213:032058.461 using configuration file: /etc/zabbix/zabbix_server.conf
1706:20190213:032058.510 current database version (mandatory/optional): 03040000/03040006
1706:20190213:032058.510 required mandatory version: 03040000
1706:20190213:032058.736 server #0 started [main process]
1710:20190213:032058.737 server #1 started [configuration syncer #1]
1711:20190213:032058.738 server #2 started [alerter #1]
1712:20190213:032058.738 server #3 started [alerter #2]
1713:20190213:032058.739 server #4 started [alerter #3]
1714:20190213:032058.740 server #5 started [housekeeper #1]
1715:20190213:032058.740 server #6 started [timer #1]
1716:20190213:032058.741 server #7 started [http poller #1]
1717:20190213:032058.742 server #8 started [discoverer #1]
1719:20190213:032058.743 server #9 started [history syncer #1]
1721:20190213:032058.743 server #10 started [history syncer #2]
1722:20190213:032058.744 server #11 started [history syncer #3]
1728:20190213:032058.747 server #14 started [proxy poller #1]
1729:20190213:032058.748 server #15 started [self-monitoring #1]
1732:20190213:032058.749 server #17 started [poller #1]
1725:20190213:032058.750 server #13 started [escalator #1]
1723:20190213:032058.752 server #12 started [history syncer #4]
1733:20190213:032058.754 server #18 started [poller #2]
1736:20190213:032058.759 server #20 started [poller #4]
1730:20190213:032058.759 server #16 started [task manager #1]
1741:20190213:032058.759 server #23 started [trapper #1]
1735:20190213:032058.760 server #19 started [poller #3]
1743:20190213:032058.761 server #24 started [trapper #2]
1737:20190213:032058.762 server #21 started [poller #5]
1747:20190213:032058.763 server #28 started [icmp pinger #1]
1739:20190213:032058.764 server #22 started [unreachable poller #1]
1748:20190213:032058.765 server #29 started [alert manager #1]
1745:20190213:032058.767 server #26 started [trapper #4]
1744:20190213:032058.772 server #25 started [trapper #3]
1756:20190213:032058.773 server #30 started [preprocessing manager #1]
1746:20190213:032058.774 server #27 started [trapper #5]
1757:20190213:032059.090 server #31 started [preprocessing worker #1]
1759:20190213:032059.101 server #33 started [preprocessing worker #3]
1758:20190213:032059.182 server #32 started [preprocessing worker #2]
1739:20190213:032552.084 enabling Zabbix agent checks on host "xxx-Server": host became available
1732:20190213:032737.608 Zabbix agent item "net.if.in[Adaptador de Rede do Microsoft Hyper-V #2-WFP Native MAC Lay$
1739:20190213:032756.640 Zabbix agent item "net.if.out[Adaptador de Rede do Microsoft Hyper-V-QoS Packet Scheduler$
1739:20190213:032815.700 Zabbix agent item "service.info[ShellHWDetection,state]" on host "xxx-Server" failed$
1739:20190213:032834.790 temporarily disabling Zabbix agent checks on host "xxx-Server": host unavailable
MIB search path: $HOME/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/shar$
Cannot find module (SNMPv2-SMI): At line 1 in (none)
1739:20190213:033352.044 enabling Zabbix agent checks on host "xxx-Server": host became available
zabbix_server [1729]: [file:'selfmon.c',line:434] lock failed: [22] Invalid argument
zabbix_server [1729]: [file:'selfmon.c',line:434] lock failed: [22] Invalid argument
zabbix_server [1706]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1716]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1713]: [file:'log.c',line:245] lock failed: [22] Invalid argumentzabbix_server [1715]:
[file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1710]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1748]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1712]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1711]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1714]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1719]: zabbix_server [1717]: [file:'log.c',line:245] lock failed: [22] Invalid argument[file:'log.c',$

zabbix_server [1722]: zabbix_server [1732]: [file:'log.c',line:245] lock failed: [22] Invalid argument[file:'log.c',$

zabbix_server [1728]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1741]: zabbix_server [1733]: [file:'log.c',line:245] lock failed: [22] Invalid argument
[file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1736]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1721]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1735]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1758]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1757]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1745]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1725]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1747]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1759]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1743]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1746]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1744]: [file:'log.c',line:245] lock failed: [22] Invalid argument
zabbix_server [1756]: [file:'log.c',line:245] lock failed: [22] Invalid argument

If I reboot the zabbix machine, it starts working again, for a certain period of time(1 or 2 days).

[dimir]

Could you check if it's this one? https://support.zabbix.com/browse/ZBX-11544

[marafado88]

Almost forgot that, if I reboot the zabbix machine, it starts working again, for a certain period of time(1 or 2 days) till it stops.

Doesnt seem to be related because in that situation there are issues when a SSH session logout is made. In this case seems to be a loop till it stops working.