zabbix_agntd 1.8.4rc1 zabbix_agentd issue

[myzabbix168]

/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf

7834:20101102:134627.450 Zabbix Agent started. Zabbix 1.8.4rc1 (revisi on 14708).
2 7834:20101102:134627.450 Listener failed with error: socket() for [[-] :10050] failed with error 22: Invalid argument.


the fix is to change back to use zabix_agentd 1.8.3.

[fratotec]

Zabbix Agent socket() failed with error 22

Strange solution...
I have zabbix_agent 1.8.4 RC1 and RC2 running on various flavours of Linux ( mainly Gentoo and SLES 9 and 10 ) ...
all compiled from the source ..
only on 1 Server ( Gentoo Kernel 2.6.24-R8 - x86_64 ) I got this error... for both RC1 and RC2 agents and proxy.

I think there is any other think missing ...
I tried also to run the zabbix_agentd with debug 4... also without any more info.

Franz

[Anton]

Can confirm this issue too - unfortunately there seems no schema for this bug.


Hostsys with Debian Lenny amd64 with XEN 3.2 and 2.6.26-amd64, multiple vhosts, also with Debian Lenny and 2.6.26-2-xen-amd64, show the reported issue.

One host is Debian Squeeze.


DPKG for lenny and squeeze has been built from latest zabbix-1.8.4.tar.gz.

root@ne14:~# tail /var/log/zabbix-agent/zabbix_agentd.log
  8306:20110107:170722.359 Zabbix Agent started. Zabbix 1.8.4 (revision 16604).
  8306:20110107:170722.361 Listener failed with error: socket() for [[-]:10050] failed with error 22: Invalid argument.

same log-output in Squeeze and Lenny.

Another vhost, also with same kernel and Lenny, works as expected:

root@hermes:~# /etc/init.d/zabbix-agent restart && tail /var/log/zabbix-agent/zabbix_agentd.log
Stopping Zabbix agent: zabbix_agentd.
Starting Zabbix agent: zabbix_agentd.
 17455:20110107:170856.400 Got signal [signal:15(SIGTERM),sender_pid:17768,sender_uid:0,reason:0]. Exiting ...
 17450:20110107:170856.401 One child process died (PID:17455,exitcode/signal:255). Exiting ...
 17450:20110107:170856.403 Got signal [signal:15(SIGTERM),sender_pid:17768,sender_uid:0,reason:0]. Exiting ...
 17450:20110107:170856.403 Zabbix Agent stopped. Zabbix 1.8.4 (revision 16604).
 17776:20110107:170858.419 Zabbix Agent started. Zabbix 1.8.4 (revision 16604).
 17777:20110107:170858.419 zabbix_agentd collector started
 17779:20110107:170858.422 zabbix_agentd listener started
 17781:20110107:170858.424 zabbix_agentd listener started

changing ListenIP to something different that "" oder "0.0.0.0" doesn't work either.

shortened strace-output of non working vhost:

(...)
[pid  3456] stat("/etc/zabbix/zabbix_agentd.conf", {st_mode=S_IFREG|0644, st_size=5053, ...}) = 0
[pid  3456] semget(0x7a0284de, 7, IPC_CREAT|IPC_EXCL|0666) = -1 EEXIST (File exists)
[pid  3456] semget(0x7a0284de, 0, 0666) = 229376
[pid  3456] semctl(229376, 0, IPC_RMID, 0) = 0
[pid  3456] semget(0x7a0284de, 7, IPC_CREAT|IPC_EXCL|0666) = 327680
[pid  3456] semctl(327680, 0, SETVAL, 0x1) = 0
[pid  3456] semctl(327680, 1, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semctl(327680, 2, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semctl(327680, 3, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semctl(327680, 4, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semctl(327680, 5, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semctl(327680, 6, SETVAL, 0x1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] semop(327680, 0x7fffbbaeca70, 1) = 0
[pid  3456] open("/var/log/zabbix-agent/zabbix_agentd.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 4
[pid  3456] close(4)                    = 0
[pid  3456] open("/var/log/zabbix-agent/zabbix_agentd.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 4
[pid  3456] gettimeofday({1294416086, 45586}, NULL) = 0
[pid  3456] open("/etc/localtime", O_RDONLY) = 5
[pid  3456] fstat(5, {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
[pid  3456] fstat(5, {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
[pid  3456] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff21a350000
[pid  3456] read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\0\0"..., 4096) = 2309
[pid  3456] lseek(5, -1467, SEEK_CUR)   = 842
[pid  3456] read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\t\0\0\0\0\0"..., 4096) = 1467
[pid  3456] close(5)                    = 0
[pid  3456] munmap(0x7ff21a350000, 4096) = 0
[pid  3456] fstat(4, {st_mode=S_IFREG|0640, st_size=1916, ...}) = 0
[pid  3456] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff21a350000
[pid  3456] write(4, "  3456:20110107:170126.045 Zabbix"..., 80) = 80
[pid  3456] close(4)                    = 0
[pid  3456] munmap(0x7ff21a350000, 4096) = 0
[pid  3456] stat("/var/log/zabbix-agent/zabbix_agentd.log", {st_mode=S_IFREG|0640, st_size=1996, ...}) = 0
[pid  3456] socket(PF_NETLINK, SOCK_RAW, 0) = 4
[pid  3456] bind(4, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
[pid  3456] getsockname(4, {sa_family=AF_NETLINK, pid=3456, groups=00000000}, [13523969038547419148]) = 0
[pid  3456] time(NULL)                  = 1294416086
[pid  3456] sendto(4, "\24\0\0\0\26\0\1\3\3268'M\0\0\0\0\0\0\0\0"..., 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
[pid  3456] recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\3268'M\200\r\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1\10"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108
[pid  3456] recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\3268'M\200\r\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 192
[pid  3456] recvmsg(4, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\3268'M\200\r\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
[pid  3456] close(4)                    = 0
[pid  3456] socket(PF_INET, 0x80001 /* SOCK_??? */, IPPROTO_TCP) = -1 EINVAL (Invalid argument)
[pid  3456] open("/var/log/zabbix-agent/zabbix_agentd.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 4
[pid  3456] gettimeofday({1294416086, 48594}, NULL) = 0
[pid  3456] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
[pid  3456] fstat(4, {st_mode=S_IFREG|0640, st_size=1996, ...}) = 0
[pid  3456] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff21a350000
[pid  3456] write(4, "  3456:20110107:170126.048 Listen"..., 128) = 128
[pid  3456] close(4)                    = 0
[pid  3456] munmap(0x7ff21a350000, 4096) = 0
[pid  3456] stat("/var/log/zabbix-agent/zabbix_agentd.log", {st_mode=S_IFREG|0640, st_size=2124, ...}) = 0
[pid  3456] exit_group(1)               = ?
Process 3456 detached
[pid  3455] --- SIGCHLD (Child exited) @ 0 (0) ---
[pid  3455] exit_group(0)               = ?
Process 3455 detached
--- SIGCHLD (Child exited) @ 0 (0) ---
exit_group(0)                           = ?

Any hints on this?

Additional data can be provided if needed.


best regards,
Anton

[richlv]

is agent compiled with or without ipv6 support ? is ipv6 enabled or disabled on those systems ?

[Anton]

is agent compiled with or without ipv6 support ? is ipv6 enabled or disabled on those systems ?

Both: Compiled with ipv6, running on ipv6-enabled hosts with working ipv6 addresses.

Specifying a specific listenip (v4 or V6) doesn't work neither.

[richlv]

hmm... just to test things it would be useful to compile agent w/o ipv6 support and see whether that helps any.

commenting out listenip directive has been tried, right ?

[Anton]

Hi,

hmm... just to test things it would be useful to compile agent w/o ipv6 support and see whether that helps any.

no, doesn't:

./configure --enable-agent --disable-ipv6 --prefix=/usr

23988:20110108:200529.284 Zabbix Agent started. Zabbix 1.8.4 (revision 16604).
23988:20110108:200529.284 Listener failed with error: socket() for [[-]:10050] failed with error 22: Invalid argument.

System is still with ipv6 enabled.

commenting out listenip directive has been tried, right ?

Yes, has been - see "[-]:10050" in log-output.

[richlv]

hmm. no ideas then. the closest i could recall was having ipv6 enabled daemon on a non-ipv6 system (https://support.zabbix.com/browse/ZBX-2964)

would be pretty cool to figure out the pattern when this happens...

[Anton]

would be pretty cool to figure out the pattern when this happens...

Which additional information are needed?

Direct access to the systems is not possible, they belong to customers.


I created an issue at the bug tracker - would be great to solve this issue.

https://support.zabbix.com/browse/ZBX-3395

Thanks!

[Sasha]

This error arises at attempt to run Zabbix agent compiled on version 2.6.27 or above on a platform with a kernel 2.6.26 and lower.

[fratotec]

Kernel 2.6.34 + Zabbix 1.8.4rc4

Hi ...
not shure what resolved my issue....
but I upgraded my kernel from 2.6.24 to 2.6.34 and compiled zabbix agentd and proxy from 1.8.4rc4 ..
which resolved 2 issues for me:
- the Listener failed with error: socket() for 10050 failed with error 22
- net.if.in[eth0,bytes]] error: Type of received value [7537381020711] is not suitable for value type [Numeric (float)]

the 'not suitable value' error, which occures on 1.8.3 when the network bytes reach a certain amount of data, seems to be solved in the 1.8.4rc1 and above...
but this versions ( rc1, rc2 and rc3 ) gave me the 'socked failed with error 22' failure on 2 ( of about 20 ) monitored servers...
Yesterday I tried out rc4, and the error went away ... but, as mentioned.. in the meantime the kernel was upgraded to 2.6.34 also.
Anyway... it work now

Regards
Fratotec Informática

[pdwalker]

This error arises at attempt to run Zabbix agent compiled on version 2.6.27 or above on a platform with a kernel 2.6.26 and lower.

Confirmed. I had the same problem: compiled on 2.6.32 and run on 2.6.26. Once I recompiled on the 2.6.26 kernel machine, the problem went away.

[develima]

This error arises at attempt to run Zabbix agent compiled on version 2.6.27 or above on a platform with a kernel 2.6.26 and lower.

Sorry, but that seems not to be the reason: I compiled Zabbix 1.8.4 from scratch (server and agent) on a Debian/Squeeze on a Xen DomU with kernel 2.6.18.8-xen0 (both Dom0 and DomU).

Neither agent nor server starts: Listener failed with error: socket() for [[-]:10051] failed with error 22: Invalid argument.

IPv6 support ist disabled.

[develima]

Addendum: I just compiled Zabbix 1.8.3 with the same configuration on th same machine and it works.

[develima]

Addendum 2: Zabbix 1.9.1 produces the same error as 1.8.4.