segfault in 1.6.4

[wax66]

I'm getting a segfault in my master Zabbix server.

Apr 22 10:46:25 monitor-dc1 kernel: zabbix_server[8633]: segfault at fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007ffffed60710 error 4

Within a few seconds of server startup I get a message saying that a child process died and it's exiting.

mysql Ver 14.12 Distrib 5.0.45, for redhat-linux-gnu (x86_64) using readline 5.0
Zabbix 1.6.4, configure options:
./configure --enable-server --enable-agent --with-mysql --with-net-snmp --with-jabber --with-libcurl --with-ldap

I'm going to try to downgrade to 1.6.3 for now, since this is a production server, but I would love it if someone could take a look at the log file (I don't see anything useful though). I'd attach it, but it's too large.

Thanks!
-Ron

[Alexei]

Please could you send the gzipped log file to s u p p o r t @ z a b b i x . c o m. Please remove spaces in the email. Thank you.

[wax66]

Downgraded

Downgrading to 1.6.3 failed with the same issue, but downgrading to 1.6.2 worked.

Logfile is on its way.

Thanks!
-Ron

[igor]

I'm getting a segfault in my master Zabbix server.

Apr 22 10:46:25 monitor-dc1 kernel: zabbix_server[8633]: segfault at fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007ffffed60710 error 4

Within a few seconds of server startup I get a message saying that a child process died and it's exiting.

mysql Ver 14.12 Distrib 5.0.45, for redhat-linux-gnu (x86_64) using readline 5.0
Zabbix 1.6.4, configure options:
./configure --enable-server --enable-agent --with-mysql --with-net-snmp --with-jabber --with-libcurl --with-ldap

I'm going to try to downgrade to 1.6.3 for now, since this is a production server, but I would love it if someone could take a look at the log file (I don't see anything useful though). I'd attach it, but it's too large.

Thanks!
-Ron

Here is part of the /var/log/messages file received from Ron:

Apr 22 04:02:02 monitor-dc1 syslogd 1.4.1: restart.
Apr 22 09:31:12 monitor-dc1 kernel: device eth1 entered promiscuous mode
Apr 22 09:59:38 monitor-dc1 kernel: zabbix_server[17579]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff7b346cf0 error 4
Apr 22 10:12:12 monitor-dc1 kernel: zabbix_server[17666]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff33650fe0 error 4
Apr 22 10:12:22 monitor-dc1 kernel: device eth1 left promiscuous mode
Apr 22 10:12:41 monitor-dc1 kernel: zabbix_server[17747]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff308421f0 error 4
Apr 22 10:12:45 monitor-dc1 kernel: zabbix_server[17792]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff257e9190 error 4
Apr 22 10:12:49 monitor-dc1 kernel: zabbix_server[17834]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fffb6a78420 error 4
Apr 22 10:12:52 monitor-dc1 kernel: zabbix_server[17881]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff8e3ccd70 error 4
Apr 22 10:12:57 monitor-dc1 kernel: zabbix_server[17927]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff8a1d8b80 error 4
Apr 22 10:20:47 monitor-dc1 kernel: zabbix_server[18366]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff4601d9c0 error 4
Apr 22 10:20:55 monitor-dc1 kernel: zabbix_server[18409]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff56791140 error 4
Apr 22 10:21:03 monitor-dc1 kernel: zabbix_server[18457]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff98698040 error 4
Apr 22 10:21:13 monitor-dc1 kernel: zabbix_server[18505]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff1185c200 error 4
Apr 22 10:21:20 monitor-dc1 kernel: zabbix_server[18551]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fffa68481f0 error 4
Apr 22 10:21:29 monitor-dc1 kernel: zabbix_server[18595]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff4cf7e920 error 4
Apr 22 10:21:38 monitor-dc1 kernel: zabbix_server[18644]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff2fb114b0 error 4
Apr 22 10:21:45 monitor-dc1 kernel: zabbix_server[18692]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fffb225bc00 error 4
Apr 22 10:21:52 monitor-dc1 kernel: zabbix_server[18750]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff59467e10 error 4
Apr 22 10:22:01 monitor-dc1 kernel: zabbix_server[18795]: segfault at
fffffffffffffff9 rip 0000003fb7a74c7b rsp 00007fff3732bcd0 error 4
Apr 22 10:26:52 monitor-dc1 shutdown[18912]: shutting down for system reboot
Apr 22 10:26:52 monitor-dc1 init: Switching to runlevel: 6

[garumph]

Any resolution on this? We are getting the same segfault messages in 1.6.5

[Calimero]

I've experienced the same problem recently (or at least it looked similar).

After a MySQL restart (while zabbix was running) zabbix_server started crashing.

I set Debug to 4 and found out that the process that segfaulted (check PID in /var/log/pid) always crashed on the same query (same function/item). I found out that items.lastvalue for that item was NULL which seemed strange. As I didn't have time to do more debugging (production zabbix !) I set item.lastvalue to 0 for all items with that key.

It worked ...

Edit: zabbix 1.6.5

[wax66]

Sorry it's taken me so long to respond.

After upgrading to 1.6.5, my master server no longer crashes. So now all 3 of my servers are at 1.6.5 with no issues that I can see.
-Ron

[Calimero]

Well I've had a few segfaults today on my 1.6.5 install.

I had previously removed all of my ugly patches not wanting to blame my poorly written hacks on zabbix developers

It looks like zabbix_server crashes when encountering "uninitialized" records in DB (just after you've added a new host). Maybe having items.lastvalue being set to NULL by default ... I don't know. That's just my guess from what I see in log file.

[garumph]

see ticket ZBX-1001. Zabbix provided a patch that fixed it for me.

[Calimero]

Ha ha ! Sounds promising. I was about to start "diffing" between 1.6.2 and 1.6.5 but as changes were pretty big it would probably have taken a lot of time.

I'll try the patch tomorrow. Thanks for the hint !