Server 2.2.3 crash

[Zaniwoop]

I upgraded from version 2.2.2 to 2.2.3 yesterday. since the the zabbix_server has randomly stopped several time.

The log file (debug level 4) just says

Zabbix Server stopped.

I have since rolled back to version 2.2.2 and it seems fine.

[aib]

When I installed Zabbix Server 2.2.2, it also stopped two or more times for unpredictable reason.

After a couple of week it works like a charm.

I don't know - may be he was settling down and crashing because missed my attention?

[Colttt]

can you please post a few entries before?

[mma]

Maybe the same thing :

20625:20140414:030029.938 In free_ipmi_handler()
20625:20140414:030029.938 End of free_ipmi_handler()
20625:20140414:030029.938 In zbx_vmware_destroy()
20625:20140414:030029.938 In zbx_mem_destroy() descr:'vmware cache size'
20625:20140414:030029.938 End of zbx_mem_destroy()
20625:20140414:030029.938 End of zbx_vmware_destroy()
20625:20140414:030029.938 In free_selfmon_collector() collector:0x7f3ca0edb000
20625:20140414:030029.938 End of free_selfmon_collector()
20625:20140414:030029.938 In unload_modules()
20625:20140414:030029.938 Zabbix Server stopped. Zabbix 2.2.3 (revision 44105).

EDIT : If it's bug, it's very critical...

[Zaniwoop]

Here is quite a chunk from the log

  7124:20140410:140345.304 End of DCflush_trends()
  7124:20140410:140345.304 query [txnlev:1] [commit;]
  7124:20140410:140345.307 syncing trends data done
  7124:20140410:140345.307 End of DCsync_trends()
  7124:20140410:140345.307 End of DCsync_all()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'history cache'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'history text cache'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'trend cache'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 End of free_database_cache()
  7124:20140410:140345.307 In free_configuration_cache()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'configuration cache'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 In zbx_strpool_destroy()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'string pool'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 End of zbx_strpool_destroy()
  7124:20140410:140345.307 End of free_configuration_cache()
  7124:20140410:140345.307 In zbx_vc_destroy()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'value cache size'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 End of zbx_vc_destroy()
  7124:20140410:140345.307 In free_ipmi_handler()
  7124:20140410:140345.307 End of free_ipmi_handler()
  7124:20140410:140345.307 In zbx_vmware_destroy()
  7124:20140410:140345.307 In zbx_mem_destroy() descr:'vmware cache size'
  7124:20140410:140345.307 End of zbx_mem_destroy()
  7124:20140410:140345.307 End of zbx_vmware_destroy()
  7124:20140410:140345.307 In free_selfmon_collector() collector:0x2b35ced35000
  7124:20140410:140345.307 End of free_selfmon_collector()
  7124:20140410:140345.307 In unload_modules()
  7124:20140410:140345.307 Zabbix Server stopped. Zabbix 2.2.3 (revision 44105).

[Colttt]

ok, thats all in a less 1second..
please post more.. mabybe the complete log to pastebin.com or somthing else

[mma]

Thanks for your return, I have no more now. My log make a rotate...
I make a script to send log and restart zabbix quickly (in cron every minute) :

PS : Seems not work
Update 2014/04/30 works with crontab part

#!/bin/bash
#by mma 20140430
HOME=/root
LOGNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:
SHELL=/bin/bash
TO_ADDRESS="*************"
FROM_ADDRESS="*********"
SUBJECT="Zabbix restarted"
EMAIL=$(cat /var/log/zabbix/zabbix_server.log)
MAILX="/usr/bin/mailx"
logfile="/var/log/zabbix/zabbix_server.log"

nbproc=$(ps aux | grep "/usr/local/sbin/zabbix_server" | grep -v "grep" | grep -v "processus-zabbixserver" | wc -l)
if [ "$nbproc" -eq "0" ]
then
   echo "Zabbix server stopped !"
   tail -n 50 $logfile | $MAILX -s "$SUBJECT" $TO_ADDRESS
   /etc/init.d/zabbix-server start
else
        echo "Zabbix server running"
fi

in crontab

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# m h dom mon dow command
* * * * * . /root/.profile;/root/scripts/processus-zabbixserver.sh

[Zaniwoop]

Unfortunately I don't have a relevant logfile anymore; it has been overwritten, but as soon as the problem reoccurs I will send it.

In the mean time what I have done is (as a last resort) re-installed Linux (Centos 6.5) and Zabbix (2.2.3) on the server. But the problem still existed.

Subsequent to that I have disabled the VMware LLD, as that was the last major addition since the upgrade from 2.0. I have not had a re-occurrence since then. It has been 2 days and the zabbix_server has not crashed.

Not sure how long I have to wait before I can consider it fixed.

[mma]

It's not really fixed... The issue persists.
I can't stop VMWare LDD for my part.

[Zaniwoop]

I'll give it a few days, if it is still stable, I'll re-enable the LLD to test.

[mma]

I have this issue again, I have no more logs...
I would to stop LDD for VMWare

[Zaniwoop]

After not having the issue for a week, I have restarted the VMware discovery and monitoring. for the past 5 days I still have had no further problems.

So, it still remains a mystery.

[mma]

Ok, I quickly update my script.
I restart Vmware LLD. I "hope" a crash to understand...

[Palmertree]

I found that if you run 2.2.3 with VMware pollers it will crash if you use Debug_Level=4. It will not crash on Debug_Level=3.

[kloczek]

I found that if you run 2.2.3 with VMware pollers it will crash if you use Debug_Level=4. It will not crash on Debug_Level=3.

Try to use patch attached in here https://support.zabbix.com/browse/ZBX-8060