zabbix_server dies if mysql dies

[dataman]

If the MySql daemon that zabbix_server uses dies, zabbix_server also dies with

3676:20070115:101827 Query failed:MySQL server has gone away [2006]
3669:20070115:101827 One child process died. Exiting ...
3669:20070115:101829 ZABBIX Server stopped

That's just not right. Zabbix, must not quit, it must generate a log error, send email alert, then retry until the server is up again. Simple prophylactic reboot of the DB server brings down the entrire surveillance system, that's ain't right.

[gpostaire]

I had this problem appear in a more dramatic way.
Mysql generated a big replication log that left no space available. So mysql was still alive but it refuse every stetement like update or insert. When Zabbix try to insert the new space, it crash even before sending the alert.
We lost our supervision for some hours.
I had to add a (not perfect) cronjob to detect when zabbix crash and send mail :
0 */1 * * 1-5 root ps -edf|grep zabbix_server|grep -v grep > /dev/null 2>&1 || (echo "zabbix KO" | mailx [email protected] -s "zabbix KO" ) > /dev/null 2>&1

It would be better if zabbix could fire a disaster alert and stand by trying to connect to mysql instead of dying.

[eger]

I just had the same problem and was kind of shocked that zabbix did not just retry the connection when it failed. Also, I would think that zabbix would hold the connection once it was initially started using a persistent connection so it wouldn't have to try to reconnect.

Hope this might be looked into for future releases. For now I will have to modify mysql max connections to be a sensible number.

[Alexei]

This will be addressed in 1.4. An user or a group of users will be notified in case if backend database engine is down or other DB related problems and ZABBIX will try to reconnect periodically.

[stritec]

Recommended Fix/ Work Around

What is a good "recommended" workaround/fix in the meantime for those of us running version 1.1.4?

Thanks,

[qix]

I have setup a (very simple) script that greps the last 2 loglines of the zabbix server log for the string "One child process died. Exiting ...".
When it finds this string, it will restart Zabbix and send an email to me.

I do not heva access to the script right now, but if you would want to know how to exactly set it up, let me know.

It's a bit of a hack, but works for me (at the moment).