Zabbix server is down due to failed to connect to database

[rootlinux]

My zabbix 1.1 suddenly down due to failed connection to database, why?

Below is the message from zabbix.log: -

031097:20060611:223307 Timeout while connecting to [10.60.4.124:161]
031097:20060611:223308 Host [cansm.a024]: first network error, wait for 15 seconds
031090:20060612:183421 Failed to connect to database: Error: Lost connection to MySQL server during query
031091:20060612:183421 Failed to connect to database: Error: Lost connection to MySQL server during query
031084:20060612:183421 One server process died. Shutting down...
031084:20060612:183421 ZABBIX server is down.

[rootlinux]

It happened again...

013225:20060614:131355 Failed to connect to database: Error: Lost connection to MySQL server during query
013218:20060614:131355 One server process died. Shutting down...
013218:20060614:131355 ZABBIX server is down.

How can I solve this?

[rootlinux]

This error is from my Zabbix #2

Zabbix #2 - using 1.1
Dual Intel Pentium Pro 166MHz
256MB RAM (512MB Swap)
php 4.4.2, mysql 3.23, and httpd 2.0.40
Monitor 39 Hosts, 113 Items, 20 Triggers

Is the server specs sufficient?

[dantheman]

I am having the same issue with 1.1... several times a day..... anyone have any ideas? I started having a few issues with the server shutting down starting with 1.1Beta12 but didn't have any time to trace it out... anyone have any ideas how we can figure out what is going on?

[dcrandall]

I was having a similar problem, except that zabbix would die almost immediately.
But very similar errors.

It turned out to be database corruption. For me, my history table was corrupt, so as soon as I would start the zabbix server, it would start updating everything, hit the history table, mysql would bounce and zabbix would crash.

It could be that you have corruption in a less frequently accessed table.
Turn the debugging up to 4 and see if it consistently dies while updating a particular table. You can also go into mysql and do a SHOW TABLE STATUS on each table. If one is corrupt, you'll get very similar errors to what you see in the zabbix log (if debug is set to 4)

I hope this helps, I was banging my head on this for two days before I figured it out.

Daniel

[dantheman]

Hi Daniel

I ran the checks and did have some corruption in the database, in multiple tables. I think it's all cleared up now.. I'm hoping that takes care of it.

Thanks!
Daniel

[dcrandall]

Ouch.
Well hopefully either they arent critical tables (hosts) or you have backups. With innodb repairing the table doesn't work. you have to drop the table and restore it from backup. At least that was the way that worked for me.

I'm actually quite concerned about the database corruption. It seems like certain combinations of zabbix manipulation causes this.

[dantheman]

I was thinking that it might have had some corruption going between so many different versions of Zabbix, I started on Beta9 and moved up with every beta until the final release. With any luck it will be ok now. History was one of my tables that was corrupted, and I know also sessions was corrupted as well, I don't remember any others.

[rootlinux]

Which command to use to check the mysql database corruption?

I am a mysql newbie...

[kurt]

Database Consistency Check

There are a few ways to determine if your database is corrupt.

Since im using MyISAM tables in my sql database
i use "mysqlcheck"

[rootlinux]

I run the following command below but it seems hang there: -

[root@nms01 root]# mysqlcheck -u zabbix -p -c zabbix
Enter password:
zabbix.acknowledges OK
zabbix.actions OK
zabbix.alarms OK
zabbix.alerts OK
zabbix.applications OK
zabbix.auditlog OK
zabbix.autoreg OK
zabbix.conditions OK
zabbix.config OK
zabbix.escalation_log OK
zabbix.escalation_rules OK
zabbix.escalations OK
zabbix.functions OK
zabbix.graphs OK
zabbix.graphs_items OK
zabbix.groups OK
zabbix.help_items OK

[kurt]

It might appear that its hanging, but its doing its thing in the background.

My History Database is 2.9Gb and took about an hour to complete.

Patients is a virtue....

[stever]

I have this little script running every 15 mins or so on our zabbix server. A couple of times we've done maint. on our mysql server, and zabbix was down for the whole weekend. This isn't the greatest, but it works.

#!/bin/bash

ps -ef | grep zabbix_server | grep -q -v grep
if [ $? -eq "0" ]
then
echo "Zabbix running"
else
echo "Zabbix_server not running!!"
#/etc/init.d/zabbix_server start
mailx -s "Restarted zabbix_server on `hostname`" [email protected] < /dev/null
fi