'Zabbix server is not running' shows on the web page

[dell3]

Hi all,

I am not a Linux guy. So it's really hard for me to troubleshoot. It’s greatly appreciated if anyone can help me on this.

CentOS 6.5 + Zabbix 2.2 + MySQL 5.1.71

I followed the online guide to install zabbix. Everything looks good. But when I access the web page, 'Zabbix server is not running' shows on the web page.


/etc/zabbix/zabbix_server.conf
=====================
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_server.pid
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
DBSocket=/var/lib/mysql/mysql.sock
=====================

/var/log/zabbix/zabbix_server.log
=====================
3065:20140213:141232.534 using configuration file: /etc/zabbix/zabbix_server.conf
3065:20140213:141232.537 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3065:20140213:141232.537 Database is down. Reconnecting in 10 seconds.
3065:20140213:141242.538 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3065:20140213:141242.539 Database is down. Reconnecting in 10 seconds.
3065:20140213:141252.570 current database version (mandatory/optional): 02020000/02020000
3065:20140213:141252.570 required mandatory version: 02020000
3255:20140213:141252.653 server #10 started [trapper #2]
3247:20140213:141252.654 server #2 started [db watchdog #1]
3257:20140213:141252.654 server #12 started [trapper #4]
3258:20140213:141252.655 server #13 started [trapper #5]
3259:20140213:141252.655 server #14 started [icmp pinger #1]
3261:20140213:141252.656 server #16 started [housekeeper #1]
3261:20140213:141252.656 executing housekeeper
3262:20140213:141252.659 server #17 started [timer #1]
3265:20140213:141252.662 server #20 started [history syncer #1]
3263:20140213:141252.663 server #18 started [http poller #1]
3266:20140213:141252.664 server #21 started [history syncer #2]
3254:20140213:141252.664 server #9 started [trapper #1]
3267:20140213:141252.664 server #22 started [history syncer #3]
3256:20140213:141252.664 server #11 started [trapper #3]
3260:20140213:141252.665 server #15 started [alerter #1]
3275:20140213:141252.666 server #24 started [escalator #1]
3065:20140213:141252.667 server #0 started [main process]
3268:20140213:141252.668 server #23 started [history syncer #4]
3276:20140213:141252.671 server #25 started [proxy poller #1]
3246:20140213:141252.672 server #1 started [configuration syncer #1]
3277:20140213:141252.672 server #26 started [self-monitoring #1]
3261:20140213:141252.742 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.063857 sec, idle 1 hour(s)]
3250:20140213:141252.812 server #5 started [poller #3]
3251:20140213:141252.813 server #6 started [poller #4]
3264:20140213:141252.810 server #19 started [discoverer #1]
3248:20140213:141252.821 server #3 started [poller #1]
3253:20140213:141252.821 server #8 started [unreachable poller #1]
3252:20140213:141252.821 server #7 started [poller #5]
3249:20140213:141252.823 server #4 started [poller #2]
3261:20140213:151252.743 executing housekeeper
3261:20140213:151252.749 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.003233 sec, idle 1 hour(s)]
3261:20140213:161252.751 executing housekeeper
3261:20140213:161252.755 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.002472 sec, idle 1 hour(s)]
3275:20140213:170644.090 [Z3005] query failed: [2006] MySQL server has gone away [select escalationid,actionid,triggerid,eventid,r_eventid, nextcheck,esc_step,status,itemid from escalations order by actionid,triggerid,itemid,escalationid]
3275:20140213:170644.091 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3275:20140213:170644.091 Database is down. Reconnecting in 10 seconds.
=====================

/etc/my.cnf
=====================
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[client]
port=3306
socket=/var/lib/mysql/mysql.sock
=====================

I have tried a lot of things which I found in other thread. Zabbix account can log on with MySQL and open Zabbix DB. Any other hints?

Thanks,

John

[Linwood]

Maybe MySql is crashing?

Try looking in the MySql logs, depending on flavor maybe in /var/log/mysql.log (it rolls periodically so there may also be mysql.log.1.gz, mysql.log.2.gz, etc.)

It looks below as though it works for a while, then dies, right? That's why I would suspect mysql.

[dell3]

Hi,

Thanks for your reply. I always saw the error message on the web page after clean install. Here is the MySQL log. Looks good?

================================
140214 11:20:47 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140214 11:20:48 InnoDB: Initializing buffer pool, size = 8.0M
140214 11:20:48 InnoDB: Completed initialization of buffer pool
140214 11:20:48 InnoDB: Started; log sequence number 0 5895453
140214 11:20:48 [Note] Event Scheduler: Loaded 0 events
140214 11:20:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140214 14:47:01 [Note] /usr/libexec/mysqld: Normal shutdown

140214 14:47:01 [Note] Event Scheduler: Purging the queue. 0 events
140214 14:47:03 InnoDB: Starting shutdown...
140214 14:47:07 InnoDB: Shutdown completed; log sequence number 0 5899789
140214 14:47:07 [Note] /usr/libexec/mysqld: Shutdown complete

140214 14:47:07 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140214 14:47:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140214 14:47:08 InnoDB: Initializing buffer pool, size = 8.0M
140214 14:47:08 InnoDB: Completed initialization of buffer pool
140214 14:47:08 InnoDB: Started; log sequence number 0 5899789
140214 14:47:08 [Note] Event Scheduler: Loaded 0 events
140214 14:47:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140214 17:09:15 [Note] /usr/libexec/mysqld: Normal shutdown

140214 17:09:15 [Note] Event Scheduler: Purging the queue. 0 events
140214 17:09:15 InnoDB: Starting shutdown...
140214 17:09:17 InnoDB: Shutdown completed; log sequence number 0 6031550
140214 17:09:17 [Note] /usr/libexec/mysqld: Shutdown complete

140214 17:09:17 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140214 21:09:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140214 21:09:30 InnoDB: Initializing buffer pool, size = 8.0M
140214 21:09:30 InnoDB: Completed initialization of buffer pool
140214 21:09:30 InnoDB: Started; log sequence number 0 6031550
140214 21:09:30 [Note] Event Scheduler: Loaded 0 events
140214 21:09:30 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140214 22:41:33 [Note] /usr/libexec/mysqld: Normal shutdown

140214 22:41:33 [Note] Event Scheduler: Purging the queue. 0 events
140214 22:41:35 InnoDB: Starting shutdown...
140214 22:41:37 InnoDB: Shutdown completed; log sequence number 0 6155499
140214 22:41:37 [Note] /usr/libexec/mysqld: Shutdown complete

140214 22:41:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
================================


Thanks,

John

[Linwood]

I've got nothing, sorry. I assume the shutdown at 17:09 was in response to this (i.e. someone rebooted or reset)?

3275:20140213:170644.090 [Z3005] query failed: [2006] MySQL server has gone away

And not the cause?

Sorry, no idea. Hopefully someone else does. You could try increasing the logging level of zabbix temporarily and see if there's any other clues that show up, but it really looks more like there's an issue with mysql connectivity.

[dell3]

Thanks, buddy.

I think this was casued by manually showdown and restart MySQL service for some tests.

John

[Spoonman]

Hi all,

I am not a Linux guy. So it's really hard for me to troubleshoot. It’s greatly appreciated if anyone can help me on this.

CentOS 6.5 + Zabbix 2.2 + MySQL 5.1.71

I followed the online guide to install zabbix. Everything looks good. But when I access the web page, 'Zabbix server is not running' shows on the web page.
[ATTACH]6664[/ATTACH]

/etc/zabbix/zabbix_server.conf
=====================
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_server.pid
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
DBSocket=/var/lib/mysql/mysql.sock
=====================

/var/log/zabbix/zabbix_server.log
=====================
3065:20140213:141232.534 using configuration file: /etc/zabbix/zabbix_server.conf
3065:20140213:141232.537 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3065:20140213:141232.537 Database is down. Reconnecting in 10 seconds.
3065:20140213:141242.538 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3065:20140213:141242.539 Database is down. Reconnecting in 10 seconds.
3065:20140213:141252.570 current database version (mandatory/optional): 02020000/02020000
3065:20140213:141252.570 required mandatory version: 02020000
3255:20140213:141252.653 server #10 started [trapper #2]
3247:20140213:141252.654 server #2 started [db watchdog #1]
3257:20140213:141252.654 server #12 started [trapper #4]
3258:20140213:141252.655 server #13 started [trapper #5]
3259:20140213:141252.655 server #14 started [icmp pinger #1]
3261:20140213:141252.656 server #16 started [housekeeper #1]
3261:20140213:141252.656 executing housekeeper
3262:20140213:141252.659 server #17 started [timer #1]
3265:20140213:141252.662 server #20 started [history syncer #1]
3263:20140213:141252.663 server #18 started [http poller #1]
3266:20140213:141252.664 server #21 started [history syncer #2]
3254:20140213:141252.664 server #9 started [trapper #1]
3267:20140213:141252.664 server #22 started [history syncer #3]
3256:20140213:141252.664 server #11 started [trapper #3]
3260:20140213:141252.665 server #15 started [alerter #1]
3275:20140213:141252.666 server #24 started [escalator #1]
3065:20140213:141252.667 server #0 started [main process]
3268:20140213:141252.668 server #23 started [history syncer #4]
3276:20140213:141252.671 server #25 started [proxy poller #1]
3246:20140213:141252.672 server #1 started [configuration syncer #1]
3277:20140213:141252.672 server #26 started [self-monitoring #1]
3261:20140213:141252.742 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.063857 sec, idle 1 hour(s)]
3250:20140213:141252.812 server #5 started [poller #3]
3251:20140213:141252.813 server #6 started [poller #4]
3264:20140213:141252.810 server #19 started [discoverer #1]
3248:20140213:141252.821 server #3 started [poller #1]
3253:20140213:141252.821 server #8 started [unreachable poller #1]
3252:20140213:141252.821 server #7 started [poller #5]
3249:20140213:141252.823 server #4 started [poller #2]
3261:20140213:151252.743 executing housekeeper
3261:20140213:151252.749 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.003233 sec, idle 1 hour(s)]
3261:20140213:161252.751 executing housekeeper
3261:20140213:161252.755 housekeeper [deleted 0 hist/trends, 0 items, 0 events, 0 sessions, 0 alarms, 0 audit items in 0.002472 sec, idle 1 hour(s)]
3275:20140213:170644.090 [Z3005] query failed: [2006] MySQL server has gone away [select escalationid,actionid,triggerid,eventid,r_eventid, nextcheck,esc_step,status,itemid from escalations order by actionid,triggerid,itemid,escalationid]
3275:20140213:170644.091 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
3275:20140213:170644.091 Database is down. Reconnecting in 10 seconds.
=====================

/etc/my.cnf
=====================
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[client]
port=3306
socket=/var/lib/mysql/mysql.sock
=====================

I have tried a lot of things which I found in other thread. Zabbix account can log on with MySQL and open Zabbix DB. Any other hints?

Thanks,

John

try to disable SELinux, set it to 'SELINUX=disabled' (see www.google.com)