Zabbix server won't start

[domiflichi]

Hello,

I just installed Zabbix 1.6.1 on my CentOS 5.2 system using the awesome script/instructions found here:
http://blog.brendon.com/unix/install...-the-easy-way/.

I'm so happy that that script was on the wiki because I was running into problem after problem trying to do it by the instructions in the manual with 1.6.5...and I'm not ripping on Zabbix software or the manual...I'm just not a Linux guy, so the script made it so easy for me.
Anyway, everything seems like it got installed, and the Agent starts, but the Server does not want to start.

Here's the last few lines of my /var/log/zabbix-server/zabbix_server.log:

4446:20090422:104535 Query failed:MySQL server has gone away [2006]
  4448:20090422:104535 Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock' (2) [2002]
  4464:20090422:104536 Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock' (2) [2002]
  4429:20090422:104541 Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock' (2) [2002]
  2128:20090422:104633 Starting zabbix_server. ZABBIX 1.6.1.
  2128:20090422:104633 **** Enabled features ****
  2128:20090422:104633 SNMP monitoring:       YES
  2128:20090422:104633 WEB monitoring:        YES
  2128:20090422:104633 Jabber notifications:   NO
  2128:20090422:104633 ODBC:                   NO
  2128:20090422:104633 IPv6 support:          YES
  2128:20090422:104633 **************************
  2128:20090422:104633 Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock' (2) [2002]
  2125:20090422:105835 Starting zabbix_server. ZABBIX 1.6.1.
  2125:20090422:105835 **** Enabled features ****
  2125:20090422:105835 SNMP monitoring:       YES
  2125:20090422:105835 WEB monitoring:        YES
  2125:20090422:105835 Jabber notifications:   NO
  2125:20090422:105835 ODBC:                   NO
  2125:20090422:105835 IPv6 support:          YES
  2125:20090422:105835 **************************
  2125:20090422:105835 Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock' (2) [2002]

Maybe this is something simple, but again, I'm not a Linux guy, so at least a point in the right direction would be greatly appreciated. The only other post that I found on this forum that had a similar problem suggested that the person turn off SELinux if they had it enabled. I checked mine, and it looks like it's already disabled:

# grep SELINUX /etc/selinux/config
# SELINUX= can take one of these three values:
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
SELINUXTYPE=targeted

Thanks!

[rue]

Failed to connect to database: Error: Can't connect to local MySQL server through sock
et '/var/lib/mysql/mysql.sock'

Do you have mysql installed and configured to be reachable on /var/lib/mysql/mysql.sock? (have a look at your global my.conf, which usually lives in /etc)

cheers

Rü

[domiflichi]

Hello,

Thanks for the reply. Yes, mySQL is installed and seems to be working - I can connect to it using the command line - using the mysql command.
The contents of my my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Thanks.

[rue]

Hello,
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
Thanks.

This one looks suspicious.. . You might want to disable it and try restarting zabbix_server. But I'm not sure, because mysqld should give "access denied", if you use wrong password format while connecting... .

Try:

shell> mysql
mysql> show databases

and, if you see "zabbix" database:

mysql>use zabbix
mysql>show tables

...to make sure db was initialized correctly during installation.

cheers

Rü

[domiflichi]

rue,

thanks for the quick reply again. I wasn't sure what you meant by:

This one looks suspicious.. . You might want to disable it and try restarting zabbix_server.

But here are the results of me connecting to the db and showing the databases and tables in zabbix:

mysql> show databases
    -> \g
+--------------------+
| Database           |
+--------------------+
| information_schema |
| asterisk           |
| asteriskcdrdb      |
| asteriskrealtime   |
| endpoints          |
| meetme             |
| mysql              |
| test               |
| zabbix             |
+--------------------+
9 rows in set (0.01 sec)

mysql> use zabbix
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables
    -> \g
+-----------------------+
| Tables_in_zabbix      |
+-----------------------+
| acknowledges          |
| actions               |
| alerts                |
| applications          |
| auditlog              |
| conditions            |
| config                |
| dchecks               |
| dhosts                |
| drules                |
| dservices             |
| escalations           |
| events                |
| functions             |
| graphs                |
| graphs_items          |
| groups                |
| help_items            |
| history               |
| history_log           |
| history_str           |
| history_str_sync      |
| history_sync          |
| history_text          |
| history_uint          |
| history_uint_sync     |
| hosts                 |
| hosts_groups          |
| hosts_profiles        |
| hosts_profiles_ext    |
| hosts_templates       |
| housekeeper           |
| httpstep              |
| httpstepitem          |
| httptest              |
| httptestitem          |
| ids                   |
| images                |
| items                 |
| items_applications    |
| mappings              |
| media                 |
| media_type            |
| node_cksum            |
| nodes                 |
| opconditions          |
| operations            |
| profiles              |
| proxy_dhistory        |
| proxy_history         |
| rights                |
| screens               |
| screens_items         |
| scripts               |
| service_alarms        |
| services              |
| services_links        |
| services_times        |
| sessions              |
| slides                |
| slideshows            |
| sysmaps               |
| sysmaps_elements      |
| sysmaps_link_triggers |
| sysmaps_links         |
| trends                |
| trends_uint           |
| trigger_depends       |
| triggers              |
| users                 |
| users_groups          |
| usrgrp                |
| valuemaps             |
+-----------------------+
73 rows in set (0.00 sec)

mysql>

Thanks.

[rue]

Well, what I meant was: If you have a DB-Server with "old_password" setting enabled, zabbix might not be able to connect to it, even if you can connect to it without a problem from the commandline.

Password format in mysql was changed between mysql major versions 3 and 4, so nowadays this should not be a problem anymore, even if you run a 4.x MySQL-Server. And mysql should complain about a wrong password (access denied...), if this was the case.

Nevertheless, your DB looks OK, so we can be sure, that this part of the installation worked fine.

Also just tested this setting in my local setup, no problem there.

What does

ls -l /var/lib/mysql/mysql.sock

give?

Rü

[domiflichi]

Oh. Yeah, I don't know why that setting is like that. The system I have is actually trixbox. Maybe some component of Asterisk requires that setting? I don't know.
Well, here's the output:

[trixbox1.localdomain etc]# ls -l /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Apr 22 10:58 /var/lib/mysql/mysql.sock

[rue]

[trixbox1.localdomain etc]# ls -l /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Apr 22 10:58 /var/lib/mysql/mysql.sock

[/QUOTE]

Looks good (again... ). Hm. I'm running out of ideas, perhaps the GUI-Installer might be of help. Could you try to access your zabbix GUI and navigate to: Administration -> Installation and just click trough the wizard until point 3 (Check of pre-requisites). The GUI lists some stuff there, that needs to be set or installed. If everything is OK there, proceed to Check 4, which tries to establish database connection.... . Hopefully we'll see something helpful.

cheers

Rü

[domiflichi]

I'm running out of ideas

Uh oh, that's not what I wanted to hear. I appreciate you helping me all this way regardless of whether you run out of ideas or not though.

Anyway, Step 3 failed at first on the timezone check even though I had a valid time zone entered. I followed the step to add a '!' in this thread (post #6) - http://www.zabbix.com/forum/showthread.php?p=44872. After I did that, then Step 3 passed:



Then Step 4 failed at first:



But then I entered the password, and it passed:



Is that normal?

I don't understand why this isn't working for me. I even tried this all on a completely other pc. I installed the same Asterisk-based Linux distro, trixbox (same version), and installed zabbix the same way, and have the exact same problem...so it's not like a fluke or something it seems.
Is there something about the version of trixbox (and maybe the specific version of mySQL) that Zabbix is not liking or something?

Thanks.

[rue]

Hi domiflichi,

some adjustments in screen Nr.3 and Nr.4 (DB-password) are expected and nothing unusual.

What MySQL server version does trixbox provide? Couldn't find it directly on their website... . But zabbix should have no problem with mysql > 3.22 (s.a. Zabbix Manual), since they don't use any "fancy" stuff, just "plain vanilla" SQL ... .

Did you try to finish the "wizard" and install the custom config-file yet?

cheers

Rü

[domiflichi]

Hello rue,

I checked my version of MySQL - 5.0.45.

No, I don't think I tried to install the custom config file. I'm not sure 100% what you are referring to, so I looked at the v1.6 Zabbix manual (again) and noticed on page 60, it refers to Step 7 and says to 'Download configuration file and place it under conf/'. Is that what you're referring to? I don't seem to get that option of downloading any kind of file in the wizard.

Did I mention how much I appreciate you sticking with me all this way, rue? I really do.

[rue]

Hello rue,

I checked my version of MySQL - 5.0.45.

No, I don't think I tried to install the custom config file. I'm not sure 100% what you are referring to, so I looked at the v1.6 Zabbix manual (again) and noticed on page 60, it refers to Step 7 and says to 'Download configuration file and place it under conf/'. Is that what you're referring to? I don't seem to get that option of downloading any kind of file in the wizard.

Yes, that's exactly what I'm refering to... . At the end of the wizard it tries to copy a custom config.php file to the appropriate place in your webserver zabbix directory. This usually fails, due to access rights, so you are prompted to download the file and copy it to a certain place manually.

This installer feature should be present, for quite some time now, so I certainly assume it to be present in 1.6.1.

MySQL should be perfectly fine. Again ... .

BTW: Since we are still not able to debug the automated installation, and it already took quite some time now, I'd suggest that you just download the most recent zabbix version and try to install that "the usual way".
It's actually not that hard and the problems you have might be due to a corrupt installer script, wrong zabbix version etc... . And the newer versions have quite a lot of bugfixes included.

Might be faster than chasing ghosts.

cheers

Rü

[domiflichi]

Ok, I think it's working now, but still there's some weirdness possibly. Here's what I did. I downloaded zabbix-1.6.4.tar.gz, and followed the instructions in the manual, except I skipped the following steps:
1. (The zabbix user already exists)
3. (The zabbix db already exists)
6. (the manual says it's not a real requirement, and it's probably there from when I did it when I was first trying to install zabbix)
7. (I don't plan on using zabbix_agent instead of zabbix_agnetd)
8. (not using zabbix_agent)
9. (didn't configure/make the agentd yet)
10. (zabbix_server.conf already seems to be correct from the previous install)

Ok, so then I came to step 11, and so I ran ./zabbix_server. Then in the log, I got the 2nd 2 lines:

(Ignore the 1st line - i just threw that in there from the last time to show where we left off)

 2128:20090427:113117 Failed to connect to database: Error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) [2002]
./zabbix_server [17508]: Semaphore [0] error in semctl(SETVAL)
./zabbix_server [17508]: Unable to create mutex for log file
 17590:20090428:075713 Starting zabbix_server. ZABBIX 1.6.4.
 17590:20090428:075713 **** Enabled features ****
 17590:20090428:075713 SNMP monitoring:       YES
 17590:20090428:075713 WEB monitoring:        YES
 17590:20090428:075713 Jabber notifications:   NO
 17590:20090428:075713 ODBC:                   NO
 17590:20090428:075713 IPv6 support:           NO
 17590:20090428:075713 **************************
 17597:20090428:075713 server #6 started [Trapper]
 17599:20090428:075713 server #7 started [Trapper]
 17600:20090428:075713 server #8 started [Trapper]
 17602:20090428:075713 server #9 started [Trapper]
 17604:20090428:075713 server #10 started [Trapper]
 17607:20090428:075713 server #11 started [ICMP pinger]
 17609:20090428:075713 server #12 started [Alerter]
 17611:20090428:075713 server #13 started [Housekeeper]
 17611:20090428:075713 Executing housekeeper
 17613:20090428:075713 server #14 started [Timer]
 17616:20090428:075713 server #16 started [Node watcher. Node ID:0]
 17618:20090428:075713 server #17 started [HTTP Poller]
 17621:20090428:075713 server #19 started [Escalator]
 17590:20090428:075713 server #0 started [Watchdog]
 17590:20090428:075713 In main_watchdog_loop()
 17592:20090428:075713 server #1 started [Poller. SNMP:YES]
 17596:20090428:075713 server #5 started [Poller. SNMP:YES]
 17615:20090428:075713 server #15 started [Poller for unreachable hosts. SNMP:YES]
 17620:20090428:075713 server #18 started [Discoverer. SNMP:YES]
 17594:20090428:075713 server #3 started [Poller. SNMP:YES]
 17593:20090428:075713 server #2 started [Poller. SNMP:YES]
 17595:20090428:075713 server #4 started [Poller. SNMP:YES]
 17611:20090428:075716 Deleted 0 records from history and trends

(my exact ./configure line was like this: ./configure --enable-server --with-mysql --with-libcurl --with-net-snmp)

Then I just decided to try running ./zabbix_server again, and that's when I got the rest of the log entries! It seems to be running! I even went into the web interface and confirmed this on the 'dashboard' - it says that ZABBIX server is running!

However, I went through the Administration - Installation wizard again, and it never prompted me to download a file still.

And in my Monitoring - Triggers screen, it's giving me an error near the bottom that says:

Undefined index: show_events_status[/usr/local/share/zabbix/tr_status.php:199]

Additionally, I forgot to use the --enable-agent option when running /.configure, but I want the agent to run on this server too, so what's the best way to get the agent on? Just run:

./configure --enable-agent

all by itself like that?

I just don't want to break anything.

So thank you so far for your help again, rue. I'm almost there, huh?

[domiflichi]

ok, this is weird. Maybe I was wrong about ignoring that line of the log, because I just rebooted my server to see what would happen, and checked the log again because zabbix didn't seem to start:

2127:20090428:082315 Starting zabbix_server. ZABBIX 1.6.4.
  2127:20090428:082315 **** Enabled features ****
  2127:20090428:082315 SNMP monitoring:       YES
  2127:20090428:082315 WEB monitoring:        YES
  2127:20090428:082315 Jabber notifications:   NO
  2127:20090428:082315 ODBC:                   NO
  2127:20090428:082315 IPv6 support:           NO
  2127:20090428:082315 **************************
  2127:20090428:082315 [Z3001] Connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
./zabbix_server [2809]: Semaphore [0] error in semctl(SETVAL)
./zabbix_server [2809]: Unable to create mutex for log file
  2825:20090428:082435 Starting zabbix_server. ZABBIX 1.6.4.
  2825:20090428:082435 **** Enabled features ****
  2825:20090428:082435 SNMP monitoring:       YES
  2825:20090428:082435 WEB monitoring:        YES
  2825:20090428:082435 Jabber notifications:   NO
  2825:20090428:082435 ODBC:                   NO
  2825:20090428:082435 IPv6 support:           NO
  2825:20090428:082435 **************************
  2832:20090428:082435 server #6 started [Trapper]
  2833:20090428:082435 server #7 started [Trapper]
  2835:20090428:082435 server #8 started [Trapper]
  2837:20090428:082435 server #9 started [Trapper]
  2839:20090428:082435 server #10 started [Trapper]
  2841:20090428:082435 server #11 started [ICMP pinger]
  2843:20090428:082435 server #12 started [Alerter]
  2845:20090428:082435 server #13 started [Housekeeper]
  2845:20090428:082435 Executing housekeeper
  2847:20090428:082435 server #14 started [Timer]
  2827:20090428:082435 server #1 started [Poller. SNMP:YES]
  2852:20090428:082435 server #16 started [Node watcher. Node ID:0]
  2853:20090428:082435 server #17 started [HTTP Poller]
  2857:20090428:082435 server #19 started [Escalator]
  2825:20090428:082435 server #0 started [Watchdog]
  2825:20090428:082435 In main_watchdog_loop()
  2828:20090428:082435 server #2 started [Poller. SNMP:YES]
  2829:20090428:082435 server #3 started [Poller. SNMP:YES]
  2849:20090428:082435 server #15 started [Poller for unreachable hosts. SNMP:YES]
  2855:20090428:082435 server #18 started [Discoverer. SNMP:YES]
  2830:20090428:082435 server #4 started [Poller. SNMP:YES]
  2831:20090428:082435 server #5 started [Poller. SNMP:YES]
  2845:20090428:082438 Deleted 0 records from history and trends

I had to run the ./zabbix_server 2x like I did right after I installed it, then it started. But it seems like it had the MySQL socket error again, the first time. I'm confused on how this log file works and the date/time stamps.

[Icenode]

On a server reboot, I also get the following error:

Connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/tmp/mysql.sock'

However, if I wait a bit after the server comes up, I can start Zabbix from it's startup script just fine. I suspect that MySQL has not finished initializing when the zabbix_server startup script runs. I've tried using "REQUIRE: mysql" in the startup script, but it didn't seem to help. I notice mysql processes running from a ps -aux immediatly after reboot, but I usually have to wait a minute or so to start zabbix or it will fail again. I think this behavior tricks the "REQUIRE" in the script.

Is there anyone out there with scripts working to successfully start both MySQL and Zabbix after a reboot? (My System is FreeBSD 7.1, MySQL Server 5.0.77_1, Zabbix Server 1.6.4)