Ad Widget

Collapse

Zabbix server is down due to failed to connect to database

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • rootlinux
    Member
    • Dec 2004
    • 32

    #1

    Zabbix server is down due to failed to connect to database

    My zabbix 1.1 suddenly down due to failed connection to database, why?

    Below is the message from zabbix.log: -

    031097:20060611:223307 Timeout while connecting to [10.60.4.124:161]
    031097:20060611:223308 Host [cansm.a024]: first network error, wait for 15 seconds
    031090:20060612:183421 Failed to connect to database: Error: Lost connection to MySQL server during query
    031091:20060612:183421 Failed to connect to database: Error: Lost connection to MySQL server during query
    031084:20060612:183421 One server process died. Shutting down...
    031084:20060612:183421 ZABBIX server is down.
  • rootlinux
    Member
    • Dec 2004
    • 32

    #2
    It happened again...

    013225:20060614:131355 Failed to connect to database: Error: Lost connection to MySQL server during query
    013218:20060614:131355 One server process died. Shutting down...
    013218:20060614:131355 ZABBIX server is down.

    How can I solve this?

    Comment

    • rootlinux
      Member
      • Dec 2004
      • 32

      #3
      This error is from my Zabbix #2

      Zabbix #2 - using 1.1
      Dual Intel Pentium Pro 166MHz
      256MB RAM (512MB Swap)
      php 4.4.2, mysql 3.23, and httpd 2.0.40
      Monitor 39 Hosts, 113 Items, 20 Triggers

      Is the server specs sufficient?

      Comment

      • dantheman
        Senior Member
        • May 2006
        • 209

        #4
        I am having the same issue with 1.1... several times a day..... anyone have any ideas? I started having a few issues with the server shutting down starting with 1.1Beta12 but didn't have any time to trace it out... anyone have any ideas how we can figure out what is going on?

        Comment

        • dcrandall
          Member
          • Apr 2006
          • 59

          #5
          I was having a similar problem, except that zabbix would die almost immediately.
          But very similar errors.

          It turned out to be database corruption. For me, my history table was corrupt, so as soon as I would start the zabbix server, it would start updating everything, hit the history table, mysql would bounce and zabbix would crash.

          It could be that you have corruption in a less frequently accessed table.
          Turn the debugging up to 4 and see if it consistently dies while updating a particular table. You can also go into mysql and do a SHOW TABLE STATUS on each table. If one is corrupt, you'll get very similar errors to what you see in the zabbix log (if debug is set to 4)

          I hope this helps, I was banging my head on this for two days before I figured it out.

          Daniel

          Comment

          • dantheman
            Senior Member
            • May 2006
            • 209

            #6
            Hi Daniel

            I ran the checks and did have some corruption in the database, in multiple tables. I think it's all cleared up now.. I'm hoping that takes care of it.

            Thanks!
            Daniel

            Comment

            • dcrandall
              Member
              • Apr 2006
              • 59

              #7
              Ouch.
              Well hopefully either they arent critical tables (hosts) or you have backups. With innodb repairing the table doesn't work. you have to drop the table and restore it from backup. At least that was the way that worked for me.

              I'm actually quite concerned about the database corruption. It seems like certain combinations of zabbix manipulation causes this.

              Comment

              • dantheman
                Senior Member
                • May 2006
                • 209

                #8
                I was thinking that it might have had some corruption going between so many different versions of Zabbix, I started on Beta9 and moved up with every beta until the final release. With any luck it will be ok now. History was one of my tables that was corrupted, and I know also sessions was corrupted as well, I don't remember any others.

                Comment

                • rootlinux
                  Member
                  • Dec 2004
                  • 32

                  #9
                  Which command to use to check the mysql database corruption?

                  I am a mysql newbie...

                  Comment

                  • kurt
                    Junior Member
                    • Aug 2005
                    • 21

                    #10
                    Database Consistency Check

                    There are a few ways to determine if your database is corrupt.

                    Since im using MyISAM tables in my sql database
                    i use "mysqlcheck"

                    Comment

                    • rootlinux
                      Member
                      • Dec 2004
                      • 32

                      #11
                      I run the following command below but it seems hang there: -

                      [root@nms01 root]# mysqlcheck -u zabbix -p -c zabbix
                      Enter password:
                      zabbix.acknowledges OK
                      zabbix.actions OK
                      zabbix.alarms OK
                      zabbix.alerts OK
                      zabbix.applications OK
                      zabbix.auditlog OK
                      zabbix.autoreg OK
                      zabbix.conditions OK
                      zabbix.config OK
                      zabbix.escalation_log OK
                      zabbix.escalation_rules OK
                      zabbix.escalations OK
                      zabbix.functions OK
                      zabbix.graphs OK
                      zabbix.graphs_items OK
                      zabbix.groups OK
                      zabbix.help_items OK

                      Comment

                      • kurt
                        Junior Member
                        • Aug 2005
                        • 21

                        #12
                        It might appear that its hanging, but its doing its thing in the background.

                        My History Database is 2.9Gb and took about an hour to complete.

                        Patients is a virtue....

                        Comment

                        • stever
                          Member
                          • Jun 2006
                          • 36

                          #13
                          I have this little script running every 15 mins or so on our zabbix server. A couple of times we've done maint. on our mysql server, and zabbix was down for the whole weekend. This isn't the greatest, but it works.

                          #!/bin/bash

                          ps -ef | grep zabbix_server | grep -q -v grep
                          if [ $? -eq "0" ]
                          then
                          echo "Zabbix running"
                          else
                          echo "Zabbix_server not running!!"
                          #/etc/init.d/zabbix_server start
                          mailx -s "Restarted zabbix_server on `hostname`" [email protected] < /dev/null
                          fi

                          Comment

                          Working...