Ad Widget

Collapse

Database is down. Retrying in 10 seconds.

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • joshuamcdo
    Member
    • Nov 2013
    • 76

    #1

    Database is down. Retrying in 10 seconds.

    Man where do I start....

    Zabbix server 2.0.13
    Agents have a mix..

    Database is RDS mysql instance.


    So I had been getting these messages off and on for a while.
    20087:20150309:091135.023 [Z3005] query failed: [2006] MySQL server has gone away [begin;]

    Restarting zabbix would always clear them.

    They started becoming much more frequent as in rolling into the log files.. So I tried to follow this guide..
    h t t p : / / zabbixzone dot . c o m/zabbix/mysql-performance-tips-for-zabbix/

    At first things seemed like they were working and I wasn't getting anymore messages.. Then I started getting the " 20087:20150309:091135.023 [Z3005] query failed: [2006] MySQL server has gone away [begin;]" messages again.. So I restarted Zabbix as I didn't have anymore time to deal with it.
    This was a mistake..
    Now I am getting these.

    ......
    0113:20150309:090825.371 [Z3005] query failed: [2006] MySQL server has gone away [select distinct t.triggerid,t.type,t.value,t.value_flags,t.error from items i,functions f,triggers t,hosts h where i.itemid=f.itemid and f.triggerid=t.triggerid and i.hostid=h.hostid and i.status=0 and i.type in (0) and f.function not in ('nodata','d
    ate','dayofmonth','dayofweek','time','now') and t.status=0 and h.hostid=10522 and h.status=0 and not exists (select 1 from functions f2,items i2,hosts h2 where f2.triggerid=f.triggerid and f2.itemid=i2.itemid and i2.hostid=h2.hostid and (f2.function in ('nodata','date','dayofmonth','dayofweek','time',' now') or (i2.type not in (0) an
    d (i2.type not in (0,1,4,6,12,16) or (i2.type in (0) and h2.available=1) or (i2.type in (1,4,6) and h2.snmp_available=1) or (i2.type in (12) and h2.ipmi_available=1) or (i2.type in (16) and h2.jmx_available=1)))) and i2.status=0 and h2.status=0) order by t.triggerid]
    .......
    22478:20150312:202519.178 [Z3005] query failed: [2006] MySQL server has gone away [update triggers set value_flags=1,error='Zabbix was restarted.' where triggerid=20583;

    ....

    22478:20150312:202539.205 Database is down. Retrying in 10 seconds.


    I have been rolling back what changes I can to the mysql config but nothing seems to be working.


    J
  • joshuamcdo
    Member
    • Nov 2013
    • 76

    #2
    Half way solved this...

    The max_allowed_packet size was somehow too low... I maxed it out until i figure out what it's supposed to be.

    This has been an on and off problem in the past.

    Comment

    • joshuamcdo
      Member
      • Nov 2013
      • 76

      #3
      So it starts again...

      And so as the world of Zabbix can sometimes do... The problem starts again..

      I woke up this morning to thousands of these messages..

      24330:20150313:113215.001 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
      24340:20150313:113215.006 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
      24365:20150313:113215.007 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
      24334:20150313:113215.007 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
      24325:20150313:113216.007 [Z3005] query failed: [2006] MySQL server has gone away [begin;]

      Does anyone understand what is going on here? This problem as to "go away" or something is going to have to give.. :/

      Thanks,
      J

      Comment

      • joshuamcdo
        Member
        • Nov 2013
        • 76

        #4
        I increased the wait_timeout...

        I increased the wait_timeout to 1200 seconds.. Still in the same situation..

        Comment

        • c.mammoli
          Member
          Zabbix Certified Specialist
          • Feb 2012
          • 48

          #5
          Originally posted by joshuamcdo
          I increased the wait_timeout to 1200 seconds.. Still in the same situation..
          God bless the day I dumped MySQL out of the windows. Sorry for not being helpful

          Comment

          • joshuamcdo
            Member
            • Nov 2013
            • 76

            #6
            A little more information..

            This is ridiculous.. I can't even post a piece of the log file without this board accusing me of posting links and images.. Loosen the rules for your verified users would ya? This is EXTREMELY counter productive and frustrating.

            Comment

            • ingus.vilnis
              Senior Member
              Zabbix Certified Trainer
              Zabbix Certified SpecialistZabbix Certified Professional
              • Mar 2014
              • 908

              #7
              Hi,

              I don't know what went wrong with the forum rules for you but here is what I can tell you about MySQL.

              max_allowed_packet = 32M should be enough. If not, go for 64M. 1GB as max possible is too much.

              wait_timeout = 28800 seconds is default for MySQL and should remain so. Having it at 1200 seconds is the reason why you have so many "MySQL server has gone away" errors.

              Also check Zabbix server performance graphs, especially "Zabbix data gathering process busy" and see if the processes don't idle a lot. Compare it with your zabbix_server.conf file. If you have specified there like StartPollers = 1000 and you have only 5 monitored hosts, then the pollers will lose the connection to DB anyways since they are not utilized properly. Having the pollers used at ~30% on average is a good value.

              Hope this helps!

              Best Regards,
              Ingus

              Comment

              • joshuamcdo
                Member
                • Nov 2013
                • 76

                #8
                Re: Help!

                Is it normal for the Zabbix busy node watcher process to report no data.

                I cannot attach a screenshot due to an insane attachment policy..

                Thanks,
                J

                Comment

                • ingus.vilnis
                  Senior Member
                  Zabbix Certified Trainer
                  Zabbix Certified SpecialistZabbix Certified Professional
                  • Mar 2014
                  • 908

                  #9
                  Originally posted by joshuamcdo
                  Is it normal for the Zabbix busy node watcher process to report no data.
                  Yes, it is normal if you are not using the distributed monitoring model with nodes as described here: https://www.zabbix.com/documentation...nitoring/nodes

                  Best Regards,
                  Ingus

                  Comment

                  Working...