Zabbix database down.

[zabbixfk]

Hi,

How do i check why i am getting 'zabbix database down' emails? I can locate the settings under 'Administration->General-Others' - but quite not able to figure out the reason.
My Zabbix log files are huge , close to 250Mb +, and rotates every 15mins, and having difficulties in doing tail to figure out ( yes debuglevel is 4 - and i have about 100G left in /var/lib/mysql so no disk issue, ram about 20G free including swap/buffer, io seems fine, cpu is <30% used ). I did run grep for

 'database down', 'server gone away', 'not reachable'

- but queries didn't return results for grep from the logs.

Zabbix - 3.0.13, mysql 5.5.57

At what condition this database down email comes ? What is the exact term to be looked at logs for this?

Any pointers are greatly helpful.

Thanks

[zabbixfk]

*bump*
Can some one reply please? I am getting emails every 15mins, and queite not able to figure out what's wrong.

Thanks

[zabbixfk]

Zabbix database is down

*bump*
Can some one reply please? I am getting emails every 15mins, and quite not able to figure out what's wrong.

Thanks

[vesper1978]

Have you looked at the mysql logs? Those should point you in the right direction as to why the database is down.

[zabbixfk]

zabbix database is down

Thanks for the reply.
I did look at the logs, i don't see any errors pointing down to zabbix or database. It seems normal. MySQL db started normally (when the upgrade was done - about a month+ back)

[root@zabbix ~]# tailf /var/log/mysqld.log
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
171222  8:33:13  InnoDB: Waiting for the background threads to start
171222  8:33:14 InnoDB: 5.5.57 started; log sequence number XXXXXXXX
171222  8:33:14 [Note] Server hostname (bind-address): '0.0.0.0'; port: XXXX
171222  8:33:14 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
171222  8:33:14 [Note] Server socket created on IP: '0.0.0.0'.
171222  8:33:14 [Note] Event Scheduler: Loaded 0 events
171222  8:33:14 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.57'  socket: '/var/lib/mysql/mysql.sock'  port: XXXX  MySQL Community Server (GPL)
^C
[root@zabbix ~]#

[vesper1978]

Are you able to connect to the MySQL server from the cli using the user+pass for the zabbix server? If so, are you able to query any of the tables in the zabbix database itself?

[zabbixfk]

zabbix database is down.

Thanks for the reply.

Ofcourse yes. Without this, my zabbix server won't run. What i mean to say is zabbix server is runnig fine, i don't see any load as explained in first post, but every 15mins i am getting this email.
How do i figure out why this email is coming?
Where do i see actually this particular logs?

Any pointers are greatly helpful.
Thanks

[vesper1978]

You'd likely need to increase the DebugLevel in your zabbix_server.conf to 4, and then restart the zabbix-server service to get more information out of the zabbix_server.log.

[zabbixfk]

Its already 4. And i don't see any logs pertaining to 'Database down' or 'Database not reachable' or 'server gone away' -
My log files per 15 min are >250Mb and its possible to post entire log here.
- this was clearly mentioned in first post.

The question was more of -
why would this error comes?
How do i figure out or locate from where this error originated?
What's the location of the file which is pointing out this error?
There's no mention on manuals why this error comes?
I don't see any events or trigger or action to send email to this particular stuff - where's this configured? - I know theres's a place under Administrator->general->others which says whom all to send email in case of database gone down - but now queitly why db is going down.


Does Zabbix team looks these forums or this is just community driven? - the main reason is , i had also posted about issues with external scripts broken in my case after upgrade in another thread and it remains same even after 4-5 days. I know enterprise support stuff and all, but in case if basic things doesn't work i don't know how to proceed with this software.

Any pointers are greatly helpful.

Thanks.

[vesper1978]

Those zabbix emails saying "database is down" are caused when the zabbix server cannot connect to the database. There can be a great many reasons as to why. Most are related to MySQL and not to Zabbix itself.

Your best bet is to search the zabbix_server.log for mysql/MySQL.

I'd suggest looking at optimizing MySQL. You may be maxing out connections at time, or queries may be timing out.

Here's a good guide on getting more performance out of MySQL for Zabbix.

https://www.percona.com/blog/2014/11/14/optimizing-mysql-zabbix/

The Zabbix team does look at these forums, however it is mostly community driven. If you're looking for quick technical support from the Zabbix team, you'll need to purchase a support package: https://www.zabbix.com/support

[zabbixfk]

Thanks for the reply.

I coudn't get anything out of the logs though debug level is 4, other than mysqldatabase polling parameters.

 35895:20180210:090511.163 In substitute_key_macros() data:'mysqld_cpu.pc'
 35895:20180210:090511.163 End of substitute_key_macros():SUCCEED data:'mysqld_cpu.pc'
 35895:20180210:090511.163 In get_value() key:'mysqld_cpu.pc'
 35895:20180210:090511.163 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysqld_cpu.pc' conn:'unencrypted'
 35895:20180210:090511.163 Sending [mysqld_cpu.pc
 36001:20180210:090528.432 In substitute_key_macros() data:'mysql.ping'
 36001:20180210:090528.432 End of substitute_key_macros():SUCCEED data:'mysql.ping'
 36001:20180210:090528.432 In get_value() key:'mysql.ping'
 36001:20180210:090528.432 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.ping' conn:'unencrypted'
 36001:20180210:090528.432 Sending [mysql.ping
 36271:20180210:090528.440 In evaluate_function() function:'Zabbix server:mysql.ping.last(0)'
 35867:20180210:090529.066 In substitute_key_macros() data:'mysql.status[Bytes_received]'
 35867:20180210:090529.066 End of substitute_key_macros():SUCCEED data:'mysql.status[Bytes_received]'
 35867:20180210:090529.066 In get_value() key:'mysql.status[Bytes_received]'
 35867:20180210:090529.066 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Bytes_received]' conn:'unencrypted'
 35867:20180210:090529.066 Sending [mysql.status[Bytes_received]
 35862:20180210:090530.532 In substitute_key_macros() data:'mysql.status[Bytes_sent]'
 35862:20180210:090530.532 End of substitute_key_macros():SUCCEED data:'mysql.status[Bytes_sent]'
 35862:20180210:090530.532 In get_value() key:'mysql.status[Bytes_sent]'
 35862:20180210:090530.532 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Bytes_sent]' conn:'unencrypted'
 35862:20180210:090530.532 Sending [mysql.status[Bytes_sent]
 36263:20180210:090531.010 In evaluate_function() function:'genie-mysql-1:agent.ping.nodata(10m)'
 35948:20180210:090531.567 In substitute_key_macros() data:'mysql.status[Com_begin]'
 35948:20180210:090531.567 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_begin]'
 35948:20180210:090531.567 In get_value() key:'mysql.status[Com_begin]'
 35948:20180210:090531.567 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_begin]' conn:'unencrypted'
 35948:20180210:090531.567 Sending [mysql.status[Com_begin]
 36025:20180210:090532.596 In substitute_key_macros() data:'mysql.status[Com_commit]'
 36025:20180210:090532.596 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_commit]'
 36025:20180210:090532.596 In get_value() key:'mysql.status[Com_commit]'
 36025:20180210:090532.596 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_commit]' conn:'unencrypted'
 36025:20180210:090532.596 Sending [mysql.status[Com_commit]
 36014:20180210:090533.646 In substitute_key_macros() data:'mysql.status[Com_delete]'
 36014:20180210:090533.646 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_delete]'
 36014:20180210:090533.646 In get_value() key:'mysql.status[Com_delete]'
 36014:20180210:090533.646 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_delete]' conn:'unencrypted'
 36014:20180210:090533.646 Sending [mysql.status[Com_delete]
 35867:20180210:090534.661 In substitute_key_macros() data:'mysql.status[Com_insert]'
 35867:20180210:090534.661 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_insert]'
 35867:20180210:090534.661 In get_value() key:'mysql.status[Com_insert]'
 35867:20180210:090534.661 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_insert]' conn:'unencrypted'
 35867:20180210:090534.661 Sending [mysql.status[Com_insert]
 36018:20180210:090535.677 In substitute_key_macros() data:'mysql.status[Com_rollback]'
 36018:20180210:090535.677 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_rollback]'
 36018:20180210:090535.677 In get_value() key:'mysql.status[Com_rollback]'
 36018:20180210:090535.677 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_rollback]' conn:'unencrypted'
 36018:20180210:090535.677 Sending [mysql.status[Com_rollback]
 36018:20180210:090536.698 In substitute_key_macros() data:'mysql.status[Com_select]'
 36018:20180210:090536.698 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_select]'
 36018:20180210:090536.698 In get_value() key:'mysql.status[Com_select]'
 36018:20180210:090536.698 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_select]' conn:'unencrypted'
 36018:20180210:090536.698 Sending [mysql.status[Com_select]
 35850:20180210:090537.385 In substitute_key_macros() data:'mysql.status[Com_update]'
 35850:20180210:090537.385 End of substitute_key_macros():SUCCEED data:'mysql.status[Com_update]'
 35850:20180210:090537.385 In get_value() key:'mysql.status[Com_update]'
 35850:20180210:090537.385 In get_value_agent() host:'Zabbix server' addr:'127.0.0.1' key:'mysql.status[Com_update]' conn:'unencrypted'
 35850:20180210:090537.385 Sending [mysql.status[Com_update]
 36020:20180210:090538.753 In substitute_key_macros() data:'mysql.status[Questions]'
 36020:20180210:090538.753 End of substitute_key_macros():SUCCEED data:'mysql.status[Questions]'

Processlist from mysql shows only ~355 connections, not maxing out at any time.

[root@zabbix ~]# mysql -uroot -p --skip-column-names -e 'SHOW PROCESSLIST;' | wc -ll
Enter password: 
347
[root@zabbix ~]#

pstree shows this output. Looks like from /etc/my.cnf its taking max_connextions and setting to 450.

[root@zabbix ~]# pstree
init─┬─abrtd
     ├─acpid
     ├─atd
     ├─auditd───{auditd}
     ├─automount───4*[{automount}]
     ├─certmonger
     ├─console-kit-dae───63*[{console-kit-da}]
     ├─crond───crond─┬─logrotate───bzip2
     │               └─sendmail───postdrop
     ├─cupsd
     ├─dbus-daemon
     ├─hald─┬─hald-runner─┬─hald-addon-acpi
     │      │             └─hald-addon-inpu
     │      └─{hald}
     ├─httpd───17*[httpd]
     ├─irqbalance
     ├─login───bash
     ├─master─┬─pickup
     │        └─qmgr
     ├─mcelog
     ├─5*[mingetty]
     ├─mysqld_safe───mysqld───451*[{mysqld}]
     ├─ntpd
     ├─rpc.statd
     ├─rpcbind
     ├─rsyslogd───3*[{rsyslogd}]
     ├─snmpd
     ├─sshd─┬─3*[sshd───sshd───bash]
     │      └─sshd───bash───pstree
     ├─udevd───2*[udevd]
     ├─zabbix_agentd───5*[zabbix_agentd]
     └─zabbix_server─┬─418*[zabbix_server]
                     └─30*[zabbix_server───sh───fping]
[root@zabbix ~]#

Some gotcha issues which i am not able to figure out from any points.
Let me look at perkona presentation.
Oh Finally, so i also need to look at support package. Thanks for that.
Strange issue is, though i set Group for database down message to None, i am still getting zabbix down emails

Thanks

[Arpishke]

Guys, check the server SPACE, for me i ran out of space and that was the reason for this error. try to ssh to the Zabbix server and do: df -h and for me was Used 100%

[tienpt]

Today I ground this issue again. The cause is OutOfMemory which declares in /var/log/syslog.
I also question: where is the trigger: "Zabbix database is down." when receiving an email from Zabbix. Then I could not find that in Configuration/ Hosts/ Triggers.

After that, I still receive another email with the same subject. Then I check the status of mysql service, and find mysql has restarted. Go /var/log/syslog, I find "OutOfMemory", so mysql service have been killed. Hope this help.

Also why this email "Zabbix database is down" doesn't show in Report/ Action Log? Because the database was down, so it could not write this "Zabbix database is down" to MySQL, then show it in Zabbix Portal.

[Osdansua]

Please, I need help. My Zabbix server is down:

● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: deactivating (stop-sigterm) since Sat 2021-01-02 21:42:07 -05; 22min ago
Process: 938971 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 569486 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 569489 (zabbix_server)
Tasks: 70 (limit: 49053)
Memory: 721.0M
CGroup: /system.slice/zabbix-server.service
├─569489 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─569497 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.088416 sec, syncing configuration]
├─569500 /usr/sbin/zabbix_server: housekeeper [connecting to the database]
├─569501 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.000393 sec, processing maintenances]
├─569502 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000600 sec, getting values]
├─569503 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000650 sec, performing discovery]
├─569504 /usr/sbin/zabbix_server: discoverer #2 [processed 0 rules in 0.000683 sec, performing discovery]
├─569506 /usr/sbin/zabbix_server: discoverer #3 [processed 0 rules in 0.000477 sec, performing discovery]
├─569508 /usr/sbin/zabbix_server: discoverer #4 [processed 0 rules in 0.000637 sec, performing discovery]
├─569509 /usr/sbin/zabbix_server: discoverer #5 [processed 0 rules in 0.000666 sec, performing discovery]
├─569510 /usr/sbin/zabbix_server: history syncer #1 [processed 11 values, 18 triggers in 0.001515 sec, syncing history]
├─569512 /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000113 sec, syncing history]
├─569513 /usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 0 triggers in 0.009424 sec, syncing history]
├─569514 /usr/sbin/zabbix_server: history syncer #4 [processed 0 values, 0 triggers in 0.000033 sec, syncing history]
├─569515 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000816 sec, processing escalations]
├─569516 /usr/sbin/zabbix_server: proxy poller #1 [terminated]
├─569518 /usr/sbin/zabbix_server: self-monitoring #1 [terminated]
├─569519 /usr/sbin/zabbix_server: task manager [processing tasks]
├─569520 /usr/sbin/zabbix_server: poller #1 [got 4 values in 12.007649 sec, getting values]
├─569522 /usr/sbin/zabbix_server: poller #2 [got 68 values in 5.081793 sec, getting values]
├─569523 /usr/sbin/zabbix_server: poller #3 [got 4 values in 12.007478 sec, getting values]
├─569524 /usr/sbin/zabbix_server: poller #4 [got 75 values in 15.204384 sec, getting values]
├─569525 /usr/sbin/zabbix_server: poller #5 [got 76 values in 14.802647 sec, getting values]
├─569526 /usr/sbin/zabbix_server: poller #6 [got 63 values in 3.690285 sec, getting values]
├─569527 /usr/sbin/zabbix_server: poller #7 [got 126 values in 5.755994 sec, getting values]
├─569528 /usr/sbin/zabbix_server: poller #8 [got 230 values in 5.326867 sec, getting values]
├─569529 /usr/sbin/zabbix_server: poller #9 [got 219 values in 5.853566 sec, getting values]
├─569530 /usr/sbin/zabbix_server: poller #10 [got 4 values in 12.007095 sec, getting values]
├─569531 /usr/sbin/zabbix_server: poller #11 [got 4 values in 12.012434 sec, getting values]
├─569532 /usr/sbin/zabbix_server: poller #12 [got 68 values in 13.774684 sec, getting values]
├─569533 /usr/sbin/zabbix_server: poller #13 [got 64 values in 4.577797 sec, getting values]
├─569534 /usr/sbin/zabbix_server: poller #14 [got 348 values in 4.523472 sec, getting values]
├─569536 /usr/sbin/zabbix_server: poller #15 [got 4 values in 12.008245 sec, getting values]
├─569537 /usr/sbin/zabbix_server: poller #16 [got 86 values in 6.010990 sec, getting values]
├─569538 /usr/sbin/zabbix_server: poller #17 [got 639 values in 4.721309 sec, getting values]
├─569539 /usr/sbin/zabbix_server: poller #18 [got 4 values in 12.009918 sec, getting values]
├─569540 /usr/sbin/zabbix_server: poller #19 [got 4 values in 12.011971 sec, getting values]
├─569541 /usr/sbin/zabbix_server: poller #20 [got 147 values in 14.441408 sec, getting values]
├─569542 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000137 sec, getting values]
├─569543 /usr/sbin/zabbix_server: unreachable poller #2 [got 5 values in 2.763210 sec, getting values]
├─569544 /usr/sbin/zabbix_server: unreachable poller #3 [got 0 values in 0.000103 sec, getting values]
├─569546 /usr/sbin/zabbix_server: unreachable poller #4 [got 0 values in 0.000021 sec, getting values]
├─569547 /usr/sbin/zabbix_server: unreachable poller #5 [got 0 values in 0.000100 sec, getting values]
├─569549 /usr/sbin/zabbix_server: unreachable poller #6 [got 0 values in 0.000076 sec, getting values]
├─569550 /usr/sbin/zabbix_server: unreachable poller #7 [got 0 values in 0.000236 sec, getting values]
├─569551 /usr/sbin/zabbix_server: unreachable poller #8 [got 0 values in 0.000024 sec, getting values]
├─569552 /usr/sbin/zabbix_server: unreachable poller #9 [got 0 values in 0.000031 sec, getting values]
├─569553 /usr/sbin/zabbix_server: unreachable poller #10 [got 0 values in 0.000032 sec, getting values]
├─569554 /usr/sbin/zabbix_server: trapper #1 [terminated]
├─569555 /usr/sbin/zabbix_server: trapper #2 [processing data]
├─569556 /usr/sbin/zabbix_server: trapper #3 [processing data]
├─569557 /usr/sbin/zabbix_server: trapper #4 [processing data]
├─569559 /usr/sbin/zabbix_server: trapper #5 [processing data]
├─569560 /usr/sbin/zabbix_server: icmp pinger #1 [pinging hosts]
├─569561 /usr/sbin/zabbix_server: icmp pinger #2 [pinging hosts]
├─569562 /usr/sbin/zabbix_server: icmp pinger #3 [pinging hosts]
├─569563 /usr/sbin/zabbix_server: icmp pinger #4 [pinging hosts]
├─569564 /usr/sbin/zabbix_server: icmp pinger #5 [pinging hosts]
├─569565 /usr/sbin/zabbix_server: alert manager #1 [terminated]
├─569566 /usr/sbin/zabbix_server: alerter #1 [sent 0, failed 0 alerts, idle 899.947355 sec during 899.949409 sec]
├─569567 /usr/sbin/zabbix_server: alerter #2 [sent 0, failed 0 alerts, idle 899.008745 sec during 899.010000 sec]
├─569568 /usr/sbin/zabbix_server: alerter #3 [sent 0, failed 0 alerts, idle 898.946624 sec during 899.011286 sec]
├─569569 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 15 values, idle 5.001800 sec during 5.001905 sec]
├─569570 /usr/sbin/zabbix_server: preprocessing worker #1 started
├─569571 /usr/sbin/zabbix_server: preprocessing worker #2 started
├─569573 /usr/sbin/zabbix_server: preprocessing worker #3 started
├─569574 /usr/sbin/zabbix_server: lld manager #1 [terminated]
├─569575 /usr/sbin/zabbix_server: lld worker #1 [processed 3 LLD rules, idle 5.144417 sec during 5.193607 sec]
├─569576 /usr/sbin/zabbix_server: lld worker #2 [processed 3 LLD rules, idle 7.123376 sec during 7.210671 sec]
└─569577 /usr/sbin/zabbix_server: alert syncer [queuing alerts]

dic 31 10:29:43 localhost.localdomain systemd[1]: Starting Zabbix Server...
dic 31 10:29:43 localhost.localdomain zabbix_server[569486]: /usr/sbin/zabbix_server: /usr/pgsql-12/lib/libpq.so.5: no version information available (required by /usr/>
dic 31 10:29:43 localhost.localdomain systemd[1]: zabbix-server.service: Supervising process 569489 which is not our child. We'll most likely not notice when it exits.
dic 31 10:29:43 localhost.localdomain systemd[1]: Started Zabbix Server.
ene 02 21:42:07 localhost.localdomain systemd[1]: Stopping Zabbix Server...