Zabbix-server died, no database exists.

[mahmood]

Hi
I have noticed the zabbix chart has been dead since the past 12 hours ago.


َThe zabbix-server is dead as below:

# service zabbix-server status
● zabbix-server.service - Zabbix Server
Loaded: loaded (/lib/systemd/system/zabbix-server.service; disabled; vendor preset: enabled)
Active: deactivating (stop-sigterm) (Result: exit-code) since Fri 2020-08-07 04:48:51 +0430; 16h ago
Process: 71311 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 18798 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 18808 (code=exited, status=1/FAILURE)
Tasks: 1 (limit: 11889)
CGroup: /system.slice/zabbix-server.service
└─18815 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.012809 sec, idle

Aug 07 04:48:51 courseware systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 4
Aug 07 04:48:51 courseware systemd[1]: Stopped Zabbix Server.
Aug 07 04:48:51 courseware systemd[1]: Starting Zabbix Server...
Aug 07 04:48:51 courseware systemd[1]: zabbix-server.service: Supervising process 18808 which is not our chi
Aug 07 04:48:51 courseware systemd[1]: Started Zabbix Server.
Aug 07 04:48:51 courseware systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAI
root@courseware:~# cd /var/log/zabbix/

In the log, I see many records like

84345:20200807:034559.314 using configuration file: /etc/zabbix/zabbix_server.conf
84345:20200807:034559.321 [Z3001] connection to database 'zabbixdb' failed: [1040] Too many connections
84345:20200807:034559.321 Cannot connect to the database. Exiting...
84458:20200807:034609.566 Starting Zabbix Server. Zabbix 4.4.6 (revision 8cc702429d).
84458:20200807:034609.566 ****** Enabled features ******
84458:20200807:034609.566 SNMP monitoring: YES
84458:20200807:034609.566 IPMI monitoring: YES
84458:20200807:034609.566 Web monitoring: YES
84458:20200807:034609.566 VMware monitoring: YES
84458:20200807:034609.566 SMTP authentication: YES
84458:20200807:034609.566 ODBC: YES
84458:20200807:034609.566 SSH support: YES
84458:20200807:034609.566 IPv6 support: YES
84458:20200807:034609.566 TLS support: YES
84458:20200807:034609.567 ******************************

I checked previous logs and see this state transition from healthy to dead:

 5751:20200730:031356.757 executing housekeeper
5751:20200730:031359.241 housekeeper [deleted 5115 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 2.481735 sec, idle for 1 hour(s)]
5751:20200730:041359.907 executing housekeeper
5751:20200730:041403.087 housekeeper [deleted 5117 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 3.177776 sec, idle for 1 hour(s)]
5751:20200730:051403.765 executing housekeeper
5751:20200730:051405.172 housekeeper [deleted 5114 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 1.405024 sec, idle for 1 hour(s)]
5751:20200730:061406.094 executing housekeeper
5751:20200730:061408.065 housekeeper [deleted 5116 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 1.969078 sec, idle for 1 hour(s)]
5757:20200730:065317.141 [Z3005] query failed: [2006] MySQL server has gone away [insert into history (itemid,clock,ns,value) values (29176,1596075796,827330017,2101.317358);
]
5757:20200730:065317.142 [Z3001] connection to database 'zabbixdb' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
5757:20200730:065317.142 database is down: reconnecting in 10 seconds
5753:20200730:065317.164 [Z3005] query failed: [1053] Server shutdown in progress [select h.hostid,h.host,h.name,t.httptestid,t.name,t.agent ,t.authentication,t.http_user,t.http_password,t.ht tp_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file, t.ssl_key_password,t.verify_peer,t.verify_host,t.d elay from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1596075797 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]
5753:20200730:065317.166 [Z3001] connection to database 'zabbixdb' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
5753:20200730:065317.166 database is down: reconnecting in 10 seconds
5800:20200730:065317.381 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select a.alertid,a.mediatypeid,a.sendto,a.subject,a.messa ge,a.status,a.retries,e.source,e.object,e.objectid ,a.parameters,a.eventid from alerts a left join events e on a.eventid=e.eventid where alerttype=0 and a.status=3 order by a.alertid]

That is weird because, there are few databases on the system and the number of connections are fine.

Now, as I login to the database, I DON'T see the zabbix database.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| moodle |
+--------------------+
2 rows in set (0.00 sec)

What has happened then? Any idea for further debug?

[mahmood]

Excuse me, the database exists. I have to login to the DB with the root account.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| moodle |
| mysql |
| performance_schema |
| sys |
| zabbixdb |
+--------------------+
6 rows in set (0.68 sec)

But the problem still exits....

@Admin: I can't edit the title for removing "no database exists" part.