Zabbix watchdog: database is down

[mislav-eu]

Hello everyone.

Recently I'm having tremendous problems with zabbix - so far I've probably tried everything suggest and what I've found, but still, no result/improvements.

Basically the problem is the following:

26859:20170405:070435.209 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (12)
26859:20170405:070435.209 watchdog: database is down

And because of that, zabbix trigger my other events and sends unwanted/not needed SMS messages (since it execute my other scripts).

Question is why zabbix report database as being down if I connect to server few seconds later and:
1) first of all, mysql server is being UP and it's running already XX amount of days
2) I can connect with same zabbix credentials
3) socket and process is alive
4) there is no high load on the server, CPU/memory is fine

There is no recovery status, so I guess this is something hard-coded in zabbix. Zabbix reconnects after few seconds back to mysql and everything is fine.

I also tried this:

http://www.michaelfoster82.co.uk/zabbbix-database-is-down/

but it didn't help. This started to happen recently, for a year, it was fine. I've tried multiple things like restarting server, mysql server, repair/optimize on zabbix database, checking mysql for optimizations - it's not even that I have too many mysql connections and that new ones are somehow blocked/rejected, I'm only using around 23% of connections. Other parameters for innodb/reads/writes are also fine.

What could be the problem?

[batchenr]

Hello everyone.

Recently I'm having tremendous problems with zabbix - so far I've probably tried everything suggest and what I've found, but still, no result/improvements.

Basically the problem is the following:


And because of that, zabbix trigger my other events and sends unwanted/not needed SMS messages (since it execute my other scripts).

Question is why zabbix report database as being down if I connect to server few seconds later and:
1) first of all, mysql server is being UP and it's running already XX amount of days
2) I can connect with same zabbix credentials
3) socket and process is alive
4) there is no high load on the server, CPU/memory is fine

There is no recovery status, so I guess this is something hard-coded in zabbix. Zabbix reconnects after few seconds back to mysql and everything is fine.

I also tried this:

http://www.michaelfoster82.co.uk/zabbbix-database-is-down/

but it didn't help. This started to happen recently, for a year, it was fine. I've tried multiple things like restarting server, mysql server, repair/optimize on zabbix database, checking mysql for optimizations - it's not even that I have too many mysql connections and that new ones are somehow blocked/rejected, I'm only using around 23% of connections. Other parameters for innodb/reads/writes are also fine.

What could be the problem?

when did it stated to happen ? did you try to upgrade or somthing ?
check df -h DB size.
check selinux is disabled
can you see mysql error log?
are you sure zabbix credential is correct at zabbix.conf file ?

[mislav-eu]

when did it stated to happen ? did you try to upgrade or somthing ?

It started to happen recently, it was fine maybe a year. No upgrades performed, I'm on 2.4.8

check df -h DB size.

Since I've truncated tables as mentioned above, it's not big:
~# du -h /var/lib/mysql/zabbix/
1.4M /var/lib/mysql/zabbix/

check selinux is disabled

It's not even installed.

can you see mysql error log?

No errors there - as mentioned, mysql server itself does not crash, but zabbix watch reports it's down and then after few seconds it's up again.

are you sure zabbix credential is correct at zabbix.conf file ?

Yes, they are. Zabbix server is up-and-running for X amounts of days. This happens maybe once a day. I've tried also to log memory / CPU usage every minute and during this period, when zabbix watchdog reports mysql is down - t here is normal/low memory usage, as well as the CPU.

[batchenr]

when you go to /etc/selinux/config you dont have anything there?

try runing a repair for mysql.

mysqlcheck --auto-repair -o --all-databases

[mislav-eu]

No, there is no selinux config file.

I've already tried optimize & repair.

[batchenr]

No, there is no selinux config file.

I've already tried optimize & repair.

>>mv /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock.bak
>> /etc/init.d/mysqld restart

try

[mislav-eu]

I've tried that as well, didn't help.

[batchenr]

I've tried that as well, didn't help.

what if you mysqldump zabbixdb , truncate DB and then incert it back ?

[mislav-eu]

That's the next step indeed - to drop database completely and insert backup (or even new installation of zabbix).

Thanks for everything, I appreciate it.

[batchenr]

That's the next step indeed - to drop database completely and insert backup (or even new installation of zabbix).

Thanks for everything, I appreciate it.

i think its also related to version- maybe upgrade also help.
update us