Zabbix database went down, only some users received an alert.
Hello. A couple of days ago our Zabbix database went down for 3 days. Some people were on vacation, myself included, but we have a small team that processes the emails sent by Zabbix 24/7 and notifies specific people when necessary. The problem is that during the time when the Zabbix database went down I started receiving this alert in my mailbox every 30 minutes
That team of people responsible for processing the emails did not get this email. Now that the problem is fixed, I tried to determine why some people received this alert and others didn't. The first thing that came to my attention is that it is clearly not an action created by us, because in every action we created and is active we have a template message. In problem history, this problem does not appear, which isn't too surprising because the database was not working. Same with Action Log. I dug through Zabbix and could not find anything that would point me to how this alert is configured and who is receiving it. It honestly looks like some kind of emergency alert. I tried googling this but I did not find anything helpful. The only idea that comes to my mind is that only Super Admins are receiving this kind of emails, but I'm not exactly 100% sure about this. If Zabbix has this kind of functionality that it can send emergency emails even when the database is down, then I would prefer to configure it properly for our use case so that this kind of issue doesn't appear again.
Is anyone able to help with this?
Zabbix database is not available.
MySQL database "zabbix" on "localhost" is not available: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
MySQL database "zabbix" on "localhost" is not available: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Is anyone able to help with this?
Comment