Zabbix 1.8.11 Suddenly Stops Sending Notifications

[jashley]

Correct. I have never had that happen before. Wasn't prepared for that.

[tchjts1]

If you take regular DB backups, you could try restoring to a period prior to when you started having issues... and get the majority of your history back.

[jashley]

I'm not that cool. I really need to set that up. That is for another thread

[Arsen]

I have found the root cause.

Database corruption

This corruption caused me to have to unlink and clear every host with an agent installed. As I did this and linked them back to the template I started receiving alerts immediately. I have lots of clean up work to do.

I have the same problem now. Could You please give some more information on how You diagnosed and solved the problem? What do you exactly mean by "unlink and clear every host with an agent installed"?

[jashley]

I have the same problem now. Could You please give some more information on how You diagnosed and solved the problem? What do you exactly mean by "unlink and clear every host with an agent installed"?

Well... I have not determined the root cause as I still from time to time stop receiving Actions that worked before. But what I have done to regain full functionality is go into Hosts Configuration> Select the host that is not sending Actions>On the right hand side under Linked Templates select the template associated with the missing Action and hit Unlink and clear. Then save it and add the template back then save it again.

I wish I could get and give more help on this. I have used Zabbix for 10 years and have never had this issue and any real issue that I could not fix with the help of the forums.

Good luck.

[Arsen]

Unfortunately this doesn't seem to help.
It looks like i may have some problems with database, for example when I just tried to update an action I got an error message:

ERROR: Cannot update action

Error in query [UPDATE actions SET name='Free disk space low (/)', eventsource='0', evaltype='0', status='0', esc_period='120', def_shortdata='Free disk space on {HOST.NAME1} low!', def_longdata='Free disk space on {HOST.NAME1} is low. I tried to remove some stuff but there\'s still less than 1GB left. :,(\r\n\r\nYou can see the log of my attempts to clean up the disk space in file: /home/zabbix/zabbix-clear-node.log', recovery_msg='1', r_shortdata='Free disk space on {HOST.NAME1} OK', r_longdata='Now there\'s more than 1GB left on {HOST.NAME1}. ' WHERE actionid='4'] [Lock wait timeout exceeded; try restarting transaction]
SQL statement execution has failed "UPDATE actions SET name='Free disk space low (/)', eventsource='0', evaltype='0', status='0', esc_period='120', def_shortdata='Free disk space on {HOST.NAME1} low!', def_longdata='Free disk space on {HOST.NAME1} is low. I tried to remove some stuff but there\'s still less than 1GB left. :,(\r\n\r\nYou can see the log of my attempts to clean up the disk space in file: /home/zabbix/zabbix-clear-node.log', recovery_msg='1', r_shortdata='Free disk space on {HOST.NAME1} OK', r_longdata='Now there\'s more than 1GB left on {HOST.NAME1}. ' WHERE actionid='4'".

I had the same problem when I tried to remove a host. I finally manged to remove it by manualy executing the SQL commands mentioned in the error message on database, after killing zabbix_server process. Before stopping it I was getting the same error in MySQL console.

I use Zabbix 2.0.5 and MySQL on a Linux machine. Does anyone has any idea about it?

[jashley]

I'm still using 1.8 because 2.x just would not work for me. Unfortunately your issue seems to be far worse then mine. I wish I could help more but MySQL is not my forte.