Ad Widget

Collapse

Messages delayed for a long time

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • ToomasAas
    Junior Member
    • Apr 2012
    • 14

    #1

    Messages delayed for a long time

    We are running Zabbix 1.8.11. Two days ago we had a major power outage that affected many systems and caused a lot of triggers to fire. The outage was resolved two days ago, but I am still receiving e-mail alerts about affected systems going down and coming up. The systems indicated by these messages do not currently have any problems, as seen in Dashboard. Looking at headers of these e-mails I can see that they haven't been held in some mail queue but are freshly generated by Zabbix server. I've looked at Administration > Queue, but there is nothing that has been queued for more than 1 minute. I've also checked the database server (separate PostgreSQL server) but the load there is normal and there are not any long-running database client processes. I have restarted the Zabbix server and PostgreSQL server (daemons, not operating systems) but this didn't solve the problem. Zabbix server log shows nothing unusual.

    Where else should I look in order to further troubleshoot the problem? Or if I can't find the root cause, can I somehow tell Zabbix to forget about triggers that were fired more than 24h ago?
  • ToomasAas
    Junior Member
    • Apr 2012
    • 14

    #2
    Solved

    Looks like it was somehow related to 'escalations' table as described in this thread:



    Even though we don't actually have escalations defined for any actions, there were ~200 rows in the escalations table yesterday morning, and by today it had dropped to ~100. I emptied the table, and there have been no more delayed messages for 12 hours now.

    Code:
    zabbix=> select count (*) from alerts where status=0 and alerttype=0;
     count
    -------
         0
    (1 row)
    
    zabbix=> delete from escalations;
    DELETE 97

    Comment

    Working...