Trigger still fires after it was deleted

[SunF1re]

Hi there, last Friday we had a trigger that was "flapping". It was the default "80% os Process CPU Load on {HOST.NAME}" trigger from the "Generic JMX" template.
This was ok because the we had some funny load patterns on the box. So at first to at least disable the flapping I rewrote the trigger condition to be more forgiving and also make use of hysteresis mechanisms.

But I still got the same amount of PROBLEM/OK messages. So I disabled the trigger in the template. After an hour it would still send the same amount of notifications out. Lastly I deleted the trigger from the template.... to no effect. 3 days later we still get the notifications. So there is no trigger in the template and also not in the actual hosts. I ran a query "select * from triggers where description like '%CPU%'" and it came back empty. Restarting the Zabbix Server process also doesn't help.

Also when I look into the alerts table the last alert that shows up for this trigger has a "clock" time of right before I modified the template trigger expressions last Friday. None of the few hundred emails that were sent out since then about this deleted trigger show up in the table.

Does anybody know what could cause this or where in the DB these "ghost" trigger are stored so I can maybe delete them manually. In the UI there is not trace of them in any place yet they keep on spamming us with alerts.

Our Zabbix version is 2.4.3.

Thanks alot!

[SunF1re]

Update: we tried to delete the CPU items history for a few of the hosts that still report the error hoping that this would stop it. No luck there.
Then as a last resort we deleted and recreated the entire host. That seemed to have stopped it - for all but 1 host. So the entire host is now deleted and we still get this stupid Process CPU notifications for it...

Does anybody have an idea what is going on and how to purge these phantom notifications? I am really lost since I can hardly do more than just delete the entire host.

Thanks

[kloczek]

Does anybody have an idea what is going on and how to purge these phantom notifications? I am really lost since I can hardly do more than just delete the entire host.

triggers table has primary key with triggerid.
Probably you had another for example untemplated trigger with similar expression and description but with different triggerid. Ergo: it was not a zabbix issue.

[SunF1re]

triggers table has primary key with triggerid.
Probably you had another for example untemplated trigger with similar expression and description but with different triggerid. Ergo: it was not a zabbix issue.

Thanks but your conclusion is premature and it seems you didn't read my initial post.
I already checked the "triggers" table and there is not trigger - either with or without template - that has the same or similar trigger description.

select count(*) from triggers where description like '%CPU%' or comments like '%CPU%';

returns 0

Yet the emails that I still keep on getting looks like this:

Problem report:
===========
Trigger: 80% os Process CPU Load on jvm-lsc1-1-prod
Trigger status: PROBLEM
Trigger severity: Average
Trigger URL:

Item values:

1. os Process CPU Load (jvm-lsc1-1-prod:jmx["java.lang:type=OperatingSystem",ProcessCpuLoa d]): 100 %
2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*
3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*

Original event ID: 21027308
===========

Clear report
===========
Trigger: 80% os Process CPU Load on jvm-lsc1-1-prod
Trigger status: OK
Trigger severity: Average
Trigger URL:

Item values:

1. os Process CPU Load (jvm-lsc1-1-prod:jmx["java.lang:type=OperatingSystem",ProcessCpuLoa d]): 0.27 %
2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*
3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*

Original event ID: 21027308
=================

Also note also that host "jvm-lsc1-1-prod" has already been deleted.

If this is not a Zabbix issue I would be happy because then it should be easy to fix but I am pretty sure this is a Zabbix issue. I just don't know where to look at to get to the bottom of it.

Thanks