Triggers with no host being displayed

[mikesphar]

I've noticed some strange situation where if I go to Monitoring -> Triggers I see a list of many triggers in Problem status that are several months old, and have no Host listed. It looks like they are mostly/all from triggers that were part of low-level discovery.

So for example, with show details enabled, the Host column is empty and this is displayed as the trigger Name:

	Free disk space is less than 10% on volume C:\
*ERROR*<10

The age of all the triggers is 4-7 months old, and in some cases it looks like the same trigger replicated several times.

Any advice on how I can clean these up? I was poking around the schema--particularly the triggers table--trying to see if I could these in the tables but have not been able to isolate them yet. The only option I'm given in the web interface is to Bulk Acknowledge them.


Edit: Forgot to add: I'm running Zabbix 2.0.4 currently, though obviously an earlier version back at the time these were triggered originally.

[heaje]

I seem to recall seeing something like this before. When I saw it, I think it was while looking g at triggers under "configuration"->"triggers". just look at all triggers in the combo box and see if the weird triggers show up in the list on some page

[mikesphar]

Yeah if I scroll around in Configuration -> Hosts -> Triggers and select ALL for group and Host, I can see several triggers where the Host column is just not there, so everything to the right of it is shifted to the left and under the Expression column it just says things like *ERROR*=1. I'm assuming that's because the trigger expression refers to an itemid that doesn't exist in the items table, and thus that's why it's not getting linked back to the hosts table.

However, when I select those triggers and delete them, the UI says "Trigger Deleted" but they're still listed. I don't see any errors logged in the zabbix server log either.

Poking around in the database, I see there's no direct connection between a trigger entry and a host or item entry. It looks like maybe the UI just tries to link up triggers to hosts based on parsing item ids out of the expression field?

What I also see when poking around the database is I apparently have almost 600,000 triggers where the templateid is null, and only about 5,000 where it is not null.

I'm wondering if all the triggers with a NULL for templateid are bogus? I only ever create triggers via templates. I'm now wondering if I can just delete all the triggers with a null for the templateid. Are there legitimate cases where triggers have NULL for the template id other than triggers manually created by hand for a host? What about triggers created by low level discovery templates?

Otherwise I'm not sure how to go about generating a list of all triggers that refer to nonexistent itemids. At this point it seems like I may have several hundred thousand bogus trigger entries in my database.

[mikesphar]

So, after more searching and poking around it looks like it is expected for the templateid to be null for triggers that are created by LLD rules, even if that LLD rule is from a template. Confusing, but okay.

With further exploring I've found that there are almost 600,000 triggers in the triggers table where there is no function in the functions table that references that trigger. Maybe that is or isn't normal, but it doesn't appear normal so far from my random testing of other triggers and functions. (And the db consistency check script on the wiki appears to explicitly search for triggers that do not contain a function reference, but not the reverse case of a trigger containing a function reference that does not exist in the functions table.)

This query (yes, a sloppy inefficient one):

select * from triggers where triggerid NOT IN (select triggerid from functions)

Returns 595,288 rows. Note my total number of hosts is only 378, and total number of items is 35,693.

So far it appears based on the descriptions that all the orphaned triggers were created by LLD rules. I don't know if there is/was some edge case where LLD rules would clean up items/functions but somehow leave the triggers behind. I've been searching like mad through forums and bug reports but if this is a known problem I haven't hit on the right combination of keywords. (Although maybe it's similar to what was reported in ZBX-5713.)

If I could figure out some reproducible case I would file it as a bug. As it stands I feel stuck just trying to figure out a way to clean up my apparently corrupted database. I could delete all triggers that don't have a reference in the functions table but I don't know if that's a valid approach or if that would just corrupt my database in other ways.

[mikesphar]

(Updating this in case others find it while searching for a similar problem)

Still haven't confirmed best safe way to remove several hundred thousand apparently orphaned triggers.

I saw this bug: https://support.zabbix.com/browse/ZBX-5968

And had hopes that it was related to my problems, so updated to 2.0.5 and have waited several days to see if the trigger count went down, but no luck, so 2.0.5 for me at least did not resolve the problem.

[syndeysider]

did you come right?

hey mate

did you ever come right with this? Im on 2.4.5 and in the same boat as you. It's only started happening in the past 3 months but our Zabbix instance has been live for almost 3 years now...

I'm stuck with a bunch of orphaned Triggers and some "No host" in trigger display triggers...