Zabbix 2.4: Database internal trigger events amount is too big

[fanatique]

Hello,

95% of our events table in our Zabbix database is filled up with internal trigger events, resulting in some very slow SQL queries, despite it having indexes.
I researched extensively and understood the concept of those events, why and when are they being generated. However, because there are at least 3 reasons because of which those events can be generated (or rather 3 reasons why a trigger would change its state to unknown) - I cannot pinpoint the source of the problem.
We currently have a total of 410 hosts, ~248k items, and ~116k triggers. Many of those hosts go unavailable for some time, because they are in other countries and sometimes there are connectivity issues, so all their triggers go into an unknown state - which generates a lot of events. However, I'm not quite sure if that is really the reason behind this number, this is just a guess at best.
In the events table there currently are ~10,000,000 events - ~9,500,000 of which are internal trigger events.

My questions:
1. Is there a proper way to see what is the root of this huge amount of internal trigger events?
2. Can we safely delete those internal trigger events without causing Zabbix any issues?
3. Is there a way to disable the functionality of Zabbix generating these internal trigger events? I read that it has been added in v2.2 so we can get reports on unknown trigger states, however, if we do not want that - can we disable it somehow, since it fills up the entire table?

The server this is on is a really busy server, so we need to optimize it as much as we can - 95% of our events table being filled up with something we don't use seems bad.

Thanks in advance.

[fanatique]

Since nobody answered and I somewhat managed to fix all of this myself, I will answer my own questions, if anybody else is interested and/or struggling:
1. There is no proper way. Keep in mind that the causes for generating those internal events can be a few - triggers with errors from used functions or triggers of not supported items (which go into the unknown state), items with errors (that go into the not supported state), items that are disabled but have triggers attached that are enabled (in this scenario the triggers go into the unknown state). These are the main causes. And to see what type of events you have the most you can just play around with the events table: the column source is the type of event (internal or some else), for internal the value is always 3; the column object shows the object type - for internal events the values can be 0 (trigger), 4 (item), or 5 (LLD rule). And if you know that they are triggers or items, you can then just have a look at your triggers/items tables - items have status 3 if they are not supported, you can read their errors column; triggers have state 1 if they are in an unknown state. By reading their errors you can understand more why certain items/triggers have errors.

2. I've deleted the whole events table a few times and I've had no problems (besides having no trigger history), so it's completely safe. Your Zabbix will not crash or anything. However, I suggest you just delete the internal events (source == 3) - that way you just delete the problematic ones and leave the useful ones still intact.

3. Still no idea about that. I've searched a ton and still haven't been able to find a way to disable the generation of internal events or events at all.

[hyapt]

Just created this account to give a shoutout. We are facing a similar issue and good information is hard to come by.

I am really shocked that here, 10 years later, this is still an issue. I understand it can be due to improper setup of triggers, but its unnerving that this is the only solution- drop the events table and start over. I do wish either housekeeping was more reliable or there was some less drastic way to deal with this.

At this rate, it would take probably 10 days of a script running to clear out only the 'bad' events.