Nodata + agent.ping triggers ghost alerts and excessive emails

[Velko]

Hello,
We’ve been experiencing a strange and inconsistent issue with the nodata() function in Zabbix.
Zabbix is generating alerts for some nodata() triggers, but these alerts appear to be false or ghost events — we receive hundreds of email notifications, yet when we click the link in the email, we get:

"No permissions to referred object or it does not exist!"

The problem is not visible in the Zabbix frontend — not in the Problems section or under the host.
Here's what we've noticed:
- We are using zabbix proxies and it happens with more then one of them (may be all)
- The issue happens only during certain time windows, usually at night.
- It’s not always related to agent.ping, although many of the affected triggers are using it.
- We have many other nodata()-based triggers that work correctly and do not cause any issues.
- The affected triggers send emails, but the problems cannot be found in the UI.
- All systems (Zabbix server, proxies, monitored hosts) have correct NTP time sync.
- Zabbix server and proxy logs show no critical errors during the affected times.

We’ve tried:
- Checking Zabbix server logs – no obvious errors (debug level)
- We also tried disabling automatic problem recovery by setting OK event generation = None for the affected triggers. Our idea was to force the events to stay visible in the Zabbix UI, so we could analyze them. Didn't work.
- Ensuring the items are not unsupported or in an error state.

Has anyone experienced something similar?
Could this be a bug related to nodata() handling?
Is it possible these are orphaned or expired events that are somehow still generating actions?
We're starting to think this might be related to:
- Corrupted event table?
- Housekeeping or caching problem?
- High-load race condition at night?
Any advice or suggestions would be greatly appreciated.

Best regards

[NgRox]

If it occurs at a specific time, it would be interesting to observe what is executed at that time...

Another thing would be to validate the retention time of these tables. If the problem "disappeared" from the problems tab, it doesn't mean it doesn't exist, but rather that it was deleted from the problems table in the database. However, there is another table (Events) that stores these problems.

I recommend querying the database by searching for the event ID to validate.

Honestly, I recommend troubleshooting at the database level.

In short:

1 - Validate what is executed at that time.

2 - Validate the database... retention time of the problems table, search for these events by eventid, among others...

[Velko]

If it occurs at a specific time, it would be interesting to observe what is executed at that time...

Another thing would be to validate the retention time of these tables. If the problem "disappeared" from the problems tab, it doesn't mean it doesn't exist, but rather that it was deleted from the problems table in the database. However, there is another table (Events) that stores these problems.

I recommend querying the database by searching for the event ID to validate.

Honestly, I recommend troubleshooting at the database level.

In short:

1 - Validate what is executed at that time.

2 - Validate the database... retention time of the problems table, search for these events by eventid, among others...

Thanks for your reply!
We tried the first option, but didn't find anything.
I guess we'll need to check the database.
Thanks again!