long delay between trigger and action

[billjam]

I'm seeing a long delay between when the trigger is triggered and when the action (email) happens, 3 hours!
Event Time 2010.Mar.01 13:23:42
Action Time 01 Mar 2010 16:16:09

The queue is empty (at least >1 minute).
server load is not high.

Any ideas on why delay is so long?


Zabbix server 1.8.1
Zabbix server is running Yes -
Number of hosts (monitored/not monitored/templates) 572 485 / 3 / 84
Number of items (monitored/disabled/not supported) 46684 31445 / 15011 / 228
Number of triggers (enabled/disabled)[true/unknown/false] 19624 17231 / 2393 [40 / 498 / 16693]
Number of users (online) 19 1
Required server performance, new values per second 465.099 -

[billjam]

getting worse

now delay is over 4 hours.

Anyone now what the process flow is for actions?
Is it zabbix_server receives event from host, looks up all triggers and sets the trigger and logs to mysql database. Then something else checks triggers for actions and applies them?
I have tried restarting zabbix_server and it didn't help.
Maybe I need to bounce both zabbix_server and mysql?

System is 96% idle so I don't think its a hardware resource issue.

Any suggestions?

[alixen]

Hi,

Anyone now what the process flow is for actions?

I don't know enough about Zabbix internals to answer to your question.

However, are you sure that the delay is due to Zabbix and not to your mail server ?
Have you checked Zabbix Audit log to see at what time Zabbix sent these emails ?

Regards,
Alixen

[billjam]

event details shows delay

Thank you for your reply, I do appreciate some response.
Yes I'm sure its not my mail system, these timing that I listed are from the event details page within Zabbix.

Here's the latest action it finally got around to doing:
Event Time: 2010.Mar.02 05:02:43
Action Time: 02 Mar 2010 19:02:41

These times are from the tr_events.php page within Zabbix.

As you can see we are now 14 hours behind!!!
At this point I'm going to turn off actions because they are doing more harm then good.

It would be really helpful if someone could give some sort of insight in to how to trouble shoot this!

[billjam]

checked Zabbix Audit log

Actually your response was helpful.
I had been going by just the events page and hadn't checked the Audit log.
Turns out I had an action set that didn't have any conditions set, so zabbix was very busy doing that action, which didn't really do anything, just a test, instead of my critical email action.

Thank you for the reply!!
Now we'll see if the email action works.... (other action is disabled now)