log file trigger status change

[tekknokrat]

I need some enlightenment with the status change of triggers based on log item.
When I have set a trigger with

{scg_logging:log[/var/log/10.4.190.16/syslog,lua_debug.*~=].count(600)}>0
Event Generation: Normal

I only get one event raised and afterwards nothing. I wonder how the trigger status will ever change. Formerly I had accidently count(600)}>1 and there was a status change because there was the count(600)=1 state. At the moment I can go with

Event Generation: Normal + Multiple TRUE Events

but I wonder how to do that with Normal only.

[Calimero]

Add a dummy .now() (or any other time related function) to the trigger so that zabbix_server evaluates it every time ?

[tekknokrat]

Add a dummy .now() (or any other time related function) to the trigger so that zabbix_server evaluates it every time ?

Thanks for the tip, do you think of something like that?

{hostname:trap.match[exp].count(600)}>0 & now()

Can I also use nodata() for that?

{hostname:trap.match[exp].count(600)}>0 & {hostname:trap.match[exp].nodata(600)#1

[Calimero]

For log monitoring, we use that kind of items/triggers:

{host:log[/path/to/log,ERROR|FATAL].count(300,%unable to connect%)}>5
&
{host:log[/path/to/log,ERROR|FATAL].now(0)}>0

Trigger is true if over the last 5 minutes we had more than five "(ERROR|FATAL) ... unable to connect ...." messages in the logs (and we're past midnight which is always true unless it's 00:00:00).

As we use a 300sec range, the dummy now() condition creates no race condition.

And yes, .nodata() is also one of the "time based functions" along with 'date','dayofweek','time','now'.


Anyway such trigger means that trigger will switch back to false after 5 minutes, unless new (matching) error messages are sent to zabbix_server.

I know some people prefer to "clear" errors by hand... Depends on what you want.

[tekknokrat]

I know some people prefer to "clear" errors by hand... Depends on what you want.

Sounds like an interesting job I'll ask my manager

[tekknokrat]

Strange thing, as soon as i set the dummy value in the trigger the server crashed:

16847:20090615:170206 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR: deadlock detected
DETAIL: Process 16848 waits for ShareLock on transaction 93499508; blocked by process 28226.
Process 28226 waits for ShareLock on transaction 93499513; blocked by process 16848.
[update functions set lastvalue='32' where itemid=28462 and function='count' and parameter='600']
16847:20090615:170206 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR: current transaction is aborted, commands ignored until end of transaction block
[select distinct t.triggerid,t.expression,t.description,t.url,t.com ments,t.status,t.value,t.priority,t.type from triggers t,functions f,items i where i.status<>3 and i.itemid=f.itemid and t.status=0 and f.triggerid=t.triggerid and f.itemid=28462]
16838:20090615:170206 One child process died. Exiting ...

Trigger:

{hostname:trap.matchSCG[unsuccessful].count(600)}>0 & {hostname:trap.matchSCG[unsuccessful].now(0)}>0

Do you have some clue

EDIT: The trigger was active and in PROBLEM state so perhaps this caused the issue (which shouldn't). I reverted the settings, disabled trigger and applied the modified trigger and it seems to work now.

Calimero, thanks for your idea with the dummy condition.

[tekknokrat]

There's an issue when I have two triggers with the same item (hostname:trap.matchSCG[unsuccessful]).

When one trigger's Event Generation is set to PROBLEM and using the now() dummy condition and the other trigger's Event Generation is set to PROBLEM + Multiple True Events.

Trigger 1:

{hostname:trap.matchSCG[unsuccessful].count(600)}>5 & {hostname:trap.matchSCG[unsuccessful].now(0)}>0

Event Generation: Normal

Trigger 2:

{hostname:trap.matchSCG[unsuccessful].count(600)}>0

Event Generation: Normal + Multiple TRUE Events

Trigger 2 generates events also there's no match in the logfile.
It seems this is due to the Dummy condition in Trigger 1.

2009.Jun.16 12:40:46 Logwatch - Match "unsuccessful" PROBLEM Average 30s No
3 1
2009.Jun.16 12:40:15 Logwatch - Match "unsuccessful" PROBLEM Average 31s No
3 1
2009.Jun.16 12:39:45 Logwatch - Match "unsuccessful" PROBLEM Average 30s No

3 1
....

[Calimero]

I have to admit I'm not familiar with "Multiple True Events" so I won't be of any help.

[tekknokrat]

I have to admit I'm not familiar with "Multiple True Events" so I won't be of any help.

Multiple True Events also generate events when the trigger condition is TRUE but the TRIGGER.STATUS doesn't change so it sends exactly one message for each match.

In my case the dummy condition introduced in Trigger 1 generates the events of Trigger 2. I don't think thats by design. What do you think is it worth a bug report?

[Eric]

logfile trigger status change

For log monitoring, we use that kind of items/triggers:

{host:log[/path/to/log,ERROR|FATAL].count(300,%unable to connect%)}>5
&
{host:log[/path/to/log,ERROR|FATAL].now(0)}>0

Trigger is true if over the last 5 minutes we had more than five "(ERROR|FATAL) ... unable to connect ...." messages in the logs (and we're past midnight which is always true unless it's 00:00:00).

As we use a 300sec range, the dummy now() condition creates no race condition.

And yes, .nodata() is also one of the "time based functions" along with 'date','dayofweek','time','now'.


Anyway such trigger means that trigger will switch back to false after 5 minutes, unless new (matching) error messages are sent to zabbix_server.

I know some people prefer to "clear" errors by hand... Depends on what you want.

Calimero,

Could you suggest a method to change/clear the trigger status above by hand?

[danrog]

What you could do is create a "Script" under Administration that looks like this.

/home/zabbix/bin/zabbix_sender -z zabserver -s {HOSTNAME} -k trap.netint -o "Reset: Manually Reset"

As long as your trigger will reset on anything thats not equal to your match, your trigger will be cleared (but manually).