Event / log monitoring - multiple instances of a search string?

[DSon]

Hi,

I'm trying to setup a trigger to create alert notifications based on the occurrence of a few strings of text within windows event logs (as well as other log files). Here is my trigger for a windows event log search:

(((({TRIGGER.VALUE}#2 | {Template_APP_Live:eventlog[My Event Log,,error].nodata(300)}#1) & {Template_APP_Live:eventlog[My Event Log,,error].change(0)}=1) & ({Template_APP_Live:eventlog[My Event Log,,error].str("Data content error")}=1 | {Template_APP_Live:eventlog[My Event Log,,error].str("File Corrupt")}=1 | {Template_APP_Live:eventlog[My Event Log,,error].str("Could not find file")}=1)) | ({TRIGGER.VALUE}=1 & {Template_APP_Live:eventlog[My Event Log,,error].str("Previous error event(s) resolved by")}#1))

What I expect this trigger to do is create an alert notification (with an appropriate action, of course) when it finds either of the following strings:

"Data content error"
"File Corrupt"
"Could not find file"

The trigger will auto-resolve when the following string is found:

"Previous error event(s) resolved by"

This works - sort of!

Unfortunately, even though the trigger says ".change(0)=1" - which I think means "if the latest event log value is different to the previous value" - the trigger is set to true ONLY FOR THE FIRST EVENT containing one of the search strings. I've also tried using ".diff(0)=1" but without success - it seems that once the trigger is true, no more alerts are generated.

I thought that using "Normal + Multiple TRUE events" would be a way of doing what I wanted, however this just repeatedly alerts for the latest event in the log (even when using either ".change" and ".diff" ?!).

I would really like to receive notification of EACH OCCURRENCE of the search strings, so our support engineers are notified when each new instance of a problem appears within the event log (or log file).

Has someone else got this working? For info, I'm on Zabbix 1.8.1.

Hope someone can help, this is a great product and the community support is invaluable to guys like me.

Danny.


NOTE: this may be irrelevant, but I'm not 100% sure what the start of the trigger is doing

i.e.

"({TRIGGER.VALUE}#2 | {Template_APP_Live:eventlog[My Event Log,,error].nodata(300)}#1)"

.. could this be this causing problems?

[richlv]

oh my. i don't even want to try to read your trigger, especially at this hour

even you don't understand it yourself, so my best suggestion would be to create the very basic trigger first, and test it with all possible combinations. when you are sure what it does - and that it does it correctly - add something more.

this might be easiest to do with zabbix trapper items that will allow you to send in arbitrary data.

[DSon]

Still not working...

Thanks for your reply Richlv

I've since performed quite a bit of testing, using only sections of the trigger as you suggested, however the fact remains that:

a) multiple matches of a trigger (such as event log entry that contains a search string) are NOT reported by Zabbix; only the first one that sets the trigger ON*
b) if new events appear within the event log, the notification reports these instead of the ORIGINAL event that caused the trigger to be true**

* even when using ".change(0)=1" in the trigger condition
** ITEM.VALUE is supposed to be "historical" isn't it?

The start (the bit I was unsure about) is only to prevent the trigger from:

a) clearing itself automatically (regardless of whether the event is resolved)
b) staying ON when no new event log entries appear

In other words, the bit I was unsure about is nothing to do with the bit that I expect to work (if that makes sense?!).

Has anyone else on this forum managed to get working what I am trying to achieve?

Help!?

Danny.

[richlv]

this all looks awfully complicated. what exactly is the problem if you try to use a simpler trigger - only check that the last string is one of the error ones, and enable multiple problem/true events ?

[DSon]

Question..

Hello again Richlv, thanks for posting.

I'd be greatful if you could give me an example of the sort of trigger you talk about. All I want to know is:

1) When an event log entry appears containing a specific string of text
2) When any subsequent log entries appear, again containing this text string

The problem that I had with "+ multiple TRUE" is that I get an alert for the last entry found every 30 seconds (the refresh rate of the item in question).

If only + multiple TRUE reported ONCE for EACH true trigger match - I'm sure that would solve this problem!

Unless I'm doing something really silly?

Danny.

[richlv]

hmm. this looks like a bit hard to resolve currently...

adding a time based function like nodata() makes triggers recalculate every 30 seconds, and enabling multiple problem events will lead to a message every 30 seconds.

so my understanding is that you want it to trigger on every new entry, but still time out after a while - is that correct ?

[DSon]

Hello again Richlv, thanks for the continued assistance.

The trigger needs to be able to generate a notification for each new instance of a few different search strings, and the notification message needs to report each new value (to enable the independant reporting of problems as they occur).

I'm not sure what you mean by time out, but I only use the nodata() function to prevent the trigger from clearing by itself.

The idea is that our support techies will run a script that writes a "problem event(s) now cleared" message to the event log in question, which in turn resolves the trigger. This is to show evidence of the fact that the problem(s) have been attended to.

Whilst I appreciate that this approach would resolve any and all outstanding event log problems, having zabbix alert on each occurrence of a search string would at least give the support team awareness of all current issues (rather than the original string that fired the trigger, repeated every x seconds!).

Does this sound do-able with Zabbix?

If not, until the even log processing part of Zabbix can be utilitised in a practical way, it would be very helpful if you could provide me with a simple way of capturing specific event log entries - and have zabbix notify them as unique problem events?

Thanks again,
Danny.

thanks,
Danny.

[richlv]

The trigger needs to be able to generate a notification for each new instance of a few different search strings, and the notification message needs to report each new value (to enable the independant reporting of problems as they occur).

i believe that referencing last value with a macro should work for notifications

The idea is that our support techies will run a script that writes a "problem event(s) now cleared" message to the event log in question, which in turn resolves the trigger. This is to show evidence of the fact that the problem(s) have been attended to.

Whilst I appreciate that this approach would resolve any and all outstanding event log problems, having zabbix alert on each occurrence of a search string would at least give the support team awareness of all current issues (rather than the original string that fired the trigger, repeated every x seconds!).

then i'm slightly confused. you can send out alert on each occurrence of the problem string, you don't need the nodata() function for that.

what is the actual functional requirement ? maybe you can give a specific scenario, including incoming values and exact results desired ?
because i'm getting a feeling that i don't really understand what you are trying to achieve

If not, until the even log processing part of Zabbix can be utilitised in a practical way, it would be very helpful if you could provide me with a simple way of capturing specific event log entries - and have zabbix notify them as unique problem events?

you can filter on specific text, and supposedly also on eventlog id (although i haven't dealt with win systems for many years by now, others should be better suited to help with that)

[DSon]

My event log monitoring requirement

Hello again Rich,

Thanks again for the continued support on this.

What I need is a trigger that fires every time one of a select few strings of text is found within the description of an event log entry.

If there are several instances within, say, 1 hr, I need the trigger to fire for each occurrence. The idea is that each time the trigger fires, an appropriate action creates a notification that contains the problem event log entry details (problem type, filename(s) affected etc).

The support team would then have a means of tracking individual occurrences of problem event log entries. They would resolve any outstanding issues, then run a simple script that writes a "Problems resolved by <username>" to the event log being monitored. This particular event log entry would then cause the trigger to resolve.

What I'm struggling with is that the trigger only fires for the first instance of an event log entry matching the trigger. Also, the action notification only reports the details of the first problem event log entry that caused the trigger to fire (these notificaions are set to repeat every hr, by the way). So if any new problem events appear, they could get missed.

The only way I can see this working is if I let Zabbix auto-resolve the trigger a few seconds after it fires. This way, any newer problem event entries should also create unique action notifications. And our support team have an audit of individual events, too.

The problem I have with making the trigger auto-resolve is that the triggers screen (used as a visual aid by our support team) will say everything is okay. This could mean that alert notifications are ignored, since we get another action notification when triggers are resolved.

Now, bearing all the above in mind, I've pasted in an example of the trigger I'm using, which fires when it sees the first event with a problem (i.e. ONLY the first event), amd it clears when a "Resolved" event entry is written to the event log being monitored.

EXAMPLE TRIGGER:

((({TRIGGER.VALUE}#2 | {Template_APP_Live:eventlog[MyEventLog,,error].nodata(300)}#1) & {Template_APP_Live:eventlog[MyEventLog,,error].str("Corrupt record found in file <FILENAME>")}=1) | ({TRIGGER.VALUE}=1 & {Template_APP_Live:eventlog[MyEventLog,,error].str("Previous error event(s) resolved by")}#1))

If you think some / all of the above trigger can be improved, please let me know.

I just need a way to achieve INDIVIDUAL problem event log tracking.

regards,
Danny.

[richlv]

What I'm struggling with is that the trigger only fires for the first instance of an event log entry matching the trigger.

ok...this is only getting more complicated, so let's try to break down the problem in small pieces

first, and i might be going in circles, but why doesn't "multiple problem events" work for you ? the problem you had with it notifying every 30 seconds was because of nodata() function - leaving it out should deal with this problem.

Also, the action notification only reports the details of the first problem event log entry that caused the trigger to fire (these notificaions are set to repeat every hr, by the way). So if any new problem events appear, they could get missed.

does this mean you are using notifications ? if yes, just use macro reference to last(0) value of the item in question

I just need a way to achieve INDIVIDUAL problem event log tracking.

not sure about this one... do you mean that for each problem entry you need to match a specific "resolved" entry, or that "resolved" should clear the currently active trigger, no matter how many problem entries were there before ?

first should be doable, second might be very hard.

ps. it sounds like a better solution would be to fix the issues that cause "Corrupt record found in file", although i guess not always it is that easy

[jayalps]

Was there ever a resolution to this? I am looking for something similar.

I want to be able to search a log file and receive an alert for every time a certain string appears.

I do not want multiple alerts for one search. I want one alert for each instance of that string.

Right now when I create a basic trigger it alerts for the first time and if the same string appears in the file after that it does not alert again.

Any info would be helpful

[elvar]

Was there ever a resolution to this? I am looking for something similar.

I want to be able to search a log file and receive an alert for every time a certain string appears.

I do not want multiple alerts for one search. I want one alert for each instance of that string.

Right now when I create a basic trigger it alerts for the first time and if the same string appears in the file after that it does not alert again.

Any info would be helpful

I have a need for this as well.