trigger - 'universal', related to string found in log file

[serwanski]

Hello,

I assume it is not possible but let me ask anyway.

I have template created by myself - working fine, does what expected. It monitores one log file (item) looking for several strings (so have several triggers). To have more log entries monitored, I have to create more triggers in template.

One sample triger is visible below:

-----
uuid: 61794942efb14cfca172f166f08ab2b7
expression: 'find(/DSKSYS linux custom events collector/logrt[/var/log/zabbix/events_zabbix.log,custom_alert,,,all],180,"like","HIGH custom_alert mariadb error")=1'
name: 'DSKSYS linux - MariaDB service down on host'
event_name: 'DSKSYS linux - MariaDB service down on host'
priority: HIGH
manual_close: 'YES'
-----

As shown, if string 'HIGH custom_alert mariadb error' would appear in monitored log file, error with sev 'HIGH' and provided description will be created - which is fine; I wonder however if there is existing / known option to use variables; I would love to set somehow string 'HIGH custom_alert mariadb error' as $var, and the use it in template, for example:

-----
uuid: 61794942efb14cfca172f166f08ab2b7
expression: 'find(/DSKSYS linux custom events collector/logrt[/var/log/zabbix/events_zabbix.log,custom_alert,,,all],180,"like","HIGH custom_alert mariadb error")=1'
set $var as 'string logrt is loking for' (here: 'HIGH custom_alert mariadb error')
name: 'DSKSYS linux - $var'
event_name: 'DSKSYS linux - $var'
priority: `$var | awk '{print($1)}`
manual_close: 'YES'
-----

If there would be such option, I could use one trigger to raise several types of alerts based on one combination: item+trigger. Is it / does it exists?

regards
Tomasz

[cyber]

User Macros? https://www.zabbix.com/documentation..._location_user
Covers most but you cannot set priority like that..

[ISiroshtan]

To note with what Cyber said: use the Multiple even generation mode. From documentation:

PROBLEM event generation mode

Mode for generating problem events: Single - a single event is generated when a trigger goes into the 'Problem' state for the first time; Multiple - an event is generated upon every 'Problem' evaluation of the trigger.

[serwanski]

User Macros? https://www.zabbix.com/documentation..._location_user
Covers most but you cannot set priority like that..

not sure to be honest, but may be that I'm missing the point; reading/checking about macros, I would say that:

- having item key as below:



- I have related entry in database:



- is it anyhow possible to use macro (predefined? new to be created?) to extract 'AVERAGE' or 'HIGH' string from log file / zabbix database, and use it as parameter for alert creation? As I understood I could use {ITEM} macro over item key value, not over entry fetched into database, am I right?

regards
Tomasz

[cyber]

Nevermind, I think I did not think too clearly that day...

One thing is, severity cannot be adjusted based on info from item data, its hardcoded... so that "priority: `$var | awk '{print($1)}`" is out of the window..
I guess minimal setup would be an item per severity, just picking up whatever contains HIGH or AVERAGE or ... And then triggers on top of that... with multiple event generation, as mentioned...
I think , what I meant with user macros there, was that you can them as item parameters, so you can use same template on many hosts, but define those patterns locally on each host... I think...

[serwanski]

Nevermind, I think I did not think too clearly that day...

One thing is, severity cannot be adjusted based on info from item data, its hardcoded... so that "priority: `$var | awk '{print($1)}`" is out of the window..
I guess minimal setup would be an item per severity, just picking up whatever contains HIGH or AVERAGE or ... And then triggers on top of that... with multiple event generation, as mentioned...
I think , what I meant with user macros there, was that you can them as item parameters, so you can use same template on many hosts, but define those patterns locally on each host... I think...

thank you for the answer anyway if anyone would have similar problem then, here is what I've done:

1) I have one item to monitor log file, looking for string 'custom_event'; if such string will be found in line, it if fetched to database
2) based on above lines, triggers are fired; in template I have one item (looking for 'custom_event') and several triggers (searching for, for example, 'HIGH mariadb error' string, where monitoring scripts has to enter same error info in monitored log file
3) based on above, I have more triggers like 'CRITICAL mongodb error' ot 'AVERAGE nfs error' base on which particular trigger is raising alert

Above works, but each type of alert: severity+event, needs to have separate trigger. If anyone would be able to simplify it and share, would appreciate.

regards
Tomasz

[ISiroshtan]

You can do a one trigger (kinda) to rule them all style but it will bring multiple sub-tasks to tackle. As was properly noted, you can not really dynamically control event severity (unless you would go super custom not out-of-the-box shenanigans). So you could do one trigger for each severity and just go with expressions like 'find(/DSKSYS linux custom events collector/logrt[/var/log/zabbix/events_zabbix.log,custom_alert,,,all],,"like","HIGH")=1' for high severity, 'find(/DSKSYS linux custom events collector/logrt[/var/log/zabbix/events_zabbix.log,custom_alert,,,all],,"like","AVERAGE")=1' for average etc.

Set the name of the trigger using macro functions like "{{ITEM.VALUE}.regsub("CEST (HIGH|AVERAGE) (.*)", \2)}" (Maybe even better to name each trigger using that specific severity, like {{ITEM.VALUE}.regsub("CEST HIGH (.*)", \1)} for high etc).
With the above you will get triggers firing with name of actual entry of log that follows after Severity. I.e. for log entry

09:15:14 AM CEST AVERAGE customer_alerts mailq error port 0000 19 mails in queue

you would get alert named

customer_alerts mailq error port 0000 19 mails in queue

Set trigger to "multiple event generation mode".

So now one trigger rule will be opening different problems based on what actually ends up in database from the log. So you only need to properly control logs ensure proper styling, size limiting and duplicate alerts handling.

The problem with such approach in your case:
there is no reliable way to automatically close these event. If you search through multiple entries over the history (like using "find(/host/key,180,...)" ), it will be generating you shitload of duplicate alert. So you can only evaluate latest value with find under such approach. The viable approaches are either you go "only manual closing of alerts" or you need to add recovery record to the log file. If the 2nd - you also need to introduce event tags + recovery expression and set this tag for matching in event recovery.
Additionally, in your 3 line example you have "AVERAGE customer_alerts mailq error port 0000 19 mails in queue" and 'AVERAGE customer_alerts mailq error port 0000 21 mails in queue". For this approach it would be two different events generating two different alerts. So you might need to think on handling duplicate alerts as well (which is possible with Zabbix as well).

I did somewhat similar cases in the past, it is possible, but it requires careful planning and understanding what is logged and how it's processed. All in all I would not call it a solution you're looking for, just another approach that can be taken for this kind of scenarios. And maybe will give you some ideas to test things out.