Zabbix 6.0.19: how to get all contents of .log file for pre-processing?

[paxas]

I need to monitor a .log file that is formatted in xml.

Basically all i need is to detect the word "Failed". If i use this Item Key:

logrt["C:\temp\myfile.log",Failed,,,skip]

the results are unpredictable - for certain files i do see the "Failed" keyword in Monitoring - Latest Data, but sometimes there is nothing captured.

Therefore, i would like to capture the entire contents of the .log file and then do some preprocessing. The .log file is generated 2 times per day and its maximum size never exceeds 8KB, so i believe it won't cause any strain for the Zabbix server.

But i dont know how to do it - if i use this simple Key:

logrt["C:\temp\myfile.log"]

then there is nothing in Monitoring - Latest Data.

I also tried to capture entire xml content with regex (confirmed that it works ok in regex101.com):

logrt["C:\temp\myfile.log","(<log.*?<\/log>)",,,,\1]

but again - nothing in Monitoring - Latest Data.

So how do i get the entire contents then?

Here is a sample log entry for reference:

<log status="0" start="638385804099562112" end="638385804283916928" taskname="MyTask - Table from ABC to XYZ DB_09212023_0439PM"><messages><msg code="2" type="0">Start Time: Tuesday, December 19, 2023 1:00:27 PM</msg><msg code="1" type="0">Executing (Delete Rows): DeleteRows MyTask-XYZ</msg><msg code="2" type="0">Start Time: Tuesday, December 19, 2023 1:00:27 PM</msg><msg code="100" type="160083">ERROR 160083: Failed to delete relationships for object</msg><msg code="100" type="-2147467259">Failed to execute (Delete Rows).</msg><msg code="3" type="0">Failed at Tuesday, December 19, 2023 1:00:28 PM (Elapsed Time: 0.73 seconds)</msg><msg code="100" type="-2147467259">Failed to execute (MyTask).</msg><msg code="3" type="0">Failed at Tuesday, December 19, 2023 1:00:28 PM (Elapsed Time: 0.96 seconds)</msg></messages></log>

[cyber]

"Powershell maybe with Userparameter to read it?"


Can you elaborate how that would work? I write a standalone Powershell script to parse the .log file and then somehow redirect the script's output to Zabbix agent to consume? Is there some online guide on how to do it?

pretty much yes, but you can call that script from agent... https://www.zabbix.com/documentation...userparameters
You probably need to handle the file choosing in PS script, it is easier there, than in Zabbix directly as you can use all kind of time and date related functions, that are not available in zabbix items.. After that use the same script to "read" the file and its contents will be sent over to Zabbix...

[cyber]

logrt monitors for changes in file and picks them up line by line. It will not pull whole file for you at once.. use "vfs.file contents".

[paxas]

logrt monitors for changes in file and picks them up line by line. It will not pull whole file for you at once.. use "vfs.file contents".

OK, my problem description was not entirely clear.

I have to use logrt because the new log file is being generated everyday. Furthermore, in the same directory other log files from another process are also generated there, once per week.

Therefore, I need to grab a single .log file which was generated most recently.

For that purpose i use logrt + regex.

When i try to use vfs.file.contents["C:\temp\<some_regex>.log"] i get error "Cannot open file: [2] No such file or directory"

[cyber]

I have to use logrt because the new log file is being generated everyday.

but you cannot use it as it works differently... it does not return file contents as one value... it works line by line...

When i try to use vfs.file.contents["C:\temp\<some_regex>.log"] i get error "Cannot open file: [2] No such file or directory"

That logical, as this key requires hardcoded name and does not accept regex.

Then maybe you need to experiment with vfs.dir.get and some discoveries based on that...

[PeterZielony]

Powershell maybe with Userparameter to read it?
write the logic for rotating logs (last modified and regex of the file name). Within PowerShell, you can do preprocessing and grab the necessary info you need more clearly

it should grab the whole file this way.

with vfs .dir.get you might take the file name and then create discovery based on each file - but you will end up with tons of items in Zabbix - this doesn't scale very well

[paxas]

"Powershell maybe with Userparameter to read it?"


Can you elaborate how that would work? I write a standalone Powershell script to parse the .log file and then somehow redirect the script's output to Zabbix agent to consume? Is there some online guide on how to do it?

[cyber]

"Powershell maybe with Userparameter to read it?"


Can you elaborate how that would work? I write a standalone Powershell script to parse the .log file and then somehow redirect the script's output to Zabbix agent to consume? Is there some online guide on how to do it?

pretty much yes, but you can call that script from agent... https://www.zabbix.com/documentation...userparameters
You probably need to handle the file choosing in PS script, it is easier there, than in Zabbix directly as you can use all kind of time and date related functions, that are not available in zabbix items.. After that use the same script to "read" the file and its contents will be sent over to Zabbix...

[paxas]

Thanks, i set up the custom script and received an email notification when testing. The only problem is with Trigger expression - I am not sure how to write it so that it evaluates to FALSE after certain period.

Basically the Trigger should fire when in the script's output there is a word "Failed", and the problem should be regarded as resolved when in the script's output a word "Failed" is not found.

After i made changes in the Item key, the Trigger Expression was automatically changed to this:

count(/myhost/ps1.script,1m)>0 and nodata(/myhost/ps1.script,5m)=0

The script always return some output with each run, so i attempted to change the Trigger Expression to this:

count(/myhost/ps1.script.str(Failed),1m)>0 and count(/myhost/ps1.script.str(Failed),5m)<1

But Zabbix frontend does not allow such expression.

What would be the correct one then?

[paxas]

nevermind my latest question - i solved it by using this expression:

find(/myhost/ps1.script,5m,"like","Failed")

[z0nk]

How can I parse output from vfs.dir.get? or just how to select fields from it? I need only basename and size, when for each file there is a lot informations:

{
    "basename": "...",
    "pathname": "...",
    "dirname": "...",
    "type": "...",
    "user": "...",
    "group": "...",
    "permissions": "...",
    "uid": ...,
    "gid": ...,
    "size": ...,
    "time": {
        "access": "...",
        "modify": "...",
        "change": "..."
    },
    "timestamp": {
        "access": ...,
        "modify": ...,
        "change": ...
    }
}

[ISiroshtan]

dependent items (https://www.zabbix.com/documentation...ndent_items)  + preprocessing(https://www.zabbix.com/documentation...rocessing) of JSONPath type(https://www.zabbix.com/documentation..._functionality)

[z0nk]

Preprocessing failed for: [{"basename":"[...]","pathname":"[...]","dirname":"[...]
1. Failed: cannot extract value from json by path "$.basename": no data matches the specified path

What I do wrong?

JSON is like above in previous post, but many accumulated so

[JSON,JSON,JSON]

[z0nk]

Ok I found - need double dot on begin: $..basename etc, because of this accumulation:

JSON is like above in previous post, but many accumulated so

[JSON,JSON,JSON]

[ISiroshtan]

I would argue the topic we discussing right is absolutely not related to this thread, but oh well... 

Can you describe the end goal you trying to achieve? Like show the actual JSON you have (replace sensitive values with some gibberish) and explain what you want to get as a result.
Then I might be able to write you a preprocessing rule you looking for. Or will explain how to do what you want.

[z0nk]

When I have



then reasults are:

[2518876,"<filename there>",16177428,"<filename there>",[...]

is possible to do size in MB/GB unit?

[ISiroshtan]

Easily - no

In a bit complicated way? - Add extra pre-processing step of javascript type and write a code to parse the sting, do the conversion and then return desired result.