Logfile device discovery

[PliusZabbixMan]

Hello,

I have a log file, that contains statistics of devices connected to a WiFi interface. The log file a device ID followed by the statistics.

I'm trying to find a way to monitor these statistics, but can't find a solution how to assign statistics to the device ID's.

Example log file:

Dec 29 10 ID:3085 Flows: 4083570 Packets: 417889869 Bytes: 542003885889 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3088 Flows: 8209950 Packets: 730187808 Bytes: 905311900208 Sequence Errors: 0 Bad Packets: 0

So far I have created separate dependent items from the log file:

ID regex - ID:*([0-9]+)

Regex for Flows, Packets, Bytes.... - Flows. ([0-9]+) Packets. ([0-9]+) Bytes. ([0-9]+) Sequence Errors. ([0-9]+) Bad Packets. ([0-9]+)

I'm having trouble assigning these items to their ID's for monitoring.

I've been looking into discovery rules, but don't see how to implement them for this use case.

[irontmp]

Hello,

I have a log file, that contains statistics of devices connected to a WiFi interface. The log file a device ID followed by the statistics.

I'm trying to find a way to monitor these statistics, but can't find a solution how to assign statistics to the device ID's.

Example log file:

Dec 29 10 ID:3085 Flows: 4083570 Packets: 417889869 Bytes: 542003885889 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3088 Flows: 8209950 Packets: 730187808 Bytes: 905311900208 Sequence Errors: 0 Bad Packets: 0

So far I have created separate dependent items from the log file position:

ID regex - ID:*([0-9]+)

Regex for Flows, Packets, Bytes.... - Flows. ([0-9]+) Packets. ([0-9]+) Bytes. ([0-9]+) Sequence Errors. ([0-9]+) Bad Packets. ([0-9]+)

I'm having trouble assigning these items to their ID's for monitoring.

I've been looking into discovery rules, but don't see how to implement them for this use case.

To effectively monitor and assign statistics to device IDs from your log file, you can consider using regular expressions and creating a parsing script or using a tool that supports log parsing and monitoring. Here's a general approach:

1. Regex for Device ID:
   • Create a regex to extract the device ID from each log entry. It seems like you've already done this: ID[0-9]+)
2. Parsing Script:
   • Develop a script (Python, Bash, etc.) to read the log file, extract the device ID using the ID regex, and then extract the statistics using the regex for Flows, Packets, Bytes, etc.
   • Associate the statistics with the corresponding device ID, and you can store or display them as needed.

Example Python script:
pythonCopy code
import re log_file_path = "your_log_file.log" with open(log_file_path, "r") as file: for line in file: device_id_match = re.search(r'ID[0-9]+)', line) if device_id_match: device_id = device_id_match.group(1) flows_match = re.search(r'Flows: ([0-9]+)', line) packets_match = re.search(r'Packets: ([0-9]+)', line) bytes_match = re.search(r'Bytes: ([0-9]+)', line) sequence_errors_match = re.search(r'Sequence Errors: ([0-9]+)', line) bad_packets_match = re.search(r'Bad Packets: ([0-9]+)', line) if flows_match and packets_match and bytes_match: flows = flows_match.group(1) packets = packets_match.group(1) bytes_data = bytes_match.group(1) sequence_errors = sequence_errors_match.group(1) if sequence_errors_match else "0" bad_packets = bad_packets_match.group(1) if bad_packets_match else "0" # Now you have device_id, flows, packets, bytes_data, sequence_errors, bad_packets # You can process or store this information as needed.

1. Monitoring System Integration:
   • If you're using a monitoring system (like Zabbix, Nagios, etc.), you can use the script or create a custom integration to feed these metrics into your monitoring system.

Remember to adapt the script based on your specific needs and the structure of your log file. This approach allows you to process the log entries, associate statistics with device IDs, and integrate them into your monitoring system for ongoing analysis.

[cyber]

I've been looking into discovery rules, but don't see how to implement them for this use case.

Discovery needs json. if you could somehow format your initial data as decent CSV, commas et al... then you can use preprocessing, csv to json, and then use that json also to extract all data for each device by giving correct jsonpath...

[PliusZabbixMan]

So it's not possible to just extract the data form the log file? I've been playing around with discovery rules, what if (and I'm not sure if it works like this) I set up a discovery rule do get device IDs:

Discovery rule

Name: Device Discovery
Type: Zabbix agent (active)
Key: log[/var/log/zabbix-agent/flow.log,,,,,,,,]

Filters

Macro: {#DEVICEID} matches ID:*([0-9]+)

Then I would just need a way for zabbix to assign the variables from each line by the #DEVICEID macro.

I apologize if this is a stupid idea, I'm still early on in the learning process and am not sure how a lot of things work.

[PeterZielony]

Discovery works with initial json format that will create "devices" and then you can feed data to it.

Without Discovery you could assign "hardcode" data to each item/host with log and filter by name (or id of device) but it doesnt scale very well and in general bad practice.


https://www.zabbix.com/documentation...stom-lld-rules

[PliusZabbixMan]

Update:

After consulting some other people, this is what I achieved:

1.Set up a discovery rule and used preprocessing JS script to make TXT format into JSON. Example: [{"{#DEVICE_ID}":"3085"},{"{#DEVICE_ID}":"3088"},{" {#DEVICE_ID}":"3089"},{"{#DEVICE_ID}":"3090"},{"{# DEVICE_ID}":"3091"}]
2.Set up Item prototypes for each variable that I want to monitor.

The thing I'm stuck on is how to tell zabbix to use {#DEVICE_ID} to discover the IDs. Should I use LLD macros section? If so what would be the JSONPath "$.path.to.node" that I should use?

[PeterZielony]

look at this thread - quite similar to what you want. I've done an example template with which you can download and see the structure of how it can be done:

https://www.zabbix.com/forum/zabbix-...ule#post475958
you have preprocessing that converts data into json - add values too in pre-processing to feed them through (and you don't need userparameters scripts) as a normal item and then based on that create a dependent item for discovery. Then item prototype will take json path for values for each item prototype (dependent item from the original JSON you are pooling originally) for {#DEVICE_ID}.


your processed json (this is formatted ver) could look like this:
[
{
"{#DEVICE_ID}": "3085",
"{#FLOWS}": "4083570",
"{#PACKETS}": "417889869",
"{#BYTES}": "542003885889 ",
"{#SEQ_ERRORS}": "0",
"{#BAD_PACKETS}": "0",
}, {
"{#DEVICE_ID}": "3086",
"{#FLOWS}": "4083570",
"{#PACKETS}": "417889869",
"{#BYTES}": "542003885889 ",
"{#SEQ_ERRORS}": "0",
"{#BAD_PACKETS}": "0",
}
]

[PliusZabbixMan]

I tried looking over the forum post you provided, but did not find a download link for the template you mentioned.

I only found stuff uploaded by the user ijonjic.

I made a Discovery rule with this preprocessing:

"
ids = value.match(/[0-9])+/g);

output = "[";

for (i = 0; i < ids.length; i++) {
if (i == ids.length - 1) {
output += "{"DEVICE_ID": "" + ids[i].split(":").pop() + ""}";
} else {
output += "{"DEVICE_ID": "" + ids[i].split(":").pop() + ""},";
}
}

output += "]";
return output;
"





Now I receive my outputs in JSON format

Example data:

Dec 29 10 ID:3085 Flows: 4083570 Packets: 417889869 Bytes: 542003885889 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3088 Flows: 8209950 Packets: 730187808 Bytes: 905311900208 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3089 Flows: 10523340 Packets: 755708556 Bytes: 871960643141 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3090 Flows: 7645350 Packets: 542571788 Bytes: 614900210651 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3091 Flows: 6350100 Packets: 475710746 Bytes: 523682115121 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3092 Flows: 4719180 Packets: 560727428 Bytes: 617551340411 Sequence Errors: 0 Bad Packets: 0
Dec 29 10 ID:3093 Flows: 8953380 Packets: 669000717 Bytes: 773656274868 Sequence Errors: 0 Bad Packets: 0

Example preprocessed:

[{"DEVICE_ID": "3085"},{"DEVICE_ID": "3088"},{"DEVICE_ID": "3089"},{"DEVICE_ID": "3090"},{"DEVICE_ID": "3091"},{"DEVICE_ID": "3092"},{"DEVICE_ID": "3093"}]

Then I used LLD macros to try and assign the values to a macro:

LLD macro________________JSONPath

{#DEVICE_ID}_____________$.DEVICE_ID






I also tried this code to assign variables instantly to the macro like this with preprocessing:

ids = value.match(/[0-9])+/g);
"
output = "[";

for (i = 0; i < ids.length; i++) {
if (i == ids.length - 1) {
output += "{"{#DEVICE_ID}":"" + ids[i].split(":").pop() + ""}";
} else {
output += "{"{#DEVICE_ID}":"" + ids[i].split(":").pop() + ""},";
}
}

output += "]";
return output;
"



Example preprocessed:

[{"{#DEVICE_ID}":"3085"},{"{#DEVICE_ID}":"3088"} ,{" {#DEVICE_ID}":"3089"},{"{#DEVICE_ID}":"3090"},{"{# DEVICE_ID}":"3091"}]





I'm having trouble with Item prototypes. No items are being created when using either preprocessing option. No errors thrown, but also nothing, only the log item in "Latest data" for the host...


Here is the Item prototype I'm trying:

Name: Flows {#DEVICE_ID}

Type: Dependent item

Key: log.flows

Master item: myhost: Log File

Type of information: Numeric (unsigned)



Preprocessing:
Name__________________________Parameters__________ _________Output

Regular expression_______________{#DEVICE_ID}\sFlows:\s(\d +)_____\1

[PliusZabbixMan]

Stuck at Item prototypes, any help would be appreciated. I feel so close, yet so far....

[PeterZielony]

Stuck at Item prototypes, any help would be appreciated. I feel so close, yet so far....

it is right there:


btw you need all info in this json to make it properly.

in template you should have:
1) item that read file - and with preprocessing produces json with all data including metrics - yours doesn't need it as you do read it and transform as json but you need to tweak it to produce all values with each read to make it work properly (there are other ways but will be too complex at this stage)
2) discovery - dependent on item in 1) with filtering etc. (look at example template)
3) item prototype based on macro with json path for values (again look at this example) - dependent item again from 1)
4) any additional triggers prototypes


note: when you use preprocessing this way as you did - you lose all metric data and you keep only device ID. You need to have (ideally) all metrics available - otherwise you would need to read them again

[PliusZabbixMan]

Ahh I missed that there is a 2cnd page. Sorry for the goof. Also I didn't restart my restart my agent. I'll start with that. Anyway working hours are done, so I won't have access to my server. I'll try it out tomorrow and post an update.

Thank you for your help!

[PeterZielony]

you don't have to restart the agent - this is only when you use a user parameter which you are not if you can take data without it (userparametr) and preprocessing.

but you might have to use user parameters anyway with the custom script - when creating custom LLD. It needs to go to zbx proxy as a full JSON - not just line by line.
Otherwise, it won't work

[PliusZabbixMan]

Is it possible the template is "private" or something of sorts? I only see the photo in the attachments.

[PeterZielony]

https://we.tl/t-EM1PbFi0uL

I think is because you are new to this forum

[PliusZabbixMan]

Update:

Here is what I've done based on the tips you provided in your last comment:

Converted the log item from TXT format to JSON, instead of converting it in discovery rule.







Then I made a discovery rule to discover ID's

[PliusZabbixMan]

Then I made Item prototypes for each variables. I'll only post "Flows" since the differences are minimal:








Full $.path.to.node:

$.devices[?(@.DEVICE_ID == {#DEVICE_ID})].DEVICE_FLOWS


The other prototypes are the same, just change FLOWS for PACKETS, etc.



Sadly no Items are being picked up / discovered:

[PliusZabbixMan]

Could it be related to the item key reading the log 1 line at a time?



The discovery rule throws this error as well:

Cannot find the "data" array in the received JSON object.

[cyber]

Remember, item key has to be unique. When you add prototype, please make that key also unique, not just name... "device.flows[{#DEVICE_ID}]"

[PliusZabbixMan]

Doesn't #DEVICE_ID make it unique?

If not how would I make it unique?

[PeterZielony]

also i might be wrong on this one as I never did it with a formatted JSON (not sure if it matters - always worked on non formatted ones)

Formatted means - easy to read but I don't know if zabbix can read it. Json might need to be in format without spaces and no new lines - something like this (just example) :

[{"{#DEVICE_ID}":"3085","{#FLOWS}":"4083570","{# PAC KETS}":"417889869","{#BYTES}":"542003885889 ","{#SEQ_ERRORS}":"0","{#BAD_PACKETS}":"0",},{ "{#D EVICE_ID}":"3086","{#FLOWS}":"4083570","{#PACKETS} ":"417889869","{#BYTES}":"542003885889 ","{#SEQ_ERRORS}":"0","{#BAD_PACKETS}":"0",}]

not like this:
[{
"{#DEVICE_ID}": "3085",
"{#FLOWS}": "4083570",
"{#PACKETS}": "417889869",
"{#BYTES}": "542003885889 ",
"{#SEQ_ERRORS}": "0",
"{#BAD_PACKETS}": "0",
}, {
"{#DEVICE_ID}": "3086",
"{#FLOWS}": "4083570",
"{#PACKETS}": "417889869",
"{#BYTES}": "542003885889 ",
"{#SEQ_ERRORS}": "0",
"{#BAD_PACKETS}": "0",
}
]

[PliusZabbixMan]

I used the example provided in the zabbix documentation, so it should be correct:

https://www.zabbix.com/documentation/current/en/manual/config/items/preprocessing/jsonpath_functionality