One log file, multiple regex - best practice?

[djp210]

What's the best way to monitor a single log file (e.g. /var/log/messages, but plenty of other non-default logs fit the same concept) for multiple patterns?

For example, if I need to monitor /var/log/messages for 20 different patterns, I could have a single Item with "log[/var/log/messages,"(pattern1|pattern2|...)]", but only if the entire key is 256 characters or less. Of course, the chances of fitting 20 regex's into that small a space are pretty much nil.

I could also create 20 different Items with "log[/var/log/messages,"(pattern1)"]" (and so on).

I could also create one Item with "log[/var/log/messages]", but that would cause Zabbix to store every line of /var/log/messages in the database. With a large number of systems, that seems like a bad idea.

If I create 20 different Items looking in the same file, does that cause the Zabbix agent to make 20 different connections to the file?

Is there any difference in these answers if the log file is a Windows event log?

We're using Zabbix 3.0 with Linux agents at 3.0.4 and Windows agents at 3.0.0.

Thanks.

Dan

[waardd]

We are also monitoring logfiles for multiple expressions.
We use zabbix_sender to send the found data to the zabbix server.

In this way you can count as many regexs as you want over an x period and send them as one file to zabbix.

Stress on zabbix is less this way.

What we do is take the last x minutes of a log file and with a script grep for multiple strings and count them.
These are put into a file and send to zabbix.

An example of such a file is:

lsrv1374 KCALC.ERROR 1471863601 0
lsrv1374 KCALC.KADIKB 1471863601 0
lsrv1374 KCALC.SigningServiceO 1471863601 0
lsrv1374 KCALC.SigningServiceP 1471863601 0
lsrv1374 KCALC.SigningServiceA 1471863601 0
lsrv1374 KCALC.PollingTriggerManager 1471863601 0
lsrv1374 KCALC.TriggerPoller 1471863601 0
lsrv1374 KCALC.BEB_Err_Snd 1471863601 0
lsrv1374 KIDUPD.ERROR 1471863601 2
lsrv1374 KIDUPD.QueryEdoIdToSiebelId 1471863601 1
lsrv1374 KIDUPD.XML 1471863601 1
lsrv1374 KIDUPD.NotMapped 1471863601 0
lsrv1374 KIDUPD.InvalidTA 1471863601 0

where the setup of the file is:
HOST ITEM TIMESTAMP(UNIX) COUNT

[andris]

If I create 20 different Items looking in the same file, does that cause the Zabbix agent to make 20 different connections to the file?

Each of 20 items will be checking log file independently. Each check tries to be efficient (regexp is compiled once, file is opened once and read in chunks, not line-by-line).

[djp210]

Each of 20 items will be checking log file independently. Each check tries to be efficient (regexp is compiled once, file is opened once and read in chunks, not line-by-line).

OK, so each item that references the same log file (e.g. log[/var/log/messages,"(pattern1)"], log[/var/log/messages,"(pattern2)"]) will cause the agent to make a connection to the file? I was trying to determine that using lsof, but so far couldn't prove it one way or another.

Any idea if that's also true with Windows event log monitoring? This would seem like a very common scenario -- monitoring, say, the Application event log for multiple specific Event IDs, strings, and so on -- so knowing the most efficient way to accomplish this would be most welcome.

Obviously we could take the easy way out and have the item simply grab the entire log file and have triggers that match the specific items you want to find, but that means every line would get stored in the database. With thousands of hosts, that just seems very inefficient.

[andris]

OK, so each item that references the same log file (e.g. log[/var/log/messages,"(pattern1)"], log[/var/log/messages,"(pattern2)"]) will cause the agent to make a connection to the file? I was trying to determine that using lsof, but so far couldn't prove it one way or another.

Any idea if that's also true with Windows event log monitoring? This would seem like a very common scenario -- monitoring, say, the Application event log for multiple specific Event IDs, strings, and so on -- so knowing the most efficient way to accomplish this would be most welcome.

Obviously we could take the easy way out and have the item simply grab the entire log file and have triggers that match the specific items you want to find, but that means every line would get stored in the database. With thousands of hosts, that just seems very inefficient.

Unlikely you will see it with 'lsof' - log[]/logrt[] on UNIX platforms tries to be very efficient, file is not kept open by 20 log[] item checks.
Each check compares size of logfile, if it has not changed, the file is not opened at all. If file has grown, then yes - each of 20 log[] item checks will open the file, go straight to 'lastlogsize' position and analyze only the new lines in the log file.
You can try to test is it noticeable to have 20 log[] checks on 1 file.
Also there are default and configurable limits on maximum number of analyzed lines in one check and max lines to be sent to server to prevent overload of agent and server CPU and I/O if you have many log[]/logrt[] checks.

Not sure about Windows eventlog - that is a different code path.

[djp210]

Each check compares size of logfile, if it has not changed, the file is not opened at all. If file has grown, then yes - each of 20 log[] item checks will open the file, go straight to 'lastlogsize' position and analyze only the new lines in the log file.

OK. Since the 20 items would all be looking at the same file, any change in the size of the file would cause them to all process. (FYI, the "20 items" is an example. I think I may have as many as 39 different things to look for in /var/log/messages.)

Anyone know if you can use a macro in an Item definition? Such as "log[/var/log/messages,{$MYMACRO}]" ? Since an Item can only be 255 characters log, maybe I can work around that by creating a macro with all the regex's I want to look for? (Update ... Looks like macros have the same 255-character limit, so that doesn't help.)

[djp210]

On a related note, I'm finding that Zabbix can't seem to parse the time stamp of a log file (/var/log/messages, in this case) that has the month in a non-numeric format.

For example:

Aug 22 12:12:58 server1234 xinetd[2583]: START: blah pid=3813 from=10.10.10.22

The Zabbix help lists "M" as "M: Month (01-12)". "MMM" doesn't seem to work.

I see at least one enhancement request (ZBXNEXT-2288) that revolves around this, and it mentions ZBXNEXT-487, which asks for "textual month support".

This particular date format is very commonly used in Unix log files. Anyone know when this will be possible in Zabbix? Or if it's there somewhere in 3.0.0 (or above) and just not documented?

[glebs.ivanovskis]

Anyone know if you can use a macro in an Item definition? Such as "log[/var/log/messages,{$MYMACRO}]" ? Since an Item can only be 255 characters log, maybe I can work around that by creating a macro with all the regex's I want to look for? (Update ... Looks like macros have the same 255-character limit, so that doesn't help.)

You can, with macros you can "bloat" item key almost up to 16Kchars (255 characters per key / 4 characters per macro name * 255 characters per macro). But especially for log items and regexps there is a better option - global regular expressions.

On a related note, I'm finding that Zabbix can't seem to parse the time stamp of a log file (/var/log/messages, in this case) that has the month in a non-numeric format.

You are right, support for date formats is very scarce.

[djp210]

You can, with macros you can "bloat" item key almost up to 16Kchars (255 characters per key / 4 characters per macro name * 255 characters per macro). But especially for log items and regexps there is a better option - global regular expressions.

Seems like the web front end still has a 255-character limit for a global regex ...

[djp210]

Looks like I may have been able to stitch together my massive regex by way of a global regular expression. Not terribly obvious -- mainly because I took as much as I could from my giant regex (the 30-something conditions all in one giant regex) in each line -- but I'm testing how it works now.

Maybe it would make more sense to take each of the individual regex's for the 30-something conditions and make them their own entry in the global regular expression. That would make management of them easier later on, but still allow the Item to have a single reference within it.

Thanks for the hint ...

[djp210]

One challenge I found with global regular expressions is that each one appears to be handled as an "AND". Meaning pattern1 AND pattern2 AND pattern3 ... If the string you're looking for is only in pattern1, it won't match, because it's not in all the patterns.

I've now set up the 30-plus conditions I'm looking for in /var/log/messages in 30-plus Items. Hopefully there's no performance issues with that.

[andris]

I've now set up the 30-plus conditions I'm looking for in /var/log/messages in 30-plus Items. Hopefully there's no performance issues with that.

It would be interesting to hear about performance of 30 items. What update interval for them ?

[djp210]

Right now I'm using the default interval of 30 seconds. So far I can't see a difference in performance, but I'll have to keep an eye on it.

[vigneshn]

Right now I'm using the default interval of 30 seconds. So far I can't see a difference in performance, but I'll have to keep an eye on it.

Is the log monitoring still active in the same manner? If yes, how is the performance of Zabbix agent and the Zabbix server? If no, what did you change to improve the performance of the Zabbix components?

Thanks.

[bhakimi]

i have a performance issue, i needed to monitor 10 instances of a application and 30 different alerts which i couldnt consolidate into one item and once zabbix ran the checks the CPU sky rocketed.. so yes there is a big performance hit