Zabbix can be used for centralized monitoring and analysis of log files with/without log rotation support.
Notifications can be used to warn users when a log file contains certain strings or string patterns.
To monitor a log file you must have:
Make sure that in the agent configuration file:
Configure a log monitoring item.
All mandatory input fields are marked with a red asterisk.
Specifically for log monitoring items you enter:
|Type||Select Zabbix agent (active) here.|
|Key||Use one of the following item keys:
log or logrt:
These two item keys allow to monitor logs and filter log entries by the content regexp, if present.
log.count or logrt.count:
These two item keys allow to return the number of matching lines only.
See supported Zabbix agent item key section for details on using these item keys and their parameters.
|Type of information||Select:
For log or logrt items -
For log.count or logrt.count items -
If optionally using the
Note that choosing a non-Log type of information will lead to the loss of local timestamp.
|Update interval (in sec)||The parameter defines how often Zabbix agent will check for any changes in the log file. Setting it to 1 second will make sure that you get new records as soon as possible.|
|Log time format||In this field you may optionally specify the pattern for parsing the log line timestamp.
If left blank the timestamp will not be parsed.
* y: Year (0001-9999)
* M: Month (01-12)
* d: Day (01-31)
* h: Hour (00-23)
* m: Minute (00-59)
* s: Second (00-59)
For example, consider the following line from the Zabbix agent log file:
“ 23480:20100328:154718.045 Zabbix agent started. Zabbix 1.8.2 (revision 11211).”
It begins with six character positions for PID, followed by date, time, and the rest of the line.
Log time format for this line would be “pppppp:yyyyMMdd:hhmmss”.
Note that “p” and “:” chars are just placeholders and can be anything but “yMdhms”.
logrtitem and Zabbix agent is following the most recent of them and this most recent log file is deleted, a warning message
“there are no files matching ”<regexp mask>“ in ”<directory>“is logged. Zabbix agent ignores log files with modification time less than the most recent modification time seen by the agent for the
logrtitem being checked.
logrtitem has Update interval of 1 second, by default the agent will analyze no more than 200 log file records and will send no more than 20 matching records to Zabbix server in one check. By increasing MaxLinesPerSecond in the agent configuration file or setting maxlines parameter in the item key, the limit can be increased up to 10000 analyzed log file records and 1000 matching records sent to Zabbix server in one check. If the Update interval is set to 2 seconds the limits for one check would be set 2 times higher than with Update interval of 1 second.
logrtare supported in filename only, directory regular expression matching is not supported.
logrtitem becomes NOTSUPPORTED if a directory where the log files are expected to be found does not exist.
logrtitem does not make it NOTSUPPORTED. Errors of reading log files for
logrtitem are logged as warnings into Zabbix agent log file but do not make the item NOTSUPPORTED.
logrtitem became NOTSUPPORTED. Zabbix can monitor its agent log file except when at DebugLevel=4.
Sometimes we may want to extract only the interesting value from a target file instead of returning the whole line when a regular expression match is found.
Since Zabbix 2.2.0, log items have the ability to extract desired values from matched lines. This is accomplished by the additional output parameter in
Using the 'output' parameter allows to indicate the subgroup of the match that we may be interested in.
So, for example
log[/path/to/the/file,"large result buffer allocation.*Entries: ([0-9]+)",,,,\1]
should allow returning the entry count as found in the content of:
Fr Feb 07 2014 11:07:36.6690 */ Thread Id 1400 (GLEWF) large result buffer allocation - /Length: 437136/Entries: 5948/Client Ver: >=10/RPC ID: 41726453/User: AUser/Form: CFG:ServiceLevelAgreement
The reason why Zabbix will return only the number is because 'output' here is defined by \1 referring to the first and only subgroup of interest: ([0-9]+)
And, with the ability to extract and return a number, the value can be used to define triggers.
The 'maxdelay' parameter in log items allows ignoring some older lines from log files in order to get the most recent lines analyzed within the 'maxdelay' seconds.
By default items for log monitoring follow all new lines appearing in the log files. However, there are applications which in some situations start writing an enormous number of messages in their log files. For example, if a database or a DNS server is unavailable, such applications flood log files with thousands of nearly identical error messages until normal operation is restored. By default, all those messages will be dutifully analyzed and matching lines sent to server as configured in
Built-in protection against overload consists of a configurable 'maxlines' parameter (protects server from too many incoming matching log lines) and a 4*'maxlines' limit (protects host CPU and I/O from overloading by agent in one check). Still, there are 2 problems with the built-in protection. First, a large number of potentially not-so-informative messages are reported to server and consume space in the database. Second, due to the limited number of lines analyzed per second the agent may lag behind the newest log records for hours. Quite likely, you might prefer to be sooner informed about the current situation in the log files instead of crawling through old records for hours.
The solution to both problems is using the 'maxdelay' parameter. If 'maxdelay' > 0 is specified, during each check the number of processed bytes, the number of remaining bytes and processing time is measured. From these numbers the agent calculates an estimated delay - how many seconds it would take to analyze all remaining records in a log file.
If the delay does not exceed 'maxdelay' then the agent proceeds with analyzing the log file as usual.
If the delay is greater than 'maxdelay' then the agent ignores a chunk of a log file by “jumping” over it to a new estimated position so that the remaining lines could be analyzed within 'maxdelay' seconds.
Note that agent does not even read ignored lines into buffer, but calculates an approximate position to jump to in a file.
The fact of skipping log file lines is logged in the agent log file like this:
14287:20160602:174344.206 item:"logrt["/home/zabbix32/test[0-9].log",ERROR,,1000,,,120.0]" logfile:"/home/zabbix32/test1.log" skipping 679858 bytes (from byte 75653115 to byte 76332973) to meet maxdelay
The “to byte” number is approximate because after the “jump” the agent adjusts the position in the file to the beginning of a log line which may be further in the file or earlier.
Depending on how the speed of growing compares with the speed of analyzing the log file you may see no “jumps”, rare or often “jumps”, large or small “jumps”, or even a small “jump” in every check. Fluctuations in the system load and network latency also affect the calculation of delay and hence, “jumping” ahead to keep up with the “maxdelay” parameter.
Setting 'maxdelay' < 'update interval' is not recommended (it may result in frequent small “jumps”).
logrt with the
copytruncate option assumes that different log files have different records (at least their timestamps are different), therefore MD5 sums of initial blocks (up to the first 512 bytes) will be different. Two files with the same MD5 sums of initial blocks means that one of them is the original, another - a copy.
logrt with the
copytruncate option makes effort to correctly process log file copies without reporting duplicates. However, things like producing multiple log file copies with the same timestamp, log file rotation more often than logrt item update interval, frequent restarting of agent are not recommended. The agent tries to handle all these situations reasonably well, but good results cannot be guaranteed in all circumstances.
Each matching line from
logrt item and a result of each
logrt.count item check requires a free slot in the designated 50% area in the agent send buffer. The buffer elements are regularly sent to server (or proxy) and the buffer slots are free again.
While there are free slots in the designated log area in the agent send buffer and communication fails between agent and server (or proxy) the log monitoring results are accumulated in the send buffer. This helps to mitigate short communication failures.
During longer communication failures all log slots get occupied and the following actions are taken:
logrtitem checks are stopped. When communication is restored and free slots in the buffer are available the checks are resumed from the previous position. No matching lines are lost, they are just reported later.
logrt.countchecks are stopped if
maxdelay = 0(default). Behavior is similar to
logrtitems as described above. Note that this can affect
logrt.countresults: for example, one check counts 100 matching lines in a log file, but as there are no free slots in the buffer the check is stopped. When communication is restored the agent counts the same 100 matching lines and also 70 new matching lines. The agent now sends count = 170 as if they were found in one check.
maxdelay > 0: if there was no “jump” during the check, then behavior is similar to described above. If a “jump” over log file lines took place then the position after “jump” is kept and the counted result is discarded. So, the agent tries to keep up with a growing log file even in case of communication failure.