Problem with log monitoring using logrt

[whitepaper.ink]

Hi Guys,

I am using Zabbix 3.2 version for our application and trying to configure the log monitoring with logrt.

I have tried to define items in several manner to monitor number of logs in a directory using regular expression but they are not working as I expected:

1) logrt["/apps/opt/weblogic/config/project/logs/[A-Z]*.log","ORA-","UTF-8",,skip] : Monitoring only one log continuously along with archived logs having datestamp after .log. And returning the same lines repeatedly in Latest data.

2) logrt["/apps/opt/weblogic/config/project/logs/^[A-Z]*.log$","ORA-","UTF-8",,skip] : Not working. No lines captured from the log though we have required string in log file.

3) logrt["/apps/opt/weblogic/config/project/logs/[A-Z]*.log$","ORA-","UTF-8",,skip] : Monitoring only one log but returning same lines repeatedly.

4) logrt["/apps/opt/weblogic/config/project/logs/Test.log$","ORA-","UTF-8",,skip] : I have hard coded the log name and this case it is working fine.

How can I setup log monitoring with regular expression to monitor all my files in directory. Please help !!!

[vso]

Hi Guys,

I am using Zabbix 3.2 version for our application and trying to configure the log monitoring with logrt.

I have tried to define items in several manner to monitor number of logs in a directory using regular expression but they are not working as I expected:

1) logrt["/apps/opt/weblogic/config/project/logs/[A-Z]*.log","ORA-","UTF-8",,skip] : Monitoring only one log continuously along with archived logs having datestamp after .log. And returning the same lines repeatedly in Latest data.

2) logrt["/apps/opt/weblogic/config/project/logs/^[A-Z]*.log$","ORA-","UTF-8",,skip] : Not working. No lines captured from the log though we have required string in log file.

3) logrt["/apps/opt/weblogic/config/project/logs/[A-Z]*.log$","ORA-","UTF-8",,skip] : Monitoring only one log but returning same lines repeatedly.

4) logrt["/apps/opt/weblogic/config/project/logs/Test.log$","ORA-","UTF-8",,skip] : I have hard coded the log name and this case it is working fine.

How can I setup log monitoring with regular expression to monitor all my files in directory. Please help !!!

Could you please try out 3.4 ? It should solve you issue

[whitepaper.ink]

Could you please try out 3.4 ? It should solve you issue

Is it standardized? I see it is still a Vanilla code. Why the regular expression for file name in logrt is not working ?

[vso]

Is it standardized? I see it is still a Vanilla code. Why the regular expression for file name in logrt is not working ?

Please be so kind and provide file names that you expect to be monitored
Note that [A-Z] are probably case sensitive

[whitepaper.ink]

Please be so kind and provide file names that you expect to be monitored
Note that [A-Z] are probably case sensitive

Hello vso -

Thanks for looking into this. File names are as below:

AFR.log
VTP_GATEWAY_SERVICE.log
SECURITY_DETAILS.log
GATEWAY_SERVICE_SAP.log
GATEWAY_SERVICE.log
INV_GATEWAY_SERVICE.log
INTRFACE_SAP_MOVEMENT.log
CUST_ORDER.log
INTERNAL.log
INTRFACE_STB_UPD.log
INTRFACE_ACTIVITY.log

These matches ^[A-Z]*.log$ regular expression but only monitoring one log file with repeated log errors.

[Atsushi]

Use logrt[] in an environment where one log file is rotated.

example1 : logrt[/var/log/syslog]
Files to be checked

/var/log/syslog
/var/log/syslog.1

example2 : logrt[/var/log/messages]
Files to be checked

/var/log/messages
/var/log/messages-20170821

It is not intended to monitor multiple different log files matching in regular expressions.

[whitepaper.ink]

Use logrt[] in an environment where one log file is rotated.

example1 : logrt[/var/log/syslog]
Files to be checked

/var/log/syslog
/var/log/syslog.1

example2 : logrt[/var/log/messages]
Files to be checked

/var/log/messages
/var/log/messages-20170821

It is not intended to monitor multiple different log files matching in regular expressions.

Hi Atsushi -

Thanks for quick reply. So Zabbix do not support multiple log files monitoring with single item key ? Do you have any idea if this is possible in 3.4 ?

[Atsushi]

It is not possible to monitor log files of different types and contents simultaneously with one item.
It will not be possible even in 3.4.
It is not listed in the new function list.

https://www.zabbix.com/documentation/3.4/manual/introduction/whatsnew340

In order to prevent rereading of the file, Zabbix manages how far it reads in the file, but that information can only have one data with one item.
Zabbix agent lists candidates to read updated files at the file update date, but if it becomes a different file from what was previously loaded, in the worst case, it reads from the beginning of the log file.

[whitepaper.ink]

It is not possible to monitor log files of different types and contents simultaneously with one item.
It will not be possible even in 3.4.
It is not listed in the new function list.

https://www.zabbix.com/documentation/3.4/manual/introduction/whatsnew340

In order to prevent rereading of the file, Zabbix manages how far it reads in the file, but that information can only have one data with one item.
Zabbix agent lists candidates to read updated files at the file update date, but if it becomes a different file from what was previously loaded, in the worst case, it reads from the beginning of the log file.

I see the below notes in Zabbix documentation to monitor multiple log files:

#################
The item must be configured as an active check.
Log rotation is based on the last modification time of files.

If output is left empty - the whole line containing the matched text is returned. Note that all global regular expression types except 'Result is TRUE' always return the whole matched line and the output parameter is ignored.

Content extraction using the output parameter takes place on the agent.

Examples:
⇒ logrt["/home/zabbix/logs/^logfile[0-9]{1,3}$",,,100] → will match a file like "logfile1" (will not match ".logfile1")
⇒ logrt["/home/user/^logfile_.*_[0-9]{1,3}$","pattern_to_match","UTF-8",100] → will collect data from files such "logfile_abc_1" or "logfile__001".

The mode parameter is supported since Zabbix 2.0.
The output parameter is supported since Zabbix 2.2.
The maxdelay parameter is supported since Zabbix 3.2.

See also additional information on log monitoring.
##################

Is this something not to monitor multiple log files? Also, There is one keyword skip to escape previous data in file and start monitoring from the current line.

[Atsushi]

This is a case when the log file of the same role and content is rotated.
Only one file can output logs.

The system log file I mentioned as an example is periodically changed to a different file name so that the size of the log file does not become too large and a new empty log file is generated.
This process is called rotation.

Normally, only log files are output as follows.

/home/zabbix/logs/logfile

This file is rotated at regular intervals.

new log file : /home/zabbix/logs/logfile
old log file : /home/zabbix/logs/logfile.1

Depending on the setting, it may be rotated by size restriction.

Zabbix checks the timestamp of the file before it is rotated and checks the timestamp of the file and if the timestamp of the log file older than the checked time is new, it continues from the place where the old log file was recorded.

Strictly there is more detailed logic.

[whitepaper.ink]

This is a case when the log file of the same role and content is rotated.
Only one file can output logs.

The system log file I mentioned as an example is periodically changed to a different file name so that the size of the log file does not become too large and a new empty log file is generated.
This process is called rotation.

Normally, only log files are output as follows.

/home/zabbix/logs/logfile

This file is rotated at regular intervals.

new log file : /home/zabbix/logs/logfile
old log file : /home/zabbix/logs/logfile.1

Depending on the setting, it may be rotated by size restriction.

Zabbix checks the timestamp of the file before it is rotated and checks the timestamp of the file and if the timestamp of the log file older than the checked time is new, it continues from the place where the old log file was recorded.

Strictly there is more detailed logic.

Hi Atsushi,

Thanks for more informative here. I need to accept that I misunderstood logrt function in Zabbix.

But do you think it is good idea to monitor multiple log files in a directory with regular expression to reduce the items and have an item key to display the respective file name when problem triggered. I would like to post it in Ideas section if it can be implemented.

[andris]

The model for which logrt[] is designed is a single log file (e.g. myapp.log) where new records are appended. Sometimes the log file may be rotated, i e. renamed to arbitrary name which still matches the file name regexp (for example, myapp.log.1, myapp.log.old, myapp-20170822-2359.log).
Among all log files matching the file name regexp the agent takes the best effort to recognize which log files have been already fully analyzed, which ones are partially analyzed, which ones are new log files.
The agent tracks file sizes, timestamps, current positions, inode numbers, checksum of a small initial block - to ensure recognizing log file renaming/rotation and truncation.
Now if you use a file name regexp which matches mutiple independent log files written by different applications or instances then logrt[] will get totally confused. It will fluctuate between log files, assuming they are appended and rotated extremely often. This is quite a different model, logrt[] is not suitable for.
So, you need a separate logrt[] item for each independent log file.

[Atsushi]

I agree with the desire to simplify the log monitoring setup.
However, if the output log file is divided by function, items and triggers should also be separated.

For example, even if an error of the same ORA code occurs, there may be cases where the actions are different.

Specifically, if the same error code is another function as below, the recovery procedure of the failure will be different.
- Login processing
- Purchasing processing
- Batch processing

I think that it is good to consider a design that can identify what service affected the occurred failure and can decide what to do for the failure that has occurred.

[whitepaper.ink]

Thank you Atsushi and andris


I understand that logrt[] is only for one log file monitoring with log rotation capability. I will create item and triggers for each file monitoring.

[kasik]

Auto Discovery Item

Hey, so i came to same problem and I'm wondering if there is any chance of using auto discover? because i have like 20 different logs in one folder and make it manualy would be ***** so can you guys help ?