Two questions:
#1) Is it better to read all log lines into zabbix and filter by trigger, or have multiple log items with regex filters?
Right now I am using an item of:
log[/var/log/messages,,,,skip,]
followed by one or more triggers of the form:
{server:log[/var/log/messages,,,,skip,].iregexp(pattern,4000)}=1
I know that can result in a lot of unnecessary log line getting pulled into zabbix, however I limit the history to 2 days. On a smaller scale the database should not be an issue. Is it better for me to do the above, or to have a separate item per regex such as:
log[/var/log/messages,error,,,skip,]
log[/var/log/messages,fail,,,skip,]
The latter will pull in much less data but I think is more difficult to troubleshoot. Also, I'm not sure how zabbix active checks on log files process multiple redundant requests with differing regex. In the simple case I could try a regex like (error|fail), but as complexity increases so does likelihood of item error lead to missing important events to trigger. I need to keep the item and trigger logic simple for multiple people to support.
Just wondering what others here have successfully used for log monitoring
#2) Should I assume that if I have a short retention on historical data that my use of trigger functions with time scale is limited to how much I retain? For example, if the item limits retention to 2 days (as above), will the trigger function only within that time scale ... so the iregexp(pattern,4000) will work fine, but something like iregexp(pattern,216000) (for 2.5 days) would not work as expected.
Thanks for any suggestions you may have. I am interested to hear from people an see examples of how you have log[] and logrt[] items with triggers working in your production environments.
#1) Is it better to read all log lines into zabbix and filter by trigger, or have multiple log items with regex filters?
Right now I am using an item of:
log[/var/log/messages,,,,skip,]
followed by one or more triggers of the form:
{server:log[/var/log/messages,,,,skip,].iregexp(pattern,4000)}=1
I know that can result in a lot of unnecessary log line getting pulled into zabbix, however I limit the history to 2 days. On a smaller scale the database should not be an issue. Is it better for me to do the above, or to have a separate item per regex such as:
log[/var/log/messages,error,,,skip,]
log[/var/log/messages,fail,,,skip,]
The latter will pull in much less data but I think is more difficult to troubleshoot. Also, I'm not sure how zabbix active checks on log files process multiple redundant requests with differing regex. In the simple case I could try a regex like (error|fail), but as complexity increases so does likelihood of item error lead to missing important events to trigger. I need to keep the item and trigger logic simple for multiple people to support.
Just wondering what others here have successfully used for log monitoring
#2) Should I assume that if I have a short retention on historical data that my use of trigger functions with time scale is limited to how much I retain? For example, if the item limits retention to 2 days (as above), will the trigger function only within that time scale ... so the iregexp(pattern,4000) will work fine, but something like iregexp(pattern,216000) (for 2.5 days) would not work as expected.
Thanks for any suggestions you may have. I am interested to hear from people an see examples of how you have log[] and logrt[] items with triggers working in your production environments.