Zabbix log file monitoring

[newton_chris]

I think I've read through all relevant forum posts on Zabbix log file monitoring, and it seems to be lacking some things that are very important for me. It is possible to monitor a log file and notify when a pattern is matched. This is helpful for some situations, but I need it to both filter on match pattern(s) and then ignore anything that matches ignore pattern(s).
One example where this is useful is an Oracle alert.log file. You want to know whenever "ORA-" is matched. However, if your instance is in archive log mode you need to ignore both "ORA-308" and "ORA-279"(maybe more) or you are going to get notified whenever there is a log switch.
Similarly, you might want to monitor when "ERROR" or "SEVERE" is matched in a jboss log, but you want to ignore several known errors and only notify on "unknown" new errors.

I read about the syslog_ng a bit and the other log monitoring tools mentioned in the forums, but what I want to do is soooo simple.

I wrote what started as a simple Perl script to do this. Here is the 5 minute pilot.

#!/usr/bin/perl
# this is just the pilot...see below for more advanced one
open (FGREPED_TAIL, "tail -1f /var/log/jboss/server.log | fgrep --line-buffered -f /zabbix/log_watcher/fgrep_MATCH_THESE | fgrep --line-buffered -v -f /zabbix/log_watcher/fgrep_IGNORE_THESE|");
while (<FGREPED_TAIL>)
{
open (JBOSS_ERRORS_TO_LOOK_INTO, ">>/zabbix/log_watcher/log_watcher.log");
print JBOSS_ERRORS_TO_LOOK_INTO $_;
close JBOSS_ERRORS_TO_LOOK_INTO;
}
close FGREPED_TAIL;


Zabbix always watches the file:
/zabbix/log_watcher/log_watcher.log and notifies whenever there is a change to it.

The UserParameter line in zabbix_agentd.conf:
UserParameter=log_watcher,/usr/bin/tail -15 /zabbix/log_watcher/log_watcher.log | /usr/bin/perl -ne 'chop;print reverse <>'|/usr/bin/perl -ne 'chop;print "$_ ____"'
This just gives the last 15 lines of log_watcher.log reversed(newest first) with "_______" in place of each newline. This is necessary because the zabbix log file monitor separates lines by newlines. It is very handy to get the last 15 lines sent to your pager instead of the last 1.

I'm not sure if the UNIX "tail -f" is busy-waiting, but it doesn't seem to slow anything down even on machines with very active log files.

Make sense? It seems to work perfectly for my purpose.

Below is the real script and associated helper files/scripts that took a couple of weeks to get right:

This is the real Perl based script from a couple weeks after the above pilot:
zab_log_watcher.pl

http://docs.google.com/View?id=dg74qh8b_183hsrpsgd9

This is a watchdog that runs in cron every hour to make sure the above instances are running on a box:
zab_log_watcher_watchdog.pl

http://docs.google.com/View?id=dg74qh8b_184c7z597gw

This is the config file that gives you one central location to identify which log files on which machines are monitored:
master_zab_log_watcher_config.txt

http://docs.google.com/View?id=dg74qh8b_185hrm3cqdt

This generates the ignore and match files(just the empty file) from the master config file:
/zabbix/log_watcher/create_match_and_ignore_files_from_master_zabbix_l og_watcher_config.pl

http://docs.google.com/View?id=dg74qh8b_187c385hjgr

and if you happen to be using Puppet:
/zabbix/log_watcher/create_puppet_pp_file_from_master_zabbix_log_watch er_config.pl

http://docs.google.com/View?id=dg74qh8b_188hhp98gf6

Please let me know if there is an obvious better way of doing this directly with Zabbix that I missed. And please don't attack my code too much...my therapist is so expensive these days and I think this is my first posting.

Thanks Alexei for a great app. Hopefully see you or some of your team in Brazil.