Check occurences of a regular expression in a file

[frater]

I couldn't find a way to do this with standard commands, so I wrote a little shell script to accomplish this.
I wanted to check my /var/log/auth.log to see if someone did a failed login.

I'm using the perl script 'logtail'
Only recently I found out it existed. Before I used my own script for that.
logtail keeps an offset-file and will only output the tail of a file it didn't show previously.

I did need to add it to the /etc/sudoers file

zabbix ALL=(ALL) NOPASSWD: /usr/sbin/logtail

This is the line in /etc/zabbix/zabbix_agentd.conf:

UserParameter=vfs.file.regcount[*],/usr/local/sbin/regcount "$1" "$2"

/usr/local/sbin/regcount

#!/bin/bash
#####################################################
# regcount
# returns the occurences of a regular expression in
# a file since its last run
#####################################################
# Uses logtail & readlink
# http://sourceforge.net/projects/logtail/
#####################################################
# 08-12-2010 by Frater
#
# Use at your own risk!
#####################################################

export PATH=/usr/local/sbin:$PATH

[ ! -h "$1" ] && [ ! -f "$1" ] && exit 1
[ -z "$2" ]   && exit 1

fname="`readlink -f "$1"`"
expression="`echo "$2" | tr -cd '0-9A-Za-z'`"
offset="/tmp/`basename "${fname}"`.${expression}.offset"
sudo logtail -f "$1" -o $offset | grep -ciE "$2"
exit $?

I created this item

vfs.file.regcount[/var/log/auth.log,"Failed password for"]
interval: 1200
numeric decimal

And this trigger:

/var/log/auth.log shows failed SSH logins on server {HOSTNAME}
{Template_Linux:vfs.file.regcount[/var/log/auth.log,"Failed password for"].last(0)}>100

[frater]

I have modified regcount, because I want to parse the whole file in some cases.
That's why I introduced a 3rd optional parameter.

I also thought it would be better to put the script itself inside the /etc/sudoers file. Inside the script are the line you have to add to /etc/sudoers and to /etc/zabbix/zabbix_agentd.conf

#!/bin/bash
#####################################################
# regcount
# returns the occurences of a regular expression in
# a file since its last run
#####################################################
# Uses logtail & readlink
# http://sourceforge.net/projects/logtail/
#####################################################
# echo 'zabbix ALL =(ALL) NOPASSWD: /usr/local/sbin/regcount' >>/etc/sudoers
#
# echo 'UserParameter=vfs.file.regcount[*],sudo /usr/local/sbin/regcount "$1" "$2" "$3" "$4" "$5"' >>/etc/zabbix/zabbix_agentd.conf
#####################################################
# 08-12-2010 by Frater
#
# The 3rd parameter (minutes) is optional.
# When 0 or empty, it will use 'logtail' which only checks the portion which hasn't been parsed before
# When minutes is 1, it will take the whole file
# When it's greater than 1, it will use 'lastmins' that will only output the last x minutes
#####################################################
# Use at your own risk!
#####################################################
export PATH=${PATH}:/usr/local/sbin:/sbin:/usr/sbin:/bin:/usr/bin
offset=/tmp/regcount.

v=
while getopts v name
do
  case $name in
    v)   v='-v ';;
    ?)   printf "Usage: %s: [-v] <file> <Eregexp>, [ <boolean whole file> ]\n" $0
    exit 2;;
  esac
done
shift $(($OPTIND - 1))

[ ! -h "$1" ] && [ ! -f "$1" ] && exit 1
# [ -z "$2" ]   && exit 1

minutes=`echo "$3" | awk -F. '{print $1}' | tr -cd '0-9'`
[ -z "${minutes}" ] && minutes=0

if [ -z "$2" ] && [ ${minutes} -eq 1 ] ; then
  wc -l "$1" | awk '{print $1}'
  exit 0
fi

file2parse="$1"
if [ ${minutes} -eq 0 ] ; then
  fname="`readlink -f "$1"`"
  fname="`basename "${fname}"`"
  expression="`echo "$fname.${v}$2$3" | tr -cd '.0-9A-Za-z-'`"
  offset="${offset}${expression}.offset"
  ftmp1=`mktemp`
  logtail -f "$1" -o $offset >${ftmp1}
  file2parse=${ftmp1}
elif [ ${minutes} -gt 1 ] ; then
  ftmp1=`mktemp`
  cat "$1" | lastmins ${minutes} >${ftmp1}
  file2parse=${ftmp1}
fi

if [ -z "$2" ] ; then
  wc -l "$file2parse" | awk '{print $1}'
else
  grep $v -cE "$2" "$file2parse"
fi
rm -f ${ftmp1} 2>/dev/null
exit 0

[frater]

I made yet another function because all my input was received with great enthousiasm...

It's 'regtop' which parses a (log)file for a certain expression and will make a toplist....

#!/bin/bash
#####################################################
# regtop
#####################################################
# Uses logtail & readlink
# http://sourceforge.net/projects/logtail/
#####################################################
# echo 'zabbix ALL =(ALL) NOPASSWD: /usr/local/sbin/regtop' >>/etc/sudoers
#
# echo 'UserParameter=vfs.file.regtop[*],  sudo /usr/local/sbin/regtop   "$1" "$2" "$3" "$4" "$5" "$6"' >>/etc/zabbix/zabbix_agentd.conf
#####################################################
# 08-12-2010 by Frater
#
# The 4th parameter (minutes) is optional.
# When 0 or empty, it will use 'logtail' which only checks the portion which hasn't been parsed before
# When minutes is 1, it will take the whole file
# When it's greater than 1, it will use 'lastmins' that will only output the last x minutes
#####################################################
export PATH=${PATH}:/usr/local/sbin:/sbin:/usr/sbin:/bin:/usr/bin
offset=/tmp/regtop.

v=
while getopts v name
do
  case $name in
    v)   v='-v ';;
    ?)   printf "Usage: %s: [-v] <file> <Eregexp>,  [<Eregexp>] , [ <minutes> ]\n" $0
    exit 2;;
  esac
done
shift $(($OPTIND - 1))

[ ! -h "$1" ] && [ ! -f "$1" ] && exit 1
[ -z "$2" ]   && exit 1
file2parse="$1"

ftmp1=`mktemp`
ftmp2=`mktemp`

minutes=`echo "$4" | awk -F. '{print $1}' | tr -cd '0-9'`
[ -z "${minutes}" ] && minutes=0

if [ ${minutes} -eq 0 ] ; then
  fname="`readlink -f "$1"`"
  fname="`basename "${fname}"`"
  expression="`echo "$fname.${v}$2$3" | tr -cd '.0-9A-Za-z-'`"
  offset="${offset}${expression}.offset"
  logtail -f "$1" -o $offset >${ftmp2}
  file2parse=${ftmp2}
elif [ ${minutes} -gt 1 ] ; then
  cat "$1" | lastmins ${minutes} >${ftmp2}
  file2parse=${ftmp2}
fi

if [ ! -z "$3" ] ; then
  ftmp3=`mktemp`
  grep $v -E "$3" ${file2parse} >${ftmp3}
  file2parse=${ftmp3}
fi

grep -oE "${2}" ${file2parse} | sort -o ${ftmp1}

total=`wc -l ${ftmp1} | awk '{print $1}'`
if [ $total -gt 0 ] ; then

  rows=`echo "${5}" | tr -cd '0-9'`
  [ -z "${rows}" ] && rows=5
  expression="${2}"

  [ -z "$3" ] || expression="${expression} && $3"
  uniq -c ${ftmp1} | sort -rn | head -n${rows}
  echo "Total: $total lines with \"${expression}\""
else
  echo '-'
fi

rm -f ${ftmp1} 2>/dev/null
rm -f ${ftmp2} 2>/dev/null
rm -f ${ftmp3} 2>/dev/null

This is what I'm getting as output for this expression:

vfs.file.regtop[/var/log/messages,"SRC=[0-9.]*",INVALID]

[frater]

The latest regtop needs 'lastmins'

#!/bin/sh
#
FORMAT=""
while getopts f: name
do
  case $name in
    f)   FORMAT="$OPTARG";;
    ?)   printf "Usage: %s: [-f FORMAT]\n" $0
         exit 2
         ;;
  esac
done
shift $(($OPTIND - 1))

EXECDIR="`dirname $0`"
MINS=
[ -z "$1" ]  || MINS=`echo $1 | tr -cd 0-9`
[ -z "$MINS" ] && MINS=10
SCRATCH=`mktemp`

NOW=`date +%s`
TIME=$(( ${NOW} - $((${MINS} * 60)) ))

while [ ${TIME} -lt ${NOW} ] ; do
  date -d "1970-01-01 00:00 UTC ${TIME} sec" "+:%Y%m%d:%H%M"    >>${SCRATCH}
  date -d "1970-01-01 00:00 UTC ${TIME} sec" "+^%b %e %H:%M"    >>${SCRATCH}
  date -d "1970-01-01 00:00 UTC ${TIME} sec" "+^%b %d %H:%M"    >>${SCRATCH}
  date -d "1970-01-01 00:00 UTC ${TIME} sec" "+^%Y-%m-%d %H:%M" >>${SCRATCH}
  date -d "1970-01-01 00:00 UTC ${TIME} sec" "+^%b-%d-%y %H:%M" >>${SCRATCH}

  [ -z "${FORMAT}" ] || date -d "1970-01-01 00:00 UTC ${TIME} sec" "+${FORMAT}" >>${SCRATCH}
  let TIME+=60
done

cat - | grep -f ${SCRATCH}
rm -f ${SCRATCH}

[frater]

This UDF is more suited in real-life situations than regmatch
Could this be implemented (maybe partial) natively???