Log based alerts triggering repeatedly

[pridetechdesign]

Hi I'm new to Zabbix and I think it's quite likely I'm doing something wrong, but I'm not certain. Perhaps someone here can help?

I have some string searches on my web access logs, to watch for attempted hacking activity. They work great, but every now and then Zabbix will trigger for old log entries. This creates a lot of spam and makes it difficult to follow the activity.

Here is the trigger expression I'm using:
{hostname:log[/etc/httpd/logs/access_log].str(htaccess)} or
{hostname:log[/etc/httpd/logs/access_log].str(c99.php)} or
{hostname:log[/etc/httpd/logs/access_log].str(r57.php)} or
{hostname:log[/etc/httpd/logs/access_log].str(filesman.php)} or
{hostname:log[/etc/httpd/logs/access_log].str(filemanager)} or
{hostname:log[/etc/httpd/logs/access_log].str(passthru)} or
{hostname:log[/etc/httpd/logs/access_log].str(shell_exec)} or
{hostname:log[/etc/httpd/logs/access_log].str(system)} or
{hostname:log[/etc/httpd/logs/access_log].str(phpinfo)} or
{hostname:log[/etc/httpd/logs/access_log].str(base64_decode)} or
{hostname:log[/etc/httpd/logs/access_log].str(edoced_46esab)} or
{hostname:log[/etc/httpd/logs/access_log].str(chmod)} or
{hostname:log[/etc/httpd/logs/access_log].str(mkdir)} or
{hostname:log[/etc/httpd/logs/access_log].str(fopen)} or
{hostname:log[/etc/httpd/logs/access_log].str(fclose)} or
{hostname:log[/etc/httpd/logs/access_log].str(w00tw00t)} or
{hostname:log[/etc/httpd/logs/access_log].str(setup.php)} or
{hostname:log[/etc/httpd/logs/access_log].str(readfile)} or
{hostname:log[/etc/httpd/logs/access_log].str(act=edit)} or
{hostname:log[/etc/httpd/logs/access_log].str(action=edit)}

Does Zabbix have a way to identify and ignore log entries which alerted previously? If so I think that must be broken.. if not, is there something I can do with my trigger expression to fix this? Help is appreciated.

[ingus.vilnis]

Hi,

Not sure why you sometimes have triggers for old log lines but here is what I can recommend you.

Add optional parameter "skip" to your item. When creating new log items it will not process old log data but only start reading from the moment when item was created.

log[/etc/httpd/logs/access_log,,,,skip,]

https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/zabbix_agent

And as the next thing I think you can optimize your trigger expression by using regexp function instead of str. Thus you can have all your matching words in one line.

{hostname:log[/etc/httpd/logs/access_log,,,,skip,].regexp(htaccess|c99.php|r57.php|...)}=1

https://www.zabbix.com/documentation/2.4/manual/appendix/triggers/functions

Hope this helps!

Best Regards,
Ingus

[pridetechdesign]

Thanks! I think that will help.

I will convert my expression to regexp at some point, but since it's currently functional it's not a top priority.

[pridetechdesign]

Update on my situation, I added the 'skip' mode to my log items, however zabbix is still reporting on old log entries. For example:

1. Apache Access Logs (server:log[/etc/httpd/logs/access_log,,,,skip,]): ###.###.###.### - - [08/Jan/2015:12:20:51 -0700] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 2026

That triggered today, Feb 18th, at 11:45am. It should have been skipped, shouldn't it?

[pridetechdesign]

Update,

I've added the skip argument and changed my trigger expression to use regexp, but on two servers I monitor they are still reporting on old log entries every time the agent restarts.

I believe this is resulting in an abnormally large database, which is becoming a problem.

Any guidance on fixing this issue is appreciated!

[pridetechdesign]

I am bumping this, I believe this is a bug in zabbix, since the 'skip' option is being ignored.

If anyone has a solution please share it.

[ingus.vilnis]

Hi,

If it is possible to force this scenario to reproduce, you can enable DebugLevel=4 in zabbix_agentd.conf file, restart the agent and see in the log what exactly the agent does. Maybe the httpd access log file gets rewritten every time from beginning instead of just adding new lines thus the agent thinks it is a completely new file.

Best Regards,
Ingus

[pridetechdesign]

Hi,

If it is possible to force this scenario to reproduce, you can enable DebugLevel=4 in zabbix_agentd.conf file, restart the agent and see in the log what exactly the agent does. Maybe the httpd access log file gets rewritten every time from beginning instead of just adding new lines thus the agent thinks it is a completely new file.

Best Regards,
Ingus

I finally got a moment to try this, and here is the log result. I have no idea how to read this so I am hoping someone can help translate:

 30077:20150501:094613.297 buffer: new element 1
 30077:20150501:094613.297 End of process_value():SUCCEED
 30077:20150501:094613.297 In process_value() key:'server.com:log[/etc/httpd/logs/error_log,,,,skip,]' value:'[Wed Apr 15 19:57:40 2015] [error] [client ##.##.##.##] LOOK general_comments_switch was submitted via objectid 429, referer: https://server.com/index.php'
 30077:20150501:094613.297 In send_buffer() host:'##.##.##.##' port:10051 values:2/100
 30077:20150501:094613.297 Will not send now. Now 1430495173 lastsent 1430495173 < 5
 30077:20150501:094613.297 End of send_buffer():SUCCEED
 30077:20150501:094613.297 buffer: new element 2
 30077:20150501:094613.297 End of process_value():SUCCEED
 30077:20150501:094613.297 In process_value() key:'server.com:log[/etc/httpd/logs/error_log,,,,skip,]' value:'[Wed Apr 15 19:57:40 2015] [error] [client ##.##.##.##] LOOK general_comments was submitted via objectid 429, referer: https://server.com/index.php'

This is only a segment of the log, but there are hundreds or thousands of entries just like that, and they are all new from restarting the agent, but the items reported are not new, as you can see from the dates.