Log file monitoring not working

[heybails]

I cannot get /var/log/messages monitored in Zabbix. I've set up an item as follows
NAME: System Event log
Type: Zabbix agent (active)
Key: log[/var/log/messages,"err|crit|alert|emerg"]
Type of information: Log
Update interval (in sec): 30
History storage period (in days): 7
Applications: Log
Enabled: <checked>

Trigger:
Name: /var/log/messages has errors {ITEM.VALUE1}
Expression: {crt-vsam-01.itf.centurylink.net:log[/var/log/messages,"err|crit|alert|emerg"].str(err|crit|alert|emerg,30)}=0
Severity: High
Enabled: <checked>

I've set /var/log/messages permissions to 644 to get the zabbix user to be able to read the log. I run a logger test to a node I'm building this on specifically (not via templates right now). Not getting any messages. Any ideas? Is this a bug?

[andris]

Hi!

You can start Zabbix agentd with "DebugLevel=4" in zabbix_agentd.conf. Check in zabbix_agentd.log whether agent is getting a list of active checks from server, is process_log or process_logrt function invoked from time to time.

Andris

[heybails]

Thanks, I've done that already. It didn't reveal anything. I also set up a test log with wide open permissions and got nothing. I'm giving up on this as I've seen other users have the same issue. This is a basic thing that Zabbix should be able to perform and it does not. Why? There should already be an OOB item with triggers built for monitoring syslog and messages logs for servers. We shouldn't have to create it.

[kloczek]

Thanks, I've done that already. It didn't reveal anythin

What you mean "reveal nothing"?
Do you see in debug output that agent is opening log file?
What exactly you've done?

[heybails]

-I've done everything including dbug and there is nothing revealed as to a possible problem. Would it be possible for anyone to show me how they set up a Zabbix event log monitor for 2.4 and then I could trace that against what I've done to see what I might be missing? Greatly appreciated.

[kloczek]

-I've done everything including dbug and there is nothing revealed as to a possible problem. Would it be possible for anyone to show me how they set up a Zabbix event log monitor for 2.4 and then I could trace that against what I've done to see what I might be missing? Greatly appreciated.

Look .. number of possible variants of diagnostics is probably infinite. You not been able to do infinite number of diagnostics in finite time.
People are using log[] years and it would be really miracle if suddenly only for you this would be not working.

So .. what exactly you've done?

1) https://www.zabbix.com/documentation...gers/functions
str() function does not support rexexp in first parameter.
2) https://www.zabbix.com/documentation...ypes/log_items
Are you really sure that you are using active items? Do you see in debug output that agent receives monitoring cfg to monitor /var/log/messages using log?
3) using log[] you've already extracted all log lines with "err|crit|alert|emerg" so why do you want to check using the same regexp using str() in last 30 lines?
4) /var/log/messages is usually rotated so you should be using logrt[] instead log[]

[heybails]

I'm pretty sure I'm not the only one having issues with log monitoring. There are a lot of threads out there from Zabbix users who are also having the issue. Also, I've worked on a ton of other monitoring tools and have never seen the log monitor be so badly designed. However, if you have a log file monitor that works, send me the basics and I'll integrate with my setup, restart debug and see what happens.

Right now, everything I try fails. Also, I agree with "logrt" that it needs to be set, but, then why bother even providing "log"?

Also, I have set another log monitor up both looking for a str and a regex and neither worked. The other log monitor was OWNED BY ZABBIX and had 666 permissions, and was set as a user parameter. I was able to see data when I ran zabbix_get, but nothing in the dashboard. This monitor was only looking for the word "Error". Bottom line is, the documentation is weak at best in making sure every angle of log monitoring is covered. So the best thing would be to see an example setup of a log monitor that is currently working for someone. Then I can set that up in my env. Thanks for your help.

[andris]

Hi !

Log monitoring is widely used and currently no experienced users are complaining about unfixed bugs.

Maybe there is a some small thing configured wrong ?
Log file monitring requires properly configured and working active checks on agent.
Try to configure some item (not log monitoring) as 'active check' and see does it work.
If yes, then try to make the simplest 'log[/var/log/syslog]' work: enable "DebugLevel=4" in zabbix_agentd.conf, observe in agent log whether a list of active checks is requested from server, how agent executes active checks. Most probably you will see where it stops.

[andris]

Have you set 'ServerActive=' parameter in zabbix_agentd.conf ?
And 'Hostname=' parameter ? Does it match the host name as shown in Zabbix frontend ?
These are prerequisites for active checks, without them log monitoring won't work.

[heybails]

Thanks Andris...yes you're right, there is probably something I'm missing, but I can't think of what it is. I've checked the config file and ServerActive and Hostname are set correctly. As I've stated, I've tried a minimal log in various ways. Nothing. I've got other things that are a priority right now. I will circle back and try again and let you all know what I see.

[kloczek]

Thanks Andris...yes you're right, there is probably something I'm missing, but I can't think of what it is. I've checked the config file and ServerActive and Hostname are set correctly. As I've stated, I've tried a minimal log in various ways. Nothing. I've got other things that are a priority right now. I will circle back and try again and let you all know what I see.

Next time when you will be setting up anything (not only zabbix) really read first documentation.
On the https://www.zabbix.com/documentation.../activepassive is:
"Active checks

Active checks require more complex processing. The agent must first retrieve from the server(s) a list of items for independent processing.

The servers to get the active checks from are listed in the 'ServerActive' parameter of the agent configuration file. The frequency of asking for these checks is set by the 'RefreshActiveChecks' parameter in the same configuration file. However, if refreshing active checks fails, it is retried after hardcoded 60 seconds. "

[andris]

Hi again!

To make sure that active checks are working, you can set 'DebugLevel=4' in zabbix_agentd.conf and restart agent. Then in zabbix_agentd.log look for lines like:

 13955:20160726:142126.208 In refresh_active_checks() host:'xxxxxxx' port:nnnnn
...
 13955:20160726:142126.213 sending [{"request":"active checks","host":"aaaaaa","port":mmmm}]
....
 13955:20160726:142126.214 got [{"response":"success","data":[{"key":"log[/home/zabbix/test1.log,ERROR]","delay":1,"lastlogsize":0,"mtime":0}]}]
 13955:20160726:142126.214 In parse_list_of_checks()
 13955:20160726:142126.214 In add_check() key:'log[/home/zabbix//test1.log,ERROR]' refresh:1 lastlogsize:0 mtime:0
 13955:20160726:142126.214 End of add_check()
 13955:20160726:142126.214 End of parse_list_of_checks():SUCCEED
 13955:20160726:142126.214 End of refresh_active_checks():SUCCEED

What else could be wrong ? Check your log item in the frontend - does it have "Type: Zabbix agent (active)" and "Type of information: Log" ?

Once active checks are working then look for lines in zabbix_agentd.log about analyzing the log file:

 13955:20160726:142126.214 In process_active_checks() server:'127.0.0.1' port:10092
 13955:20160726:142126.214 In process_logrt() is_logrt:0 is_count:32 filename:'/home/zabbix/test1.log' lastlogsize:0 mtime:0
....
 13955:20160726:142126.214 process_logrt() old file list:
 13955:20160726:142126.214    file list empty
 13955:20160726:142126.214 process_logrt() new file list: (mtime:0 lastlogsize:0 start_idx:0)
 13955:20160726:142126.214    nr:0 filename:'/home/zabbix/test1.log1' mtime:1465206873 size:892 processed_size:0 seq:0 incomplete:0 dev:65026 ino_hi:0 ino_lo:9914 md5size:512 md5buf:978f3e1bb67e4aa70765ad6c7759f2e7
 13955:20160726:142126.214 In process_log() filename:'/home/zabbix/test1.log1' lastlogsize:0 mtime:0
...

I hope we'll get it working

[heybails]

I have never been able to fix this. However, we have decided to focus on Splunk as our log monitoring tool, and that doesn't require root access for the Zabbix user. So for now, I'm bagging log monitoring in Zabbix.