Unable to monitor log files

[ben.bressler]

Hello!

I'm setting up zabbix 1.8 and I've got most things set up. I've set up and tweaked the default Linux template and I've created my own templates to monitor the variety of custom processes we run on our boxes but I'm having trouble with the last thing I'm trying to set up; monitoring inside log files.

Here is what I've set up:

Item:
Type - zabbix agent (active)
Key - log["/opt/log/error.log","running",,120]
Type of information - Log
Update interval - 1 sec

Trigger:
Expression - {Log Template:log["/opt/log/error.log","running",,120].str(running)}=1

zabbix_agentd.conf:
Hostname=Server (this is the same as what the name of the server is in zabbix server)
DisableActive=0

Log File Info:
-rwxrwxrwx 1 root root 2023650255 Dec 6 20:28 error.log

Log File Entry:
2012-12-06 20:28:35,656 INFO task=running,last post=2012-12-06 20:28:35.656007,total posts=82560000,throttle=100%

The log file is generally updated every 10 to 20 seconds with an entry that is very similar to the above.

The way I think this should work is that I should receive an e-mail every 10 to 20 seconds every time zabbix detects the word "running" inside the error.log file. Obvious this isn't working and at this point I'm boggled as to why. I'm sure its just something small I've missed but I've run out of ideas. Any help is appreciated. Thanks!

[BDiE8VNy]

Does only the action not be executed, or doesn't the trigger get fired, or doesn't even the item get values?

[ben.bressler]

Does only the action not be executed, or doesn't the trigger get fired, or doesn't even the item get values?

Sorry I should have clarified that in the OP. From what I can tell I'm not receiving any data whatsoever from it - the item isn't getting values.

[ben.bressler]

Also, when I try and use zabbix_get to see if anything comes back I get "Accessible only as active check!".

This is what I'm running:
./zabbix_get -s Server -k "log["/opt/log/error.log","running",,120]"

[ben.bressler]

I am not receiving anything from what I can tell. The item does not return values.

[BDiE8VNy]

Does zabbix have the necessary priviliges to access the file?
Issuing this command as root:

# sudo -u zabbix test -r /opt/log/error.log && echo OK || echo NOK

shoud return 'OK'.

Do you already have taken a look into the log file of the zabbix-agent (e.g. /var/log/zabbix/zabbix_agentd.log)?
Be sure to have the configuration option 'DebugLevel' set to 3.

[ben.bressler]

I ran what you suggested:

ubuntu@Server:~$ sudo -u zabbix test -r /opt/log/error.log && echo OK || echo NOK
OK
ubuntu@Server:~$

And it returned OK.

I enabled debug level 3 and restarted the zabbix agent and here is all I'm seeing in the log file:

25545:20121207:221733.479 Starting Zabbix Agent [Server]. Zabbix 1.8.11 (revision 26205).
25546:20121207:221733.487 agent #0 started [collector]
25547:20121207:221733.488 agent #1 started[listener]
25548:20121207:221733.488 agent #2 started[listener]
25550:20121207:221733.501 agent #3 started[listener]
25551:20121207:221733.501 agent #4 started [active checks]

I also made sure I can telnet from the agent to the server:
ubuntu@Server:/var/log/zabbix-agent$ sudo telnet 10.72.126.220 10051
Trying 10.72.126.220...
Connected to 10.72.126.220.
Escape character is '^]'.

Connection closed by foreign host.
ubuntu@Server:/var/log/zabbix-agent$

[BDiE8VNy]

Please try this item key instead:
log["/opt/log/error.log","running",,120,skip]

[ben.bressler]

I made that change and I'm still not seeing anything come through.

[BDiE8VNy]

Just to be sure: There is no Zabbix-Proxy in between and after making changes and restarting the Zabbix-Agent came new log entries containing "running".

Then please change the 'DebugLevel' to 4 and inspect the output fore entries belongen to the log file item.

[ben.bressler]

No proxy.

After changing debug level to 4 and restarting the zabbix agent I found this in the log:

26104:20121207:224954.657 Active check [log["/opt/log/error.log","running",,120,skip]] is not supported. Disabled.

[BDiE8VNy]

Sorry, my mistake. The "skip" parameter is supported since 2.0.
Please 'touch' a file beside the real log file with same permissions and reconfigure the item (without 'skip') to the new file. After restarting the agent again, please write some lines containing 'running' into the new file.

[ben.bressler]

So here is what I've done. I created a 2nd log file called test.log and did what you said, wrote running into it manually and zabbix picked it up without any issue and I can see it in "latest data". I even copy/pasted lines directly from the real log file, error.log, into test.log and zabbix picked it up no problem. But zabbix still doesn't see anything from the real log file.

[ben.bressler]

I think I finally figured out what is going on. For some reason I can't apply a log file monitoring item to a template; it must be applied directly to a host.

When I set it to be applied directly to a host it works; that was the only difference between the 2 items I tested with above.

Is this standard behavior?

Also, can you have multiple items all monitoring a single log file?

I have one item set up now looking for "running" which is working. I cloned the item and changed "running" to "stopped" but it doesn't seem to be finding anything.

[BDiE8VNy]

I believe it has been worked from the very beginning.

The log file you want to monitor has a size of ~2GB and Zabbix tries to catch up while taken into account not to overload the system.
Set max lines to 1000 and restart the zabbix-agent again. However it will take some time.

Yes, one can have multiple items parsing the same log file.