Cannot monitor syslog - missing something obvious?

[pablos]

Hi,

I am relatively new to Zabbix and I'm having a difficult time implementing the monitoring of syslogs. I think I must be doing something incredibly dumb. I'm hoping a second set of eyes from the forum members might help me.

Environment

To keep things simple I have two servers: the one to monitor and the Zabbix server.

• CentOS 7
• SELinux set to Permissive
• The Agent and Zabbix Server are running as root
• The backend store is Postgres
• Version 2.4.4.1 of the Agent and Zabbix Server
• Firewalls disabled

Zabbix Object Configuration

For test puropses my Item is defined as follows:

log[/var/log/secure,sshd]

Any time someone logs in via ssh, /var/log/secure has a line with sshd in it therefore the above Item should match anything.

Next, I have a very simple Trigger:

{syslog:log[/var/log/secure,sshd].nodata(60)}=0

Agent .conf

• ServerActiove=10.200.0.61
• Hostname=Zabbix server

Problem

There are a couple of problems. First, unless I add the Zabbix Server to my "syslog" Template, the Agent on the monitored box does not report any data. I've confirmed by both using the GUI's Last data screen as well as enabling DebugLevel=4 on the Agent box.

If I add the Zabbix Server to my "syslog" Template, I see data like the following in the Agent's log:

 31508:20150317:112919.540 setup_old2new: is_same_file(/var/log/secure, /var/log/secure) = 1
 31508:20150317:112919.540 process_logrt() old file list:
 31508:20150317:112919.540    nr:0 filename:'/var/log/secure' mtime:1426605493 size:18131 processed_size:18131 seq:1 incomplete:0 dev:64769 ino_hi:0 ino_lo
:67969294 md5size:512 md5buf:4e5c42406fc0603a8578276968d0b327
 31508:20150317:112919.540 process_logrt() new file list: (mtime:0 lastlogsize:18131 start_idx:0)
 31508:20150317:112919.540    nr:0 filename:'/var/log/secure' mtime:1426605493 size:18131 processed_size:18131 seq:1 incomplete:0 dev:64769 ino_hi:0 ino_lo:67969294 md5size:512 md5buf:4e5c42406fc0603a8578276968d0b327
 31508:20150317:112919.540 End of process_logrt():SUCCEED error_count:0
 31508:20150317:112919.540 End of process_active_checks()
 31508:20150317:112919.540 In get_min_nextcheck()

but then I get this error in the Zabbix Server's log:

  3976:20150317:122834.998 item "Zabbix server:log[/var/log/secure,sshd]" became supported
  3975:20150317:122955.061 item "Zabbix server:log[/var/log/secure,sshd]" became not supported: Cannot open file "/var/log/secure": [13] Permission denied

Again, I'm running both the Agent and the Server as root, SELinux is set to Permissive so I'm not sure why I'm getting a Permission denied error.

Any help would be greatly appreciated!

Thx!
-pablo

[HaveDill]

yeah i had a similar issue with /var/log/messages

What i ended up doing to resolve it was i created a symbolic link to that logfile, with the name ending in .log

so now zabbix watches /var/log/messages4zabbix.log

Also make sure the user "Zabbix" has permissions to read that file.

[aib]

Again, I'm running both the Agent and the Server as root, SELinux is set to Permissive so I'm not sure why I'm getting a Permission denied error.

Typically, zabbix agent starts under "zabbix" user account, not "root"

#  ps -ef | grep zabbix_agent
zabbix   15806     1  0 03:07 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   15808 15806  0 03:07 ?        00:00:09 zabbix_agentd: collector [idle 1 sec]    
zabbix   15809 15806  0 03:07 ?        00:00:01 zabbix_agentd: listener #1 [waiting for connection]
zabbix   15810 15806  0 03:07 ?        00:00:01 zabbix_agentd: listener #2 [waiting for connection]
zabbix   15811 15806  0 03:07 ?        00:00:01 zabbix_agentd: listener #3 [waiting for connection]
zabbix   15812 15806  0 03:07 ?        00:00:42 zabbix_agentd: active checks #1 [idle 1 sec]

check if you have different settings

### Option: AllowRoot
#       Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent
#       will try to switch to the user specified by the User configuration option instead.
#       Has no effect if started under a regular user.
#       0 - do not allow
#       1 - allow
#
# Mandatory: no
# Default:
# AllowRoot=0

### Option: User
#       Drop privileges to a specific, existing user on the system.
#       Only has effect if run as 'root' and AllowRoot is disabled.
#
# Mandatory: no
# Default:
# User=zabbix

[pablos]

yeah i had a similar issue with /var/log/messages

What i ended up doing to resolve it was i created a symbolic link to that logfile, with the name ending in .log

so now zabbix watches /var/log/messages4zabbix.log

Also make sure the user "Zabbix" has permissions to read that file.

Hi HaveDill,

Thank you for your response.

Do you recall whether your Item's definition was something like log[/var/log/messages4zabbix.log,...? Mine is defined without .log

Because it's easy to test, I went ahead and created the symlink's in case Zabbix has some bug which causes it to search for .log. Unfortunately, I have the same permission error.

Next, I tried using the symlink name in the Item definition: log[/var/log/secure.log,...]. I ended up with a different error - see below.

Also, I've purposely configured both the Agent and the Zabbix Server to run as root to rule out permission issues. And furthermore, I've disabled SELinux too. I'm striving to simply the problem.

Error when using /var/log/secure

  4009:20150318:115831.946 item "Zabbix server:log[/var/log/secure,sshd]" became supported
  4009:20150318:115946.995 item "Zabbix server:log[/var/log/secure,sshd]" became not supported: Cannot open file "/var/log/secure": [13] Permission denied

Error when using /var/log/secure.log - a symlink to /var/log/secure

 15294:20150318:120830.496 item "Zabbix server:log[/var/log/secure.log,sshd]" became supported
 15292:20150318:120930.531 item "Zabbix server:log[/var/log/secure.log,sshd]" became not supported: Cannot obtain information for file "/var/log/secure.log": [2] No such file or directory

[pablos]

Typically, zabbix agent starts under "zabbix" user account, not "root"

Hi aib,

Thank you for your response. Yes, for the purposes of simplifying the problem, I am purposely running as root. This should mitigate any permission issues.

Thx!
-pablo

[pablos]

*** Bump .. please... ***

[HaveDill]

*** Bump .. please... ***

? We answered your questions. if that still isn't working do my solution. create a symbolic link ending in .log and point zabbix at it..

[pablos]

? We answered your questions. if that still isn't working do my solution. create a symbolic link ending in .log and point zabbix at it..

Hi,

I responded with the results of your suggestion. Unfortunately it didn't work.

The other suggestion wasn't on-key and I responded to it as well.

Any other thoughts?

Cheers,
-pablo

[HaveDill]

Can you send a screenshot of the item configuration?

Also, let's remove the sshd flag, and have it just collect the entire log file - test to see if that works.

[aib]

His problem not in wrong Item configuration.
He asked, why process, started from root, cannot read a file, which have a permission to read (for root) ?

[pablos]

Can you send a screenshot of the item configuration?

Hi,

Please see the attachment. At the moment the Item is disabled.

Also, let's remove the sshd flag, and have it just collect the entire log file - test to see if that works.

Dumb question ... ... it's currently:

log[/var/log/secure,sshd]

and you'd like me to set it to .. is that right?

log[/var/log/secure]

Thank you again for your help. I appreciate it.

-pablo

[pablos]

His problem not in wrong Item configuration.
He asked, why process, started from root, cannot read a file, which have a permission to read (for root) ?

Hi,

I apolgize for not being clearer. Here's the history.

I initially started as the zabbix Linux user and encountered the permission errors. Next, I changed the file permissions so the zabbix Linux user could read the file. When that didn't work, I disabled SELinux. When that didn't work, I set everything up to run as "root" This didn't work either and this is the current state of my environment. I figure by leaving everything as "root" for now, we should not have any permission issues. But we do. I've confirmed (via "ps") that the Zabbix Agent "and" Server are running as "root" So, I'm very puzzled.

I hope the above helps.

Thank you for your assistance!

Cheers,
-pablo

[pablos]

Resolved!

Hi,

It turns out the issue was a bit rudimentry. I hadn't configured my Agent's Hostname properly. *sigh* The Hostname has to match the name configured under Configuration > Hosts

The clue was I wasn't getting any data reported for the specific Host test case: Monitoring > Latest data

I hope by posting the above, someone else can learn from my mistake.

Cheers,
-pablo

[mkash28]

thanks pablos,

you may have helped me with my issue. seems like hostname wasn't matching with how it was added on the zabbix server. now old triggers generate errors still but atlast I can create new triggers and they are operational (green enabled).
amazing job with troubleshooting!!
thanks!

[boatcoder]

On Debian you can add zabbix to the adm group (same as syslog) to get read access to /var/log/syslog