Better syslog message handling for Zabbix

[zabbix-x]

Thu Feb 24 15:01:26 2011 Database error 'DBI:mysql:database=zabbix;host=localhos
t;port=3306': Table 'zabbix.interface' doesn't exist (retries:0/10)
Thu Feb 24 15:01:26 2011 Database connection 'DBI:mysql:database=zabbix;host=loc
alhost;port=3306' : OK

what shall i do？

[JBo]

Hi,

Thu Feb 24 15:01:26 2011 Database error 'DBI:mysql:database=zabbix;host=localhos
t;port=3306': Table 'zabbix.interface' doesn't exist (retries:0/10)
Thu Feb 24 15:01:26 2011 Database connection 'DBI:mysql:database=zabbix;host=loc
alhost;port=3306' : OK

what shall i do？

Check zabbix_version parameter in zbxlog.conf and set it to 1.8:

# Zabbix server version
# Supported values:
#   zabbix_version=1.8
#   zabbix_version=2.0
[B]zabbix_version=1.8[/B]

Hope this helps,
JBo

[zalex_ua]

Thu Feb 24 15:01:26 2011 Database error 'DBI:mysql:database=zabbix;host=localhos
t;port=3306': Table 'zabbix.interface' doesn't exist (retries:0/10)
Thu Feb 24 15:01:26 2011 Database connection 'DBI:mysql:database=zabbix;host=loc
alhost;port=3306' : OK

what shall i do?

zabbix-x, it seems you use a zbxlog for Zabbix trunk branche.

Hi,

I have just released zbxlog v0.7
This release :

• adds preliminary support for upcoming Zabbix 2.0 (tested on 1.9.1)
  ...

It works with zabbix 1.9.1 (and of course with 1.8).

JBo, how you control the zbxlog versions for a different Zabbix branches?
I noticed this early but forgot to pay your attention.

[zalex_ua]

Bah,
JBo, is 1 minute difference in posts

[zabbix-x]

Hi,



Check zabbix_version parameter in zbxlog.conf and set it to 1.8:

# Zabbix server version
# Supported values:
#   zabbix_version=1.8
#   zabbix_version=2.0
[B]zabbix_version=1.8[/B]

Hope this helps,
JBo

OK! It works right!
thank u so much. and thanks zalex_ua.

[zalex_ua]

JBo, take into account - 1.9.2 released.
It has some interest changes - Configurable colors and names for trigger severities ZBXNEXT-631

btw, i tried and it seems patches from 1.9.1 are compatible with 1.9.2

[JBo]

Hi zalex,

JBo, take into account - 1.9.2 released.
It has some interest changes - Configurable colors and names for trigger severities ZBXNEXT-631

btw, i tried and it seems patches from 1.9.1 are compatible with 1.9.2

Thanks for the link to this new feature.

JBo

[JBo]

zbxlog release 0.9

Hi,

I have just released zbxlog v0.9 (http://www.alixen.org/attachments/do...bxlog-r0.9.tgz).

This release includes:

• Zabbix 1.9.2 frontend patches
• CentOS startup script (etc/init.d/zbxlog.centos) contributed by tof233
• connection to PostgresSQL backend documentation
• zabbix_version set to 1.8 in zbxlog.conf.example

Happy syslog monitoring !
JBo

[Blinkiz]

I was a against implementing this because it's not a supported feature from Zabbix. Seeing the flow of new versions, like support for Zabbix 2.0, it seems safe to implement this. Providing a clear uninstall procedure and you have everything!

I have not tried your syslog monitor yet but I would like to thank you for this contribution. It's a great feature that Zabbix currently does not have.
Thanks JBo!

[Blinkiz]

Hi there again.
Started to try this syslog patches. Working well against the sysloghost. Am happy
Anyway, I can not get the syslog entries go to a specific host. Enabling debug and I can see this from your script:

Tue Mar 29 07:25:46 2011
Zbxlog::SyslogMsg::new buf=er3<30>Mar 29 09:25:46 er3 zebra[7774]: Router advertisement received

Zbxlog::SyslogMsg::new $VAR1 = bless( {
                 '_timestamp' => 1301383546,
                 '_host' => 'er3',
                 '_facility' => 'daemon',
                 '_message' => 'zebra[7774]: Router advertisement received',
                 '_severity' => 'info',
                 '_priority' => '30',
                 '_ip' => ''
               }, 'Zbxlog::SyslogMsg' );

Interesting part here is _host. It's the hostname of the machine. Like this, er3.mydomain.com. I guess it's standard for linux to supply hostname like this.

$template host_syslog,"%HOSTNAME%%rawmsg%\n"
*.*     @77.xx.xx.xx:1514;host_syslog

Tried also $template host_syslog,"%fromhost%%rawmsg%\n" and $template host_syslog,"%fromhost-ip%%rawmsg%\n". %fromhost-ip% works well. Problem is that majority of our machines use IPv6 and not IPv4. So your script needs to known both the IPv6 address and IPv4 address for this to work.
IPv6 is currently disabled on our Zabbix host because it does not support it fully until 1.8.5.

[JBo]

Hi,

Anyway, I can not get the syslog entries go to a specific host. How can I troubleshoot this? How is this mapping done?

IP address of syslog message sender is used to map to Zabbix hostname.
Basically, this is how it is done:

• Get sender IP address
• Reverse DNS lookup of IP to get DNS name
• Search a Zabbix host that match either «IP address» or «DNS name» in Zabbix host configuration AND has a syslog[..] item defined.
• If a zabbix host is found with corresponding item, send syslog message to it
• If no zabbix host found, send syslog message to sysloghost.

Messages sent to sysloghost are prefixed with sender IP address.
Make sure that your Zabbix host has:

• at least one syslog item (syslog[] is a good start, it will catch all syslog messages)
• either its «IP address» or «DNS name» defined

Is it possible to turn on a debug mode or something in your script so I can see what it does?

Debug messages are sent to /usr/local/zbxlog/var/zbxlog.log
You can set:

• $DEBUG = 1 in /usr/local/zbxlog/lib/Zbxlog/Hosts.pm

It will periodically (every minute by default) dump an hash table starting with
«LoadHosts hosts=» describing the mapping between Zabbix hostnames and DNS and IP addresses

• $DEBUG = 5 in /usr/local/zbxlog/lib/Zbxlog/Items.pm

It will periodically (every minute by default) log messages starting with:
LoadItems current time
followed by lines of the form:
h=zabbix_hostname e=syslog_key

With this setup, you'll get also a lot of debug messages.
I am afraid that debug messages are right now more developer than user friendly.

Hope this helps,
JBo

[JBo]

Hi,

You have updated your post while I was answering to it so I missed your update.

There are two ways to setup zbxlog.

1. zbxlog is listening on port 514 and receives directly syslog messages from remote hosts
2. zbxlog is listening on a non standard port (1514 for instance) and syslog messages are received by another syslog daemon that forward them to zbxlog.

IP and DNS name mapping to zabbix hostname as described in my previous post only works in first setup.

From your updated post, I understand that you are using second configuration.
If this is the case, Zbxlog has no way to know real sender IP address.
So you need to make sure that sender identification (name or IP) prefixed to syslog message matches what is defined in Zabbix host configuration.

If the sender is using IPv6 and you don't set IPv6 address in Zabbix configuration, the only way to match is to use DNS name.
So you need to make sure that DNS name is 'er3' not 'er3.mydomain.com' in Zabbix host configuration.

Regards,
JBo

[Blinkiz]

I removed my middle rsyslog service and pointed one machine directly against port 1514. I have also enabled IPv6 on Zabbix host with a workaround to have IPv6 and IPv4 working at the same time.

First problem is that your script can not listen (listen_ip=) on a IPv6 address. Not a biggi because I will work around that later. Okay, so IPv4 then..

Zbxlog::SyslogMsg::new buf=<86>Mar 29 11:06:01 feng CRON[9922]: pam_unix(cron:session): session closed for user root
Zbxlog::SyslogMsg::new $VAR1 = bless( {
                 '_timestamp' => 1301389561,
                 '_host' => '94.128.32.14.c.mydomain.com',
                 '_facility' => 'authpriv',
                 '_message' => 'CRON[9922]: pam_unix(cron:session): session closed for user root',
                 '_severity' => 'info',
                 '_priority' => '86',
                 '_ip' => '94.128.32.14'
               }, 'Zbxlog::SyslogMsg' );

Tue Mar 29 09:06:01 2011

In Zabbix GUI:
DNS name: service.mydomain.com
IP address: 0.0.0.0
Connect to: DNS name

LoadHosts hosts=$VAR1 = {
          '0.0.0.0' => 'service.mydomain.com',
          'service.mydomain.com' => 'service.mydomain.com'
        };

service.mydomain.com has IP number 94.128.32.14 and 2a21:410:0:3:5034:ff:fed9:1b6e.

It does not seems the IP number mapping is working in my case? Reverse DNS will in most cases don't match and it's not required.
Are you doing dns lookup (A and/or AAAA) of the DNS name fetched from Zabbix? That will of course match with the real IP number of the machine.

Don't get me wrong, you have done a amazing script for Zabbix. It's almost enough for me to just use the special sysloghost. This is like a bonus if mapping will work

[Blinkiz]

I guess my solution at the moment is to get hostname -> ip mapping always working, is setting up reverse DNS for every host that should fetch syslog entries. Reverse DNS should of course match DNS name in Zabbix.
Not the best solution but it will probably work

[JBo]

Hi,

In Zabbix GUI:
DNS name: service.mydomain.com
IP address: 0.0.0.0
Connect to: DNS name

Zbxlog uses IP address as defined in Zabbix.
If you want Zbxlog to match 94.128.32.14 you need to set IP address to 94.128.32.14.

Are you doing dns lookup (A and/or AAAA) of the DNS name fetched from Zabbix?

No, I am doing a reverse DNS lookup of IP address with Perl gethostbyaddr() function.
It may be interesting to do a DNS lookup with A and/or AAAA when IP address is not defined in Zabbix (0.0.0.0). It seems to be a good idea for a future evolution.

I have checked IPv6 status of Perl modules I am using (specially IO::Socket::INET). I'm afraid that I'll need to do some work for IPv6 support

Thanks for your reports.
JBo