Better syslog message handling for Zabbix

[Blinkiz]

It may be interesting to do a DNS lookup with A and/or AAAA when IP address is not defined in Zabbix (0.0.0.0). It seems to be a good idea for a future evolution.

Nice. DNS lookup should always be done regardless if a IP address is set or not... IF the user has chosen "Connect to: DNS Name".
When doing dns lookups, it can reply with multiple IP addresses. If you can hook into how Zabbix lookups up "DNS name"s, then you are always safe.

[tof233]

Hello,
I have a problem with host <-> syslog association:

The problem is that the messages are not associated with the host (LAX13)

Here is my schema LAX13.syslog-ng -> Zabbix_server.syslog-ng -> Zabbix_server.zbxlog


Here is the syslog message sent by syslog-ng :

U 127.0.0.1:43385 -> 127.0.0.1:514
<43>Mar 30 17:20:01 LAX13 syslog-ng[14690]: Error opening file for writing; filename='/var/log/syslog', error='Permission denied (13)'.

and here are the logs :

Read:remote=localhost.localdomain buf=<46>Mar 30 17:05:22 LAX13 syslog-ng[14632]: Termination requested via signal, terminating;
3)'

Zbxlog::SyslogMsg::new $VAR1 = bless( {
'_timestamp' => 1301497522,
'_host' => 'localhost.localdomain',
'_facility' => 'syslog',
'_message' => 'syslog-ng[14632]: Error opening file for writing; filename=\'/var/log/syslog\', error=\'Permission denied (13)\'',
'_severity' => 'err',
'_priority' => '43',
'_ip' => '127.0.0.1'
}, 'Zbxlog::SyslogMsg'

Do you have an idea on what I should modify to have the association?

[Blinkiz]

H there tof233
The script is doing a reverse lookup on a IP number to match host tag in Zimbra config and the real machine name.

Doing reverse lookup on 127.0.0.1 gets you localhost.localdomain.
You can modify this in /etc/hosts file. Just make sure it says "localhost" somewhere on the same line. Like "myfinemachinename.domain.com localhost".
It's not ideal but creator if this script though reverse dns was a good idea

[JBo]

Hi,

Hello,
I have a problem with host <-> syslog association:

The problem is that the messages are not associated with the host (LAX13)

Here is my schema LAX13.syslog-ng -> Zabbix_server.syslog-ng -> Zabbix_server.zbxlog

Do you have an idea on what I should modify to have the association?

Since zbxlog is receiving the message from local syslog-ng server, it has no way to know the IP address of the real server (LAX13).

You need to modify the message format used by syslog-ng on Zabbix server in order to prefix it with real server name or IP.

With rsyslog, it is done with one of the following format specifications:

$template host_syslog,"%fromhost%%rawmsg%\n"
$template host_syslog,"%fromhost-ip%%rawmsg%\n"

I don't have any experience with syslog-ng.
I have checked syslog-ng documentation. It seems to have a way to customize message format but I don't know how to do it.

If you find how to do it, feel free to report it here. I will add it to zbxlog documentation.

Regards,
JBo

[tof233]

Thank you JBo,

I will try to find how to handle it with syslog-ng.

[tof233]

I found it, here is my syslog-ng configuration :

source s_zbx {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
tcp(ip(0.0.0.0) port(1514));
};
destination d_zbx {
udp("127.0.0.1", port(514)
template("$FULLHOST<$PRI>$DATE $MESSAGE\n") );
};
log { source(s_zbx); destination(d_zbx); };

I now have to find a way of using zbxlog on the Zabbix_server with hosts monitored accross a zabbix_proxy. I don't want zbxlog to run on a zabbix_proxy. I only want it to run on the zabbix_server.

[JBo]

Hi,

I found it, here is my syslog-ng configuration :

Thank you.
I will add it to Zbxlog README.

I now have to find a way of using zbxlog on the Zabbix_server with hosts monitored accross a zabbix_proxy. I don't want zbxlog to run on a zabbix_proxy. I only want it to run on the zabbix_server.

Never tested but it should work without any special configuration.
Zbxlog extracts from Zabbix DB all hosts that have a syslog[..] item defined regardless of proxy setup.

Regards,
JBo

[tof233]

I tried and it's not working
When I set the proxy in the host configuration, I get a :

Zbxlog::Sender::Send response=ZBXDW{
"response":"success",
"info":"Processed 0 Failed 4 Total 4 Seconds spent 0.002398"}

And without the proxy configuration, no Failed

[JBo]

Hi,

I tried and it's not working
When I set the proxy in the host configuration, I get a :

And without the proxy configuration, no Failed

Bad news.
Zbxlog uses Zabbix agent protocol as defined in manual to send syslog messages to zabbix server. Obviously this is not working for proxied hosts.
I'm afraid that you will need a zbxlog server per proxy.

Thank you for your feedback.
JBo

[tof233]

So I will try to find a way of specifying a proxy, even if we have to specify it in each zbxlog configuration file.

[JBo]

So I will try to find a way of specifying a proxy, even if we have to specify it in each zbxlog configuration file.

It should be possible to point zbxlog to a proxy with zabbix_server parameter in zbxlog.conf.
Let me know if it works (I don't have a proxy in my test configuration, I will probably install a VM with a proxy in the near future).

JBo

[tof233]

In fact, I don't want to overload the Proxy. That's why I want zbxlog to directly insert syslog messages into the Zabbix server.
So I am looking on the differences between a proxy and zbxlog packets.

Proxy -> Zabbix

.{.."request":"history data",.."host":"LAX13",.."data":[...{...."host":"LAX13",...."key":"vfs.fs.size[\/usr,pused]",...."clock":1301572063,...."value":"38.41602 3"}],.."clock":1301572064}

zbxlog -> Zabbix

ZBXD.E.......{"request":"sender data",."data":[.{."host":"LAX13",."key":"syslog[]",."value":"syslog-ng[16558]: Termination requested via signal, terminating;",."timestamp":"1301574761",."source": "syslog",."severity":"11",."eventid":"46",.}.].}

And the only difference I found here is the "ZBXD.E......." and the "history data" -> "sender data" .
So there should be a way to modify that...

[tof233]

I succeeded by modifying in /usr/local/zbxlog/lib/Zbxlog/Sender.pm :

# my $zbx_data = qq|{"request":"sender data",\n"data":[|;
my $zbx_data = qq|{"request":"history data","host":"Proxy_name",\n"data":[|;

So now i have to find a way to get the proxy name from the database

The main problem seems to be rewriting the Sender.pm so that it sends one packet per proxy and not only one global packet.

[JBo]

I succeeded by modifying in /usr/local/zbxlog/lib/Zbxlog/Sender.pm :


So now i have to find a way to get the proxy name from the database

The main problem seems to be rewriting the Sender.pm so that it sends one packet per proxy and not only one global packet.

Thanks.

I have started modifying SQL query in order to get proxy name from Zabbix DB.
I think, I will post a patch tomorrow for proxy support.

JBo

[tof233]

I modified a little bit Items.pm with

SELECT hostinfo.host,hostinfo.key_,hosts.host AS proxyname FROM    
	(select
		h.host,
		i.key_,
		h.proxy_hostid
	    from
		hosts h,
		items i
	    where
		h.hostid=i.hostid
	    and h.status=0
	    and i.status=0
	    and i.key_ like 'syslog[%')  AS iteminfo 
	LEFT OUTER JOIN hosts ON hostinfo.proxy_hostid=hosts.hostid;

But now I'm a bit lost with how to split items by proxy to send one global packet per proxy...

I really thank you for your reactivity and would be happy to contribute...