Better syslog message handling for Zabbix

[parcival]

Hi JBo,
on a fresh / clean installtion with FreeBSD / zbx2.0.4 i have the same Problem

Debug log zabbix Server:

 91227:20130207:131808.083 Trapper got [{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 91.xxx.xxx.xxx:60900 to 217.xxx.xxx
.xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 13:18:05)] len 301
 91226:20130207:131808.083 In process_mass_data()
 91226:20130207:131808.083 End of process_mass_data()
 91227:20130207:131808.084 Trapper got [{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 91.xxx.xxx.xxx:1196 to 217.xxx.xxx.
xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 13:18:07)] len 300

regards
parcival

[JBo]

Hi parcival,

The only difference I see is that you are on FreeBSD.
My 2.0.4 installation is on Linux Ubuntu Server 10.04.4.

There is one strange thing in all your zabbix_server logs: all trapper messages seem to be truncated around 300 bytes.

Can you capture network traffic between zbxlog and zabbix_server with tcpdump or wireshark ?

Regards,
JBo

[parcival]

JBo, the tcpdump is correct !

cleartext tcpdump

ZBXD{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 149.xxx.xxx.xxx:50955 to 217.xxx.xxx.xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 14:06:42)",
"timestamp":"1360242403",
"source":"local0",
"severity":"15",
"eventid":"130",
}
]
}

This looks like a bug in FreeBSD <-> zbx2.0.4.
Why is the rest cut off of trapper data?
Is there a parameter in zabbix_server.conf ??

regards
parcival

[JBo]

Hi parcival,

I don't know why zabbix trapper is truncating these messages and I'm afraid I will not be able to help you since I don't have any experience on FreeBSD.

I have attached a patch that you can apply to zbxlog Sender.pm:
- It reorders the fields in trapper message so that syslog message is the last one. You should now get valid source and severity, only text message should be truncated.
- I have also removed newlines between fields. It makes it harder to debug but less characters should be lost.

Hope this helps
JBo

[parcival]

JBo, big thank you for your time and help, i opened a ticket on zabbix support.

https://support.zabbix.com/browse/ZBX-6230

I will test your patch an wrote here the result.

regards
parcival

[parcival]

JBo, this look a liitle better but the value wont wrote in the database.
No values in hystory_log and items (lastvalue) database table

zbxlog logfile:

Zbxlog::Sender::Send item=$VAR1 = [
          'fwgate-1',
          '',
          'syslog[]',
          '134',
          'local0',
          11,
          1360324535,
          'fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.xx Phase 1: Retransmission limit has been
reached. (2013-02-08 12:55:35).'
        ];

Zbxlog::Sender::Send zbx_data={"request":"sender data","data":[{"host":"fwgate-1","key":"syslog[]","timestamp":"1360324535","source":"local0
","severity":"11","eventid":"134","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.xxx Phase 1
: Retransmission limit has been reached. (2013-02-08 12:55:35).",}]}Zbxlog::Sender::Send response=OK

and here zabbix server logfile:

 65021:20130208:125426.256 Trapper got [{"request":"sender data","data":[{"host":"fwgate-1","key":"syslog[]","timestamp":"1360324466","sourc
e":"local0","severity":"11","eventid":"134","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.
76 Phase 1: Retransmission limit has been reached. (2013-02-08 12:54:26)] len 312

[JBo]

Hi parcival,

I have seen the comments on the ticket about data length calculation.
I have rewritten it so that it doesn't depend on pack() Perl function.

I have attached a new patch to zbxlog Sender.pm.
Let me know if it works better.

JBo

[parcival]

Sorry JBo, nope the same behavior.
I see now all keys and values (logfile), but no data in the table....

[JBo]

Sorry JBo, nope the same behavior.
I see now all keys and values (logfile), but no data in the table....

I'm not sure I understand it correctly.
Do you mean that in zabbix_server.log you now see a line such as:

842:20130208:165036.936 Trapper got [{"request":"sender data","data":[{"host":"Zabbix server","key":"syslog[]","timestamp":"1360338634","source":"local0","severity":"14","eventid":"131","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-error-00601: DNS:QUERY: NULL-QUERY has been detected from 91.XXX.XXX.XXX/5976 to 217.XXX.XXX.XXX/53 through policy 114 1 times. (2013-02-06 16:56:19).",}]}] len 366

that contains the full message including ending }]} but you still don't get anything in Web frontend or zabbix database ?

[parcival]

yes JBo exact.

After your first patch i see in the logfile this:

842:20130208:165036.936 Trapper got [{"request":"sender data","data":[{"host":"Zabbix server","key":"syslog[]","timestamp":"1360338634","source":"local0","severity":"14","eventid":"131","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-error-00601: DNS:QUERY: NULL-QUERY has been detected from 91.XXX.XXX.XXX/5976 to 217.XXX.XXX.XXX/53 through policy 114 1 times. (2013-02-06 16:56:19).",}]}] len 366

but with the first und your second patch, i got no data in database and nothing show up in the Webintreface.

hystory_log and item table with this key "syslog[]" allway empty, no values.

[jussipo]

Zbxlog for no dns

For those ones who might wonder why zbxlog is not working with rsyslog and situation is hosts are not put to DNS (so hosts are named on zabbix and rsyslog sends msg with ip) I give some my cents:

Make sure rsyslog is forwarding messages using rfc format, so put rsyslog.conf a template like:

$template host_syslog,"%fromhost-ip%<%PRI%>%TIMESTAMP% %syslogtag%%msg%"
or something similar.

To make sure Zabbix is having names, not IP of hosts, do one trick to Hosts.pm:

sub GetHost {
# Try to deduce zabbix hostname from syslog host or IP
my $self = shift;
my $h = shift;
my $ip = shift;
my $host;
if ($host ne '' and defined $self->{_hosts}->{$h}) {
$host = $self->{_hosts}->{$h};
} elsif ($ip ne '' and defined $self->{_hosts}->{$ip}) {
$host = $self->{_hosts}->{$ip};
} else {
# $host = $h;
$host = $self->{_hosts}->{$h}; # <--- this one
}
return $host;
}

Would be good to also disable host queries from Controller.pm to lower DNS load (because hosts are not added to dns, no need to query).

-jussipo

[ZeeD]

Thanks a lot jussipo!
Just FYI, 1.5 works fine on FreeBSD 8.3 with Zabbix 2.0.6

[gregtompkins]

I want to utilize this script for monitoring my dd-wrt and Ubuntu syslogs. Where does the template statement go in the rsyslog.conf file? Before or after the "rules" section?

Thank You!

[gregtompkins]

@JBo,

I am running CentOS..... I found a few whoopsies in your README file....

Startup at boot
---------------

for CentOS:
# cp etc/init.d/zbxlog.centos /etc/rc.d/init.d/zbxlog
# chmod +x /etc/init.d/rc.d/zbxlog <<-------- SHOULD BE /etc/rc.d/init.d no?
# chkconfig --add zbxlog
# chkconfig --level 2345 zbxlog on



Zabbix 2.0.4
------------
zabbix/patches/2.0.4/frontends/php/items.inc.php.patch
zabbix/patches/2.0.4/frontends/php/defines.inc.php.patch
zabbix/patches/2.0.4/frontends/php/CScreenHistory.php.patch

# cd /usr/share/zabbix
# patch -p0 < /usr/local/zbxlog/zabbix/patches/2.0.0/frontends/php/defines.inc.php.patch
# patch -p0 < /usr/local/zbxlog/zabbix/patches/2.0.0/frontends/php/items.inc.php.patch
# patch -p0 < /usr/local/zbxlog/zabbix/patches/2.0.0/frontends/php/CScreenHistory.php.patch

should patches for 2.0.4 be 2.0.4 or 2.0.0 (the first three lines reference 2.0.4 but the next three say 2.0.0

Now, for the startup I did service zbxlog start I did service zbxlog start and it started but I'm getting errors in the logfile

[root@sentinel log]# more zbxlog.log
Sat Nov 23 17:39:23 2013 zbxlog.pl started
Can't locate NetAddr/IP/Util.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_per
l /usr/lib64/perl5 /usr/share/perl5 . ./lib) at lib/Zbxlog/Controller.pm line 22.
BEGIN failed--compilation aborted at lib/Zbxlog/Controller.pm line 22.
Compilation failed in require at /usr/local/zbxlog/bin/zbxlog.pl line 34.
BEGIN failed--compilation aborted at /usr/local/zbxlog/bin/zbxlog.pl line 34.

Here is my rsyslog.conf file. I don't know what I've done wrong.

[root@sentinel log]# more zbxlog.log
Sat Nov 23 17:39:23 2013 zbxlog.pl started
Can't locate NetAddr/IP/Util.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_per
l /usr/lib64/perl5 /usr/share/perl5 . ./lib) at lib/Zbxlog/Controller.pm line 22.
BEGIN failed--compilation aborted at lib/Zbxlog/Controller.pm line 22.
Compilation failed in require at /usr/local/zbxlog/bin/zbxlog.pl line 34.
BEGIN failed--compilation aborted at /usr/local/zbxlog/bin/zbxlog.pl line 34.

Thanks, this looks like a great script! I want to use it to monitor my forwarded syslogs from other hosts including Ubiquiti wifi radios, an EdgeMax router (based on Vyatta) and some DD_WRT devices.

Thanks again.

[Marek768kb]

Hello, someone test it on zabbix 3.0.3 ?