Better syslog message handling for Zabbix

[zalex_ua]

Any feedback is welcome.

from 'zbxlog_config.txt.example':
# minimum number of seconds between two connections to zabbix server
zabbix_delay=30

Why is it necessary? Even if you are, then the default value 30 - is too great. The default should be 1 or 0.

Also i propose rename "zbxlog_dbconf.example" to "zbxlog_db.conf.example"
and "zbxlog_config.txt.example" to "zbxlog.conf.example"
if agreed - need to tweak the source code, respectively.
And why they do two? We must unite in a single file.

[alixen]

new zbxlog release

Hi,

I have just released zbxlog r0.3 (http://www.alixen.org/attachments/do...bxlog-r0.3.tgz)

New feature:

• Add syslog "priority" in Zabbix EventID field (as suggested by zalex_ua)

Enhanced features:

• Integrate Zabbix DB access in main program (zbxlog.pl)
• DB access with Perl DBI (support all Zabbix DBMS via DBD)
• Reorganize configuration (everything in one file)

Cleanup:

• Remove useless external scripts accessing Zabbix DB
• Remove useless crontab

Happy syslog monitoring !

Alixen

[alixen]

Hi,

from 'zbxlog_config.txt.example':
# minimum number of seconds between two connections to zabbix server
zabbix_delay=30

Why is it necessary? Even if you are, then the default value 30 - is too great. The default should be 1 or 0.

The goal is to avoid to open a new connection to Zabbix server for each message but instead buffer them and send all of them in one connection.
I agree that 30 is too high. I am currently running with 10 and I think it can be lowered.
1 should be OK, but don't set it to 0, it will turn off all buffering (may be I should add a check for that).

Also i propose rename "zbxlog_dbconf.example" to "zbxlog_db.conf.example"
and "zbxlog_config.txt.example" to "zbxlog.conf.example"
if agreed - need to tweak the source code, respectively.
And why they do two? We must unite in a single file.

See my previous post, there is only one file now.

Regards,
Alixen

[alixen]

Hi,

The goal is to avoid to open a new connection to Zabbix server for each message but instead buffer them and send all of them in one connection.
I agree that 30 is too high. I am currently running with 10 and I think it can be lowered.
1 should be OK, but don't set it to 0, it will turn off all buffering (may be I should add a check for that).

I have done some load tests on a VMware VM with a single virtual CPU (one core from a Xeon Quad Core 2,66 GHz).
Zabbix configuration is 6 hosts, 4 syslog items per host.

I have tried:

• zabbix_delay=5 and zabbix_delay=1 (as suggested by zalex_ua) : they both work, I can process around 800 messages/s.
• zabbix_delay=0 : it works, I also saturate at 800 msgs/s but I see a lot of sockets in TIME_WAIT due to the high number of connections to zabbix_server (one per message). zabbix_delay=0 should not be used.

I have also done some code optimization (in trunk, not yet released), the new limit (with same test conditions as above) is around 4000 messages/s.

Regards,
Alixen

[zalex_ua]

Hi Alixen,

See my previous post, there is only one file now.

May still be necessary to rename zbxlog_config.txt.example to zbxlog.conf.example and switch to use zbxlog.conf ? I think so generally accepted.

I have tried:

• zabbix_delay=5 and zabbix_delay=1 (as suggested by zalex_ua) : they both work, I can process around 800 messages/s.
• zabbix_delay=0 : it works, I also saturate at 800 msgs/s but I see a lot of sockets in TIME_WAIT due to the high number of connections to zabbix_server (one per message). zabbix_delay=0 should not be used.

I have also done some code optimization (in trunk, not yet released), the new limit (with same test conditions as above) is around 4000 messages/s.

What processor load during syslog flow as above? Need complex report, maybe later

And finally, what about support new functionality of nanoseconds

https://support.zabbix.com/browse/ZBXNEXT-457

I use something as:

use Time::HiRes qw ( gettimeofday );
($seconds, $microseconds) = gettimeofday;
print "$seconds\n";
print "$microseconds\n";

http://perldoc.perl.org/Time/HiRes.html

This can be implemented in a zbxlog now and it's compatible with 1.8.x branch.
Of course it is not necessary because the server supports the older agents, but I think I still need to implement it in zbxlog.

[alixen]

Hi zalex_ua,

May still be necessary to rename zbxlog_config.txt.example to zbxlog.conf.example and switch to use zbxlog.conf ? I think so generally accepted.

Agreed, I'll do that before releasing 0.4.

What processor load during syslog flow as above? Need complex report, maybe later

processor load was 100% with roughly:

• mysqld : 37%
• zabbix_server : 35%
• zbxlog.pl : 28%

And finally, what about support new functionality of nanoseconds

https://support.zabbix.com/browse/ZBXNEXT-457

I use something as:

use Time::HiRes qw ( gettimeofday );
($seconds, $microseconds) = gettimeofday;
print "$seconds\n";
print "$microseconds\n";

http://perldoc.perl.org/Time/HiRes.html

This can be implemented in a zbxlog now and it's compatible with 1.8.x branch.
Of course it is not necessary because the server supports the older agents, but I think I still need to implement it in zbxlog.

timestamp field is based on time as defined in syslog message. It has one second resolution, so I don't see how to add subsecond granularity unless I replace syslog timestamp by my own.

Regards,
Alixen

[alixen]

zbxlog release 0.4

Hi,

I have just released v0.4 of zbxlog (http://www.alixen.org/attachments/do...bxlog-r0.4.tgz)

Evolutions included in this release:

• Performance optimization for item matching
• Configuration file renamed etc/zbxlog.conf
• Added sanity checks in configuration parser

Regards,
Alixen

[alixen]

zbxlog release 0.5

Hi,

I have just released zbxlog v0.5 (http://www.alixen.org/attachments/do...bxlog-r0.5.tgz).

It adds support for a default destination of all non matching items.

Happy syslog monitoring !
Alixen

[sire]

Great work, alixen! In my opinion it's the only right way syslog messages should be processed by monitoring/log analyzing software because it is impossible to install agent on a hardware router or switch.
I hope Zabbix team will re-implement your program in C code as a native Zabbix process.

[aic]

Is it possible apply this solution in Linux environment instead of Windows? I have implemented Zabbix with Oracle DB.

Thx,
aic

[alixen]

Hi,

Is it possible apply this solution in Linux environment instead of Windows? I have implemented Zabbix with Oracle DB.

Maybe I misunderstand your question but Zbxlog does not apply to Windows. Windows event logs are already supported by Zabbix agent with eventlog[] items.

Zbxlog runs on Linux and can collect syslog messages from any host supporting syslog protocol (almost all network devices such as routers, switches, firewalls and Unix/Linux systems).

Zbxlog uses Perl DBI to access Zabbix database. If you use Oracle, you will probably need to install perl DBD::Oracle package (Zbxlog has been currently tested with mysql only).

Regards,
Alixen

[aic]

Thank you for replying!

[tof233]

Zbxlog on a slave node

I have a problem setting up zbxlog on a slave node.

I installed it directly on the node and created the host on this node:
Name: SVIMOCAVL007
Item key: syslog[]
type: Zabbix_trapper

The problem is that everything is forwarded to the nonmatch host and not to
SVIMOCAVL007.
I looked on the packets exchanged with mysql, and everything seems to be ok.

T 2010/12/23 12:11:42.498560 127.0.0.1:53559 -> 127.0.0.1:3306 [AP]
             (   select    h.ip as address,    h.host   from    hosts h,    items i   where    h.hostid=i.hostid       and h.status=0       and h.ip <> ''
         and i.status=0       and i.key_ like 'syslog[%'      )      union      (   select    h.dns as address,    h.host   from    hosts h,    items i   
  where    h.hostid=i.hostid       and h.status=0       and h.dns <> ''       and i.status=0       and i.key_ like 'syslog[%'      )                      
###
T 2010/12/23 12:11:42.500050 127.0.0.1:53559 -> 127.0.0.1:3306 [AP]
             select   h.host,   i.key_      from   hosts h,   items i      where   h.hostid=i.hostid      and h.status=0      and i.status=0      and i.ke
  y_ like 'syslog[%'                                                                                                                                      
#
T 2010/12/23 12:11:42.500693 127.0.0.1:3306 -> 127.0.0.1:53559 [AP]
       *    def zabbix h hosts host host   @     @   *    def zabbix i items key_ key_         @                 SVIMOCAVL007 syslog[]

I did the same on the master node, and didn't face this problem.
Do you have an idea on this problem?

[alixen]

Hi,

I have a problem setting up zbxlog on a slave node.

[...]

I did the same on the master node, and didn't face this problem.
Do you have an idea on this problem?

Never tried on a slave node, I don't have this configuration.
However, if messages are sent to nonmatch host, this means that zbxlog could not find zabbix hostname based on IP address.

Since the only information zbxlog knows about syslog message sender is its IP address and it needs to know zabbix hostname, it uses following logic:
- search for an host with corresponding IP address in Zabbix host table
- reverse DNS lookup of IP address and search corresponding DNS name in Zabbix host table

So you need to make sure that at least one of (IP address, DNS name) is correct in your zabbix host configuration.

Regards,
Alixen

[zalex_ua]

Alixen, hi, have in mind that version 2.0 DB schema has changed significantly. In particular there is added support for multiple interfaces, and now the "host" table does not contain a "ip". now is added the new table "interface" that contains "ip" value linked for 'hostid'
You would not want to make now a zbxlog branch with support for Zabbix 2.0?
Maybe I'll start to use zbxlog .