Ad Widget

**James Wells** · 18-03-2007, 01:18

Greetings,

Originally posted by transitv

Does zabbix have the ability to combine these two items (databases) into one so that I don't have to store the same info twice?

Yes and no. By default Zabbix is unable to properly track changes to a syslog style log file. The reason for this is that the Agent doesn't know how to process the time format correctly. However, I have been asked by a couple of people at work to fix this, so unless Alexei fixes it soon, I will be doing so in my bulk patches. Once this is done, you will have not only the features that php-syslog-ng provides, but also, but also the features of swatch and some of the features of splunck and spelunk.

I should point out that Zabbix can support log4j, log4s, and apache style log files.

**transitv** · 18-03-2007, 22:19

I'm still getting my head around the different packages out there (I've looked at splunk but not too much), but it'll be great if zabbix eventually moves to include syslog storage, monitoring, and analysis. The raw data from my syslog (daemon) systems is already aggregating into a mysql database on this server (but this should work no matter where the db is as long as zabbix could access it). I think it would make zabbiz a much more useful/powerful system if it could do both roles. It sounds like the path is laid out, but isn't quite there yet.

If you need any help with patches let me know, I don't have a lot of time but having one system do look at instaed of a bunch of separate ones would be grand. Plus having it pull from the same "source" for triggers and such would cut down on duplicate data.

I guess the biggest challenge with this idea is to find a way for zabbix triggers to fire off when snmp data is put into the database. You'd either have to combine syslog-ng into zabbix so it's processes can watch the flow of incoming data to catch items, or perhaps zabbix could constantly watch the syslog database and read over all incoming data secondhand. The only problem I see with this whole idea is the volume of data being streamed from all my log servers (especially the firewall, it's about a gig a day right now).

I guess these ARE two separate functions (monitoring and logging), perhaps it's not enough dupliation to make it worth the effort of this idea...

**transitv** · 18-03-2007, 22:21

James,

I reread your post and am still trying to fully understand it. Are you saying the ability is already there it is just an issue with the time stamps?

If this is so, how do I get zabbix to accept an imcoming syslog stream from my firewall? I kinda understand how to do this with an agent, but not how to do it without one...

**James Wells** · 19-03-2007, 04:48

Originally posted by transitv

I reread your post and am still trying to fully understand it. Are you saying the ability is already there it is just an issue with the time stamps?

Yes. The key type for logging is log[*]. So to pull in the data from /var/log/syslog is log[/var/log/syslog]. The problem is that the syslog timestamp is a mixed strings and numbers, whereas the logtimefmt supports only numbers. BTW, for those wanting more information about the logtimefmt, it accepts the following only;

M -- Month (00-12)
m -- Minute (00-59)
h -- Hour (00-24)
y -- Year (0001-9999)
s -- Second (00-59)
d -- Day (01-31)
Blank means use unix timestamp

If this is so, how do I get zabbix to accept an imcoming syslog stream from my firewall?

By default, Zabbix cannot accept the incoming syslog data, instead, you have to either use Zabbix Agent (Active) mode on a syslog server, or using an SNMP agent to send the data to an SNMP trapper, or using Zabbix to pull the data directly from an SNMP agent. Personally, I configured my firewall to send syslog information to a syslog server, then have a Zabbix Agent running on the syslog server send the data to my Zabbix Server.

**transitv** · 19-03-2007, 15:26

Thanks for the answers. That's kinda what I gathered from other posts, but I have a few questions about the details (I kinda remember reading about your method in other posts, but didn't get an authoritative answer on exactly how to do it and what is the preferred/best method...)

1) Is using the agent better than the other methods? How would I go about using an SNMP trapper or pulling directly from an SNMP agent?

2) Using the agent sounds acceptable as it's already running on the syslog server (which is atm the zabbix server). Here's my proposed layout:

* Syslog-ng running on the (zabbix) server and configured to accept input over udp/tcp ports. Currently this is then ported into mysql separate from zabbix. I will turn this off and leave the data in syslog format and rotate using linux logrotate.

* Filters setup in syslog-ng to separate the firewall into its own "sink"/file, linux server syslogs (not apache) into another sink, and windows server syslogs into a third sink. (Apache syslogs are left on the web server(s) and monitored by awstats and local syslog rotating/etc for now, so zabbix won't see them.)

* Setup zabbix agent to watch these files and forward incoming data to the zabbix server.

* Setup the zabbix server to accept this data.

3) Where does zabbix store the incoming syslog data? I would think you'd want it separate from the other zabbix tables so that doing other reporting (normal zabbix monitoring stuff) doesn't have to sift through the gigabytes of log data...

4) Now that I have the data flowing into my zabbix database, how (well) does it handle log rotation? Based on my current setup it's looking like I'm getting a gig a day, so rotation will be important.

5) You said once you patch the data issue I'd have the best of php-syslogng and splunk (etc). I assume the code is already in place? Have you seen this in action yet? How does the current zabbix compare to phpsyslogng?

6)

I should point out that Zabbix can support log4j, log4s, and apache style log files.

What effect does this have?

7) Does zabbix just store the data, or does it act on it too (ie event triggers)?

Thankx again, I really appreciate it!

**James Wells** · 19-03-2007, 17:09

Originally posted by transitv

Is using the agent better than the other methods? How would I go about using an SNMP trapper or pulling directly from an SNMP agent?

If you want Zabbix to act on the notices in the syslog, then yes, using the Zabbix Agent is the best way to do it. The Agent will pull the data from the log, and even offers tools to perform simple expression searches to narrow the scope of the data it sends to the server as well. If instead you just want a way to organize correlate the log information, then no. At present, Zabbix is not the best choice for that. Using the Zabbix Agent method is actually better than using SNMP to push or pull the data.

Where does zabbix store the incoming syslog data? I would think you'd want it separate from the other zabbix tables so that doing other reporting (normal zabbix monitoring stuff) doesn't have to sift through the gigabytes of log data...

If you use the log[*] key, it stores the most recent entry and the previous entry in the items table for each host you have configured, from there it is also stored in history_log. This particular table is used only for log type items, as such it is fairly safe to use. Also, because of the seperation of this table, it will only be called when triggers are evaluated or when you specifically look at item history. When all is said and done this table is no more or less efficient than the table used by phpsyslog-ng, though it appears to be less efficient than the tables used by splunk.

Now that I have the data flowing into my zabbix database, how (well) does it handle log rotation? Based on my current setup it's looking like I'm getting a gig a day, so rotation will be important.

No. Personally, I use syslog-ng which has built in rotation. Each log file has the YYYYMMDD appended to it, then I have a script that runs at midnight which symlinks each date based log file to it's base name, so messages is a symlink for messages-20070318. I then have Zabbix read messages. And yes, Zabbix has code built in for when the logs are truncated.

You said once you patch the data issue I'd have the best of php-syslogng and splunk (etc). I assume the code is already in place? Have you seen this in action yet? How does the current zabbix compare to phpsyslogng?

I guess I wasn't clear enough. This is a patch I, or someone else, will have to create. Reading through the phpsyslog-ng code, it will be fairly trivial to make Zabbix more efficient, but I haven't looked close enough at splunk as of yet to determine how easy that will be.

Where I a better C programmer, I would create a new agent package called zabbix_logger, whose whole purpose would be to read syslog style logs and send the data to the server. Additionally, I would include capabilities in the agent that would allow it to perform more complex parsing of the syslog.

Once the data is in the DB, writing the UI to correlate it would e fairly simple, if a bit time consuming.

Does zabbix just store the data, or does it act on it too (ie event triggers)?

Both. Remember, when you create an item in Zabbix, it collects the data, but does nothing with it, save storing it until you create a trigger to act upon it. Log type items are no different. You can create an item without a trigger to act on the data, or you can create a trigger and actions for the log data.

**transitv** · 19-03-2007, 18:48

Ok, so I'll stick with the agent method. At this point I'm using syslog-ng to "filter" the data into the appropriate syslog files, so I don't need the agent to do this. All I want is to dump the syslog data into the database so zabbix can sort/display it and act on it (via triggers, etc).

The volume of data will be big (as I said), and in the firewall's case I don't want to narrow the data, just save it in a better form than syslog (files).

If instead you just want a way to organize correlate the log information, then no. At present, Zabbix is not the best choice for that.

Hmm, this sounds like it contradicts your earlier reply. To clarify, are you saying that if I just want to use zabbix (in addition to monitoring servers) to act as a syslog "storage" and viewing/reporting device that it won't work as well as php-syslog-ng (and syslog-ng)?

**transitv** · 19-03-2007, 18:58

If you use the log[*] key, it stores the most recent entry and the previous entry in the items table for each host you have configured, from there it is also stored in history_log.

Ok, this is good. The only issue is that my server syslog info is only megabytes a day compared to the gigs of data put out by the firewall. My current syslog-ng system puts the firewall in one table, linux in another table, and will (when I set it up) put the windows in a third table. This is on the same server, but again to ease log access I thought it good to split them up.

So my question is: does zabbix let me do something similar? It sounds like your answer says it just dumps them into a single table. While this isn't a dealbreaker, it sounds to me like it would be better to split them up. What do you think?

**James Wells** · 19-03-2007, 19:05

Originally posted by transitv

So my question is: does zabbix let me do something similar? It sounds like your answer says it just dumps them into a single table. While this isn't a dealbreaker, it sounds to me like it would be better to split them up. What do you think?

By default, no. All of the entries would go to the same table, though, if / when someone starts building the functionality we are talking about, it would be best if we broke up the tables based on some criteria. I know it wouldn't quite fit your bill, but I think breaking the tables up based on syslog facility would be the way to go overall especially when you consider that most applications allow you to specify which facility they log to.

**transitv** · 19-03-2007, 19:05

Just wanted to say thanks for being my sounding board (as well as info source). I've got a few more items, and figured it may be easier to split them into separate posts...

No. Personally, I use syslog-ng which has built in rotation.

For this part of your answer I only half followed it. I think the zabbix rotation will be fine, but wasn't sure about your "message" idea. Is it the zabbix agent which looks at this file?

How often does it look (as it is updated, or once an hour, etc)?

**James Wells** · 19-03-2007, 19:11

Originally posted by transitv

For this part of your answer I only half followed it. I think the zabbix rotation will be fine, but wasn't sure about your "message" idea. Is it the zabbix agent which looks at this file?

I have the Zabbix Agent look at /var/log/messages every 60 seconds. Remember, in my case /var/log/messages is just a symlink to /var/log/messages-YYYYMMDD. Also, in case you are curious, I also have a script that goes through at 12:10AM that gzips yesterday's messages-YYYYMMDD and moves it to an archive directory. Just a very simple script;

Code:

#!/bin/bash
#
#
YESTERDAY=`/usr/bin/date '+%Y%m%d' -d yesterday`

/bin/gzip -9 /var/log/messages-${YESTERDAY}
/usr/bin/mv /var/log/messages-${YESTERDAY}.gz /var/log/archive

And my syslog-ng.conf entry looks like;

Code:

destination messages { file("/var/log/messages-$YEAR$MONTH$DAY"); };

**transitv** · 19-03-2007, 19:14

..but I think breaking the tables up based on syslog facility would be the way to go..

That's what I thought. I'm pretty sure it would be easy to split the syslog data into facilities if my system isn't doing that already (need to check my setup to see).

On a side note... I'm mostly a Network Admin, but I've dabbled in php and c coding (at school and on my own), so time permitting this is something I wouldn't mind assisting with. I think it would be a great direction/addition to Zabbix and help take it to a new level!

**transitv** · 19-03-2007, 20:30

I have the Zabbix Agent look at /var/log/messages every 60 seconds.

That's what I suspected. I mostly followed it, but without that piece I wasn't sure. This should work for me doing the server logs as those are the ones I'm going to run triggers on (at this point) for things like disk failures, etc.

The volume from the firewall may be too much for this, so I guess I'll stick with doing that separate for now...

**transitv** · 19-03-2007, 20:34

I don't know if you missed it in my 12:48 post, but I did have one more question for you...

Hmm, this sounds like it contradicts your earlier reply. To clarify, are you saying that if I just want to use zabbix (in addition to monitoring servers) to act as a syslog "storage" and viewing/reporting device that it won't work as well as php-syslog-ng (and syslog-ng)?

Ad Widget

Can zabbiz work as a syslog to ease reporting/monitoring of logs?

Can zabbiz work as a syslog to ease reporting/monitoring of logs?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment