ZABBIX Forums  
  #1  
Old 27-11-2017, 16:06
Linwood Linwood is offline
Senior Member
 
Join Date: Dec 2013
Location: Cape Coral, FL, USA
Posts: 257
Default Large systems log filtering

Zabbix can do log files, yes. But to me it's awkward and a bit kludgy.

Then there are tools like Graylog, which are deep and thorough, but are a completely different thing to setup and maintain. Yes, they can feed to Zabbix for consistent alerting, but it's still another substantial tool.

My question is this: For those with moderate and larger systems and a need for widespread log monitoring (say all servers, network devices like switches and routers, and firewalls), what are you using?

Or put another way, how many have successfully used Zabbix without introducing another tool into the mix?

Please note I'm not asking "does Zabbix work" -- yes, I know it works. I'm asking a practical question: do you find zabbix more work to make it work, than adopting another tool like Graylog?
Reply With Quote
  #2  
Old 28-11-2017, 12:28
andris andris is offline
Senior Member
 
Join Date: Feb 2012
Location: Riga, Latvia
Posts: 127
Default

May I ask what features are missing in Zabbix log[], log.count[], logrt[], logrt.count[] items ?
What would you recommend to add ?
(disclosure: I'm involved in their development)
Reply With Quote
  #3  
Old 28-11-2017, 17:04
Linwood Linwood is offline
Senior Member
 
Join Date: Dec 2013
Location: Cape Coral, FL, USA
Posts: 257
Default

That's a fair question and honestly I do not know, now. I remember trying to deal with windows system logs a while back, and giving up as almost unmanageable, but memory fails as to the details. I also did do a trial Graylog install and had much less trouble, again memory fails as to details of what made it less trouble.

One aspect is research after the fact, my impression is Zabbix is manageable if you know what you want to look for and can trigger on a specific item. Trying to say "something went wrong about 6:18 on that date, tell me everything matching these wildcards that happened on any AD server" or some such appears out of scope of Zabbix. Emphasis on "appears".

But that's the reason for my question: I am expecting to once again delve into this (the prior project was cancelled). So I'm curious if there are people with moderately large environments who find Zabbix alone (or probably with a syslog server) adequate?
Reply With Quote
  #4  
Old 28-11-2017, 17:11
andris andris is offline
Senior Member
 
Join Date: Feb 2012
Location: Riga, Latvia
Posts: 127
Default

Ok, you're interested in MS Windows event log. It has not received so much attention recently as log[], log.count[], logrt[], logrt.count[] items which are working with text log files (on Unix and MS Windows).
Reply With Quote
  #5  
Old 28-11-2017, 17:17
Linwood Linwood is offline
Senior Member
 
Join Date: Dec 2013
Location: Cape Coral, FL, USA
Posts: 257
Default

The two biggest problems I usually have to deal with are windows event logs, and ASA (Cisco firewall) event logs, as both tend to have huge volumes, and quite often the thing you are looking for is something new, that you haven't seen before, or is a combination of several events.
Reply With Quote
  #6  
Old 30-11-2017, 12:52
GPegel GPegel is offline
Senior Member
Zabbix certified professionalZabbix certified specialist
 
Join Date: Dec 2015
Location: The Netherlands
Posts: 102
Default

Maybe this is not the answer you are looking for but I manage to have more than 50 milion messages every 24 hour by using Logstash, Elasticsearch and Kibana. With of course a output plugin to Zabbix. In this case I'm able to see ALL logging (windows event logs and network hardware logs) and I'm able to define (using Grok rules) what I want to see exactly and when a certain type of message is passing by, the message will be 'forwarded' to Zabbix to collect the metrics. And based on those metrics I've build some triggers to send alerts.

I know Zabbix is able to look into log files but for our company that is not enough.
Reply With Quote
  #7  
Old 30-11-2017, 16:02
Linwood Linwood is offline
Senior Member
 
Join Date: Dec 2013
Location: Cape Coral, FL, USA
Posts: 257
Default

Quote:
Originally Posted by GPegel View Post
Maybe this is not the answer you are looking for...
No, it is. I'm curious how many people with larger needs, using zabbix, use it for logging vs other tools, and why. I think notable in yours is you collect them all, vs. looking for specific events only.
Reply With Quote
  #8  
Old 30-11-2017, 16:16
GPegel GPegel is offline
Senior Member
Zabbix certified professionalZabbix certified specialist
 
Join Date: Dec 2015
Location: The Netherlands
Posts: 102
Default

I think your curiousity is fair but I'm afraid Zabbix is not able to compete with other tools like the Elastic products when we are talking about log file monitoring. For example, I need to see real time data because messages are generated in thousands of seconds. In case of Zabbix it is going to look into the log files based on an interval and for us that is to slow. And another thing, some logfiles are creating messages that consists of just one sentence, but other log files are creating messages that consists of 30.000 sentences and I've tested this with Zabbix and Zabbix is not going to handle that.

Last edited by GPegel; 30-11-2017 at 16:18.
Reply With Quote
  #9  
Old 30-11-2017, 16:23
Linwood Linwood is offline
Senior Member
 
Join Date: Dec 2013
Location: Cape Coral, FL, USA
Posts: 257
Default

Quote:
Originally Posted by GPegel View Post
I think your curiousity is fair but I'm afraid Zabbix is not able to compete with other tools like the Elastic products when we are talking about log file monitoring. For example, I need to see real time data because messages are generated in thousands of seconds. In case of Zabbix it is going to look into the log files based on an interval and for us that is to slow.
I don't think it is about competition per se, but trying to find what people have succeeded at with it, or failed and moved to other tools.

But your comment intrigues me - when you say it is too slow because of an interval based access, do you mean alerts come in with too much lag? Because human lag (i.e. from getting alert to being able to take action) is awfully slow itself. Are you speaking of too slow to alert?

Or that because it reads only on intervals it has too much back log and falls progressively further behind and cannot keep up?
Reply With Quote
  #10  
Old 08-12-2017, 14:46
GPegel GPegel is offline
Senior Member
Zabbix certified professionalZabbix certified specialist
 
Join Date: Dec 2015
Location: The Netherlands
Posts: 102
Default

Indeed, Zabbix can't keep it up because the messages arrives with a speed of more than 1400 messages per second. And as I said, some messages consist of just one sentence but I also have lots of messages containing a max of 30.000 sentences.

And when such a 'big' message arrives including 30.000 sentences I also need to index a whole bunch of fields who I have configured using GROK rules. And those fields are being used by different teams to gather 'their' data to put it into a dashboard like Grafana.

To be honest, I do use Zabbix loging capabilities but only for small log files who are altered just once every minute or so. But not for real time data.

About the alerts, you are right... a human is the slowest factor in this case. And no monitoring system will fix that ;-) Or, in some cases, I use some extra actions so the human factor is becoming obsolete after a trigger fires ;-)

Last edited by GPegel; 08-12-2017 at 14:54. Reason: I was wrong about the speed, re-calculated it again at my TI-84+
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 19:44.