Large systems log filtering

[Linwood]

Zabbix can do log files, yes. But to me it's awkward and a bit kludgy.

Then there are tools like Graylog, which are deep and thorough, but are a completely different thing to setup and maintain. Yes, they can feed to Zabbix for consistent alerting, but it's still another substantial tool.

My question is this: For those with moderate and larger systems and a need for widespread log monitoring (say all servers, network devices like switches and routers, and firewalls), what are you using?

Or put another way, how many have successfully used Zabbix without introducing another tool into the mix?

Please note I'm not asking "does Zabbix work" -- yes, I know it works. I'm asking a practical question: do you find zabbix more work to make it work, than adopting another tool like Graylog?

[andris]

May I ask what features are missing in Zabbix log[], log.count[], logrt[], logrt.count[] items ?
What would you recommend to add ?
(disclosure: I'm involved in their development)

[Linwood]

That's a fair question and honestly I do not know, now. I remember trying to deal with windows system logs a while back, and giving up as almost unmanageable, but memory fails as to the details. I also did do a trial Graylog install and had much less trouble, again memory fails as to details of what made it less trouble.

One aspect is research after the fact, my impression is Zabbix is manageable if you know what you want to look for and can trigger on a specific item. Trying to say "something went wrong about 6:18 on that date, tell me everything matching these wildcards that happened on any AD server" or some such appears out of scope of Zabbix. Emphasis on "appears".

But that's the reason for my question: I am expecting to once again delve into this (the prior project was cancelled). So I'm curious if there are people with moderately large environments who find Zabbix alone (or probably with a syslog server) adequate?

[andris]

Ok, you're interested in MS Windows event log. It has not received so much attention recently as log[], log.count[], logrt[], logrt.count[] items which are working with text log files (on Unix and MS Windows).

[Linwood]

The two biggest problems I usually have to deal with are windows event logs, and ASA (Cisco firewall) event logs, as both tend to have huge volumes, and quite often the thing you are looking for is something new, that you haven't seen before, or is a combination of several events.

[GPegel]

Maybe this is not the answer you are looking for but I manage to have more than 50 milion messages every 24 hour by using Logstash, Elasticsearch and Kibana. With of course a output plugin to Zabbix. In this case I'm able to see ALL logging (windows event logs and network hardware logs) and I'm able to define (using Grok rules) what I want to see exactly and when a certain type of message is passing by, the message will be 'forwarded' to Zabbix to collect the metrics. And based on those metrics I've build some triggers to send alerts.

I know Zabbix is able to look into log files but for our company that is not enough.

[Linwood]

Maybe this is not the answer you are looking for...

No, it is. I'm curious how many people with larger needs, using zabbix, use it for logging vs other tools, and why. I think notable in yours is you collect them all, vs. looking for specific events only.

[GPegel]

I think your curiousity is fair but I'm afraid Zabbix is not able to compete with other tools like the Elastic products when we are talking about log file monitoring. For example, I need to see real time data because messages are generated in thousands of seconds. In case of Zabbix it is going to look into the log files based on an interval and for us that is to slow. And another thing, some logfiles are creating messages that consists of just one sentence, but other log files are creating messages that consists of 30.000 sentences and I've tested this with Zabbix and Zabbix is not going to handle that.

[Linwood]

I think your curiousity is fair but I'm afraid Zabbix is not able to compete with other tools like the Elastic products when we are talking about log file monitoring. For example, I need to see real time data because messages are generated in thousands of seconds. In case of Zabbix it is going to look into the log files based on an interval and for us that is to slow.

I don't think it is about competition per se, but trying to find what people have succeeded at with it, or failed and moved to other tools.

But your comment intrigues me - when you say it is too slow because of an interval based access, do you mean alerts come in with too much lag? Because human lag (i.e. from getting alert to being able to take action) is awfully slow itself. Are you speaking of too slow to alert?

Or that because it reads only on intervals it has too much back log and falls progressively further behind and cannot keep up?

[GPegel]

Indeed, Zabbix can't keep it up because the messages arrives with a speed of more than 1400 messages per second. And as I said, some messages consist of just one sentence but I also have lots of messages containing a max of 30.000 sentences.

And when such a 'big' message arrives including 30.000 sentences I also need to index a whole bunch of fields who I have configured using GROK rules. And those fields are being used by different teams to gather 'their' data to put it into a dashboard like Grafana.

To be honest, I do use Zabbix loging capabilities but only for small log files who are altered just once every minute or so. But not for real time data.

About the alerts, you are right... a human is the slowest factor in this case. And no monitoring system will fix that ;-) Or, in some cases, I use some extra actions so the human factor is becoming obsolete after a trigger fires ;-)