Item for finding duplicate lines in a log file

[erezsandel]

Hello everyone,
The use case I have in a log file that is being inflated in size due to multiple ERROR entries. I am trying to monitor this with the log.count item but it does not work. It does not detect new lines being written, or pops the trigger made by this item.

A couple questions:
1. Is the item configured correctly?
2. Will this item detect input that is not strictly ERROR, but also a full log entry which contains the ERROR header?
3. What is the limitations of new lines per second zabbix can handle?
4. Is this the optimal way of doing what I want? Is there a different item which is better suited(log, system.run, etc.)

[Hamardaban]

1) it will show the number of lines added to the log in 30 seconds that contain the word ERROR. Is this result expected?
2) No - each line is processed separately
3) i dont know... try asking for documentation ...

[erezsandel]

1)So, lets say the log input is the following line:
00:31:23.335 ERROR [Erez_Puller] SBEBinaryFixMessageDecoder - Couldn't find message schema for message ErezMessageHeader{, msgSize=0, blockLength=0, templateID=0, schemaID=0, version=0}
Will Zabbix identify this entry?

[cyber]

1. No syntax error there.. Just for some parameters there, "all" and "0".. those are defaults anyway, so you don't need to enter them and can just close parameter list after 5000.

3. it said in item docs...
maxproclines - maximum number of new lines per second the agent will analyze (cannot exceed 10000). Default value is 10*'MaxLinesPerSecond' in zabbix_agentd.conf.
default in config is 20, so 200 lines. You have set it to 5000 in parameters, overriding defaults. I actually do not know, will it multiply it by check interval also or not...:P But I have never tried even to increase the maxproclines, as I never needed it.. Usually logfile checks are left with 1s interval...

But this just find ERROR-s, not the duplicates... Only, if you are certain that all errors are duplicates...

[Hamardaban]

The essence of the problem is to determine what "duplication" is.
For example, a situation with 2 identical line in a row can be detected by comparing the current and previous value of the log key (by extracting the necessary information via regex) in the trigger.

But if the same lines are separated by other lines, then you can't do it without using the Event correlation mechanism. Or write some kind of external handler.

[cyber]

erezsandel commented
Today, 10:14
1)So, lets say the log input is the following line:
00:31:23.335 ERROR [Erez_Puller] SBEBinaryFixMessageDecoder - Couldn't find message schema for message ErezMessageHeader{, msgSize=0, blockLength=0, templateID=0, schemaID=0, version=0}
Will Zabbix identify this entry?

yes it should find this, as it contains ERROR, what you set in parameters as regex to be found.