Agent log file limit

[Manish]

Hi there,

I am using zabbix version 2.4.7 and monitoring agent log file but when log file size incresed it stop processing and say zabbix_not supported. I can see in zabbix doc that from version 2.0 they have added large file support greater than 2Gb then why this is happening, any other setting are there to enable large file support?

Thanks fast reply really appreciated.

[BDiE8VNy]

Is it for sure that you've used Zabbix agent binaries compiled for the amd64 architecture?

[Manish]

Hi, Thanks for your reply. Yes , I have used compiled binaries.

[BDiE8VNy]

The question was actually about whether a binary compiled for the right platform (amd64 aka 64-Bit) has been used. Not whether it was (pre-)compiled

[Manish]

Hi,
sorry I was not specific on answer. yes it was compiled for 64 bit system.

[Manish]

Here it is showing on zabbix agent
Installed Packages
zabbix.x86_64 1.8.22-1.el6 @epel

[BDiE8VNy]

Zabbix 1.8 is not supported anymore. If I recall right the mentioned fix/improvement in the Zabbix agent was implemented for >= 2.0 only.

[Manish]

Oh you are right. on server 2.4.7 installed and on agent this version. This was the reason. Thanks I will upgrade agent and let you know.

[Manish]

Hi, Thanks I have upgraded zabbix-agent to version > 2.0 and its reading file more that 2GB size.

Thanks again.

I need one more help. Since I am monitoring log file I need trigger which will count occurance of specified string in 5min and if its more than 50 then alert high. How do I acheive this?

[BDiE8VNy]

Well there is a rather resource intensive supported way and a very resource efficient but not yet supported way.

The former would be to configure a Log item to send e.g. the value "hit" for each occurrence of the specified string:
log[/path/to/log/file,specified string,,,,hit]

A Trigger may then check the count of values received during a period of time:
{example.com:log[/path/to/log/file,specified string,,,,hit].count(5m)} > 50

The obvious downside of doing it that way is the matter of fact that calculation happens on Zabbix server. So, for each specified string occurrence has a value to be send to Zabbix server which also consumes database resources. When there are only a few lines expected, then this should not matter. But if there are many "hits" at constant rate, this may become a problem in respects to database resources. To mitigate the database impact a bit one could think of setting the Item's history storage period to 0. But this comes at the expense of not having any value available via the frontend/API anymore.

A much more efficient approach is to let the Zabbix agent itself do the calculation and return the result in the frequency of the Item update interval.
See ZBXNEXT-2430 for a patch that implements that.

When using a Item update interval of 5m the Item key and Trigger expressionmay look like this:
log[/path/to/log/file,specified string,,,,,count]
{example.com:log[/path/to/log/file,specified string,,,,,count].last()} > 50

Or when wanting to use a more frequent update interval like 60s, then the Trigger expression may look like that:
{example.com:log[/path/to/log/file,specified string,,,,,count].sum(5m)} > 50

[Manish]

Hi,

Thanks for reply. I tried your suggesion like log[/path/to/log/file,specified string,,,,,count] log[/path/to/log/file,specified string,,,,hit] but both are saying not supported item by zabbix. So I tried like {log[/path/to/log/file,skip].count(300,ERROR,eq)}>50 and it accepted by zabbix.

But problem is its not giving me alert. I can see there are many error in last 5 min in my log file but I didnt get alert high.

I have not tried patching log file may be after patch the option which u have suggested will work but I guess patch has to be applied on agent level and not on zabbix server level so if I patch still same error message like item not supported will come.

[BDiE8VNy]

Sorry, I possibly should have pointed that out more clearly.

The latter mentioned "much more efficient approach" was the not yet supported one and has indeed to be patched into the Zabbix agent first to be used.
Btw, consider to vote for ZBXNEXT-2430 to push its implementation.

Futher the "Ability to extract matching part of a regular expression" is supported since Zabbix agent 2.2. So when the Item from my first example gets "Not supported", then the agent is likely of a lower version. If so, then just upgrade the Zabbix agent and it should work then.

The reason why your Trigger does not fire even though log messages including the string "ERROR" could be the fact that the way you used the count() trigger function it counts values that match exactly "ERROR".
Assuming the returned values by your Log item include "ERROR" but do not consist of it entirely, you should probably try either count(300,ERROR,like) or even shorter count(5m,ERROR) - as 'like' is the default for string, text and log items.

However, doing it the way you currently try causes the Zabbix agent to send the entire log file to Zabbix server/database. What is not necessary to just fulfill your requirement.
You can avoid sending unrelated log messages by moving the "ERROR" condition from the trigger function to the Item.

Anyway, I suggest to upgrade the Zabbix agent instead as this allows to let the Zabbix agent send a very short value like "hit" or even only "1" independent from the pattern to consider for sending.

[Manish]

Hello,

As per your suggesion I upgraded zabbix agent to 2.4.4 and its working. I can use both hit and count in item parameter. But now I am confused.

As you said downside with using hit is it consume DB resource. Just wanted to know in my example "log[/path/to/file,ERROR,skip].count(300)}>50" what would be impact on DB. Reason why I am asking this gives me proper alert.

So I wonder should I patch or continue using above trigger.

One more thing I would like to highlight is since I moved string to search for alert in item parameter I get only that thing in latest data. Means I can see only error and not other thing in latest data.

[BDiE8VNy]

The 'hit' parameter was supposed to be the 6th one while the 'count' parameter was supposed to be the 7th one. The latter one is not native supported by Zabbix agent but became supported after applying the mentioned patch to the source code and compiling the Zabbix agent afterwards.

When using the approach with the un-patched Zabbix agent (the 'hit' resp. '0' approach), then a value ('hit'/'0') is send from the agent to the Zabbix server/database for each and every matching log line. The idea behind this is to not send the entire log line (tenth or hundreds of Bytes) to Zabbix server/database just for counting occurrences.

For instance, think of a log item that is configured to investigate a log file every 5 seconds. The log file gets 1000 'ERROR' lines per second written.

With the 'hit' approach, the Zabbix agent may send 5000 'hit' strings per item interval to the Zabbix server which, finally stores these values also to the Zabbix database.

With the 'count' approach, the Zabbix agent will send just one value per interval - the sum of matching 'ERROR' lines already counted by the agent.

Since the question was "[...] count occurance of specified string in 5min and if its more than 50 then alert high" I proposed a resource efficient approach.

[Manish]

Hi,

I really appreciate your suggestion and I am working accordingly. Per your suggestion I applied patch and its working.

Now I need last help I want to search many string in one trigger. For example.
log[path/to/file,string1 or string2 or string3,,,skip,hit].count(300)}>50 then it must count occurance of each and every string in log file and trigger alert. I tried this solution but I think this trigger taking logical expression also as string.