anomaly detection on rising counters

[slyper]

Hello,

We're monitoring some custom numeric data, configured through UserParameters.
The custom data represents the number of api calls that has been used by a certain customer.
We would like to monitor the data, and get notified if some client has unusually high or low usage, compared to 'previous normal usage'.

For this we defined a UserParameter script in the Zabbix agent which can collect the usage data:

UserParameter=custom_data[*],"./custom_data.sh $1 $2"​

In the Zabbix template ApiCalls, which we attach to the customers API server, we define an item with key:

custom_data[{$SOME_MACRO},api_calls]

This data is collected as expected, and on this we would like to do anomaly detection.
Since the numeric data is always incrementing, we cannot use trend functions directly (we suppose).

We were thinking about creating a Calculated Item first based on the apicalls item, which represents the change between the current and previous value.
As key we use:

custom_data[{$SOME_MACRO},api_calls_diff]

As formula we use:

change(/ApiCalls/custom_data[{$SOME_MACRO},api_calls])

This gets item gets created, but on a host with the template configured, we see the item is not supported with the message:

Cannot evaluate function: item "/ApiCalls/custom_data[{$SOME_MACRO},api_calls]" does not exist at "change(/ApiCalls/custom_data[{$SOME_MACRO},api_calls])".

We use the same notation for some triggers on other custom_data entries, which works fine.

We would like to use trendstl or similar to detect anomalies in the usage data.

Our questions for you:
- how can we calculate the change of a custom item (including a macro and another parameter as input to UserParameter)?
- how do we monitor trends of always rising counters (do we need to calculate change first, or is there another way)?

kind regards

[LenR]

You don't need to calculate first, the same operations can be used in a trigger. Without syntax checking, something like: custom_data[{$SOME_MACRO},api_calls].avg(1d)*1.25<custom_data[{$SOME_MACRO},api_calls].last()

That would be the last value was 25% higher than the average for the past day.

[slyper]

Thanks for your reply, LenR.

The usage data of our customers depend heavily on time of the day and the day of the week (heavy traffic in the morning, almost nothing in weekends).
Also we would like to get notified more rapidly, like every ten minutes. If api calls stop working for some reason, we'll get notified this way.
As the usage is different for every customer and is very different based on the time of day, we were thinking about using the new ML algorithms like trendstl, to not have to define a fixed threshold, but we're struggling in configuring it that way.
So any help there is appreciated.