Monitoring of application-generated alerts via JSON

[4f1sh3r]

Dear Zabbix Community

I am struggling to monitor alerts created from an external application. I am able to receive the alerts as JSON via HTTP-Agent Item.

The JSON-String has the following format:

{
"AlertLog": [
{
"@Name": "Alert 1",
"@TimeStamp": "1629733872506",
"@Severity": "2",
"@Node": "node 1",
"@Description": "Description of Alert 1",
"@AlertSentTo": "",
"@AlertGroupID": "System"
},
{
"@Name": "Alert 2",
"@TimeStamp": "1629733842503",
"@Severity": "2",
"@Node": "node 1",
"@Description": "Description of Alert 2",
"@AlertSentTo": "",
"@AlertGroupID": "System"
},
{
"@Name": "Alert n",
"@TimeStamp": "1629733842503",
"@Severity": "2",
"@Node": "node 2",
"@Description": "Description of Alert n",
"@AlertSentTo": "",
"@AlertGroupID": "System"
}
]
}

My goal is to have separate Problem-Events in Zabbix for each alert.

##First Attempt##
My first attempt was via a single Item and a single Trigger with "multiple problem events generation" enabled.
I used this as the trigger expression: find(/template/item.key,5m,"regexp",""@TimeStamp":"(\d+)"")=1

I managed to extract the timestamp into a Tag via {{ITEM.VALUE}.regsub(""@TimeStamp":"([^"]*)","\1")} and I used that for comparison when to close events.
Unfortunately, this didn't work well. New alerts got created each time the item got updated. They also never got closed. I guess this "multiple problem events via a single trigger" feature only works reliably with log files.


##Second Attempt##
My second attempt was to create several dependent items, to be used as alarm "slots". I preprocessed each sub-item with jsonpath like this:
$.AlertLog[0] in the first item
$.AlertLog[1] in the second item
and so on.

I enabled "custom on fail" and set the value to "none". I then created triggers for each item, which fire, as soon as there is not "none" as value.

This worked well, but as soon as one alert disappears in the middle of the JSON, the remaining ones will move into different slots. So I had the issue that Problem-Events didn't have the right alarm anymore.

##Third Attempt##
My third attempt is to try it via discovery. Here, however, I have a knot in my brain, because I don't have anything to query using the Item-Prototypes.
All the data is in the JSON string, which I have to query in the discovery rule itself. Unfortunately, I can't create a "dependent item prototype" which uses the discovery rule as Master-Item.

I guess that the discovery is the best way. How do I have to create the item prototypes then?

Does anyone here monitor application-generated alarms via Zabbix and can give me some hints on how this works best?
Any help would be much appreciated. I love Zabbix but this problem is driving me crazy. :-)

Best regards
Alex

[Hamardaban]

See how the templates from zabbix are made.
They have a “basic” item (returning json) which is the basis for lld discovery (depended item) AND for item prototypes (depended item).

[4f1sh3r]

Wow, that was a good hint! I was totally unaware that this is actually possible. :-)
Thanks, it worked perfectly fine.