How to make JMX triggers reliable

[yuri69]

Hello,
I got an app inside a Java runtime. The runtime exposes a JMX MBean for each deployed app. My Zabbix has got JMX items configured to read the MBean's attributes for each deployed app.

My problem is the proper trigger configuration on these JMX items. Naturally, I would like to fire a trigger/action when any of the deployed apps fails. This means I gotta check the item value and availability.

---

Naively, I thought it would be enough to add a trigger with nodata(T) = 1 and it would handle all the error states (undeployed app/MBean, unavailable JMX, shutdown runtime, etc.).

Nope, it doesn't work like that... When one of the apps crashed its MBean disappeared and the JMX item simply switched to Cannot evaluate function "hostname:jmx["foo","bar"].nodata(90)": item is not supported.

This item error state does not trigger the nodata(T) = 1 so we ended up with a broken app and no notification.

---

Is there a way to handle this error state?

[dimir]

You can use trigger dependencies. Item "service availability" will for example check if service is running on a port and will have "service down" trigger attached. The jmx trigger will depend on "service down" trigger.

[yuri69]

Item "service availability" will for example check if service is running on a port and will have "service down" trigger attached. The jmx trigger will depend on "service down" trigger.

The app inside the runtime does not listen on any port. The runtime itself exposes the JMX and thus listens on a JMX port. Thus when app dies the runtime port remains open.

Sure, I could add listening functionality to the application for the Zabbix purpose only but this solution sounds rather cluttering...

[dimir]

I agree. I'm really bad at Java/JMX thingies, I hope someone else can advise. And just in case you didn't know, you can get notifications on items going notsupported/triggers getting unknown state (Action source: internal event).

[yuri69]

I agree. I'm really bad at Java/JMX thingies, I hope someone else can advise. And just in case you didn't know, you can get notifications on items going notsupported/triggers getting unknown state (Action source: internal event).

Thanks for pointing me to the Actions - Event source - Internal. I will try to setup and test an action catching the JMX-related errors.

[yuri69]

Reporting back.

Employing the Internal Action works for catching the JMX item error states!

However, it's really cumbersome having to create another Inernal Action to each Trigger Action you have. So much duplication...

[dimir]

Usually it's enough to have single Action to catch all the cases of notsupported items and unknown triggers. Why do you need one for each trigger action? Unless specific description of the problem in notification I don't see a reason for that.

[yuri69]

Usually it's enough to have single Action to catch all the cases of notsupported items and unknown triggers. Why do you need one for each trigger action? Unless specific description of the problem in notification I don't see a reason for that.

I need multiple Actions in order to cover different combinations of Host Groups and Channels.

Host_group X is catered with Slack channel X
Host_group Y is catered with Slack channel Y
Host_group Z is catered with Email channel to User Group Z
etc.

[dimir]

Host group can be used in notification using {TRIGGER.HOSTGROUP.NAME} macro. For other things you can have host-level user macros. Then, if notification needs to be strictly formatted (based on values of macros) you can execute script instead of using Zabbix built-in email notification mechanism. It might be less complicated in the end than duplicating the Actions.