Java Gateway mod for configurable endpoints, Jolokia + JMX Operations

[rrupp]

This is a project I started for my company to get the Zabbix Java Gateway to fulfill requests using Jolokia (jmx-->http webservices), link here - https://bitbucket.org/ryanrupp/zabbi...eway/wiki/Home - Jolokia has a nice set of features, essentially it makes your Java servers JMX data available via HTTP webservices (REST type requests). Additionally, it has other nice features such as bulk requests (bundling all JMX attribute requests into a single roundtrip) which is ideal for monitoring a large amount of attributes or going over a higher latency network.

Basically this is just a custom Zabbix Java Gateway that you can deploy that has the following features:

• Allows the JMX endpoint to be configured
• Allows Jolokia to be used instead of standard JMX communication
• Allows JMX operations to be used when using Jolokia
• Publishes metrics via JMX about the Java Gateway's performance

The wiki I linked has more information. This is currently a work in progress but it's functional right now if anyone wanted to check it out.

[nnoori]

Hey Ryan,

Thank you very much for sharing your project, we are doing some work along the same line but by using a Zabbix Ruby API's and and Jolokia Ruby API's to help us with auto-discovery for JMX mBeans. You mentioned on the Wiki page that your company is using JMX operations for auto-discovering JMX mbeans can share more about that topic!
Thanks.. Nadia

[rrupp]

Essentially, we modify the application server to have some JMX operations that return the expected JSON output for Zabbix discovery to help find MBeans that are dynamic. An example would be if you look at the garbage collector mbeans they have the same attributes but the JMX object name is dynamic (depends on what JVM you're using/JVM arguments that control the garbage collector used), so the Object Name is like this:

java.lang:type=GarbageCollector,name={#GC_NAME}

We use the macro {#GC_NAME} for discovery, then on the application server we have a JMX operation that when invoked returns the JSON output for discovery listing all the GC_NAMES something along the lines of:

"data": [ {"{#GC_NAME}":"ConcurrentMarkSweep"},
             {"{#GC_NAME}":"ParNew"} ]

We like using JMX operations (via the modified Java Gateway we have here) because then we don't have to deploy additional scripts on Zabbix. You could do the same thing though with external scripts like I'm guessing you're doing with Ruby + Jolokia to search for certain MBeans.

In the future I'm looking to add better support to the Zabbix Java Gateway for the key "jmx.discovery" where you can past it a wildcard to discover object names, for instance in that memory example you would do:

"jmx.discovery[java.lang:type=GarbageCollector,name=*]"

and you'd get the JSON response:

"data": [ {"{#OBJ_NAME}":"java.lang:type=GarbageCollector,name=ConcurrentMarkSweep"},
             {"{#OBJ_NAME}":"java.lang:type=GarbageCollector,name=ParNew"} ]

[rrupp]

I added a new discovery option that allows you to find MBeans based on a wildcard search - this is useful to address the situation mentioned in my previous posts (different JVMs/Garbage Collectors have different names for their Garbage Collector/Memory Pool MBeans) I've detailed it in the project wiki under "New Discovery Options" - https://bitbucket.org/ryanrupp/zabbi...eway/wiki/Home - it works for both standard JMX and Jolokia.

Also, added some other minor changes like support for array attributes - if you try to read an array attribute it will concatenate the entries together with newlines in between, so you would have a "Text" item in Zabbix. This is useful for collecting information like for Garbage Collector MBeans there is the attribute "MemoryPoolNames" which specifies which memory pools the particular garbage collector is managing.

[iantivey]

Ryan,

thanks for sharing this code.

I'm trying to use your code to connect to an alternative JMX endpoint. Unfortunately my Zabbix front end is running on port 8088, so the API is unable to connect. I had a quick look through your code and can see there's no port specified on the API call to the JSON PHP script.

How quickly would you be able to add a configuration parameter to specify host and port for the API connection? I'd download and attempt it myself, but it would take me some time to get a development environment up and running to be able to compile.

Thanks!
Ian

[rrupp]

Changes to configure the API hostname/port are available now on the repository (download link updated as well) - configured through settings.sh

[iantivey]

Brilliant, thanks Ryan. Downloading it now, will try and get it running later today and feedback.

[iantivey]

That worked perfectly, thanks Ryan.

Had another thought whilst implementing it, that it might also be useful to be able to change the URL suffix to connect to (at the moment I think "/zabbix" is hardcoded).

[rrupp]

At some point soon I'll address the hardcoding of the frontend URL and just have the full URL entered as a configuration setting. I've added some additional features to this modified Zabbix Java Gateway, most recently I've added improved security so that JMX passwords that are entered via the frontend can now be encrypted instead of existing in plain text. This is useful if some people have access to your Configuration screens which is OK but you don't want everyone to actually know the JMX password. For more information see "Improved Security" on the wiki - https://bitbucket.org/ryanrupp/zabbi...eway/wiki/Home

For example:

Plain text password in a JMX item: "mypassword"
Encrypted password in a JMX item: "|ZBX|IxjOX3RROTgxYYrR/Z0THQ==***EjvpU2J/b9Dmf03AZ6gH2w=="

[BDiE8VNy]

ZBXNEXT-1660 would improve security for credentials in general.
It's about having no clear-text passwords stored anywhere except RAM - even no key or key-encryption-key for encrypted clear-text passwords.

Just wanted to attract attention for a general solution

[kevinwalter]

Any updates on this java gateway mod?

I was still using this version but it broke after the upgrade to zabbix 3.4. I'm monitoring my applications using Jolokia.

2018-01-24 10:29:38.327 [pool-1-thread-85] WARN com.zabbix.gateway.SocketProcessor - error processing request: JSONObject["conn"] not found.

[sshaikh]

Any updates on this java gateway mod?

I was still using this version but it broke after the upgrade to zabbix 3.4. I'm monitoring my applications using Jolokia.

2018-01-24 10:29:38.327 [pool-1-thread-85] WARN com.zabbix.gateway.SocketProcessor - error processing request: JSONObject["conn"] not found.

Hi Kevin,

I ran into the same issue myself. Were you able to find a fix for this issue?

Regards,
sshaikh

[kevinwalter]

Hi Kevin,

I ran into the same issue myself. Were you able to find a fix for this issue?

Regards,
sshaikh

Nope, we have converted to full JMX support instead of Jolokia.

[MichaelAlanS]

I am attempting to monitor Apache SOLR metrics with the JMX Gateway, and have attempted many things to get it to work, but still, no data flows into the Zabbix server. The gateway is running on the SOLR server. Everything seems to be up to date, I’ve followed the instructions in the most recent manual, insights on the web, you name it, I’ve done it. Yet, still... no data comes in. I’ve put the gateway into trace mode, so the logs are as verbose as possible. Getting odd errors in the gateway log on the SOLR server, including time outs, although I’ve configured the timeout to be at the maximum.

I can find nothing about the following error:

java.io.IOException: Received an invalid message length of 1514297412

“Abrupt Connection”

Any ideas? Thank you!