Zabbix Java Gateway doesn't work with ssl

[francesco.b]

Hi guys,
I'm struggling to make the zabbix java gateway work with ssl. I've configured a tomcat to open the jmx port using the following setenv:

export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9999"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.rmi.port=9999"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.password.file=jmxrem ote.password"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.access.file=jmxremot e.access"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=true"
export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=jmxkeystore.pkcs12"
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=[OMISSIS]"
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStoreAlias=[OMISSIS]"
export JAVA_OPTS="$JAVA_OPTS -Dsun.management.jmxremote.ssl.need.client.auth=tru e"
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.trustStore=jmxtrustore.pkcs12"
export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.trustStorePassword=[OMISSIS]"

There is no firewall between the target host and the zabbix java gateway. I'm sure the jvm is correctly configured because i can connect through an Oracle Java Mission Control only if I specify both the credentials and the keystore/trustedstore to use with the right values.
In the file /etc/zabbix/zabbix_java_gateway.conf I've added the following parameters to specify which certificates the zabbix java gateway has to use (they are the same used in the oracle java mission control):

JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.trustStore=/var/lib/zabbix/jmxtrustore.pkcs12"
JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.trustStorePassword=[OMISSIS]"
JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.keyStore=/var/lib/zabbix/jmxkeystore.pkcs12"
JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.keyStorePassword=[OMISSIS]"
JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.keyStoreAlias=[OMISSIS]"
JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.debug=ssl,handshake"

Unfortunately the jmx status of the host reports the error "non-JRMP server at remote endpoint: service:jmx:rmi:///jndi/rmi://[OMISSIS]:9999/jmxrmi" and in the java gateway log i find the following errors every time I check a jmx items

2020-06-30 16:22:18.348 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2020-06-30 16:22:18.349 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2020-06-30 16:22:18.349 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length
2020-06-30 16:22:18.349 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 202 bytes of request data
2020-06-30 16:22:18.350 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request":"java gateway jmx","username":"<[OMISSIS]","password":"[OMISSIS]","jmx_endpoint":"service:jmx:rmi:///jndi/rmi://[OMISSIS]:9999/jmxrmi","keys":["jmx["java.lang:type=Threading",PeakThreadCount]"]}
2020-06-30 16:22:18.350 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
2020-06-30 16:22:18.350 [pool-1-thread-1] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://[OMISSIS]:9999/jmxrmi'
2020-06-30 16:22:18.355 [pool-1-thread-1] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx["java.lang:type=Threading",PeakThreadCount]" failed: non-JRMP server at remote endpoint: service:jmx:rmi:///jndi/rmi://[OMISSIS]:9999/jmxrmi
2020-06-30 16:22:18.356 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - error caused by
com.zabbix.gateway.ZabbixException: non-JRMP server at remote endpoint: service:jmx:rmi:///jndi/rmi://[OMISSIS]:9999/jmxrmi
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:131) ~[zabbix-java-gateway-4.4.10.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:79) ~[zabbix-java-gateway-4.4.10.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1149) [na:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) [na:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2020-06-30 16:22:18.356 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"non-JRMP server at remote endpoint: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/[OMISSIS]:9999\/jmxrmi"}
2020-06-30 16:22:18.356 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection

Can someone help?
Cheers
Francesco

[Hamardaban]

Does the connection work from zabbix without ssl?

[francesco.b]

Hi Hamardaban,
without ssl the connection works, the java gateway server passes the authentication step and all jmx items receive their data.
I suspect the java gateway server tries to connect without ssl, because the oracle jmc also gives the same error of "non-JRMP server at remote endpoint" when i try a connection without configuring the keystore/trustedstore.
Moreover, if ssl is enable and I configure a wrong certificate in the keystore of the zabbix java gateway server, I don't receive an error like SSL mismatch or similar, which imo it means the java gateway server is not even trying to negotiate the ssl fase of the connection.

Francesco

[tiguchi]

I've been fighting this problem for a whole day and here is what I found out:

TL;DR

Set com.sun.management.jmxremote.registry.ssl to false but be aware of the security implications.

1. JConsole behaves the same as Zabbix Java Gateway. It gives the exact same error message
2. I debugged SSL connections and handshakes on the server side and Zabbix Java Gateway nor JConsole even attempted to establish a SSL connection. They try to access the JMX registry endpoint using an unencrypted channel, which causes an error on the server side.
3. In JConsole the problem can be prevented by using hostname : port syntax instead of service:jmx:rmi:///jndi/rmi://..... In that case JConsole must be connecting using a different mechanism, which forces a SSL connection
4. That workaround does not work in Zabbix. Zabbix requires the service:jmx:rmi:///jndi/rmi:// URI syntax
5. The problem can be reproduced when the JMX registry is configured to be SSL encrypted (via com.sun.management.jmxremote.registry.ssl=true)
4. The problem disappears when com.sun.management.jmxremote.registry.ssl=false

So to "fix" that problem, JMX registry SSL encryption needs to be disabled, which basically exposes object information, and possibly also user name and password but I'm not sure about that.