Yet another JMX configuration issue

[jerelev]

Hi,
I've read (almost) every post here and oracle docs, and i still can't manage to have my java application connected to zabbix.
So here is my config:

Zabbix server is also java gateway
Server configuration:

JavaGateway=127.0.0.1
JavaGatewayPort=10052
StartJavaPollers=5

Java Gateway default config:

# cat /etc/zabbix/zabbix_java_gateway.conf
# LISTEN_IP=127.0.0.1
# LISTEN_PORT=10052
PID_FILE="/var/run/zabbix/zabbix_java_gateway.pid"
START_POLLERS=5
TIMEOUT=3

My application is on another server, it's quickbuild so the configuration is not the same as "simple" tomcat with JAVA_OPTS
It uses a wrapper and that's the way it's configured:

# Java Additional Parameters
wrapper.java.additional.1=-Djava.awt.headless=true
wrapper.java.additional.2=-Xverify:none
wrapper.java.additional.3=-Djavax.net.ssl.trustStore=/nfs/quickbuild/ssl/keystore
wrapper.java.additional.4=-XX:MaxPermSize=2048m
wrapper.java.additional.5=-XX:+UseParallelGC
wrapper.java.additional.6=-DmaxThreads=512
# Use 10% of cpu to garbage collect
wrapper.java.additional.7=-XX:GCTimeRatio=9
[B]wrapper.java.additional.8=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12340 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false[/B]

And we can see it in the process command:

# ps aux |grep jmx
18411    31384  9.5  3.5 11267792 575444 ?     Sl   15:59   0:49 java -Djava.awt.headless=true -Xverify:none -Djavax.net.ssl.trustStore=/nfs/quickbuild/ssl/keystore -XX:MaxPermSize=2048m -XX:+UseParallelGC -DmaxThreads=512 -XX:GCTimeRatio=9 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12340 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ....

My host is configured with the right port : 12340

No firewall configured.

And yet the error from the log:

2017-03-07 16:23:07.936 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"bad protocol header: 7B 22 72 65 71"}                                            
2017-03-07 16:23:07.937 [pool-1-thread-5] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection                                                                                                               
2017-03-07 16:24:01.155 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection                                                                                                               
2017-03-07 16:24:01.155 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header                                                                                                                         
2017-03-07 16:24:01.156 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length                                                                                                                         
2017-03-07 16:24:01.156 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 91 bytes of request data                                                                                                                       
2017-03-07 16:24:01.156 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request":"java gateway jmx","conn":"10.184.152.33","port":12340,"keys":["jmx.discovery"]}
2017-03-07 16:24:01.156 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
2017-03-07 16:24:01.156 [pool-1-thread-1] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://[10.184.152.33]:12340/jmxrmi'
2017-03-07 16:24:01.159 [pool-1-thread-1] WARN  com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.184.152.33; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)]
        at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:97) ~[zabbix-java-gateway-3.2.4.jar:na]
        at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:62) ~[zabbix-java-gateway-3.2.4.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_121]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_121]
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.184.152.33; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370) ~[na:1.7.0_121]
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268) ~[na:1.7.0_121]
        at com.zabbix.gateway.ZabbixJMXConnectorFactory$1.run(ZabbixJMXConnectorFactory.java:76) ~[zabbix-java-gateway-3.2.4.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_121]
        ... 3 common frames omitted
Caused by: javax.naming.ServiceUnavailableException: null
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[na:1.7.0_121]
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203) ~[na:1.7.0_121]
        at javax.naming.InitialContext.lookup(InitialContext.java:411) ~[na:1.7.0_121]
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1928) ~[na:1.7.0_121]
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1895) ~[na:1.7.0_121]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.7.0_121]
        ... 7 common frames omitted
Caused by: java.rmi.ConnectException: Connection refused to host: 10.184.152.33; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.7.0_121]
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.7.0_121]
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.7.0_121]
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341) ~[na:1.7.0_121]
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.7.0_121]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:128) ~[na:1.7.0_121]
        ... 12 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_121]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_121]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_121]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_121]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_121]
        at java.net.Socket.connect(Socket.java:576) ~[na:1.7.0_121]
        at java.net.Socket.connect(Socket.java:525) ~[na:1.7.0_121]
        at java.net.Socket.<init>(Socket.java:425) ~[na:1.7.0_121]
        at java.net.Socket.<init>(Socket.java:208) ~[na:1.7.0_121]
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.7.0_121]
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147) ~[na:1.7.0_121]
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[na:1.7.0_121]
        ... 17 common frames omitted
2017-03-07 16:24:01.159 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.184.152.33; nested exception is: \n\tjava.net.ConnectException: Connection refused (Connection refused)]"}
2017-03-07 16:24:01.159 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection

[jerelev]

I've tested with the script suggested here https://www.zabbix.org/wiki/ConfigureJMX

# ./test.sh localhost 10052 10.184.152.33 12340 'jmx[\"java.lang:type=Threading\",ThreadCount]'
{"response":"failed","error":"bad protocol header: 7B 22 72 65 71"}

NB: netstat on the server running quickbuild is not showing the port 12340 listening.


Voila.
If someone knows why... it will help me a lot!

[jerelev]

Thank you all for your readings, the solution was pretty "simple".
I've broken the configuration line into multiple:

wrapper.java.additional.8=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12340 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

wrapper.java.additional.8=-Dcom.sun.management.jmxremote
wrapper.java.additional.9=-Dcom.sun.management.jmxremote.port=12345
wrapper.java.additional.10=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.11=-Dcom.sun.management.jmxremote.ssl=false

Then I could see the listening port and see the connection:

# netstat -anp |grep 1234
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:12345           0.0.0.0:*               LISTEN      3190/java
tcp        0      0 10.184.152.33:12345     10.184.152.198:58299    ESTABLISHED 3190/java

You can mark it as resolved and it can be usefull for someone else... maybe.

[vishnu.dinoct]

All checks go to unknown status for jmx

2017-05-07 15:55:05.962 [pool-1-thread-2] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"\"bad protocol header: 6A 6D 78 5B 62\""}
2017-05-07 15:55:05.962 [pool-1-thread-2] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection
2017-05-07 15:55:06.067 [pool-1-thread-3] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2017-05-07 15:55:06.067 [pool-1-thread-3] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2017-05-07 15:55:06.068 [pool-1-thread-3] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: bad protocol header: 6A 6D 78 5B 62
at com.zabbix.gateway.BinaryProtocolSpeaker.getReques t(BinaryProtocolSpeaker.java:59) ~[zabbix-java-gateway-2.2.0.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:51) ~[zabbix-java-gateway-2.2.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

[kloczek]

2017-03-07 16:24:01.159 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"java.io.IOExcepti on: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.184.152.33; nested exception is: \n\tjava.net.ConnectException: Connection refused (Connection refused)]"}
2017-03-07 16:24:01.159 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection

"Connection refused" by system monitored by zabbix jmx client. Password or user issue on JVM side.

[vishnu.dinoct]

"Connection refused" by system monitored by zabbix jmx client. Password or user issue on JVM side.

Hi can you help me with this one please???



2017-05-07 15:55:05.962 [pool-1-thread-2] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"\"bad protocol header: 6A 6D 78 5B 62\""}
2017-05-07 15:55:05.962 [pool-1-thread-2] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection
2017-05-07 15:55:06.067 [pool-1-thread-3] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2017-05-07 15:55:06.067 [pool-1-thread-3] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2017-05-07 15:55:06.068 [pool-1-thread-3] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: bad protocol header: 6A 6D 78 5B 62
at com.zabbix.gateway.BinaryProtocolSpeaker.getReques t(BinaryProtocolSpeaker.java:59) ~[zabbix-java-gateway-2.2.0.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:51) ~[zabbix-java-gateway-2.2.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]