Ad Widget

Collapse

JMX monitoring: SSL peer shut down incorrectly

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • bofh666
    Junior Member
    • Jun 2021
    • 2
    #1

    JMX monitoring: SSL peer shut down incorrectly

    Hi there!

    I'm trying to monitor Java apps via JMX but got red square with mentioned error in web interface. For the sake of simplicity I've started to troubleshoot only Zabbix Java gateway itself, so uncommented:

    JAVA_OPTIONS="$JAVA_OPTIONS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345
    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.registry.ssl=false"

    at the bottom of config file and added JMX interface and Template App Generic Java JMX to Zabbix Server. The errors in /var/log/zabbix/zabbix_java_gateway.log look like:

    2021-06-25 21:14:16.528 [pool-2-thread-1] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx["java.lang:type=GarbageCollector,name=Copy",Co llec tionTime]" failed: SSL peer shut down incorrectly: service:jmx:rmi:///jndi/rmi://127.0.0.1:12345/jmxrmi

    There's two way PSK encryption between Zabbix Server and Zabbix Agent, I've tried to disable it just to ensure it doesn't affect JMX, and the error persists.

    It looks like JVM expects SSL by default, but it shouldn't because of startup parameters and management.properties contents. I've tried to connect to Java Gateway via JConsole and it warns about SSL too, but it's possible to ignore this warning and then the console shows everything just fine.

    I run out of ideas. Google is not of much help too. So any thoughts are greatly appreciated.

    My setup is:
    CentOS Linux release 7.9.2009 (SELinux and firewall are disabled)
    OpenJDK 1.8.0_292
    Zabbix server and Java gateway: 4.0.31
    Tags: jmx ssl
      😞 1
    • bofh666
      Junior Member
      • Jun 2021
      • 2
      #2
      Well, replying to myself.

      Setting -Djava.rmi.server.hostname to be equal to IP address where JMX connections from Zabbix Java Gateway are expected fixes the issue. The parameter name includes hostname but I suggest to put IP for reliability.

      After I've fixed monitoring of ZJG itself I fixed remote hosts. One of them is behind NAT, so I had to add -Dcom.sun.management.jmxremote.rmi.port parameter with value equal to jmxremote.port. And yes, it's OK that this port is used twice. In case with (D)NAT -Djava.rmi.server.hostname value should be equal to public IP, which JMX port of monitored host is being translated to.

      I don't think the latter parameter harms NAT-less setup, so the whole thing in my case looks like:

      java -Dcom.sun.management.jmxremote.port=XXXX -Dcom.sun.management.jmxremote.rmi.port=XXXX -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=A.B.C.D app.jar

      where XXXX is port number and A.B.C.D is IP address configured for JMX interface of monitored host in Zabbix.

      Hope this helps somebody

      P.S. JMX authentication and SSL could be disabled in absolutely trusted environments only!

        Comment

        • jasdeep!8
          #2.1
          jasdeep!8 commented
          02-05-2022, 13:24
          Editing a comment
          Hi bohf666,

          These changes that you mentioned have to be done on the JMX host. am i right? I need a little bit of help since I am new to this JMX monitoring. I am trying to monitor a few machines running dell boomi via JMX but I am getting the SSL shutdown incorrectly error for these servers.

          PS: These JMX hosts are monitored via Proxy, I have installed the Java Gateway on the zabbix server itself and configured the gateway IP in the proxy config file.

          Any help will be much appreciated.

          Thanks
          • dieselfluxcapacitor
            #2.2
            dieselfluxcapacitor commented
            11-07-2023, 17:44
            Editing a comment
            Thank you x1000!

            Adding the -Djava.rmi.server.hostname=A.B.C.D parameter to my Java application solved the "SSL peer shut down incorrectly" error I was receiving.
          • draken
            Junior Member
            • Jul 2021
            • 2
            #3
            Hello,
            I believe I am having the same problem as you.
            However, my problem only occurs on 1 host (I monitor a lot of applications on 1 host and I have about 10 such hosts, only 1 host has this problem across all applications).
            I have enable full debug for Zabbix Java Gateway and got the debug log:
            Code:
            2021-07-01 15:39:34.132 [pool-2-thread-59] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 157 bytes of request data
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request": "java gateway jmx", "keys": ["jmx[\"java.lang:type=Runtime\",Uptime]"], "jmx_endpoint": "service:jmx:rmi:///jndi/rmi://10.20.31.54:12352/jmxrmi"}
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG com.zabbix.gateway.SocketProcessor - RMI SSL hint cache cleanup is scheduled on 1625214620855, now is: 1625128774135
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
2021-07-01 15:39:34.135 [pool-2-thread-59] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://10.20.31.54:12352/jmxrmi'
2021-07-01 15:39:34.140 [pool-2-thread-59] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://10.20.31.54:12352/jmxrmi'
2021-07-01 15:39:34.144 [pool-2-thread-59] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx["java.lang:type=Runtime",Uptime]" failed: SSL peer shut down incorrectly: service:jmx:rmi:///jndi/rmi://10.20.31.54:12352/jmxrmi
2021-07-01 15:39:34.144 [pool-2-thread-59] DEBUG com.zabbix.gateway.SocketProcessor - error caused by
com.zabbix.gateway.ZabbixException: SSL peer shut down incorrectly: service:jmx:rmi:///jndi/rmi://10.20.31.54:12352/jmxrmi
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:168)
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-07-01 15:39:34.144 [pool-2-thread-59] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"SSL peer shut down incorrectly: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/10.20.31.54:12352\/jmxrmi"}
2021-07-01 15:39:34.144 [pool-2-thread-59] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection
            I still don't know how to fix this situation.
            I have disabled JMX monitoring for this host, and this does not happen again.
            Is this condition related to Zabbix Java Gateway performance?

              Comment

              • naxxon
                Junior Member
                • Feb 2022
                • 3
                #4
                dis you fix this issue? im still stuck.

                  Comment

                  • naxxon
                    Junior Member
                    • Feb 2022
                    • 3
                    #5
                    I've been stuck here 2 complete days.
                    I've used JMX monitoring since zabbix 4, the Tomcat and Zabbix config were almost the same for all my previous projects.

                    If you have alredy tried all internet-forum solutions for this error (SSL peer shut down incorrectly) and youre still stuck, try this.
                    (but try other solutions first, because this one is not the way i've been configuring all my previous zabbix+tomcat+jmx instalations)

                    This time im using Ubuntu 20.04 and Tomcat 9 (tried all zabbix 5 and 6 versions just in case without luck)

                    Remove from your tomcat config: -Djava.rmi.server.hostname=A.B.C.D (This is usually needed, and widely recommended but in this setup it screw up things)

                    Check in the tomcat9 machine that "hostname -i" returns the public ip address. (If it returns 127.0.0.1, or locahost or anything else, edit /etc/hosts and reboot)

                    My Tomcat config (tried zillinos of variants, this worked):

                    -Dcom.sun.management.jmxremote
                    -Dcom.sun.management.jmxremote.port=9000
                    -Dcom.sun.management.jmxremote.rmi.port=9000
                    -Dcom.sun.management.jmxremote.local.only=false
                    -Dcom.sun.management.jmxremote.authenticate=false
                    -Dcom.sun.management.jmxremote.ssl=false


                    hope it helps.

                    Best Regards, Nacho
                    Last edited by naxxon; 22-02-2022, 18:47.

                      Comment

                      • lptarik
                        Member
                        • Oct 2021
                        • 33
                        #6

                        who did solve this issue? I am trying to monitor activemq

                          Comment

                          • amanuelbm
                            Junior Member
                            • Mar 2023
                            • 1
                            #7
                            edit setenv.sh
                            nano /opt/tomcat/bin/setenv.sh and append -Dcom.sun.management.jmxremote.rmi.port=10052

                              Comment

                              • hakanozanlagan
                                Member
                                • Nov 2021
                                • 35
                                #8
                                Hello, I was getting this error on the Zabbix and Tomcat servers I set up in the test environment. I run the "hostname -i" command on tomcat server and result was 127.0.1.1 .
                                I marked the bottom line in the hosts file with #. and the problem was fixed. my operating system is ubuntu 22.04.

                                127.0.0.1 localhost
                                #127.0.1.1 zbxtom01

                                my java_opts line below.
                                JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/opt/tomcat/latest/conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/opt/tomcat/latest/conf/jmxremote.access"
                                Last edited by hakanozanlagan; 02-03-2024, 13:39.

                                  Comment

                                  • hehedemo
                                    Junior Member
                                    • Mar 2024
                                    • 5
                                    #9
                                    Thank you very much. After adding the parameter - DCom. sun. management. jmxremote. rami. port=12345, everything worked fine. However, due to the system firewall being turned on, errors kept appearing. Adding this parameter will fix the issue

                                      Comment

                                      Previous template Next
                                      Working...