Ad Widget

Collapse

Zabbix can't connect via JMX, but it works from jmxterm

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • tiber
    Junior Member
    • Dec 2017
    • 2
    #1

    Zabbix can't connect via JMX, but it works from jmxterm

    We're trying to monitor a cassandra installation here. Zabbix is complaining in the UI:
    java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
    java.net.ConnectException: Connection refused (Connection refused)]

    But, when I try it using jmxterm, I get a fine connection and can list the beans.

    [root@cassandra-01~]# java -jar ./jmxterm.jar --url localhost:7199
    Welcome to JMX terminal. Type "help" for available commands.
    $>open
    rmi://127.0.0.1 5,service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi
    $>

    I'm kinda stumped here. In the host definition in zabbix, I have my JMX stuff set to 127.0.0.1 / IP / 7199.

    Kinda stumped here. Ideas?
    Tags: jmx
    • joshuamcdo
      Member
      • Nov 2013
      • 76
      #2
      Originally posted by tiber
      We're trying to monitor a cassandra installation here. Zabbix is complaining in the UI:
      $>open
      rmi://127.0.0.1 5,service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi
      I'm kinda stumped here. In the host definition in zabbix, I have my JMX stuff set to 127.0.0.1 / IP / 7199.

      Kinda stumped here. Ideas?
      I just went through this..

      In the JAVA_OPTS for your application.

      ...
      "-Djava.rmi.server.hostname=<IP OF THE ZABBIX SERVER/JAVA GATEWAY> -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
      ...

      If you specify 127.0.0.1 for the rmi.server.hostname it will refuse all outside connections to it. I think there is a way to make it open to more if not all hosts but for security reasons you should not do that IMO. Even with the zabbix server IP specified you can still connect from the localhost.
      ...
      -Dcom.sun.management.jmxremote.rmi.port=7199
      ...

      The above setting makes sure that the port # doesn't change and is always 7199.


      Then there is the obvious.. make sure there aren't any iptables and or otherwise related firewalls blocking you. But if you're getting a connection refused it's likely that the traffic is making it to the host.

      Hope that helps.

        Comment

        • tiber
          Junior Member
          • Dec 2017
          • 2
          #3
          OK! That was the hint I needed. Writing this up for my own edification as much as anyone else's...

          So lets say you want to monitor cassandra with something like this: https://share.zabbix.com/databases/c...r-template-jmx

          The first step is connectivity - just make sure the zabbix host (in my case running java gateway on the same box) can talk to the cassandra host on port 7199.

          Download jmxterm since it gives you nice, verbose errors: http://wiki.cyclopsgroup.org/jmxterm/

          Run jmxterm:
          java -jar ./jmxterm.jar --url cassandra-01.mydomain.com:7199


          If you see something like "connection refused", but the port is open, it's a JMX problem. If you can't connect, it's a firewall/acl problem. In my case telnet cassandra-01.mydomain.com 7199 so we know whats up.

          WARNING: Before editing the next file, understand that copy and pasting this will result in an unauthenticated JMX port. Please make appropriate security considerations in your environment.

          Edit the /etc/cassandra/default.conf/cassandra-env.sh file. At the bottom is the JMX snippet we're looking for - and name resolution does not work and these have to be IPs:

          #if [ "$LOCAL_JMX" = "yes" ]; then
          # JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT -XX:+DisableExplicitGC"
          #else
          JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
          JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
          JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
          JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
          JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
          # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
          # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
          # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
          # JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"
          # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth =true"
          # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"
          # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocol s=<enabled-protocols>"
          # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.s uites=<enabled-cipher-suites>"
          #fi

          # To use mx4j, an HTML interface for JMX, add mx4j-tools.jar to the lib/
          # directory.
          # See http://wiki.apache.org/cassandra/Ope...ring_with_MX4J
          # By default mx4j listens on 0.0.0.0:8081. Uncomment the following lines
          # to control its listen address and port.
          #MX4J_ADDRESS="-Dmx4jaddress=127.0.0.1"
          #MX4J_PORT="-Dmx4jport=8081"

          # Cassandra uses SIGAR to capture OS metrics CASSANDRA-7838
          # for SIGAR we have to set the java.library.path
          # to the location of the native libraries.
          JVM_OPTS="$JVM_OPTS -Djava.library.path=$CASSANDRA_HOME/lib/sigar-bin"

          JVM_OPTS="$JVM_OPTS $MX4J_ADDRESS"
          JVM_OPTS="$JVM_OPTS $MX4J_PORT"
          JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
          JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<zabbix server IP>"
          JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1"
          JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<cassandra IP>"

          I touched /etc/cassandra/jmxremote.password just to satisfy the script. Bounce cassandra and try to use jmxterm to connect from the zabbix host again. Also ensure nodetool status works because if it doesn't, the other cassandra tools won't work either.

            Comment

            • sriramkumar
              Junior Member
              • Mar 2018
              • 3
              #4
              Zabbix monitoring with jmx not working...suggestions needed

              java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
              java.net.ConnectException: Connection refused
              Last edited by sriramkumar; 09-03-2018, 09:56.

                Comment

                • sriramkumar
                  Junior Member
                  • Mar 2018
                  • 3
                  #5
                  Now i got following error (After modify your suggestion @joshuamcdo):

                  java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.27.111; nested exception is:
                  java.net.ConnectException: Connection refused].

                  Please suggest me where i missed or make a mistake
                  My current Configurations below:

                  Zabbix server,Java gateway both installed in same machine- 192.168.27.13
                  Cassandra server installed on -192.168.27.111

                  zabbix_server.conf file:(IP:192.168.27.13)

                  DBHost=localhost
                  DBName=zabbix
                  DBUser=zabbix
                  DBPassword=zabbix
                  JavaGateway=127.0.0.1
                  JavaGatewayPort=10052
                  StartJavaPollers=5

                  Zabbix_gateway (IP: 192.168.27.13)

                  LISTEN_IP=127.0.0.1
                  LISTEN_PORT=10052
                  START_POLLERS=5
                  PID_FILE="/var/run/zabbix/zabbix_java_gateway.pid"
                  TIMEOUT=3


                  cassandra-env.sh (IP:192.168.27.111)

                  #if [ "x$LOCAL_JMX" = "x" ]; then
                  # LOCAL_JMX=no
                  #fi

                  JMX_PORT="7199"

                  #if [ "$LOCAL_JMX" = "yes" ]; then
                  #JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
                  # JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

                  #else
                  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT"
                  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
                  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

                  # jmx ssl options
                  #JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"
                  #JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth =true"
                  #JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocol s=<enabled-protocols>"
                  #JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.s uites=<enabled-cipher-suites>"
                  #JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore"
                  #JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=<keystore-password>"
                  #JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/path/to/truststore"
                  #JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=<truststore-password>"

                  etc/java-9-openjdk/Management/management.propertiesIP-192.168.27.13)

                  java \
                  -Djava.rmi.server.hostname=192.168.27.13
                  -Dcom.sun.management.jmxremote=7199
                  -Dcom.sun.management.jmxremote.rmi.port=7199
                  -Dcom.sun.management.jmxremote.port=7199
                  -Dcom.sun.management.jmxremote.authenticate=false
                  -Dcom.sun.management.jmxremote.ssl=false
                  Attached Files

                    Comment

                    • emil@hirestorm.com
                      Junior Member
                      • Sep 2019
                      • 6
                      #6
                      Hello,

                      I am having a similar issue quoted above for JMX monitoring .

                      My zabbix server and java gateway is on different server.

                      Please find the configurations performed.
                      Attached Files
                      Screenshot by Lightshot
                      https://prnt.sc
                      Captured with Lightshot
                      Last edited by [email protected]; 17-09-2019, 14:50.

                        Comment

                        Previous template Next
                        Working...