Ad Widget

Collapse

zabbix-java-gateway java.net.NoRouteToHostException: No route to host

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • kpetruk
    Junior Member
    • Jan 2011
    • 7

    #1

    zabbix-java-gateway java.net.NoRouteToHostException: No route to host

    Can't connect one server through JMX. Connection to several other servers are OK.
    Firewall is configured and port is correct. I could telnet to that port, but zabbix-java-agent can't connect. I could also connect to that server through RMI (standard ports).

    Differences between servers:
    * on problematic server installed java (build 1.6.0_17-b04). On others 1.8
    * problematic server is located on different provider (Rackspace).

    Java app started with:
    "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10107 -Dcom.sun.management.jmxremote.rmi.port=10107 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=x.x.x.x"


    Below is tracelog.
    Code:
    2014-11-17 12:53:10.520 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
    2014-11-17 12:53:10.520 [pool-1-thread-1] DEBUG com.zabbix.gateway.JMXItemChecker - connecting to JMX agent at service:jmx:rmi:///jndi/rmi://x.x.x.x:10107/jmxrmi
    2014-11-17 12:53:11.496 [pool-1-thread-1] WARN  com.zabbix.gateway.SocketProcessor - error processing request
    com.zabbix.gateway.ZabbixException: java.rmi.ConnectIOException: Exception creating connection to: x.x.x.x; nested exception is:
            java.net.NoRouteToHostException: No route to host
            at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:100) ~[zabbix-java-gateway-2.4.2.jar:na]
            at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:63) ~[zabbix-java-gateway-2.4.2.jar:na]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
    Caused by: java.rmi.ConnectIOException: Exception creating connection to: x.x.x.x; nested exception is:
            java.net.NoRouteToHostException: No route to host
            at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:631) ~[na:1.8.0_25]
            at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_25]
            at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_25]
            at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_25]
            at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) ~[na:1.8.0_25]
            at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404) ~[na:1.8.0_25]
            at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_25]
            at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_25]
            at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:92) ~[zabbix-java-gateway-2.4.2.jar:na]
            ... 4 common frames omitted
    Caused by: java.net.NoRouteToHostException: No route to host
            at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_25]
            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) ~[na:1.8.0_25]
            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_25]
            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_25]
            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_25]
            at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_25]
            at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_25]
            at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_25]
            at java.net.Socket.<init>(Socket.java:211) ~[na:1.8.0_25]
            at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_25]
            at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[na:1.8.0_25]
            at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[na:1.8.0_25]
            ... 12 common frames omitted
  • Daniel Weber
    Junior Member
    • Jan 2014
    • 6

    #2
    No Route to Host, maybe wrong IP

    Hello

    I've seen your post, as i don't know your Environment completely, i can only tell what the Message tells me.

    No Route to host means for me, that the Proxy or the Server, that is trying to connect to your x.x.x.x IP-Adress has no route to it.

    Have you checked the route list? (Command in Linux: ip route)

    Maybe the Server / Proxy misses a Default Gatway or your Network does not know where to send the packets to.

    Have you tried ping the IP-Adress on the Zabbix-Server / Proxy?
    Maybe is the IP-Adress wrong?

    Comment

    • kpetruk
      Junior Member
      • Jan 2011
      • 7

      #3
      As I mentioned in my original post: "I could telnet to that port".
      I could also connect to other services on that server.

      Comment

      • mucknet
        Member
        • Dec 2004
        • 59

        #4
        Originally posted by kpetruk
        Can't connect one server through JMX. Connection to several other servers are OK.
        Firewall is configured and port is correct. I could telnet to that port, but zabbix-java-agent can't connect. I could also connect to that server through RMI (standard ports).

        Differences between servers:
        * on problematic server installed java (build 1.6.0_17-b04). On others 1.8
        * problematic server is located on different provider (Rackspace).
        Did you ever find an answer to this? I'm having the same issue. zabbix_java.log is reporting "No Route To Host", however, I can telnet to my JMX port from my java gateway server just fine.

        I have some additional strange behavior, which is that the IP zabbix_java.log is reporting is NOT the ip I've configured as my JMX interface in my host configuration. Instead the IP it is reporting is another interface on the server I'm trying to monitor. An interface I DO NOT use for connecting to from zabbix.

        FWIW, I'm using 3.2.3 for server, agent, and java gateway.

        Comment

        • Pada
          Senior Member
          • Apr 2012
          • 236

          #5
          What does your server report when you run:
          Code:
          hostname -i
          , because if it returns 127.0.0.1 and not the JMX IP you may have problems accessing it.
          Also, can you access it with something like jConsole or jVisualVM? The above would cause those apps to fail too.

          I wrote the following script for our environments where the `hostname -i` resulted in localhost:
          Code:
          ipeth0=`ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1`
          if [[ "127.0.0.1" == "`hostname -i`" ]]; then
            if [[ "`hostname`" == "`hostname -s`" ]]; then
              sudo sed -i "s/ `hostname`\$//" /etc/hosts
            else
              sudo sed -i "s/ `hostname`//" /etc/hosts
            fi
            sudo echo "$ipeth0 `hostname`" >> /etc/hosts
            
            if [[ "$ipeth0" != "`hostname -i`" ]]; then
              echo "Error: Failed to change 'hostname -i' to '$ipeth0'. Please correct /etc/hosts file manually."
            else
              sudo service our-java-webapp-service restart
            fi
          elif [[ "$ipeth0" != "`hostname -i`" ]]; then
            echo "Error: Please correct /etc/hosts file manually."
          else
            echo "Already configured to '$ipeth0'"
          fi
          I've also taken the zabbix_get_jmx script and modified it to work with Zabbix 3.2.3 (although I could't get zabbix_get to work) - see the attached zabbix_get_jmx.txt shell script.

          And then I use it by running it on my proxy server that has the Java gateway running on port 10052 with like:
          Code:
          ./zabbix_get_jmx localhost 10052 host.example.com 12345 'jmx["Catalina:type=GlobalRequestProcessor,name=\"http-nio-8443\"",bytesReceived]'
          Perhaps also consult the Zabbix JMX wiki page: https://www.zabbix.org/wiki/ConfigureJMX
          Attached Files

          Comment

          • mucknet
            Member
            • Dec 2004
            • 59

            #6
            Thanks Pada, I didn't end up using your solution exactly, but you pointed me in the right direction, and I am now successful. Instead, I found that I could use -Djava.rmi.server.hostname=app-server.sub.domain.com to specify the interface/IP I wanted it to use.

            Here are my notes that I took for getting this all setup, in case they're useful to someone else:


            * Java gateway sits on same server as zabbix_server - It doesn't have to, but in our case it does.


            Zabbix Server Config
            * Configure /opt/zabbix/etc/zabbix_server.conf
            * * JavaGateway=127.0.0.1
            * * JavaGateWayPort=10052
            * * StartJavaPollers=4
            * Configure /opt/zabbix/sbin/zabbix_java/settings.sh
            * * LISTEN_PORT=10052
            * Configure zabbix_java to auto start. I did that by adding it to existing zabbix_server startup script:
            Code:
            ZABBIX_JAVA_DIR="/opt/zabbix/sbin/zabbix_java"
             
            if [ ! -x ${ZABBIX_BIN} ] ; then
                    echo -n "${ZABBIX_BIN} not installed! "
                    # Tell the user this has skipped
                    exit 5
            fi
             
            start() {
                    echo -n $"Starting $prog: "
                    daemon $ZABBIX_BIN
                    RETVAL=$?
                    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/zabbix_server
                    echo
                runuser -l zabbix -c "$ZABBIX_JAVA_DIR/startup.sh"
            }
             
            stop() {
                    echo -n $"Stopping $prog: "
                    killproc $ZABBIX_BIN
                    RETVAL=$?
                    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/zabbix_server
                    echo
                runuser -l zabbix -c "$ZABBIX_JAVA_DIR/shutdown.sh"
            }
            Host Config (JVM and Firewall):
            * If you have Multiple Interfaces on the host, you need to configure the "-Djava.rmi.server.hostname" option, otherwise you get a convoluted and useless "No Route To Host" error.
            * The hostname you configure, does NOT have to be in /etc/hosts - at least our JDK 6 will use DNS to get the results. Having it in /etc/hosts isn't a bad idea though, if possible.
            * Firewall Changes:
            * * For JDK6, you have to allow all ports from your java gateway server (same as zabbix server in this case):
            * * -A INPUT -m conntrack -i eth2 --ctstate NEW -m tcp -p tcp -s zabbixserver.sub.domain.com -j ACCEPT -m comment --comment "zabbixserver.sub.domain.com"
            * * For JDK7u4, you can specify an rmi port, and just allow access to that port from your Java Gateway Server
            * * -A INPUT -m conntrack -i eth2 --ctstate NEW -m tcp -p tcp --match multiport -s zabbixserver.sub.domain.com --dports 22,80,443,10050,10099,10199 -j ACCEPT -m comment --comment "zabbixserver.sub.domain.com"

            * resin.conf changes - JDK 7 supports specifying RMI port - JDK6 Does not:
            * This was for a resin server, but the same options should work for any java web server - tomcat, weblogic, etc
            * * Also note that these settings should be considered insecure, and depending on your environment you should lock down JMX access better than this
            Code:
            <jvm-arg>-Dcom.sun.management.jmxremote.port=10099</jvm-arg>
                  <jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</jvm-arg>
                  <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
                  <jvm-arg>-Djava.rmi.server.hostname=app-server.sub.domain.com</jvm-arg>
                 <jvm-arg>-Dcom.sun.management.jmxremote.rmi.port=10199</jvm-arg>
            Zabbix Host UI Config
            * JMX Interface: Can use IP or Hostname (app-server.sub.domain.com). Hostname needs to match the hostname used in the JVM config (or more accurately they both need to resolve to the same IP address)
            * Apply JMX Template to Host

            Search Keywords: zabbix_java.log zabbix_java JMX resin tomcat Java Gateway jconsole No Route To Host Exception Creating connection NoRouteToHostException Host Unreachable
            Last edited by mucknet; 05-01-2017, 05:48.

            Comment

            • vishnu.dinoct
              Junior Member
              • May 2017
              • 9

              #7
              com.zabbix.gateway.SocketProcessor - error processing request

              Can please help me with this error??? I am dead already


              WARN com.zabbix.gateway.SocketProcessor - error processing request
              com.zabbix.gateway.ZabbixException: bad protocol header: 7B 22 72 65 71
              at com.zabbix.gateway.BinaryProtocolSpeaker.getReques t(BinaryProtocolSpeaker.java:59) ~[zabbix-java-gateway-2.2.0.jar:na]
              at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:51) ~[zabbix-java-gateway-2.2.0.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) [na:1.8.0_45]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) [na:1.8.0_45]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

              Comment

              • Raghav
                Junior Member
                • May 2019
                • 1

                #8
                I Had the same problem. I solved it by using a ssh tunnel from the multiple hosts to the zabbix central server. Setting RMI port was not working for me and by default it takes a random port due to which zabbix_java_gateway was unable to get to the rmi port.

                Comment

                Working...