java gateway issue

[pradhanparas]

Hi.

I have around 10 hosts that I use to monitor jmx. It works well except I need to restart java_gateway frequently to resolve the issue. I see the following messages in log.

zabbix version is 2.0.2 built from source. In zabbix_server.conf I have, StartPollers = 50 and StartJavaPollers=50. zabbix_java's setting.sh has START_POLLERS=35. I increased the values from default but it is still the same.

log:
--
2014-01-13 10:40:46.093 [pool-1-thread-4] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.rmi.UnknownHostException: Unknown host: client01.mydomain; nested exception is:
java.net.UnknownHostException: client01.mydomain
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:100) ~[zabbix-java-gateway-2.0.2.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:63) ~[zabbix-java-gateway-2.0.2.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.rmi.UnknownHostException: Unknown host: client01.mydomain; nested exception is:
java.net.UnknownHostException: client01.mydomain
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEnd point.java:616) ~[na:1.7.0_45]
at sun.rmi.transport.tcp.TCPChannel.createConnection( TCPChannel.java:216) ~[na:1.7.0_45]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCP Channel.java:202) ~[na:1.7.0_45]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:1 29) ~[na:1.7.0_45]
at javax.management.remote.rmi.RMIServerImpl_Stub.new Client(Unknown Source) ~[na:1.7.0_45]
at javax.management.remote.rmi.RMIConnector.getConnec tion(RMIConnector.java:2392) ~[na:1.7.0_45]
at javax.management.remote.rmi.RMIConnector.connect(R MIConnector.java:308) ~[na:1.7.0_45]
at javax.management.remote.JMXConnectorFactory.connec t(JMXConnectorFactory.java:268) ~[na:1.7.0_45]
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:92) ~[zabbix-java-gateway-2.0.2.jar:na]
... 4 common frames omitted
Caused by: java.net.UnknownHostException: client01.mydomain
at java.net.AbstractPlainSocketImpl.connect(AbstractP lainSocketImpl.java:178) ~[na:1.7.0_45]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:392) ~[na:1.7.0_45]
at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_45]
at java.net.Socket.connect(Socket.java:528) ~[na:1.7.0_45]
at java.net.Socket.<init>(Socket.java:425) ~[na:1.7.0_45]
at java.net.Socket.<init>(Socket.java:208) ~[na:1.7.0_45]
at sun.rmi.transport.proxy.RMIDirectSocketFactory.cre ateSocket(RMIDirectSocketFactory.java:40) ~[na:1.7.0_45]
at sun.rmi.transport.proxy.RMIMasterSocketFactory.cre ateSocket(RMIMasterSocketFactory.java:147) ~[na:1.7.0_45]
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEnd point.java:613) ~[na:1.7.0_45]
... 12 common frames omitted
--


How do I debug what is causing this? zabbix server is healthy in terms of cpu, memory, io, network etc etc
Thanks!
Paras.

[jan.garaj]

From source code:
JMXItemChecker.java:92

jmxc = JMXConnectorFactory.connect(url, env);

This code raised a exception java.net.UnknownHostException: client01.mydomain

Are you sure that you are able to make TCP connection to client01.mydomain? Try ping client01.mydomain on your Zabbix server.

Also enable higher level of debuging in Zabbix server config file and you will see in log file, where Zabbix is making jmx connection - there is the debug code:

logger.debug("connecting to JMX agent at {}", url);

[pradhanparas]

I am 100% sure about the zabbix server can ping the clien01.mydomain.

Only jmx has issues. The rest items such has cpu, mem, disk are populated from clien01 to srver and no issues at all.

-Paras.

[jan.garaj]

Try to debug/connect to JMX interface with some jmx tool (jconsole) when you have issue. Use Zabbix jmx url:

service:jmx:rmi:///jndi/rmi://<ZABBIX_JMX_HOST>:<ZABBIX_JMX_PORT>/jmxrmi

You can also setup DNS and TCP check in Zabbix for client01.mydomain (net.tcp.dns.query and net.tcp.port). Maybe you have a momentary problems with DNS in your network. Monitor it.

[pradhanparas]

Thanks for your reply. I will monitor using jconsole but I am thinking if that will help or not because as soon as I see red or jmx unavailability I restart zabbix_java it fixes the issue.

What do you think about the pollers paramters affecting this issue? When I see this issue at the same time i see this warning.

zabbix poller processes more than 75


Monitroing dns is a good point I will do it.

-Paras.

[jan.garaj]

Read http://blog.zabbix.com/monitoring-ho...esses-are/457/ and check your graph "Zabbix data gathering process busy %".

Have you any peaks in graph "Zabbix server performance" or "MySQL operations"? Some users have a problems with Zabbix performance in large environment, when Housekeeping is executed.

[pradhanparas]

mysql operations and zabbix server perfomance is very minimal and no peaks at all. But at the time of jmx issue I see "zabbix busy unreachable poller process in %" is 100% and it stays like that untill I restart zabbix_java process.

-Paras.

[jan.garaj]

What about Zabbix CPU usage graph? Have you any peaks in iowait/interrupt/softirq times?

You have to check Zabbix server, when issue occurs: Zabbix server/agent log files (with debug level 4), process list of Zabbix database, processes, cpu utilization, connections, system log, iops, jmx connecting, ...

My forecast from remote view:
you have probably any network/DNS issue. So pollers will stuck in reconnecting (100% busy pollers) and java pollers will raise unknown host exception.

[pradhanparas]

CPU is fine.

CPU %user %nice %system %iowait %steal %idle
Average: all 1.12 0.21 1.59 0.74 0.03 96.31

I will change the debug to 4 and report it here when if happens.

Thanks for your help ! Appreciate it

-Paras.

[jtremblay]

same problem as OP

I have been researching this issue for days! I can find NO RESOLUTIONS.
zabbix 2.2.7 all updated on fresh CentOS 6.6
can anyone help?

[pradhanparas]

I had to play around a bit with

StartPollers
StartPingers
StartJavaPollers
START_POLLERS (in zabbix_java)



Now I don't have this issue.


Thanks,:q
Paras.