JMX supervision (JBOSS) - OutOfMemoryError: Java heap space

[ryuma]

Hi,

I supervised JBOSS with Zabbix JMX. For that, i used this thread https://www.zabbix.com/forum/showthread.php?p=124373 .

So, i changed JMXItemChecker.java to allow the JMX supervision of JBOSS and charged remoting-jmx module but a memory problem appears with Jboss's JVM after 1-2 days .

* Jboss log :
ERROR [org.xnio.listener] (Remoting "mcptest1:MANAGEMENT" read-1) A channel event listener threw an exception: java.lang.OutOfMemoryError: Java heap space

* Java gateway log :
[pool-1-thread-1] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.net.MalformedURLException: Unsupported protocol: remoting-jmx
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:104) ~[zabbix-java-gateway-2.0.0.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:63) ~[zabbix-java-gateway-2.0.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1110) [na:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:603) [na:1.6.0_24]
at java.lang.Thread.run(Thread.java:679) [na:1.6.0_24]
Caused by: java.net.MalformedURLException: Unsupported protocol: remoting-jmx
at javax.management.remote.JMXConnectorFactory.newJMX Connector(JMXConnectorFactory.java:345) ~[na:1.6.0_24]
at javax.management.remote.JMXConnectorFactory.connec t(JMXConnectorFactory.java:266) ~[na:1.6.0_24]
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:96) ~[zabbix-java-gateway-2.0.0.jar:na]
... 4 common frames omitted
Caused by: java.io.IOException: Too many open files
at javax.management.remote.JMXConnectorFactory.getCon nectorAsService(JMXConnectorFactory.java:456) ~[na:1.6.0_24]
at javax.management.remote.JMXConnectorFactory.newJMX Connector(JMXConnectorFactory.java:328) ~[na:1.6.0_24]
... 6 common frames omitted
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_24]
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorI mpl.java:67) ~[na:1.6.0_24]
at sun.nio.ch.EPollSelectorProvider.openSelector(EPol lSelectorProvider.java:36) ~[na:1.6.0_24]
at java.nio.channels.Selector.open(Selector.java:226) ~[na:1.6.0_24]
at org.xnio.nio.NioXnioWorker.<init>(NioXnioWorker.ja va:120) ~[xnio-nio-3.0.3.GA.jar:3.0.3.GA]
at org.xnio.nio.NioXnio.createWorker(NioXnio.java:126 ) ~[xnio-nio-3.0.3.GA.jar:3.0.3.GA]
at org.jboss.remoting3.EndpointImpl.construct(Endpoin tImpl.java:137) ~[jboss-remoting-3.2.2.GA.jar:3.2.2.GA]
at org.jboss.remoting3.Remoting.createEndpoint(Remoti ng.java:60) ~[jboss-remoting-3.2.2.GA.jar:3.2.2.GA]
at org.jboss.remoting3.jmx.RemotingConnector.<init>(R emotingConnector.java:90) ~[remoting-jmx-1.0.0.Final.jar:1.0.0.Final]
at org.jboss.remoting3.jmx.RemotingConnectorProvider. newJMXConnector(RemotingConnectorProvider.java:48) ~[remoting-jmx-1.0.0.Final.jar:1.0.0.Final]
at javax.management.remote.JMXConnectorFactory.getCon nectorAsService(JMXConnectorFactory.java:442) ~[na:1.6.0_24]
... 7 common frames omitted

I tried to increase JVM memory but no changement. So, i used MemoryAnalyser (eclipse tool) to find the leak memory and i had the following results:

3 problems suspect:

-2 889 instances of "org.jboss.remoting3.remote.RemoteConnection", loaded by "org.jboss.modules.ModuleClassLoader @ 0x92ab0420" occupy 49 737 024 (28,61%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

-5 instances of "org.jboss.as.server.deployment.DeploymentUnitImpl ", loaded by "org.jboss.modules.ModuleClassLoader @ 0x92f85830" occupy 22 485 624 (12,94%) bytes.

Biggest instances:

org.jboss.as.server.deployment.DeploymentUnitImpl @ 0x92dbcaa0 - 9 458 664 (5,44%) bytes.
org.jboss.as.server.deployment.DeploymentUnitImpl @ 0x92db4df8 - 8 533 352 (4,91%) bytes.
org.jboss.as.server.deployment.DeploymentUnitImpl @ 0x925267a0 - 4 366 592 (2,51%) bytes.


-2 889 instances of "org.jboss.remoting3.remote.RemoteConnectionChanne l", loaded by "org.jboss.modules.ModuleClassLoader @ 0x92ab0420" occupy 19 182 960 (11,04%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"


Now, i don't have idea to fix this problem. If someone can help me ^^".
Thanks.

[zychonatic]

Caused by: java.io.IOException: Too many open files
says that there are too many open connections.

i got the same problem, when an application is too long down.

i fixed this problem, by restarting the JMX gateway everyday via cronjob.
i know it´s an ugly workaround, but this prevent and outage

br matthias

[ryuma]

yes i olso restart the JMX gateway with zabbix action as workaround but i cannot continue with this solution.

So I checked a java gateway source and i didn't find problems. I explained the problem to JBOSS and they said me it's a know memory leak in JBOSS.
they going to fix this problem in 7.1.3 (and second memory leak in 7.2.0).

Jboss Thread:

https://community.jboss.org/thread/220655

Thanks for your answer ^^.

[nickwelhar]

The java.lang.OutOfMemoryError: Java heap space error will be triggered when the application attempts to add more data into the heap space area, but there is not enough room for it.

Run Java with the command-line option -Xmx, which sets the maximum size of the heap.

Increasing the heap size is not a "fix" it is a "plaster", 100% temporary. It will crash again in somewhere else. To avoid these issues, write high performance code.

Use local variables wherever possible.
Make sure you select the correct object (EX: Selection between String, StringBuffer and StringBuilder)
Use a good code system for your program(EX: Using static variables VS non static variables)
Other stuff which could work on your code.
Try to move with multy THREADING