zabbix_Java Restart Needed often

[rreilly]

I believe the problem is the java code is not reclaiming threads, I am monitoring it with jconsole and the threads just keep getting created with no reclamation.

thoughts ?
Thanks
Rob

[jan.garaj]

Can you provide list of threads with details for every thread (stack trace, ...)?

[rreilly]

Attached is the thread dump, Thanks
http://pastebin.com/h7KnDCqW
Rob

[jan.garaj]

Stats from your thread dump:

     48 Remoting
      2 RMI TCP Accept-0
    ...

All "Remoting" threads got stuck at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method), so Zabbix had to create new threads, which probably also got stuck in epollWait function. Check this hypothesis during some time period.

It looks like issue with JDK. Similar issue with another Java SW https://github.com/netty/netty/issues/327
Try another version of JDK or UTFG for this Java issue.

[rreilly]

Thanks, I knew it was spawning threads endlessly just did not know why, I upgraded my java i will monitor and report back.
Thanks
Rob

[rreilly]

so i tried sun 1.7.0.25 sun 1.7.0.51 openjdk 1.6.0 same problem ...

[rreilly]

or is it a bug with the jboss NIO library ?

[jan.garaj]

Sorry, I don't have so huge knowledge about Java thread problems - Google knows :-)

Interesting discussion http://code.google.com/p/spymemcache.../detail?id=279

Ok, I could solve the problem.
The cpu load is caused by the vm itself. 
After further profiling I found out that epollwait caused the high cpu load due to a "bad" to garbage collection setting for the vm.
 -XX:CMSInitiatingOccupancyFraction was set to 50 which lead to many gc calls which in fact caused a high cpu load on epoolwait. 
(Why this happens I don't know - this could have many causes).
Setting the value to 80 made the high cpu load disappear. 
I was mislead due to the fact that when I disabled memcacheclient cpu load dropped too.

[rreilly]

I found the problem, it's not with the zabbix_java and I never experienced high cpu, just filling the thread queue. The problem is with the version of netty that is shipped with jboss 7.1. There is a workaround on the jboss jdk add
-Dorg.jboss.netty.epollBugWorkaround=true to JAVA_OPTS we are testing it now and it looks like it will fix the problem. I will report back after 24 hours of baking.
Rob