Java Gateway loses JMX connections - polling, debugging, logging

[tmcleod]

Hello,

We are monitoring 16 servers using both the Zabbix agent and JMX. Periodically, our JMX connections to all servers fail and all the JMX indicators turn red in Zabbix (Configuration-->Hosts-->followed by Items for any given host). Our only recourse is to restart the Java Gateway.

We have turned on debugging for the Java Gateway. Unfortunately, it seems that when the log reaches 5Mb in size it automatically rolls over a maximum of three times. Because of the volume of data being logged, by the time I am aware that our JMX connections are down, the relevant log entries have been lost.

Is there anyway to increase the size of the log and the number of roll overs?

I am wondering if the number of Java pollers needs to be increased but can find no rules of thumb regarding this. We were using the default value of 5 but have increased it to 10. However, this doesn't seem to have resolved the problem.

Our Zabbix server and Java Gateway run on the same VM which runs Red Hat Enterprise 6.x.

Any assistance would be appreciated.

Trevor

[tmcleod]

This may now be resolved. I noticed that the server log had lots of messages like the one below while the gateway log had none. I theorized that there was a problem with the server talking to the gateway because once it got through to the gateway, the gateway sent a request to the host in question and retrieve the JMX values without any problems.

I had to update the number of Java pollers in both the settings.sh for the Java Gateway and the zabbix_server.conf for the Zabbix server to 300. Its original value was 5. I kept incrementing this value by 10 and finally by 50 to arrive at a value where messages such as the following stopped being logged by the server:

13312:20120918:103603.988 JMX item [jmx["org.hornetq:module=JMS,type=Queue,name=\"SIS\"",M essagesAdded]] on host [mqprod01.lms.it.ubc.ca] failed: another network error, wait for 15 seconds

I also had to increase the max_connections parameter in the my.cnf configuration file for MySQL from the default of 150 to 300.

Whether this resolves the problem that the Java Gateway has to be restarted only time will tell.
Trevor