Java Gateway - CPU utilization

[tmcleod]

We are investigating various ways of gathering and reporting on metrics from a pool of about 50 monitored hosts. Many of those hosts run a Java based learning management system. All the hosts are virtual machines. We chose Zabbix to evaluate because of the Java Gateway feature.

We installed Zabbix server and the Java Gateway on a single VM with 4 CPUs and 6 Gb of memory. The OS is Red Hat Enterprise 6. The EVC mode is that of a Intel Xeon Core 2 processor.

We initially used the generic Linux template to gather metrics from all of our monitored hosts. We then fired up the Java Gateway and collected metrics from only ONE monitored host using the generic JMX and Tomcat templates.

Our problem is that with the Java Gateway running, the Zabbix web interface is very sluggish. The "top" command reveals that the Java Gateway is running at about 55% CPU utilization even without gathering any metrics.

We are considering moving the Java Gateway to another VM or to increase the horsepower of our Zabbix server VM.

Has anyone else experienced similar performance issues when using the Java Gateway? If so, what steps were taken to correct them?

Thanks in advance,
Trevor

[tchjts1]

Our Java gateway doesn't even register in top. We also run it on a RHEL image on VM

Running the below on the pid (9321) of the java gateway reveals this:

ps -p 9321 -o %cpu,%mem
%CPU %MEM
0.0 1.1

That is with actively monitoring 2 hosts on their JMX items.

[tmcleod]

I wonder if we are comparing apples with apples.

Can you provide any specifics about the VM itself (e.g. number of CPUs and so on)?

Thanks,
Trevor

[tchjts1]

2 vCPU's, 8GB memory.

This server runs Zabbix server, java gateway and Zabbix DB.

This is a test setup and only has a total of 6 hosts being monitored. 2 of which have JMX monitoring.

[tmcleod]

So, we have 4 virtual CPUs, with 6Gb of memory while you have 2 CPUs with 8 Gb of memory.

Do you know what type of virtual CPU is being emulated and its speed?

Also, how did you build your Zabbix software? Here are my build notes:

--enable-server
--enable-agent
--with-mysql
--enable-static
--enable-java
--enable-ipv6
--with-libcurl
--with-net-snmp

The resulting binaries were compiled and installed in the default locations on our host (RHE 6.x):

/usr/local/sbin
for the server and agent binaries as well as the Java gateway JAR file

/usr/local/etc
for the server and agent configuration files

/usr/local/bin
for the zabbix_sender and zabbix_get utilities

/usr/local/share
for the Zabbix man pages

Note that the –enable-static option did not work. The following error message was displayed: “configure: error: Not found z library”. Installing all available zlib and zlibrary packages did not resolve the problem. No similar issues could be found using Google or by checking the Zabbix website.

Consequently, the binaries were compiled without the –enable-static option. This may be problematic should a Zabbix agent be deployed to a non-RHE 6.x host.

Thanks,
Trevor

[tchjts1]

I did the same compile as you, but without static and without ipv6.
This is on RHEL 6.1

I don't have any further info on the CPU speed or emulation, sorry.

[tmcleod]

What about the Java version you are using? Here's ours:

java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.6) (rhel-1.43.1.10.6.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

[tchjts1]

Almost identical
Difference is - (IcedTea6 1.10.4) (rhel-1.42.1.10.4.el6_2-x86_64)

java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.4) (rhel-1.42.1.10.4.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

Just curious - in your zabbix_server.conf file, how many java pollers are you starting?

### Option: StartJavaPollers
# Number of pre-forked instances of Java pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartJavaPollers=0
StartJavaPollers=5

[tmcleod]

We initially went with 5 pollers (default). I then dropped it to 2 pollers and there was no difference in CPU utilization.

Trevor

[tchjts1]

What version of Zabbix are you running?

[tmcleod]

We were using 2.0.0. I upgraded to 2.0.1 this morning to resolve an unrelated issue.

The upgrade has not changed the CPU utilization.

Trevor

[tchjts1]

Ok. I am currently on 2.0.2rc1

We started using the Java gateway with 2.0

I don't recall any high CPU usage with any version though.