Can Java Based Applications be monitored in ZABBIX 1.4 ?

[massoo]

Hi,

I posted this question earlier in the forums on how to monitor a JBoss Application server ... but could not get any concrete directions on how to do that. If exist any such methods please direct me on how to do this.

My Question now is that in the future release ZABBIX 1.4 monitoring of WebApps will be supported ... Does this mean that we can monitor JBoss Application server and its internals like JDBC connection usage, JVM usage, etc.

Regards
Prashant

[just2blue4u]

You can already monitor the jBoss by watching processes and if he listens on a specific port...

[massoo]

what i mean by monitoring jboss is to monitor JMS, JDBC Connects, JVM Usage, etc which are internal to JBOSS

[sauron]

use zabbix_trapper & cron.

[massoo]

how ?

can you elaborate as i am still exploring zabbix

[Calimero]

The "problem" is not really on the zabbix side I think.

You should first write a set of script that tests what you want to test and have your send results back to zabbix using zabbix_sender.

[pdwalker]

If JBOSS supports JMX, you can use JMX to retrieve the key/values you are interested in from the jvm and return them for plotting with zabbix.

I currently use JMX calls via custom user parms to retrieve information from my java servlets running under tomcat. (or application was specifically instrumented to support JMX just so we could easily control how the application runs and pull interesting stats from it)

- Paul

[massoo]

JBoss support JMX. We have our web applcations packaged as *.war files. and are using Linux, JDK 1.5.x along wih JBoss 4.0.3-SP1.

It would be nice if you can share your experience and details on how do you monitor :
1. JVM usage
2. O.S Information - default supported by Zabbix
3. JDBC Usage
4. JVM Threads
5. Alert Mechanism, etc

Thanks & Regards
Prashant

[pdwalker]

jvm monitoring via jmx for use by zabbix.

First of all, you need to configure your jvm to allow jmx monitoring. How do you know if you can do this? You can use the sun jconsole utility that comes with the jdk and point it at your machine running the jvm. If you can connect, you are good.


In my tomcat environment, I enable it by setting the following options for the jvm:

-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=xxxxx \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.password.file=/path/java/jre/lib/management/jmxremote. password"


This tells the jmx server to run on port XXXXX, to use password authentication, and to refer to the passwords stored in the jmxremote.password file. See the sun docs on jconsole for details. (You might consider enabling ssl to make the connection more secure.)


Once that is done, I can then run jconsole and see everything that is currently exposed (and to verify that I can connect properly). jconsole will also provide you the information you need to query specific jmx attributes from the information tab.


Now, since I use Tomcat, there are two ways that I can grab the jmx attribute values (or effect a jmx operation). The first way is I can use the servlet provided by Tomcat. (Don't know what jboss has). The second way is I can send well formatted requests via a jmx command line tool.


Let's say I am interested in peak threads used by the system. I browse down through the jmx objects via jconsole, find it under java.lang, Threading. After selecting Threading, I click on the info tab, and I can see the name of the mbean is "java.lang:type=Threading"


With tomcat, I can do the following:

curl -s -u<jmxusername>:<jmxpassword> 'http://<tomcat_hostname>/manager/jmxproxy/?qry=java.lang:type=Threading'

where the jmx username and password are the ones defined in the file defined in the jvm options above, the qry string is the one obtained from jconsole.


The output from this will be all the metrics from this jmx key. Parse the output and grab the number of your choice.


If you don't have a servlet that will allow you to make a http request to the jmx interface, you can use the command line tool like this

/<pathTo>/java -jar /<pathTo>/cmdline-jmxclient.jar <jmxusername>:<jmxpassword> <jvmhostname>:<jmxport> java.lang:type=Threading PeakThreadCount


The difference with the command line client is you need to specify the attribute you are interested in specifically. Leaving it out will give you a list of all the attributes available under Threading.

Again, parse the output for the data of your choice.


Once you can reliably grab the data you are interested in, you can then turn that command into a zabbix userparm.

e.g.

UserParameter=jvm.maxthreads, /usr/bin/curl -s -u<jmxusername>:<jmxpassword> 'http://<tomcat_hostname>/manager/jmxproxy/?qry=java.lang:type=Threading' | /bin/awk '/^PeakThreadCount\:/ { gsub( /[^0123456789]/, "" ); print $1 }'

or

UserParameter=jvm.maxthreads, /<pathTo>/java -jar /<pathTo>/cmdline-jmxclient.jar <jmxusername>:<jmxhostname> <jvmhostname>:<jmxport> java.lang:type=Threading PeakThreadCount | <some filter to grab just the number you need - left as an exercise to the reader>


That's it.

I prefer getting my stats from the servlet via http rather than using the java command line client as it is much "lighter" to start up and grab the information.


Need a command line jmx client? I use the one from here:

http://crawler.archive.org/cmdline-jmxclient/

Information on setting up jmx monitoring for your jvms

http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html

General Information on JMX

http://java.sun.com/j2se/1.5.0/docs/guide/jmx/overview/JMXoverviewTOC.html

Hope that helps.

- Paul

PS: apparently the 1.5 jvm also supports snmp which provides another option.

[Alexei]

Thanks for the excellent HOWTO! Don't you mind if I add it to ZABBIX Manual?

[pdwalker]

Of course!

Please feel free.

If you need any more information, I'll happily tell you what I know.

- Paul

[simeon]

Another approach for monitoring java applications would be an implementation of the zabbix_sender written in pure java. Such implementation could be embedded in any java application without a need to use JMX.

Where could I find the communication protocol used to send messages from the zabbix_sender to a server? I wouldn't mind to implement it in java and contribute it to the project.

[skaze]

For all - those wishing to use less intrusive tomcat JVM monitoring may like to take a look at my jmxproxy script that can be run either via active agents (i.e. using a UserParameter) or run via an active agent that then pushes all the data back to zabbix via the zabbix_sender and Zabbix trapper items. (Thanks goes to the user that inspired this via a similar solution to the apache server-status page polling).

In escence:-

jmxproxy.sh can be run from the cmd line or via a UserParameter thus:

# Tomcat JVM (via jmxproxy) stats, requires command (see script), host, port and, optionally, app-path
UserParameter=jmxproxy[*],jmxproxy.sh $1 $2 $3 $4

required params are:

command, host and port.

Command can be one of:

threads.max
threads.cur
heap.committed
heap.initial
heap.max
heap.used
stack.committed (yes i know its not real stack but...)
stack.initial
stack.max
stack.used
sessions.all.active
sessions.all.maxactive
sessions.all.total
sessions.all.expired

For the following commands you must also supply an application path, e.g. "/myapp"

sessions.app.active
sessions.app.maxactive
sessions.app.total
sessions.app.expired
sessions.app.age.avg
sessions.app.age.max

There is then the 'push' command of

push.all

This then sends all the above stats to Zabbix (ie. it assumes you have trappers defined listening for the values).

Additionally, the push.all command 'auto discovers' all the apps running in tomcat and sends the 'sessions.app.XXX' keys to zabbix aswell.

e.g.

The following is an example of the zabbix_sender result file created by the push.all command:


zabbix-server.com monitored-host.com 10051 jmxproxy[threads.max,localhost,8080] 354
zabbix-server.com monitored-host.com 10051 jmxproxy[threads.cur,localhost,8080] 240
zabbix-server.com monitored-host.com 10051 jmxproxy[stack.used,localhost,8080] 151865144
zabbix-server.com monitored-host.com 10051 jmxproxy[stack.max,localhost,8080] 570425344
zabbix-server.com monitored-host.com 10051 jmxproxy[stack.committed,localhost,8080] 300875776
zabbix-server.com monitored-host.com 10051 jmxproxy[stack.initial,localhost,8080] 268599296
zabbix-server.com monitored-host.com 10051 jmxproxy[heap.used,localhost,8080] 473314120
zabbix-server.com monitored-host.com 10051 jmxproxy[heap.max,localhost,8080] 1034027008
zabbix-server.com monitored-host.com 10051 jmxproxy[heap.committed,localhost,8080] 537722880
zabbix-server.com monitored-host.com 10051 jmxproxy[heap.initial,localhost,8080] 536870912
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.all.total,localhost,8080] 5891
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.all.active,localhost,8080] 52
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.all.maxactive,localhost,8080] 131
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.all.expired,localhost,8080] 7187
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.active,localhost,8080,/myapp] 18
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.maxactive,localhost,8080,/myapp] 38
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.total,localhost,8080,/myapp] 1003
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.expired,localhost,8080,/myapp] 985
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.age.avg,localhost,8080,/myapp] 3601
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.age.max,localhost,8080,/myapp] 8725
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.active,localhost,8080,/anotherapp] 21
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.maxactive,localhost,8080,/anotherapp] 46
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.total,localhost,8080,/anotherapp] 2614
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.expired,localhost,8080,/anotherapp] 3941
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.age.avg,localhost,8080,/anotherapp]
zabbix-server.com monitored-host.com 10051 jmxproxy[sessions.app.age.max,localhost,8080,/anotherapp]


You simply declare trappers in Zabbix for the appropriate keys and an active agent item to run the UserParameter every 30 seconds or so and you have a complete non-intrusive (ish) tomcat monitoring solution.

Script attached is attached, youll need to rename it and edit the contents to supply some usernames, passwords and zabbix server details .

Examples of a simple poll based item:

Getting the heap committed memory:

Simply create an item of type Zabbix agent with the key:
jmxproxy[heap.committed,localhost,8080]

Getting the total active sessions:

Simply create an item of type Zabbix agent with the key:
jmxproxy[sessions.all.active,localhost,8080]

Getting the active sessions for a specific app

Simply create an item of type Zabbix agent with the key:
jmxproxy[sessions.app.active,localhost,8080,/myapp]


All the above can be done via a push though (much more efficient) simply by creating one active item of type Zabbix agent (active) with key: jmxproxy[push.all,localhost,8080]

And then change all your Zabbix agent type jmxproxy items to Zabbix trapper types. Job done.

Enjoy!

[areplogle]

Monitoring Jboss with Zabbix

You can use twiddle as well:

twiddle is a console jmx interface that comes with jboss in $JBOSS_HOME\bin directory. Below are a few examples of info you can get. Look through your jboss jmx console for more.

# JVM Heap Usage
twiddle get "jboss.system:type=ServerInfo" FreeMemory
twiddle get "jboss.system:type=ServerInfo" TotalMemory
twiddle get "jboss.system:type=ServerInfo" MaxMemory

#AP Server Thread Usage & Configuration
twiddle get "jboss.system:type=ServerInfo" ActiveThreadGroupCount
twiddle get "jboss.system:type=ServerInfo" ActiveThreadCount
twiddle get "jboss.jca:service=WorkManagerThreadPool" QueueSize
twiddle get "jboss.jca:service=WorkManagerThreadPool" MaximumQueueSize
twiddle get "jboss.jca:service=WorkManagerThreadPool" MinimumPoolSize
twiddle get "jboss.jca:service=WorkManagerThreadPool" MaximumPoolSize

#Connection Pool Utilization
twiddle get "jboss.jca:name=HPS,service=ManagedConnectionP ool" ConnectionCount
twiddle get "jboss.jca:name=HPS,service=ManagedConnectionP ool" AvailableConnectionCount
twiddle get "jboss.jca:name=HPS,service=ManagedConnectionP ool" MinSize
twiddle get "jboss.jca:name=HPS,service=ManagedConnectionP ool" MaxSize
twiddle get "jboss.jca:name=HPS,service=ManagedConnectionP ool" InUseConnectionCount

#Get Server Thread Dump
twiddle invoke "jboss.system:type=ServerInfo" listThreadDump

# Force GC
twiddle invoke "jboss.system:type=Server" runGarbageCollector
11:27:14,208 INFO [Server] Total/free memory: 133365760/94608856
11:27:14,676 INFO [Server] Hinted to the JVM to run garbage collection
11:27:14,676 INFO [Server] Total/free memory: 133365760/95472528

[Kees Jan Koster]

Centralised Configuration

I note that most of the proposed solutions, while excellent, mean that you have a decentralised configuration. I have written a small library that adds a little Zabbix agent to the Java application. http://www.kjkoster.org/zapcat/

The advantage of Zapcat is that no configration exists on the monitored JMV, other than that you have to compile the agent into your code once.

Obviously, if you are happy to have the configuration of the agent on the machines themselves instead of centralised in Zabbix, ignore my post.