How to monitor a process's thread count?

[tim.mooney]

I need to monitor how many threads a particular Linux process (Tomcat) has spawned.

Zabbix agent doesn't seem to have a thread.num[] equivalent to proc.num[] . I also don't see any way to make proc.num[] count threads.

I've found some old bug reports that seem to indicate that proc.num[] used to count all the threads for a multi-threaded process, but that behavior wasn't what many users wanted, so the behavior was changed to be only processes.

Is a custom item using a script the recommended way to do this?

Thanks,

Tim

[gert.derouck]

HI, I'm also monitoring the number of threads of JBOSS containers. I use following UserParameter for this. You can adapt it for your needs.

UserParameter=sw.jboss_numthreads[*],if [ `ps -C java -o pid,command|grep "\-D\[$1\]"|wc -l` -ge 1 ]; then grep '^Threads:' /proc/`ps -C java -o pid,command|grep "\-D\[$1\]"|awk '{print $ 1}'`/status|awk '{print $NF}'; else echo 0; fi

[tim.mooney]

HI, I'm also monitoring the number of threads of JBOSS containers. I use following UserParameter for this. You can adapt it for your needs.

UserParameter=sw.jboss_numthreads[*],if [ `ps -C java -o pid,command|grep "\-D\[$1\]"|wc -l` -ge 1 ]; then grep '^Threads:' /proc/`ps -C java -o pid,command|grep "\-D\[$1\]"|awk '{print $ 1}'`/status|awk '{print $NF}'; else echo 0; fi

Hi Gert!

Thanks for the quick reply and for including the UserParameter that you're using!

I started to adapt one of our other UserParameter's to check thread-count via the following script:

#!/bin/sh

# Author : Tim Mooney <[email protected]>
# Purpose: Until Zabbix gets a thread.num[] monitor similar to proc.num[],
#          this script can be used as a custom item to get the number of
#          threads.

# Accepts 0, 1, or 2 arguments.  
#
# With 0 arguments, it reports the number of threads system-wide.
# With 1 argument, which must be the program name, it reports on the
#  number of threads running for that program
# With 2 arguments, program name and user name, it reports on the number of
#  threads running for that program and user.

PATH=/usr/bin:/bin; export PATH

#
# With no arguments, just output the total # of threads, for all processes:
#
if test $# -eq 0 ; then
        ps -eL | wc -l | sed -e 's/^ *//'
        exit 0
fi

#
# with 1 argument, output the number of threads for that program
#
if test $# -eq 1 ; then
        ps -eLf | egrep ":[0-9][0-9] $1" | wc -l | sed -e 's/^ *//'
        exit 0
fi

#
# with 2 arguments, output the number of threads for that program user.
#
if test $# -eq 2 ; then
        ps -u "$2" -Lf | egrep ":[0-9][0-9] $1" | wc -l | sed -e 's/^ *//'
        exit 0
fi

#
# an error state.
#
echo -1
exit 1

But then I realized that I can't easily do a UserParameter that can take a variable number of parameters this way, so I'll have to either always pass 2 parameters (program name and username) or I need to do more argument processing in the script, to check whether an argument is present but empty.

Thanks for sharing your solution to the same problem! It's good to know I'm probably headed down the best path.

Tim

[gert.derouck]

Or you could just create 3 UserParameters: 1 without arg, 1 with 1 arg, 1 with 2 arg. All pointing to the same script.

[tim.mooney]

I had that idea too Gert. It's not super elegant, but it is by far the easiest way to do it, so it might be the path I end up taking. Thanks for the suggestion!

[tim.mooney]

Just to close out this thread:

Until zabbix_agentd gets a thread.num[] builtin item key like proc.num[], I took Gert's advice and used the script I posted above with 3 different UserParameters, each accepting a different number of arguments:

#
# Zabbix doesn't currently have a thread.num that works like proc.num[],
# so create our own.
#
# Also, use 3 different names so we easily can support 3 different # of
# arguments.
#
UserParameter=ndsu.thread.num.all,/usr/local/libexec/monitoring/ndsu.thread.num
UserParameter=ndsu.thread.num.prog[*],/usr/local/libexec/monitoring/ndsu.thread.num "$1"
UserParameter=ndsu.thread.num[*],/usr/local/libexec/monitoring/ndsu.thread.num "$1" "$2"

(we prefix all our custom item keys with our organization name, hence the "ndsu." at the start of each item key).

Thanks again to Gert for the suggestions and information!