Extending zabbix to monitor apache 2.0.x

[frankcheong]

I have tried to extend zabbix 1.8 running on Linux/FreeBSD to monitor apache version 2.0.x (most of the monitoring help is on version 2.2.x instead which does not really work on 2.0.x) with the following steps and problem:-

1. Enable active check on zabbix agentd with DisableActive=0 on zabbix_agentd.conf (is this step really necessary ?)

2. Modify the zabbix_agentd.conf with the following lines:-
UserParameter=apache2.CPULoadUser,/usr/sbin/apachectl status | /bin/grep

"CPU load" | /bin/awk '{print $3}'
UserParameter=apache2.CPULoadSystem,/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $4}'
UserParameter=apache2.CPULoadPercentage,/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $8}'
UserParameter=apache2.RequestPerSecond,/usr/sbin/apachectl status | /bin/grep "requests/sec" | /bin/awk '{print $1}'
UserParameter=apache2.BytePerSecond,/usr/sbin/apachectl status | /bin/grep "B/second" | /bin/awk "'multiplier=1;if (substr($5,1,1)!="B") multiplier*=1024; if (substr($5,1,1)=="M") multiplier*=1024; if (substr($5,1,1)=="G") multiplier*=1024*1024; print $4*multiplier}'
UserParamter=apache2.BytePerRequest,/usr/sbin/apachectl status | /bin/grep "B/request" | /bin/awk '{multiplier=1;if (substr($8,1,1)!="B") multiplier*=1024; if (substr($8,1,1)=="M") multiplier*=1024; if (substr($8,1,1)=="G") multiplier*=1024*1024; print $7*multiplier}'
UserParameter=apache2.BusyWorker,/usr/sbin/apachectl status | /bin/grep "requests currently being processed," | /bin/awk '{print $1}'
UserParameter=apache2.IdleWorker,/usr/sbin/apachectl status | /bin/grep "idle workers" | /bin/awk '{print $6}'
UserParameter=apache2.WaitingForConnection,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "WaitingForConnection" | /bin/awk '{print $3}'
UserParameter=apache2.StartingUp,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "StartingUp" | /bin/awk '{print $3}'
UserParameter=apache2.ReadingRequest,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ReadingRequest" | /bin/awk '{print $3}'
UserParameter=apache2.SendingReply,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "SendingReply" | /bin/awk '{print $3}'
UserParameter=apache2.Keepalive,/usr/sbin/apachectl status | /bin/awk -f checkWorker.awk | /bin/grep "Keepalive" | /bin/awk "'print $3}'
UserParameter=apache2.DNSLookup,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "DNSLookup" | /bin/awk '{print $3}'
UserParameter=apache2.ClosingConnection,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ClosingConnection" | /bin/awk '{print $3}'
UserParameter=apache2.Logging,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "Logging" | /bin/awk '{print $3}'
UserParameter=apache2.GracefullyFinishing,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "GracefullyFinishing" | /bin/awk '{print $3}'
UserParameter=apache2.IdleCleanup,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "IdleCleanup" | /bin/awk '{print $3}'
UserParameter=apache2.OpenSlot,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "OpenSlot" | /bin/awk '{print $3}'

3. Create the /home/zabbix/checkWorker.awk as below:-

BEGIN {
  Mode="start";
  Key="_SRWKDCLGI.";
  Description[0]="WaitingForConnection";
  Description[1]="StartingUp";
  Description[2]="ReadingRequest";
  Description[3]="SendingReply";
  Description[4]="Keepalive";
  Description[5]="DNSLookup";
  Description[6]="ClosingConnection";
  Description[7]="Logging";
  Description[8]="GracefullyFinishing";
  Description[9]="IdleCleanup";
  Description[10]="OpenSlot";
  for (Position = 0 ; Position <= 10 ; Position++)
    Value[Position] = 0;
}
{
  if (Mode=="start") {
    if (/ idle workers/)
      Mode="Processing";
  }
  else if (Mode=="Processing") {
    if (/Scoreboard Key:/)
      Mode="End";
    else
      for (Post = 0; Post <= length;Post++) {
        Slot=index(Key, substr($0, Post, 1));
        if (Slot > 0)
          Value[Slot-1]+=1;
      }
  }
}

END {
  for ( Position = 0 ; Position <= 10 ; Position++)
    print Description[Position] " : " Value[Position];
}

4. Add item apache.* (CPULoadUser, CPULoadSystem, ..., OpenSlot etc) using the UI.

5. Restart the zabbix agent.

While I cannot locate those data on the zabbix. When I tried to run the following command :-

root@hostname #zabbix_agentd -t apache2.ReadingRequest
apache2.ReadingRequest                              [m|ZBX_NOTSUPPORTED]

I then tried to run the command using the zabbix user account and it works:-

zabbix@hostname #/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ReadingRequest" | /bin/awk '{print $3}'
0

I then turn on debugging by putting DebugLevel 4 inside the zabbix_agentd.conf and then restart the zabbix_agentd and found the following error message inside the log:-

 18868:20100115:094510.936 Requested [apache2.ReadingRequest]
Unable to find or create ELinks config directory. Please check if you have $HOME variable set correctly and if you have write permission to your home directory.
 18868:20100115:094513.966 Before
 18868:20100115:094513.966 Run remote command [/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ReadingRequest" | /bin/awk '{print $3}'] Result [1] [0]...
 18868:20100115:094513.966 Sending back [0]

BTW, zabbix user have write permission on /home/zabbix/ and all files inside the folder.

What have I done wrong? Any help is much appreciated.

[frankcheong]

During the troubleshooting, I spotted some typo with the zabbix_agend.conf and the correct one should be :-

UserParameter=apache2.CPULoadUser,/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $3}'
UserParameter=apache2.CPULoadSystem,/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $4}'
UserParameter=apache2.CPULoadPercentage,/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $8}'
UserParameter=apache2.RequestPerSecond,/usr/sbin/apachectl status | /bin/grep "requests/sec" | /bin/awk '{print $1}'
UserParameter=apache2.BytePerSecond,/usr/sbin/apachectl status | /bin/grep "B/second" | /bin/awk '{multiplier=1;if (substr($5,1,1)!="B") multiplier*=1024; if (substr($5,1,1)=="M") multiplier*=1024; if (substr($5,1,1)=="G") multiplier*=1024*1024; print $4*multiplier}'
UserParamter=apache2.BytePerRequest,/usr/sbin/apachectl status | /bin/grep "B/request" | /bin/awk '{multiplier=1;if (substr($8,1,1)!="B") multiplier*=1024; if (substr($8,1,1)=="M") multiplier*=1024; if (substr($8,1,1)=="G") multiplier*=1024*1024; print $7*multiplier}'
UserParameter=apache2.BusyWorker,/usr/sbin/apachectl status | /bin/grep "requests currently being processed," | /bin/awk '{print $1}'
UserParameter=apache2.IdleWorker,/usr/sbin/apachectl status | /bin/grep "idle workers" | /bin/awk '{print $6}'
UserParameter=apache2.WaitingForConnection,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "WaitingForConnection" | /bin/awk '{print $3}'
UserParameter=apache2.StartingUp,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "StartingUp" | /bin/awk '{print $3}'
UserParameter=apache2.ReadingRequest,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ReadingRequest" | /bin/awk '{print $3}'
UserParameter=apache2.SendingReply,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "SendingReply" | /bin/awk '{print $3}'
UserParameter=apache2.Keepalive,/usr/sbin/apachectl status | /bin/awk -f checkWorker.awk | /bin/grep "Keepalive" | /bin/awk '{print $3}'
UserParameter=apache2.DNSLookup,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "DNSLookup" | /bin/awk '{print $3}'
UserParameter=apache2.ClosingConnection,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ClosingConnection" | /bin/awk '{print $3}'
UserParameter=apache2.Logging,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "Logging" | /bin/awk '{print $3}'
UserParameter=apache2.GracefullyFinishing,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "GracefullyFinishing" | /bin/awk '{print $3}'
UserParameter=apache2.IdleCleanup,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "IdleCleanup" | /bin/awk '{print $3}'
UserParameter=apache2.OpenSlot,/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "OpenSlot" | /bin/awk '{print $3}'

I have tried all command line and it all returned value as expected, while the result is the same. The second strange thing I found that zabbix only execute and report error with apache2.ReadingRequest but not others. After double checking, the apache2.ReadingRequest has been configured as Zabbix Agent instead of Zabbix Agent (Active). Does it means I do not need to setup as active check ? Simply passive check with zabbix_agent can do the trick already?

[frankcheong]

Eventually, I get it partially working with the following findings and hiccup:-

Final configuration:-

Create the shell script name apache2Status.sh under /home/zabbix/ as below

#!/bin/bash

export HOME=/home/zabbix
result=0

if [ "$1" == "CPULoadUser" ] ; then
  load=`/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $3}'`
  result=${load:1}
elif [ "$1" == "CPULoadSystem" ] ; then
  load=`/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $4}'`
  result=${load:1}
elif [ "$1" == "CPULoadPercentage" ] ; then
  load=`/usr/sbin/apachectl status | /bin/grep "CPU load" | /bin/awk '{print $8}'`
  result=${load:0:${#load}-1}
elif [ "$1" == "RequestPerSecond" ] ; then
  result=`/usr/sbin/apachectl status | /bin/grep "requests/sec" | /bin/awk '{print $1}'`
elif [ "$1" == "BytePerSecond" ] ; then
  result=`/usr/sbin/apachectl status | /bin/grep "B/second" | /bin/awk '{multiplier=1;if (substr($5,1,1)!="B") multiplier*=1024; if (substr($5,1,1)=="M") multiplier*=1024; if (substr($5,1,1)=="G") multiplier*=1024*1024; print $4*multiplier}'`
elif [ "$1" == "BytePerRequest" ] ; then
  result=`/usr/sbin/apachectl status | /bin/grep "B/request" | /bin/awk '{multiplier=1;if (substr($8,1,1)!="B") multiplier*=1024; if (substr($8,1,1)=="M") multiplier*=1024; if (substr($8,1,1)=="G") multiplier*=1024*1024; print $7*multiplier}'`
elif [ "$1" == "BusyWorker" ] ; then
  result=`/usr/sbin/apachectl status | /bin/grep "requests currently being processed," | /bin/awk '{print $1}'`
elif [ "$1" == "IdleWorker" ] ; then
  result=`/usr/sbin/apachectl status | /bin/grep "idle workers" | /bin/awk '{print $6}'`
elif [ "$1" == "WaitingForConnection" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "WaitingForConnection" | /bin/awk '{print $3}'`
elif [ "$1" == "StartingUp" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "StartingUp" | /bin/awk '{print $3}'`
elif [ "$1" == "ReadingRequest" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ReadingRequest" | /bin/awk '{print $3}'`
elif [ "$1" == "SendingReply" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "SendingReply" | /bin/awk '{print $3}'`
elif [ "$1" == "Keepalive" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "Keepalive" | /bin/awk '{print $3}'`
elif [ "$1" == "DNSLookup" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "DNSLookup" | /bin/awk '{print $3}'`
elif [ "$1" == "ClosingConnection" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "ClosingConnection" | /bin/awk '{print $3}'`
elif [ "$1" == "Logging" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "Logging" | /bin/awk '{print $3}'`
elif [ "$1" == "GracefullyFinishing" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "GracefullyFinishing" | /bin/awk '{print $3}'`
elif [ "$1" == "IdleCleanup" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "IdleCleanup" | /bin/awk '{print $3}'`
elif [ "$1" == "OpenSlot" ] ; then
  result=`/usr/sbin/apachectl status | /bin/awk -f /home/zabbix/checkWorker.awk | /bin/grep "OpenSlot" | /bin/awk '{print $3}'`
fi

echo $result;

Create the awk script name checkWorkder.awk under /home/zabbix/ as below:-

BEGIN {
  Mode="start";
  Key="_SRWKDCLGI.";
  Description[0]="WaitingForConnection";
  Description[1]="StartingUp";
  Description[2]="ReadingRequest";
  Description[3]="SendingReply";
  Description[4]="Keepalive";
  Description[5]="DNSLookup";
  Description[6]="ClosingConnection";
  Description[7]="Logging";
  Description[8]="GracefullyFinishing";
  Description[9]="IdleCleanup";
  Description[10]="OpenSlot";
  for (Position = 0 ; Position <= 10 ; Position++)
    Value[Position] = 0;
}
{
  if (Mode=="start") {
    if (/ idle workers/)
      Mode="Processing";
  }
  else if (Mode=="Processing") {
    if (/Scoreboard Key:/)
      Mode="End";
    else
      for (Post = 0; Post <= length;Post++) {
        Slot=index(Key, substr($0, Post, 1));
        if (Slot > 0)
          Value[Slot-1]+=1;
      }
  }
}

END {
  for ( Position = 0 ; Position <= 10 ; Position++)
    print Description[Position] " : " Value[Position];
}

Edit the zabbix_agentd.conf append the following :-

UserParameter=apache2.CPULoadUser,/home/zabbix/apache2Status.sh CPULoadUser
UserParameter=apache2.CPULoadSystem,/home/zabbix/apache2Status.sh CPULoadSystem
UserParameter=apache2.CPULoadPercentage,/home/zabbix/apache2Status.sh CPULoadPercentage
UserParameter=apache2.RequestPerSecond,/home/zabbix/apache2Status.sh RequestPerSecond
UserParameter=apache2.BytePerSecond,/home/zabbix/apache2Status.sh BytePerSecond
UserParamter=apache2.BytePerRequest,/home/zabbix/apache2Status.sh BytePerRequest
UserParameter=apache2.BusyWorker,/home/zabbix/apache2Status.sh BusyWorker
UserParameter=apache2.IdleWorker,/home/zabbix/apache2Status.sh IdleWorker
UserParameter=apache2.WaitingForConnection,/home/zabbix/apache2Status.sh WaitingForConnection
UserParameter=apache2.StartingUp,/home/zabbix/apache2Status.sh StartingUp
UserParameter=apache2.ReadingRequest,/home/zabbix/apache2Status.sh ReadingRequest
UserParameter=apache2.SendingReply,/home/zabbix/apache2Status.sh SendingReply
UserParameter=apache2.Keepalive,/home/zabbix/apache2Status.sh Keepalive
UserParameter=apache2.DNSLookup,/home/zabbix/apache2Status.sh DNSLookup
UserParameter=apache2.ClosingConnection,/home/zabbix/apache2Status.sh ClosingConnection
UserParameter=apache2.Logging,/home/zabbix/apache2Status.sh Logging
UserParameter=apache2.GracefullyFinishing,/home/zabbix/apache2Status.sh GracefullyFinishing
UserParameter=apache2.IdleCleanup,/home/zabbix/apache2Status.sh IdleCleanup
UserParameter=apache2.OpenSlot,/home/zabbix/apache2Status.sh OpenSlot

Create corresponding item with type "zabbix_agent" but not "zabbix_agent (active)", seems like I don't exactly know what is the mean by zabbix_agent active and active check.

While it still have some hiccup as below:-

I have changed the owner of the script apache2Status.sh as well as checkWorker.awk as zabbix:zabbix. Have done a ps -ef and found that the zabbix_agent is running as zabbix while I found from the zabbix_agentd.log that whenever it invoke the apache2Status.sh, it complained "/bin/bash: /root/.bashrc: Permission denied" while the /home/zabbix/.bashrc does not refer to the /root/.bashrc directly nor indirectly which this error is kind of strange. I have then changed the owner of the script apache2Status.sh as root:zabbix and the result is the same. I dare not turn on the Sticky Bit which may turn out to be a potential loop hole. What else should I check/do?

In addition, when I issue zabbix_agentd -t apache2.BytePerSecond it complained with "apache2.BytePerSecond [m|ZBX_NOTSUPPORTED]" while in the graph I see that zabbix is able to collect figures (at least once). Why is it complained with ZBX_NOTSUPPORTED ?

[frankcheong]

BTW, I have created those item under a specific host and every things (with problem mentioned in above post) while after I have bind those item to an template instead, it failed.

Can I extend zabbix with new item that is bind to template instead of individual host?

With that I can create one generic graph as well as screen for all host under the same template which they all have apache running instead of having creating an individual graph as well as screen for each individual host. Don't mention that I have to also create those item under each host.

[frankcheong]

Strange, the graph can be shown without problem today when I back to office, while I have monitored the graph for around 15 min after making the change yesterday and I still got an empty graph.

Anyway, this problem is not existed.

Before I can contribute these scripts to the wiki or cookbook forum, can someone help with the question that I have below:-

I have changed the owner of the script apache2Status.sh as well as checkWorker.awk as zabbix:zabbix. Have done a ps -ef and found that the zabbix_agent is running as zabbix while I found from the zabbix_agentd.log that whenever it invoke the apache2Status.sh, it complained "/bin/bash: /root/.bashrc: Permission denied" while the /home/zabbix/.bashrc does not refer to the /root/.bashrc directly nor indirectly which this error is kind of strange. I have then changed the owner of the script apache2Status.sh as root:zabbix and the result is the same. I dare not turn on the Sticky Bit which may turn out to be a potential loop hole. What else should I check/do? I think I shouldn't ignore the Permission denied message, right?

In addition, when I issue zabbix_agentd -t apache2.BytePerSecond it complained with "apache2.BytePerSecond [m|ZBX_NOTSUPPORTED]" while in the graph I see that zabbix is able to collect figures (at least once). Why is it complained with ZBX_NOTSUPPORTED?

Thx very much for any help in advance.