Getting a process list

[crb]

When I collect my items from my agents every minute, I'd also like to get a process list, perhaps by a user parameter (the output of top in batch mode, or ps, etc).

That way, when I get sent a message saying "Processor load too high", instead of having to log into the machine, and guess what might have been going on from the LA and number of procs at the time, I can see exactly which processes were running.

If I collected a big table of data like this, I wouldn't want it saved for more than the last minute (it would fill up the database in no time!) - it only exists so it can be send out in the notification.

Is this possible? Does anyone have Zabbix configured like this?

[walterheck]

That does not seem like a bad idea. Thinking of how to achieve that with current means, the best way I can come up with is to use a mediatype script that sends you the email (instead of plain SMTP). Then, you couldhave that script retrieve the top list by ssh from the machined that you are getting the alert from. Nasty, but I don't see why it wouldn't work..

[crb]

That does not seem like a bad idea. Thinking of how to achieve that with current means, the best way I can come up with is to use a mediatype script that sends you the email (instead of plain SMTP). Then, you couldhave that script retrieve the top list by ssh from the machined that you are getting the alert from. Nasty, but I don't see why it wouldn't work..

That's a damned good idea, actually! I already have a script which sends the e-mail, as I needed that to use the local 'sendmail' binary. Now just to hack the output to reliably get the machine name...

[hopkinsju]

Details?

Hi there. I'm wanting to do the exact same thing - get a list of processes to clarify the cause of "CPU load too high" alerts. It would seem that you've found a solution. I was wondering if you have it up and running and can possibly provide me with more details of how it is done. This thread is at the top of Google's results for "zabbix list of processes", so if anyone else is looking this thread will likely be where they end up.

[crb]

I said the idea was good but never did any more. I encourage you to whip it up - shouldn't be hard at all - and then post it here!

[hopkinsju]

Ok, first thing is to add a UserParameter. Add this to /etc/zabbix/zabbix_agentd.conf for all of the hosts you want to get this information from.

UserParameter=cpu.tasks,top -b -n 1 |head -n 20

(The naming of this "cpu.tasks" could be changed, but that's what we went with)

*Remember to restart the agent after adding this on all of your hosts

Then you need to create a new media type called "Email Task List" of type "script" and enter "email-tasks.sh" for the script name.



Next you will want to add an action. Mine is configured like this:



Everything on the right is meaningless except for the Trigger description like "Processor load is too high" which is going to make it actually run. On the left it's pretty straightforward but worth mentioning is the subject "{HOST.DNS}" which ends up being passed as the second argument to our email-tasks.sh script.

Now go to your user profile and add a Media Type. Choose the Email Task List one and in the "send to" field enter the full path to the script we are about to create: "/etc/zabbix/alert.d/email-tasks.sh"



Then create a script on your zabbix server - /etc/zabbix/alert.d/email-tasks.sh like so:

#!/bin/bash
zabbix_get -s $2 -p 10050 -k cpu.tasks | mail -s "Task List from $2" [email protected],[email protected]

Assuming you have all of your hosts configured with dns names, then then $2 will be substituted with the correct hostnames by virtue of {HOST.DNS} being in the Subject field of your action.

Now just:

/etc/init.d/zabbix-server restart

And you should be good to go.

The other possible approach here, and this may not be worth mentioning, is rather than using zabbix_get with a UserParameter, you could just write the script to ssh to the host and run the command... The zabbix_get method is nice because you don't have to deploy ssh keys to all of your hosts and there's no need to give zabbix a shell... The only issue is that if you decide you want to add another email address, or you want head -n 30 instead of 20, you've got a lot of updating to do. There may be a more sophisticated way of doing this, but these are my only two thoughts on it. Plus, it's working

[Ori0n]

Nice work hopkins! For anyone that might be interested in the other method you mentioned, I quickly hacked something together. In this method you should only have to touch the target host for the SSH key exchange.

A potential limitation to this method is that the top script is executed once the trigger evaluates and not at the time of normal data collection. So there could be a small delay between when Zabbix detects a problem and the top output is gathered. I think this should be negligible.

First do the SSH key exchange and make sure you can execute a command with no password prompt.

Then write the script; something like:
/usr/local/bin/zabbix/topSend.sh

#!/bin/bash

HOST=$1
TOPFILE=/tmp/top-$HOST.txt

/usr/bin/ssh -p22 $HOST "top -n1 -b | head -n 13" > $TOPFILE

/usr/local/bin/sendEmail -f [email protected] -t [email protected] -u "Top output from $HOST" -s smtp.foo.com -xu user -xp pass -o message-file=$TOPFILE -m ""

exit

Make the necessary changes for your mail handler. Test the script from the command line.

In Zabbix, create an action that will fire based on your trigger and configure the action to run a remote command


That's it, you should see an email like when you enter a PROBLEM state:

top - 15:23:37 up 41 days, 14:00,  1 user,  load average: 1.38, 1.47, 1.39
Tasks: 432 total,   1 running, 431 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6%us,  0.8%sy,  0.7%ni, 96.0%id,  0.8%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   8173984k total,  6954048k used,  1219936k free,   217380k buffers
Swap: 10223608k total,      156k used, 10223452k free,  4302732k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 3612 mysql     15   0 2773m 1.5g 5052 S  5.9 19.8   4071:31 mysqld             
20199 --------  20   0 19540 2184 1096 S  3.9  0.0   0:00.02 snmpget            
19933 zabbix    20   5  1812  612  532 S  2.0  0.0   0:00.02 fping              
20048 zabbix    20   5  1812  612  532 S  2.0  0.0   0:00.01 fping              
20180 zabbix    15   0 13004 1260  720 R  2.0  0.0   0:00.01 top                
26601 zabbix    20   5  390m  46m  44m S  2.0  0.6  41:20.53 zabbix_proxy

[hopkinsju]

Yeah, that's great! I was led to believe that no arguments were passed when you used remote command instead of script Media type. I didn't consider that you could use variables like {IPADDRESS} in your remote command - this would allow you to pass so much more useful information to eventually be included in your email.

Also, sending the output of top to a local file and then sending the email from your zabbix server rather than a host known to be having problems makes a lot more sense.

After seeing what you've done, I think it's the best way. Good call.