Monitoring Xen dom0?

[cowmix]

I'm running RHEL/CENTOS Xen. Just poking around the forums I don't see a lot of talk about monitoring Xen from the dom0 perspective to get stats on the total resources used and individual domU usage (network, CPU, I/O, etc) into Zabbix.

Am I missing something obvious here?

thanks!

[fjrial]

xen dom-0 monitoring

Hi:

I'm working on monitoring Dom0 stats: cpu usage and memory usage from domains, etc... maybe we can share our knowledgement.

I'm trying to do this with the agent installed only in the dom-0 and with active checks (because all my virtualized hosts are behind nat and thus, I don't need to make new rules in my firewalls, I only need outgoing connectivity on the machines to the zabbix_server).

I'm working in the user parameters on zabbix agent, trying to cut/sed/awk the information from "xm top"

For me, this "ugly" command takes out the domain-name, cpu usage and memory usage:
echo 'quit()' | xm top | tr '\r' '\n' | sed 's/[0-9][;][0-9][0-9][a-Z]/ /g' |
col -bx | sed 1,4d | awk '{print $2,$5,$6}'

Example:
Centos 0.0 261760
Domain-0 0.3 765056


The sed command is because "xm top" print a lot of trash between columns of information so I need to strip them..

xm top information provided (see screenshot attached)

Cheers.
Javi Rial.

[fjrial]

More information:

To allow zabbix user to execute xen commands, I did this:

Modify /etc/sudoers and add:
User_Alias XENUSERS = zabbix
Cmnd_Alias XEN = /usr/sbin/xm
Cmnd_Alias XENTOP = /usr/sbin/xentop
XENUSERS ALL = NOPASSWD: XEN,XENIMG

And comment (also in /etc/sudoers) the line:
#Defaults requiretty

And, I had to change the "xm top" command from my first answer to "sudo xentop"...

echo 'quit()' | sudo xentop | tr '\r' '\n' | sed 's/[0-9][;][0-9][0-9][a-Z]/ /g' |
col -bx | sed 1,4d | awk '{print $2,$5,$6}'

Now, I have to parse this information into several items with the subelements information for each domain in xen-server.
If anyone knows a better way to do this, please, share it.

For Zabbix developers:
It would be fantastic if we could use an item that could store an array structure with all information.
xen.cpu_mem[] will be the array structure which will have the information from the command before:
Domain-0 13.0 745621
centos 3.0 245621

And we would use it to get the information in a way like this:
xen.cpu_mem[0][0]=Domain-0
xen.cpu_mem[0][1]=13.0
xen.cpu_mem[0][2]=745621
xen.cpu_mem[1][0]=centos
xen.cpu_mem[1][1]=3.0
xen.cpu_mem[1][2]=245621

[fjrial]

#User Defined Parameters:

#Number of domains
UserParameter=xen.number,echo 'quit()' | sudo xentop | tr '\r' '\n' | col -bx | sed 1,4d | wc -l

#Name of domain: * should be replaced by a number (representing the domain)
UserParameter=xen.name[*], sudo xm list | sed 1d | cut -d' ' -f 1 | sed -n '$1{p;q}'

#Mem of domain: * should be replaced by a number
UserParameter=xen.mem[*],echo 'quit()' | sudo xentop | tr '\r' '\n' | sed 1,4d | col -bx | sed -e 's/[ ][ ]*/;/g' | cut -d';' -f5 | sed -n '$1{p;q}'

#Cpu of domain: * should be replaced by a number
UserParameter=xen.mem[*],echo 'quit()' | sudo xentop | tr '\r' '\n' | sed 1,4d | col -bx | sed -e 's/[ ][ ]*/;/g' | cut -d';' -f4 | sed -n '$1{p;q}'

[fjrial]

simplicity

Finally, found something easier to deploy in zabbix_agend.conf in Dom-0

### XEN ###
UserParameter=xen.number,sudo xm list | sed 1d | wc -l

UserParameter=xen.name[*],sudo xm list | sed 1d | cut -d' ' -f1 | sed -n '$1{p;q}'

UserParameter=xen.mem[*],sudo xm list | sed 1d | sed 's/[ ][ ]*/;/g' | cut -d';' -f3 | sed -n '$1{p;q}'

UserParameter=xen.cpu_util[*],sudo xm list | sed 1d | sed 's/[ ][ ]*/;/g' | cut -d';' -f6 | sed -n '$1{p;q}'

For monitoring I use this items (based on number of vm's):
(cpu_util[] is an item with custom multiplier *100 and stored as Delta (speed per second) to get the CPU utilization in percentage.. be carefull with hosts with several CPUs, max CPU percentage could be 100%*number of cpus)
xen.cpu_util[1]
xen.cpu_util[2]
xen.cpu_util[3]
xen.mem[1]
xen.mem[2]
xen.mem[3]
xen.name[1]
xen.name[2]
xen.name[3]
xen.number

The last thing I would like (again, these are my whishes , anyway, there is noone reading this ) ) is that based on xen.number create as much as needed xen.cpu_util, xen.name and xen.mem: let me explain it:
If we have xen.number=4 we should have xen.mem[1],xen.mem[2],xen.mem[3],xen.mem[4], and the same with the other items.. but I want this to be done in an automatic way, with a trigger that looks at the value in xen.number and creates more items based on that value..

[fjrial]

this approach have a problem:

it uses the number enclosed in brackets to get the n line of xm list.. but the domains in xm list can dynamically change:

if domain number 2 goes down, the domain 3 would be the domain 2, and so on..

So the value of the item does not anymore belongs to the same domain..

[Anton]

Try the check_xen-Plugin from Nagios:

http://www.monitoringexchange.org/inventory/Check-Plugins/Operating-Systems/Linux/check_xen

[fjrial]

I know the nagios plugin, we have been using nagios for a long time.. but know we have changed network monitoring to zabbix, and now we are doing the same with network services.

What I'm trying to know is how many domu has dom-0, name of each domu, the cpu usage and memory usage foreach one, graphs with domus cpu and memory usage and all of this in the easiest way that we can find: we want to monitor the dom-0 servers in zabbix just adding a template with items preconfigured.. without knowing how many domus has the dom-0, no matter if I start a new domu, zabbix would have to know that a new domu has started, etc...

[Rotzooi]

if you change

UserParameter=xen.cpu_util[*],sudo xm list | sed 1d | sed 's/[ ][ ]*/;/g' | cut -d';' -f6 | sed -n '$1{p;q}'

to

UserParameter=xen.cpu_util[*],sudo xm list $1 | sed 1d | sed 's/[ ][ ]*/;/g' | cut -d';' -f6

you can use the xen.cpu_util key as follow:
xen.cpu_util[Domain-0] for dom0
and
xen.cpu_util[DomU] for a domu called DomU
or
xen.cpu_util[Zabbix] for a domu called Zabbix

[fjrial]

Thanks,

That's a very good approach to what I'm looking for..

Still has a problem, you have to know how many domu has Dom0 and the name of every domu.

I'll try to investigate deeper in order to make this in an automatic way:
check every X min for new domu and their names and auto create the necessary items with the parameters obtained from the check.

I need this because we have a large installations of xen servers, and domu can migrate between servers.

I think that it will be hard to achieve this behaviour, but for me, it's necessary.

I'll keep you informed on this.

Thank you again.

[jamied66]

Has anyone looked into pulling statistics using the libvirt python module?
SNMP checks also might be feasible.

[fjrial]

SNMP or any library to get stats are OK, but the main problem is how to create items automatically based on stats got from those protocols/libraries/scripts... because we have a lot of virtual machines, they are shared via NFS, and any virtual machine can belong to any physical xen server.. so the script needs the ability to create/eliminate items on the host

I mean: I have an script installed on every Xen Server, that returns all those stats (based on nagios plugin) but, I need that, based on the returned data, the script create (or not) more items for that host in Zabbix... something like a command line tool, similar to zabbix_sender, but "zabbix_create"

Other option, would be, that zabbix_sender checks if exists the item, and if not, create it.. but, for what I've seen, zabbix_sender doesn't work in this way, it checks for item, but if it not exists, then if fails..

I'll tray to work in this solution, modifying zabbix_sender source code...

Any advice on how to do this would be very appreciated

[mjsr]

#Number of domains
UserParameter=xen.number,echo 'quit()' | sudo xentop | tr '\r' '\n' | col -bx | sed 1,4d | wc -l

Hola Javier,

Just my two cents about that command. We're running Xen also and this thread looks promising.

Anyway, just remember that xentop's behaviour is like top command: it automaticly iterates keeping the screen refreshed from time to time.

The man page shows the possibility of iterating one time so the "echo 'quit()'" is not necessary (at least here).

So the:

 echo 'quit()' | xentop | tr '\r' '\n' | col -bx | sed 1,4d | wc -l

Goes to:

 xentop -i 1 | tr '\r' '\n' | col -bx | sed 1,4d | wc -l

Just, like I said, a 2 cent contribution great work there.

./mjsr

[fjrial]

thank you Marcos..

I would also add a "-b" option.. (batch mode). With -b, the output is less dirty..

I didn't have to much time to work on zabbix during last months.. but found something that could be useful.. just sharing information ...

The original post at xensource:

http://lists.xensource.com/archives/html/xen-users/2010-06/msg00604.html

The info:

http://practicalsystemadmin.blogspot.com/2010/06/monitor-xen-domus-from-dom0-with.html

Cheers

[mjsr]

I would also add a "-b" option.. (batch mode). With -b, the output is less dirty..

Didn't notice it. Thank you for that and those links.