Monitoring Xen dom0?

[Slash]

Hello everyone,

Sorry to bump an old post but I used this thread as a base to make my own UserParameters.

You can find all the command that I use on my wiki here : http://aceslash.net/index.php?title=Sysadmin:xen_zabbix

I get the following from those UserParameters :

1 Number of running domains
2 Total memory seen by Xen
3 Memory currently used by the system
4 Free memory for domUs
5 CPU load on the server in %, add all domUs + dom0 cpu load
6 CPU load on the server in %, all domUs, excluding the dom0
7 Memory used on the server, in KB, excluding the dom0
8 Memory used on the server, in %, excluding the dom0

I use it to monitor several Xen servers running RHEL or CentOS 5.x with Xen 3.

[fjrial]

great!

Looks great!!

I'll take a look..

Thanks for sharing the info

[wdingus]

FWIW, XenServer requires a license and if you use the *free* license it works for 1 year and requires renewal. Here's how we monitor that:

#!/bin/bash
#
EXPDATE=$(cat /etc/xensource/license  | grep -o "human_readable_expiry=\"[0-9]*-[0-9]*-[0-9]*" | cut -d\" -f2-)
EXPSEC=$(date -d $EXPDATE +%s)
TODAYSEC=$(date +%s)
((diff_sec=EXPSEC-TODAYSEC))
echo - | awk -v SECS=$diff_sec '{printf "%d\n",SECS/(60*60*24)}'

That outputs the number of days left until license expiry. A prerequisite is either giving the zabbix user sudo ability or changing the license file so that it can see the contents. For instance:

chmod a+r /etc/xensource/license

[jamied66]

Thanks Slash.

Looks like a great way to get some shotgun data!

-jamied66

[Jason]

I'm interested in this too. About to start on setting up some monitoring for our Xen farm.

It'd be nice to automate getting all the virtual names. I can see that it'd be easy to set up the agent such that you can just pull back the virtual information by name once you have that info (as shown above).

Obviously if the virtuals are being moved round then it's still hard to keep track on what virtuals are there and that would be a manual process.

My musings below on how i think something could work... It'll be about a month or so before I can start to look at something like this...

I'm guessing must be able to pull out a comma seperated list of all the virtuals running on each host...

Then I'm guessing using the API it should be possible to create a set of items for each virtual on each host and flag up when the number of virtuals changes. That script could then be run on the zabbix server and keep track of all the virtuals on each host and you'd just need to put the agent config on each host and a bit of tweaking to make sure permissions were ok.

Jason

[Slash]

Then I'm guessing using the API it should be possible to create a set of items for each virtual on each host and flag up when the number of virtuals changes. That script could then be run on the zabbix server and keep track of all the virtuals on each host and you'd just need to put the agent config on each host and a bit of tweaking to make sure permissions were ok.

If you can achieve something like that, this would indeed be an impressive (and useful!) improvement :-)

I'm still looking for a way to associate virtual machines to their virtual host in a logical way into Zabbix, having a stacked graph with all the load of the virtual machines for a given host would be very useful for example.

[vaq]

slash script

Dear Slash,

Thanks for your script, however I seem to have some issues with it:


# /usr/sbin/xentop -i 1 -b| grep "Mem:"|sed 's/[ ][ ]*/;/g'|cut -d';' -f2 |sed 's/k//g'

Produces no result.

I am running CentOS 5.5 with Xen 3.4

Example of xentop batch:
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID
Domain-0 -----r 2550759 26.6 1048620 2.1 no limit n/a 8 4 0 0 0 0 0 0 0
vm. --b--- 15081 1.6 262144 0.5 262144 0.5 4 0 0 0 2 3279 52139 868819 0

Any ideas?

[Slash]

Hello Vaq,

You Xen version gives a different output. The same as Xen 4 actually. I have one of those (a Debian Squeeze box).

I must admit that I've done a really ugly fix to have some result... I still have to look at the possible way to have something cleaner.

What I've basically done is set a hard value for the memory. For example on this server, I have 64GB RAM and 2GB reserved for dom0 so I changed my "UserParameters" to those:

## Number of domains
UserParameter=xen.domUs_number,sudo /usr/sbin/xm list |sed 1,2d|wc -l
## Total memory seen by Xen
UserParameter=xen.total_memory,echo "67098852"
## Memory currently used by the system
UserParameter=xen.used_memory,used=0; for mem in `sudo /usr/sbin/xm list | sed 1,2d | sed 's/[ ][ ]*/;/g' | cut -d';' -f3`; do used=`echo $used + $mem|bc`; done; echo `echo "($used + 2000) * 1024"|bc`
## Free memory for domUs
UserParameter=xen.free_memory,total=67098852;used= 0; for mem in `sudo /usr/sbin/xm list | sed 1,2d | sed 's/[ ][ ]*/;/g' | cut -d';' -f3`; do used=`echo $used + $mem|bc`; done; used=`echo "($used + 2000) * 1024"|bc`; echo `echo $total - $used|bc`
## Total load for every domUs
UserParameter=xen.domUs_load,total=0;for cpu in `sudo /usr/sbin/xentop -i 1 -b|sed 1,2d|sed 's/^......//'|sed 's/[ ][ ]*/;/g'|cut -d';' -f4`; do total=`echo $total + $cpu|bc`;done; echo $total

It's so ugly it hurts my eyes but I don't have time to look at it yet

[fjrial]

well.. after some time without taking a look at this task, last days worked hard on this.

Now, I´ve this approach:

-have a perl script which basically does: auto-discover xen server running nagios nrpe plugin (with check_xen), execute check_nrpe -h $HOST -c check_xen, parse output, auto-add this xenserver with a predefined template, auto-add items with mem,cpu and domain name for each xenserver discovered.
-also detects if one domu migrates from one xen-server to another and moves its items to new xen-server in zabbix (so history of domu remains intact).

Now, future plans: replace nrpe plugin from nagios with (if it´s possible) a one command which zabbix-agent could execute it and that will output the same that nagios plugin does.

at this moment, worst part is that my perl script needs nagios plugin output... any help on this will be really appreciated

More future: auto-create graphs (cpu usage and mem usage) from domus foreach xen-server

All automated steps are done with zabbix api (using perl json)

Tomorrow I´ll post the scripts.. now I´m at home..
Please, if I forget to post the script, please comment in this thread.. I´ll post as soon as I can.

Cheers.

[fjrial]

For those who doesn´t know how check_xen nrpe plugin works.. this is the output that it generates:

XEN OK - VM : Domain-0: hosting1: hosting2: hosting3|{totalMEM=8023} Domain-0=1065205;383 hosting1=19464540;768 hosting2=9875774;768 hosting3=11454148;512

which basically means: XEN $status - VM : $domain1: $domain2: $domainN|{$total_mem_xen=xxxx} $domain1:$cpu_usage_domain1;$mem_usage_domain1;etc ..

updated: posted a nice screenshot with info obtained from one xen-server

[fjrial]

Source Code

Source Code posted at CESGA's github:

https://github.com/CESGA/CESGA-XEN-ZABBIX

Cheers.
Also check our web mobile frontend for zabbix:

https://github.com/CESGA/CESGA-VNMS-ZABBIX

Description:

https://www.cesga.es/es/soporte_usuarios/usr-comunicaciones-IT/cesga_vnms

[Slash]

This looks very good indeed!

As soon as I got some time I will try to implement it.

Did you try it on different xen version? like xen 3 and xen 4?

[fjrial]

This looks very good indeed!

As soon as I got some time I will try to implement it.

Did you try it on different xen version? like xen 3 and xen 4?

Tomorrow I´ll try to post a new and better version, with some global vars in order to deploy it easier.. now, I´m trying to change mysql queries to api queries.

In a few days, I'll try on xen4, at this moment, we are migrating some xen 3 to new xen 4 version.

I'll post the results here, but results depends on how nagios check_xen plugins works with xen 3/4, my script only parses check_xen output..

[fjrial]

works with xen 3 and 4

tested and working with xen 3 and xen 4

[fjrial]

New files at github:
-improved code
-replace all mysql queries with api calls

still missing how to get the same output zabbix agentd as from check_nrpe plugin..

now, trying to use/build something compatible with libvirt-snmp (as somenone, mentioned before in this thread)

cheers.