HP servers monitoring via iLO

[Vengant]

Hi all,

I attached archive with scripts and template for auto-discovery and monitor hardware status parameters and more other information on HP Proliant servers via iLO IPMI interface.

Overview

Template allows to collect this types of data:
- Temperatures, FANs, power supplies for Proliants G5 and above,
- Disks and memory status for Proliants G7 and above,
- General inventory information such as serial and model numbers, firmware versions etc.

This solution requires FreeIPMI package to be installed.

Archive contains template and two perl scripts:

ilo_discovery.pl - querying iLO for items and generating standard JSON output for LLD,
ipmi_proliant.pl - querying all data for specified ipmi section (sensors, chassis, fru etc). Data will be cached in temporary file to prevent iLO to be overloaded by IPMI requests. Then specified value retrieving from cache.

Installation

1. Download archive and import template into Zabbix,

2. Place scripts to "externalscripts" directory path as configured in zabbix_server.conf, and make them executable,

3. Download and compile FreeIPMI (FAQ for installation can be found there):

# wget http://ftp.gnu.org/gnu/freeipmi/freeipmi-1.2.1.tar.gz 
# tar -xvzf freeipmi-1.2.1.tar.gz
# cd freeipmi-1.2.1
# ./configure --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man
# make install

For 64-bit systems configure string will be:

./configure --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --libdir=/usr/lib64

4. Create account with "user" rights for Zabbix in your iLO. This account not required administrative privilegies. Then replace values of $user and $pass variables in scripts to login and password of this account, that will be used to collect data from iLO.

5. Check FreeIPMI to success connect with iLO (replace IP, login and password to yours):

# /usr/sbin/ipmi-sensors -D LAN2_0 -h 192.168.0.1 -u monitor -p P@$$w0rd -l USER -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names

We should get something like this output:

0,System Chassis 1 UID Light,OEM Reserved,N/A,N/A,'OEM Event = 0000h'
1,System Chassis 2 Health LED,OEM Reserved,N/A,N/A,'OEM Event = 0000h'
2,Processor Module VRM 1,Power Unit,N/A,N/A,'Device Inserted/Device Present'
3,Power Supply Power Supply 1,Power Supply,N/A,N/A,'Presence detected'

6. Check discovery script to success parsing IPMI data (replace IP to yours):

# /usr/lib/zabbix/externalscripts/ilo_discovery.pl 192.168.0.1 sensor temp numeric

We should get something like this output:

{
        "data":[
                {
                        "{#CLASS}":"sensor",
                        "{#KEY}":"Air Inlet 01-Inlet Ambient",
                        "{#SECTION}":"Temperature",
                        "{#TYPE}":"numeric",
                        "{#MEASURE}":"C"},
                {
                        "{#CLASS}":"sensor",
                        "{#KEY}":"Processor 02-CPU",
                        "{#SECTION}":"Temperature",
                        "{#TYPE}":"numeric",
                        "{#MEASURE}":"C"},

7. In Zabbix web interface, go to our host that we want to monitor via iLO and create macros {$ILO} with address of our iLO. "IPMI interface" host field not used.

8. Attach template "Template IPMI Script iLO Discovery" to host,

9. Wait for discovery will be complete. At the end of it, we should get something like this in "lastest data":


And in "graphs":


That's all. This solution was tested with Zabbix 2.0 and 2.2 on CentOS 5 and CentOS6 (x86 and x64), for Proliants DL, ML and BL generation 5, 6, 7 and 8. For servers with lo100 it will also work, but the data will be less beacuse of lo100 restrictions.

General recomendaion: upgrade your iLO firmware to lastest version before using this.

[steveboyson]

Thanks very much, looks promising!
The preliminary tests ran fine.
Just waiting for the discovery to finish.

Would be nice to have "username" and "password" as macros defined in zabbix instead of hardcoded in the scripts, but that is just cosmetic.

Nota bene: does not work on iLO generation 1, but you have stated that already.

Anyway, good work and thank you!

[steveboyson]

Oha.
I really don't want to be offensive but a deeper look revealed that it is a bit messy. Every now and then I get

"item [XXXX-ilo:ipmi_proliant.pl['System Board 3 Fan 3',sensor,'{$ILO}',numeric]] became not supported: Received value [] is not suitable for value type [Numeric (float)]"

But in the next gathering intervall they become supported again.

The temporary files which are written have sometimes single quotes in the filename and sometimes not.

I think I will do a rewrite.

Again, thanks - you did a very good job - and left some place for improvements ;-)

[Vengant]

Hmm...
I don't see this bug in my environment - maybe it depends on perl version.

But i am not a perl programmer and used the code constructions that were available and understood to me. If you will find better and proper solution for this code, it will be fine.

[steveboyson]

Well, I rearranged the ipmi_proliant script and the params in the template. I do understand now why sometimes single quotes are used and sometimes not: otherwise there would be duplicate item keys.

I came along that issue by just appending individual dummy parameters like "__ChassisPowerState" or "__iLOPowerSuppliesLoad" to make the item keys unique and cleaning up the item prototypes and definitions.

What concerns me more is that every IPMI request gets logged in the iLO event log, resulting in filling up the eventlog and moving "relevant" records to the very end.
Thus, I decided not to use it. It is of course not your fault instead it lies in the IPMI implementation by HP.

[Vengant]

Well, I rearranged the ipmi_proliant script and the params in the template. I do understand now why sometimes single quotes are used and sometimes not: otherwise there would be duplicate item keys.

How item keys may affect temporary file name? In script used constructions like

$cache_file = '/var/tmp/ipmi_sensors_'.$server;

Only server ilo address, not item keys, used in file name generation.
What version of perl you use?

[steveboyson]

I use Perl 5.16.2 on Suse 12.3 (the appliance build).

For the files: you used sometimes '{$ILO}' while on other items you are using "{$ILO}" or even only {$ILO}. That's what causing the different filenames.
Have a look for yourself at /var/tmp/

EDIT: The predefined macro {$ILO} is part of your item keys.

[Vengant]

Ok, it's clear.

Different quotas in macros name used because of Zabbix restrictions for duplicate keys in item prototypes even if variables being used in key naming.

I use this scripts with perl 5.10 on CentOS and this bug not appears. I think it's really depend on perl version. I will edit template to exclude quotas usage for macros used in item prototypes.

[emvectra]

Error while starting Agent

Hi.
Thanks for the great script.

But if I start (or restart) the Zabbix-Agent, I get this Error:

zabbix_agentd [27964]: invalid entry [use strict;] (not following "parameter=value" notation) in config file [/etc/zabbix/zabbix_agentd.d//ilo_discovery.pl], line 8

If I move both Perl-Scripts to annother Directory, starts the Agent and then moves both Perl-Scripts back to "etc/zabbix/zabbix_agentd.d" it works.
But then my other Script ("userparameter_mysql.conf" for MySQL) doesn't work.

If I use
zabbix_agentd -t "mysql.version"
on the Command Line, I get the same Error:

zabbix_agentd [27964]: invalid entry [use strict;] (not following "parameter=value" notation) in config file [/etc/zabbix/zabbix_agentd.d//ilo_discovery.pl], line 8


Any ideas? Or is their a new Version of the script?
-----------

Zabbix 2.2.4
CentOS 6.5 x86_64
Perl 5.10.1 x86_64
MySQL 5.1.73 x86_64

----------

Thanks,
emvectra

[Vengant]

Scripts does not interact with Zabbix Agent anyway.

Check your agent configs.

[emvectra]

Thanks, I´ve solved it:

1) New Directory /etc/zabbix/zabbix_serverd.d
2) Move both Perl-Scripts to the new Directory
3) Change /etc/zabbix/zabbix_server.conf:
ExternalScripts=/etc/zabbix/zabbix_serverd.d

4) Restart Zabbix-Server: OK
5) Restart Zabbix-Agent: OK

emvectra

[m.krug]

ilo_discovery.pl

Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
Do you have any idea ?

[killout]

Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
Do you have any idea ?

I found that this error appear if ipmi offline on the host, or wrong ip specified.

Try to check if you can connect with ipmi-sensors command.

/usr/sbin/ipmi-sensors -D LAN2_0 -h _IP_ -u _USER_ -p _PASSWD_ -l _PRIV.LEVEL_ -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names

Change
_USER_
_PASSWD_
_PRIV.LEVEL_
to your values.

[copperek]

I get the same error message as m.krug

Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
Do you have any idea ?

I found that this error appear if ipmi offline on the host, or wrong ip specified.

Try to check if you can connect with ipmi-sensors command.

Change
_USER_
_PASSWD_
_PRIV.LEVEL_
to your values.

I have checked my ILO and it is online because if I run

/usr/sbin/ipmi-sensors -D LAN2_0 -h 192.168.xxx.xxx -u [USER] -p [PASSWORD] -l USER -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names

I get a reply from the ILO. I'm trying to monitor a DL380pG8 from zabbix 2.2.6.

Nevermind, found the problem, I was 100% sure I had changed the username and password in the Ilo_discovery.pl, but it seems that this change wasn't stored. When I corrected the username and password everything seems to be oke now.

Regards,

[Smersh]

In zabbix log periodically appears the following strings:

flock() on closed filehandle CACHE at /usr/local/etc/zabbix22/zabbix/externalscripts/ipmi_proliant.pl line 75.
readline() on closed filehandle CACHE at /usr/local/etc/zabbix22/zabbix/externalscripts/ipmi_proliant.pl line 76.

I have a suspicion that because of this zabbix at times receives no data...