Ad Widget

Collapse

HP servers monitoring via iLO

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    HP servers monitoring via iLO

    Hi all,

    I attached archive with scripts and template for auto-discovery and monitor hardware status parameters and more other information on HP Proliant servers via iLO IPMI interface.

    Overview

    Template allows to collect this types of data:
    - Temperatures, FANs, power supplies for Proliants G5 and above,
    - Disks and memory status for Proliants G7 and above,
    - General inventory information such as serial and model numbers, firmware versions etc.

    This solution requires FreeIPMI package to be installed.

    Archive contains template and two perl scripts:

    ilo_discovery.pl - querying iLO for items and generating standard JSON output for LLD,
    ipmi_proliant.pl - querying all data for specified ipmi section (sensors, chassis, fru etc). Data will be cached in temporary file to prevent iLO to be overloaded by IPMI requests. Then specified value retrieving from cache.

    Installation

    1. Download archive and import template into Zabbix,

    2. Place scripts to "externalscripts" directory path as configured in zabbix_server.conf, and make them executable,

    3. Download and compile FreeIPMI (FAQ for installation can be found there):
    Code:
    # wget http://ftp.gnu.org/gnu/freeipmi/freeipmi-1.2.1.tar.gz 
    # tar -xvzf freeipmi-1.2.1.tar.gz
    # cd freeipmi-1.2.1
    # ./configure --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man
    # make install
    For 64-bit systems configure string will be:
    Code:
    ./configure --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --libdir=/usr/lib64
    4. Create account with "user" rights for Zabbix in your iLO. This account not required administrative privilegies. Then replace values of $user and $pass variables in scripts to login and password of this account, that will be used to collect data from iLO.

    5. Check FreeIPMI to success connect with iLO (replace IP, login and password to yours):
    Code:
    # /usr/sbin/ipmi-sensors -D LAN2_0 -h 192.168.0.1 -u monitor -p [email protected]$$w0rd -l USER -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names
    We should get something like this output:
    Code:
    0,System Chassis 1 UID Light,OEM Reserved,N/A,N/A,'OEM Event = 0000h'
    1,System Chassis 2 Health LED,OEM Reserved,N/A,N/A,'OEM Event = 0000h'
    2,Processor Module VRM 1,Power Unit,N/A,N/A,'Device Inserted/Device Present'
    3,Power Supply Power Supply 1,Power Supply,N/A,N/A,'Presence detected'
    6. Check discovery script to success parsing IPMI data (replace IP to yours):
    Code:
    # /usr/lib/zabbix/externalscripts/ilo_discovery.pl 192.168.0.1 sensor temp numeric
    We should get something like this output:
    Code:
    {
            "data":[
                    {
                            "{#CLASS}":"sensor",
                            "{#KEY}":"Air Inlet 01-Inlet Ambient",
                            "{#SECTION}":"Temperature",
                            "{#TYPE}":"numeric",
                            "{#MEASURE}":"C"},
                    {
                            "{#CLASS}":"sensor",
                            "{#KEY}":"Processor 02-CPU",
                            "{#SECTION}":"Temperature",
                            "{#TYPE}":"numeric",
                            "{#MEASURE}":"C"},
    7. In Zabbix web interface, go to our host that we want to monitor via iLO and create macros {$ILO} with address of our iLO. "IPMI interface" host field not used.

    8. Attach template "Template IPMI Script iLO Discovery" to host,

    9. Wait for discovery will be complete. At the end of it, we should get something like this in "lastest data":


    And in "graphs":


    That's all. This solution was tested with Zabbix 2.0 and 2.2 on CentOS 5 and CentOS6 (x86 and x64), for Proliants DL, ML and BL generation 5, 6, 7 and 8. For servers with lo100 it will also work, but the data will be less beacuse of lo100 restrictions.

    General recomendaion: upgrade your iLO firmware to lastest version before using this.
    Attached Files
    Last edited by Vengant; 10-04-2014, 17:57.

    #2
    Thanks very much, looks promising!
    The preliminary tests ran fine.
    Just waiting for the discovery to finish.

    Would be nice to have "username" and "password" as macros defined in zabbix instead of hardcoded in the scripts, but that is just cosmetic.

    Nota bene: does not work on iLO generation 1, but you have stated that already.

    Anyway, good work and thank you!

    Comment


      #3
      Oha.
      I really don't want to be offensive but a deeper look revealed that it is a bit messy. Every now and then I get
      Code:
      "item [XXXX-ilo:ipmi_proliant.pl['System Board 3 Fan 3',sensor,'{$ILO}',numeric]] became not supported: Received value [] is not suitable for value type [Numeric (float)]"
      But in the next gathering intervall they become supported again.

      The temporary files which are written have sometimes single quotes in the filename and sometimes not.

      I think I will do a rewrite.

      Again, thanks - you did a very good job - and left some place for improvements ;-)

      Comment


        #4
        Hmm...
        I don't see this bug in my environment - maybe it depends on perl version.

        But i am not a perl programmer and used the code constructions that were available and understood to me. If you will find better and proper solution for this code, it will be fine.

        Comment


          #5
          Well, I rearranged the ipmi_proliant script and the params in the template. I do understand now why sometimes single quotes are used and sometimes not: otherwise there would be duplicate item keys.

          I came along that issue by just appending individual dummy parameters like "__ChassisPowerState" or "__iLOPowerSuppliesLoad" to make the item keys unique and cleaning up the item prototypes and definitions.

          What concerns me more is that every IPMI request gets logged in the iLO event log, resulting in filling up the eventlog and moving "relevant" records to the very end.
          Thus, I decided not to use it. It is of course not your fault instead it lies in the IPMI implementation by HP.

          Comment


            #6
            Originally posted by steveboyson View Post
            Well, I rearranged the ipmi_proliant script and the params in the template. I do understand now why sometimes single quotes are used and sometimes not: otherwise there would be duplicate item keys.
            How item keys may affect temporary file name? In script used constructions like
            Code:
            $cache_file = '/var/tmp/ipmi_sensors_'.$server;
            Only server ilo address, not item keys, used in file name generation.
            What version of perl you use?
            Last edited by Vengant; 11-04-2014, 16:31.

            Comment


              #7
              I use Perl 5.16.2 on Suse 12.3 (the appliance build).

              For the files: you used sometimes '{$ILO}' while on other items you are using "{$ILO}" or even only {$ILO}. That's what causing the different filenames.
              Have a look for yourself at /var/tmp/

              EDIT: The predefined macro {$ILO} is part of your item keys.

              Comment


                #8
                Ok, it's clear.

                Different quotas in macros name used because of Zabbix restrictions for duplicate keys in item prototypes even if variables being used in key naming.

                I use this scripts with perl 5.10 on CentOS and this bug not appears. I think it's really depend on perl version. I will edit template to exclude quotas usage for macros used in item prototypes.
                Last edited by Vengant; 11-04-2014, 17:19.

                Comment


                  #9
                  Error while starting Agent

                  Hi.
                  Thanks for the great script.

                  But if I start (or restart) the Zabbix-Agent, I get this Error:

                  zabbix_agentd [27964]: invalid entry [use strict;] (not following "parameter=value" notation) in config file [/etc/zabbix/zabbix_agentd.d//ilo_discovery.pl], line 8

                  If I move both Perl-Scripts to annother Directory, starts the Agent and then moves both Perl-Scripts back to "etc/zabbix/zabbix_agentd.d" it works.
                  But then my other Script ("userparameter_mysql.conf" for MySQL) doesn't work.

                  If I use
                  zabbix_agentd -t "mysql.version"
                  on the Command Line, I get the same Error:

                  zabbix_agentd [27964]: invalid entry [use strict;] (not following "parameter=value" notation) in config file [/etc/zabbix/zabbix_agentd.d//ilo_discovery.pl], line 8


                  Any ideas? Or is their a new Version of the script?
                  -----------

                  Zabbix 2.2.4
                  CentOS 6.5 x86_64
                  Perl 5.10.1 x86_64
                  MySQL 5.1.73 x86_64

                  ----------

                  Thanks,
                  emvectra

                  Comment


                    #10
                    Scripts does not interact with Zabbix Agent anyway.

                    Check your agent configs.

                    Comment


                      #11
                      Thanks, I´ve solved it:

                      1) New Directory /etc/zabbix/zabbix_serverd.d
                      2) Move both Perl-Scripts to the new Directory
                      3) Change /etc/zabbix/zabbix_server.conf:
                      ExternalScripts=/etc/zabbix/zabbix_serverd.d

                      4) Restart Zabbix-Server: OK
                      5) Restart Zabbix-Agent: OK

                      emvectra

                      Comment


                        #12
                        ilo_discovery.pl

                        Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
                        Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
                        Do you have any idea ?

                        Comment


                          #13
                          Originally posted by m.krug View Post
                          Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
                          Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
                          Do you have any idea ?
                          I found that this error appear if ipmi offline on the host, or wrong ip specified.

                          Try to check if you can connect with ipmi-sensors command.
                          /usr/sbin/ipmi-sensors -D LAN2_0 -h _IP_ -u _USER_ -p _PASSWD_ -l _PRIV.LEVEL_ -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names
                          Change
                          _USER_
                          _PASSWD_
                          _PRIV.LEVEL_
                          to your values.

                          Comment


                            #14
                            I get the same error message as m.krug
                            Hello, it is a great solution. But in my installation ( CentOS 6.5 with Zabbix 2.2.4) it works only partially. In my zabbix_server.log i got this message
                            Use of uninitialized value $results in print at /opt/zabbix/share/zabbix/externalscripts/ilo_discovery.pl line 56.
                            Do you have any idea ?
                            Originally posted by killout View Post
                            I found that this error appear if ipmi offline on the host, or wrong ip specified.

                            Try to check if you can connect with ipmi-sensors command.

                            Change
                            _USER_
                            _PASSWD_
                            _PRIV.LEVEL_
                            to your values.
                            I have checked my ILO and it is online because if I run

                            Code:
                            /usr/sbin/ipmi-sensors -D LAN2_0 -h 192.168.xxx.xxx -u [USER] -p [PASSWORD] -l USER -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names
                            I get a reply from the ILO. I'm trying to monitor a DL380pG8 from zabbix 2.2.6.

                            Nevermind, found the problem, I was 100% sure I had changed the username and password in the Ilo_discovery.pl, but it seems that this change wasn't stored. When I corrected the username and password everything seems to be oke now.

                            Regards,
                            Last edited by copperek; 23-09-2014, 11:37. Reason: problem fixed

                            Comment


                              #15
                              In zabbix log periodically appears the following strings:
                              Code:
                              flock() on closed filehandle CACHE at /usr/local/etc/zabbix22/zabbix/externalscripts/ipmi_proliant.pl line 75.
                              readline() on closed filehandle CACHE at /usr/local/etc/zabbix22/zabbix/externalscripts/ipmi_proliant.pl line 76.
                              I have a suspicion that because of this zabbix at times receives no data...

                              Comment

                              Announcement

                              Collapse
                              No announcement yet.
                              Working...
                              X