linux software RAID monitoring

[fratotec]

MDADM Perl Script

Hi, this script give false positives on a busy RAID1.
I realized that the "State" of my RAID1 array changes from "clean" to "active".. when heavy writes occures.

so I changed the the following line to
my $return = `/usr/bin/sudo /sbin/mdadm --detail /dev/$device |grep \"Active\"|cut -d \":\" -f 2`;

this hopefully alerts me if the number of Active devices not equal 2.

Here is what I use in zabbix_agentd.conf:

UserParameter=custom.raidstate.md0,/etc/zabbix/bin/custom.raidstate md0
UserParameter=custom.raidstate.md1,/etc/zabbix/bin/custom.raidstate md1

And here is the code for /etc/zabbix/bin/custom.raidstate :

#!/usr/bin/perl
#
#
#sudo /sbin/mdadm --detail /dev/md0|grep -i "State :"|cut -d ":" -f 2
#

use strict;
use warnings;

my $device = $ARGV[0];

my $return = `/usr/bin/sudo /sbin/mdadm --detail /dev/$device  |grep -i \"State :\"|cut -d \":\" -f 2`;

chomp ($return);
$return =~ s/\ //g;

if ( $return eq 'clean' ) {
  print "0";
} else {
  print "1";
}

# - The End

I use Numeric (float) to store this kind of value with no custom multiplier. For triggering, I use something like:

{MyHost:custom.raidstate.md0.last(0)}>0

To be able to use mdadm --detail as zabbix user, I use sudo with the following statements in sudoers file:

# Cmnd alias specification
Cmnd_Alias ZABBIXCMD = /sbin/mdadm --detail *
# ZABBIX special privileges
zabbix  ALL=NOPASSWD:   ZABBIXCMD

Hope this'll help you.

Cheers,

Cheers

Franz

[casshan]

How about?

sudo mdadm -D /dev/md3| grep 'Failed Devices' | cut -d ':' -f 2 | tr -d '

Just have a trigger if the value is > 0. No client side scrips needed, just remote commands enabled.

[nack]

Monitoring software raid:

Item1
To check if software raid exist

Name: Software Raid exist
Key: vfs.file.regmatch[/proc/mdstat,raid]

Item2
To check if raid is broken

Name: Software Raid broken
Key: vfs.file.regmatch[/proc/mdstat,_]

Trigger

If raid exist and the raid is broken, then send alert.

Name: Software raid broken {HOSTNAME}
Expression: {Template:vfs.file.regmatch[/proc/mdstat,raid].last(0)}>0&{Template:vfs.file.regmatch[/proc/mdstat,_].last(0)}>0

[Gav]

My script

#!/bin/sh
[ -b "/dev/$1" ] || { echo -1; exit 1; }

/sbin/mdadm -D /dev/$1 | /bin/grep '^[\t ]*State' | /bin/sed 's/^[\t ]*State :[\t ]*//g' | /usr/bin/awk 'BEGIN{a=0};/clean/{a+=1};/degraded/{a+=2};/resyncing/{a+=4};/recovering/{a+=8};/Not Started/{a+=16};END{if (NR==1) print a; else print -1 }'

usage myscript.sh mdX

return following values (coresponds to value map on Zabix)

0 ⇒ OK
1 ⇒ OK
2 ⇒ Degraded
3 ⇒ Degraded
4 ⇒ Resyncing
5 ⇒ Resyncing
6 ⇒ Degraded
7 ⇒ Degraded
8 ⇒ Recovering
9 ⇒ Recovering
10 ⇒ Degraded
11 ⇒ Degraded
12 ⇒ N/A
13 ⇒ N/A
14 ⇒ N/A
15 ⇒ N/A
16 ⇒ Not Started
99999 ⇒ Not Found

This script uses mdstat which can be run only by root so you need to edit sudoers to match with zabbix users and allow him to execute this script

zabbix ALL=(root) NOPASSWD: /etc/scripts/mdstat.sh

[Colttt]

or take a look at this site(german): http://lab4.org/wiki/Zabbix_linux_so...id_ueberwachen

[Jason]

I've a script somewhere for monitoring linux servers with MegaRaid that I found and hacked a bit which works with LSI/Dell Perc cards and can also be installed on openfiler boxes although it is quite basic and in need of updating. Will try and post it later on this week.

[frater]

Reading all the different posts I think this is a clean/efficient interpretation....

UserParameter=vfs.softraid.faulty, [ -e /proc/mdstat ] && grep blocks /proc/mdstat | egrep -vc '\[U+\]'

[jstefano]

Monitoring software raid - how to I install this?

Monitoring software raid:

Item1
To check if software raid exist

Name: Software Raid exist
Key: vfs.file.regmatch[/proc/mdstat,raid]

Item2
To check if raid is broken

Name: Software Raid broken
Key: vfs.file.regmatch[/proc/mdstat,_]

Trigger

If raid exist and the raid is broken, then send alert.

Name: Software raid broken {HOSTNAME}
Expression: {Template:vfs.file.regmatch[/proc/mdstat,raid].last(0)}>0&{Template:vfs.file.regmatch[/proc/mdstat,_].last(0)}>0

Hi guys

how to I implement this please? Goes to Zabbix server as a template or is this a script on the client? Can I get more details about this please?

Thanks for your time!!!

Jan

[casshan]

Forgot about this old thread of mine, here is my current template I use for this. You need to make sure the zabbix user has sudo access to the mdadm command. You could probably convert the system.run commands to UserParameter but I prefer to keep the zabbix_agentd.conf as basic as possible.

Items:
Name: Linux Software RAID: Any Array Degraded
Key: system.run["sudo /etc/zabbix/scripts/mdadm -D /dev/md* | grep degraded | wc -l"]

Name: Linux Software RAID: Any Array Recovering
Key: system.run["sudo /etc/zabbix/scripts/mdadm -D /dev/md* | grep recovering | wc -l"]

Name: Linux Software RAID: Info
Key: system.run[sudo /etc/zabbix/scripts/mdadm -D /dev/md* | egrep -e 'State :' -e 'Raid Level : ' -e '^/dev' -e 'Active Devices :' -e 'Working Devices :' -e 'Failed Devices :' -e 'Spare Devices :' -e 'Array Size :' -e 'Version :']

Triggers:
Severity: High
Name: Linux Software RAID: Degraded
Expression: {Template_Software RAID:system.run["sudo /etc/zabbix/scripts/mdadm -D /dev/md* | grep degraded | wc -l"].last(#1)}>0

Severity: Warning
Name: Linux Software RAID: Recovering
Expression: {Template_Software RAID:system.run["sudo /etc/zabbix/scripts/mdadm -D /dev/md* | grep recovering | wc -l"].last(#1)}>0

[jstefano]

Thank you thank you thank you!!!
Any chance to export your template so I'll be able to import it?
I am quite new to Zabbix to be honest and not familiar (yet) with creating templates...
Have a great Friday and thanks again!!!

Jan

[casshan]

Here you go

[jstefano]

Here you go

Thaaaaaaaaaaaaaaaaaaaaaaanx!!!

Going to try and keep you posted later!
You rock!!!

Jan

[jstefano]

Here you go

OPS... got this error:

##Unknown import version. [conf.import.php:190 → CFrontendApiWrapper->import() → CApiWrapper->__call() → CFrontendApiWrapper->callMethod() → CApiWrapper->callMethod() → CFrontendApiWrapper->callClientMethod() → CLocalApiClient->callMethod() → call_user_func_array() → CConfiguration->import() → CConfigurationImport->import() → CConfigurationImport->getFormatter() in /usr/share/zabbix/include/classes/import/CConfigurationImport.php:157]

any hint please?

BTW, I am on version 2.4.7 running on CentOS 6.7
The line 157 of /usr/share/zabbix/include/classes/import/CConfigurationImport.phpsays:
$this->formatter = $this->getFormatter($version);

and the line 190 of conf.import.php says:
return $importWidget;

Thank you very much for your help, really appreciate it!

Jan