[Linux] Monitoring for a read-only filesystem

[coreychristian]

The underlying cause was pretty much a mystery. Our VMware environment has both Linux and Windows on the same hosts, this problem didn't seem to effect Windows hosts.

And, as we have change VMware infrastructure, it seems to have gone away. Our storage was iscsi based, what is yours?

I am not sure what type of storage we use, the end cause though was disk latency, we saw the same thing though Linux would go read only, windows would not.

Linux I guess it a bit more sensitive to that, here is a link from VMWare for a similar issue though I think our's was just latency.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306

[Bock]

It's just like OSs are working with FS. Unfortunately, you can take from windows disk and turn it back later and it will be work, but for linux it doesn't help.
Issue for linux is appear, as I see, for xfs, because for ext4 you can put to remount disk in readonly in case of errors (errors=remount-ro), but for XFS you haven't this option.
The only option, as I see, it's only use Configuration - Actions - Event source - Internal and receive a email for unsupported items (if you set fs.readonly item with unique application).

[coreychristian]

The only option, as I see, it's only use Configuration - Actions - Event source - Internal and receive a email for unsupported items (if you set fs.readonly item with unique application).

With the script above I use the following trigger. The nice thing about running it from cron using the zabbix sender is it will still function as long as cron is running. It will also alert on no data for 30 minutes.

Trigger 1 - Read Only FS
Name: Filesystem Checker failed to write to {ITEM.VALUE} filesystems on {HOST.NAME}
Expression: {Template Baseline Linux:filesystem.ro.check.last()}>0

Trigger 2 - No data
Name: zabbix file system checker script is not sending data into zabbix for {HOST.NAME} in the past 30 minutes
Expression: {Template Baseline Linux:filesystem.ro.check.nodata(30m)}=1

[Andrey123q]

coreychristian, thanks, for your script/idea.
I've just slightly modified it (see below) and created different item prototypes (filesystem.ro.check[#FSNAME]) for each filesystem and 2 triggers for nodata and last check >0 in mounted filesystems discovery rule.

#!/bin/bash
# Variables                                                                                                                                                            
log="/var/log/fsrocheck.log"                                                                                                                                           
cur_date=$(date +%Y%m%d\-%H%M%S)                                                                                                                                       
zabbix_sender_cmd=$(which zabbix_sender)                                                                                                                               
                                                                                                                                                                       
# File List array                                                                                                                                                      
fs_list=( $(cat /etc/fstab| egrep "ext" | grep -v "^#"| awk '{ print  $2 }') )                                                                                         
                                                                                                                                                                       
# Loop foreach filesystem                                                                                                                                              
for fs in ${fs_list[*]}; do                                                                                                                                            
        if touch $fs/zabbix_test.txt 2> /dev/null; then                                                                                                                
                rm $fs/zabbix_test.txt 2> /dev/null                                                                                                                    
                fs_status=0                                                                                                                                            
                fs_status_log="$fs is writeable"                                                                                                                       
        else                                                                                                                                                           
                fs_status=1                                                                                                                                            
                fs_status_log="$fs is read-only"                                                                                                                       
        fi                                                                                                                                                             
        # Save report to $log                                                                                                                                          
        echo $cur_date $fs_status_log >> $log                                                                                                                          
        #Send the fs check to zabbix                                                                                                                                   
        $zabbix_sender_cmd -c /etc/zabbix/zabbix_agentd.conf -k filesystem.ro.check[$fs] -o $fs_status > /dev/null                                                      
done

[tyrell_c]

I'm confused as to what the item key is supposed to be when you configure the item. The way I have it in the zabbix config file is :

UserParameter=checkro[*],/etc/zabbix/scripts/checkro.sh $1

and I understand that in item key it should at least look like

custom.checkro

but not sure what the mount point would be. Any way I've tried it I've got Status unsupported.

[LenR]

I think, for example to monitor /nfs/data/, the key would be checkro[/nfs/data].

There would have to be a different key (because of the value in []) for each monitored mount.

[panchook]

Yup that is why I wrote that script, our issue was the same with VMWare, all of our disks would go read only, so the files never got updated saying the file systems were read only.

We also had to put it into cron because the zabbix agent would sometimes die when it can't write to the log file.

While the script I wrote could likely be cleaned up a bit it does work fairly well.

I can export the items/triggers I have that go along with that script if you guys want.

Long shot at this point but any chance you still have the items/triggers for this? I'm struggling to get it working. Zabbix sender is sending the item but its not processed.