Problem monitoring ZFS pool status with Ubuntu 22.04 and Zabbix 6.0

[danboid]

I used to use this template to monitor ZFS using Zabbix:

https://github.com/Cosium/zabbix_zfs-on-linux

Under Zabbix 4.0 and that worked fine but that template doesn't work for me with Zabbix 6 for monitoring ZFS pools


I've tried the Zabbix 6.0 ZFS templates that are in the zabbix community github repo but I've not been able to get either template_zfs_on_linux.yaml nor template_zfs_on_linux_active.yaml to work properly with Zabbix 6.0 and Ubuntu 22.04. Both my Zabbix server and the host with the ZFS disk are running Ubuntu 22.04. The main thing I need to work with these ZFS templates is the monitoring of the ZFS pool's health but Zabbix isn't displaying the DEGRADED status for my zabbix client2 running ZFS host when I (intentionally) put the pool into a degraded state.

The first thing I noticed that stopped these templates from working at all for me with Ubuntu is that the paths for both the zfs and zpool commands are wrong for Ubuntu users. To get these templates to work at all, Ubuntu users have to replace every instance of /sbin/zfs with /usr/sbin/zfs and /sbin/zpool with /usr/sbin/zpool. Doing that fixes most of the ZFS monitoring but it doesn't fix the monitoring of the pools health.

userparams_zol_without_sudo.conf has these lines for monitoring pool health:

# pool health
UserParameter=zfs.zpool.health[*],/sbin/zpool list -H -o health $1

I read here:

https://www.zabbix.com/documentation/4.0/en/manual/installation/upgrade_notes_400#deprecated-macros-in-item-names

That positional macros aren't used any more in Zabbix 6.0 so maybe this is part of my problem?

I tried a few things such as:

# pool health
UserParameter=zfs.zpool.health[*],/sbin/zpool list -H -o health "{#POOLNAME}"

With and without the quotes or not including a (positional) macro at the end at all but nothing has fixed the ZFS health monitoring for me yet. I have intentionally degraded the pool but Zabbix isn't alerting me of that. Zabbix works fine otherwise.

I've also tried using template_zfs_on_linux_active.yaml but I had to modify that slightly to get it to import properly. I had to change the version string from "6.2" to "6.0" (its in a dir called 6.0 so surely I should really be 6.0 or in a 6.2 dir?) and I had to change L4 from template_groups: to just groups: before Zabbix would import it.

Any tips on debugging this are appreciated, thanks!

https://github.com/zabbix/community-templates/issues/334

[cyber]

Usage of "$1" etc in user parameters is still OK. It is reference to the respective parameter in item key. That {#POOLNAME} will not work as agent does not know a thing about LLD macros..

https://www.zabbix.com/documentation/current/en/manual/config/items/userparameters

Command
Command to be executed to evaluate value of the key.
For flexible user parameters only:
You may use positional references $1…$9 in the command to refer to the respective parameter in the item key.
Zabbix parses the parameters enclosed in [ ] of the item key and substitutes $1,...,$9 in the command accordingly.
$0 will be substituted by the original command (prior to expansion of $0,...,$9) to be run.
Positional references are interpreted regardless of whether they are enclosed between double (") or single (') quotes.
To use positional references unaltered, specify a double dollar sign - for example, awk '{print $$2}'. In this case $$2 will actually turn into $2 when executing the command.

[danboid]

I have got the community ZFS for Linux template for Zabbix 6.0 template working now, including the all important pool health status alerts.

I have opened a PR to add some installation instructions to the templates README:

https://github.com/zabbix/community-templates/pull/337

[danboid]

Here's the bits I've added to the templates README:

# ZFS on Linux

## Installation

To use the Zabbix ZFS on Linux template, you must first install a correctly configured userparams file on any machines running the zabbix agent and using ZFS.

### Choosing the correct userparams file

There should be two different user parameters files in the same directory as this README, `userparams_zol_without_sudo.conf` and `userparams_zol_with_sudo.conf`. One uses `sudo` to run and thus you must give Zabbix the correct rights and the other doesn't use `sudo`.

On recent ZFS on Linux versions (version 0.7.0+), you don't need sudo to run `zpool list` or `zfs list` so you need to copy `userparams_zol_without_sudo.conf` into `/etc/zabbix/zabbix_agentd.d` or `/etc/zabbix/zabbix_agent2.d` if using zabbix-agent2.

For older ZFS on Linux versions (eg version 0.6.x), you will need to add some sudo rights with the file `userparams_zol_with_sudo.conf`. On some distributions, ZoL already includes a suitable file with all the necessary rights at `/etc/sudoers.d/zfs` but its contents are commented out, you just need to remove the comments and any user will be able to list zfs datasets and pools. An example config might look like so:

```
## Allow read-only ZoL commands to be called through sudo
## without a password. Remove the first '#' column to enable.
##
## CAUTION: Any syntax error introduced here will break sudo.
##
## Cmnd alias specification
Cmnd_Alias C_ZFS = \
/sbin/zfs "", /sbin/zfs help *, \
/sbin/zfs get, /sbin/zfs get *, \
/sbin/zfs list, /sbin/zfs list *, \
/sbin/zpool "", /sbin/zpool help *, \
/sbin/zpool iostat, /sbin/zpool iostat *, \
/sbin/zpool list, /sbin/zpool list *, \
/sbin/zpool status, /sbin/zpool status *, \
/sbin/zpool upgrade, /sbin/zpool upgrade -v

## allow any user to use basic read-only ZFS commands
ALL ALL = (root) NOPASSWD: C_ZFS
```

If you don't know where your userparameters directory is, this is usually `/etc/zabbix/zabbix_agentd.d`. If in doubt, just look at your `zabbix_agentd.conf` file for the line begining by `Include=`.

### Editing the userparams file

The path used for the `zfs` and `zpool` binaries is different under almost every Linux distro, the paths aren't even the same for Debian and Ubuntu, as one notable example. For this reason you should run `which zfs` and `which zpool` on each ZFS machine you want to monitor with Zabbix to find out the correct paths to use to run these two binaries, then you need to edit the userparams file so that every instance of `zfs` and `zpool` is called using the correct path, if your distro's path doesn't match the path used in example userparams files.