Set a disk usage threshold for a specific fileystem on a linux host. Zabbix 5.0

[jenlain]

Hello community and happy new year 2023,

I'm running Zabbix 5.0 to monitor some Linux (CentOS7, Almalinux8) hosts using the default template (Template OS Linux by Zabbix agent).

It's monitoring disk usage and the "mounted filesystem discovery rule" is creating items per filesystem and triggers like :

Mounted filesystem discovery: /tmp: Disk space is critically low (used > {$VFS.FS.PUSED.MAX.CRIT:"/tmp"}%)

where $VFS.FS.PUSED.MAX.CRIT is a macro set by default to 90.

So, I'm receiving a critical alert when one of my filesytems is reaching 90% of usage.

Now, I have on some hosts a /data filesystem that is 1TB.

On this specific filesystem, I'd like to adjust the trigger to receive an alert only if the available disk space is less than 20 Gb.

How can I do to easily set this alert to 98% of disk usage without changing all the thresholds of the other filesystems ?

Thanks for your help !

[tim.mooney]

Your question is shockingly similar to another recent one:

https://www.zabbix.com/forum/zabbix-help/456689-how-to-create-an-alert-for-a-specific-filesystem

[jenlain]

I checked on the forum before Xmas and did not find any answer, it looks like I have a doppelganger with the same issue somewhere else ! I think my problem is solved now.

Thanks a lot !

[yurtesen]

jenlain Except that solution won't work an you will never receive warnings. Because this trigger expression depends on wrong calculations. (and nobody at Zabbix team seams to care)

If you check the Zabbix 5 template -> https://git.zabbix.com/projects/ZBX/...Frelease%2F5.0
Description of trigger is very clear:

Two conditions should match: First, space utilization should be above {$VFS.FS.PUSED.MAX.CRIT:"{#FSNAME}"}.

Second condition should be one of the following:

- The disk free space is less than {$VFS.FS.FREE.MIN.CRIT:"{#FSNAME}"}.

- The disk will be full in less than 24 hours.

And the trigger expression:

{TEMPLATE_NAME:vfs.fs.size[{#FSNAME},pused].last()}>{$VFS.FS.PUSED.MAX.CRIT:"{#FSNAME}"} and (({TEMPLATE_NAME:vfs.fs.size[{#FSNAME},total].last()}-{TEMPLATE_NAME:vfs.fs.size[{#FSNAME},used].last()})<{$VFS.FS.FREE.MIN.CRIT:"{#FSNAME}"} or {TEMPLATE_NAME:vfs.fs.size[{#FSNAME},pused].timeleft(1h,,100)}<1d)

The problem is that the trigger uses TOTAL - USED to find out the FREE space. But this is absurd. Because in Linux there is reserved space(5%) which is NOT included in "USED". For 1000GB drive, 5% is 50GB. So when the disk is "full". TOTAL-USED will be 50GB, which is larger than {$VFS.FS.FREE.MIN.CRIT} and {$VFS.FS.FREE.MIN.WARN}. Therefore trigger will never work as 2nd part of the expression will never be "true".

Here the problem is described in detail. As you can see, nobody seems to care about it...

https://support.zabbix.com/browse/ZBX-19502

[cyber]

Is it something different how you set up your disks or ... I don't see any difference in numbers in Zabbix or looking directly in host (df) ... Total size minus used size equals, what is available and that is all I should care about... I would say even that df lies, as it rounds up, but zabbix shows 2 places after comma...
example of 2.5T disk...

1K-blocks       Used  Available Use% Mounted on
2683838468 1221631452 1462207016  46% /the_disk

zabbix shows 45.54% or something...
I do not use those out of the box templates, but this should not matter, as initial data is still the same..

[yurtesen]

cyber No, it is not a custom setup. This is how ext4 works. https://wiki.archlinux.org/title/ext4#Reserved_blocks

By default, 5% of the filesystem blocks will be reserved for the super-user

zabbix shows 45.54% or something...

I did not say the percent was incorrect. Actually, the percent seems to be correct in Zabbix and with df (internally df seems utilize to used and available space).

However with official Zabbix Linux template, a person who uses ext4 with a large partition, won't get warning even if it is 100% full. Because trigger does not only use the percent. Read the trigger expression from my post above (or from official Zabbix Linux template) It also calculates free space by assuming "Free/Usable space" is "Total space" minus "Used space", which is incorrect.

About your output, you seem to have modified your filesystem or you are not using ext4 perhaps?. In a filesystem which is setup using Linux defaults/ext4, the `df` output the total of "Used" + "Available" will NOT equal to "1K-blocks".

I do not use those out of the box templates, but this should not matter, as initial data is still the same..

Well if you use a different trigger expression, the results will be different. So I am not sure how you can say "this should not matter". Some data is calculated from other values in out of the box templates and it does matter.

[cyber]

"it does not matter" in a sense, that I could get the same needed numbers, If I want...Total, Used, Available and do the math... and get correct results.. The case, that I am only using percentage, is irrelevant..

But one thing you mentioned .. which I did not check and which really is not my concern as a user... Yes. that is not ext4, its xfs... I don't know what distros you are using, but in RH xfs seems to be default nowadays... So.. its not wrong everywhere..

[yurtesen]

"it does not matter" in a sense, that I could get the same needed numbers, If I want...Total, Used, Available and do the math... and get correct results.. The case, that I am only using percentage, is irrelevant..

Yes, I can also do the math but what I am saying is that the official Zabbix Linux template sometimes does the math incorrectly. Because Total = Used + Available is not always true. There is a better way which works for all cases.

But one thing you mentioned .. which I did not check and which really is not my concern as a user... Yes. that is not ext4, its xfs... I don't know what distros you are using, but in RH xfs seems to be default nowadays... So.. its not wrong everywhere..

XFS might be the default for RH, but people can change default during installation. Maybe Zabbix team should change the name of the template from "Linux by Zabbix agent" to "Linux(xfs) by Zabbix agent" ))

https://en.wikipedia.org/wiki/List_o...t_file_systems
Some of the Linux distributions which use EXT4 are Ubuntu, Debian, Slackware, openSUSE, Arch, Gentoo... so I would think that it would greatly benefit Zabbix to correct this bug in the Linux template. The fix is so simple it is stupid to let people suffer by not fixing it.

[jhboricua]

Correct me if I'm wrong but according to the link at:

https://www.zabbix.com/documentation...s/zabbix_agent

vfs.fs.size[{#FSNAME},free]

which is the key you are proposing to use in order to get around this issue, won't work since that link states the following for vfs.fs.size: 'Reserved space of a file system is taken into account and NOT included when using the free mode.'