How to remove discovered filesystems in Linux SNMP

[aspitzer]

I am doing a Zabbix POC, and initially most of the machines are using SNMP (because it is already enabled), but I am also testing the zabbix-agent2 on a bunch of boxes.

For Linux SNMP, I am running into an issue where zabbix is picking up thousands of Kubernetes transient virtual mounts:
/var/lib/docker/*
/var/lib/kubelet/*

Examples:
/var/lib/docker/containers/88f7a781c55a265587422a8771ab04da2a2a810ccf5f55b443 9048420852e402/mounts/shm
/var/lib/docker/overlay2/e30b6bc019e6d9e0a34d7a1ed80e88efcc27f4d9c4e9a75d18 71ed7bf203ea2c/merged
/var/lib/kubelet/pods/71a4dd88-51a2-4612-aa87-e3ab05dd7708/volumes/kubernetes.io~secret/redacted
/var/lib/docker/containers/72e9032a72850e222a0f8296f087cb6b5aa4aed05c65d798d7 a63727d9a86f2e/mounts/shm

Upon digging around in the templates, it seems that {$VFS.FS.FSNAME.NOT_MATCHES} can be used to exclude filesystems with a regex.

I tried extending the existing {$VFS.FS.FSNAME.NOT_MATCHES} by adding "|/var/lib/docker/.+|/var/lib/kubelet/.+", but they are still popping up.

I did the following:
Administration -> General -> Macros -> {$VFS.FS.FSNAME.NOT_MATCHES} = ^(/dev|/sys|/run|/proc|.+/shm$|/var/lib/docker/.+|/var/lib/kubelet/.+)

Is {$VFS.FS.FSNAME.NOT_MATCHES} the proper way to filter out unwanted? How do I set it globally? Otherwise what is the proper way to filter filesystem names?

Thanks!

[aspitzer]

For completeness:
version zabbix-server-mysql-6.2.7-release1.el8.x86_64 on RHEL8.
version zabbix-proxy-mysql-6.2.7-release1.el7.x86_64 on RHEL7 that feeds all the data into the server.

[aspitzer]

Is this a bug in 6.2.7? If I set the global macro, it does not seem to override the variable for the correct template, only OTHER templates.

I set the following global macros, and added XXX at the front to make it easier to spot the global version:

Global Macros - Admin -> General -> Macros:
{$VFS.FS.FSNAME.NOT_MATCHES} = ^(XXX|/dev|/sys|/run|/proc|.+/shm$|/var/lib/docker/.+|/var/lib/kubelet/.+)
{$NET.IF.IFNAME.NOT_MATCHES} = (XXX|^Software Loopback Interface|^NULL[0-9.]*$|^[Ll]o[0-9.]*$|^[Ss]ystem$|^Nu[0-9.]*$|^veth[0-9a-z]+$|^cali|^flannel|^fw|^tap|docker[0-9]+|br-[a-z0-9]{12})

Now if I go to
Configuration -> Templates -> Linux filesystems by SNMP -> Macros -> Inherited and template macros

{$VFS.FS.FSNAME.NOT_MATCHES} = ^(/dev|/sys|/run|/proc|.+/shm$)
{$NET.IF.IFNAME.NOT_MATCHES} = (XXX|^Software Loopback Interface|^NULL[0-9.]*$|^[Ll]o[0-9.]*$|^[Ss]ystem$|^Nu[0-9.]*$|^veth[0-9a-z]+$|^cali|^flannel|^fw|^tap|docker[0-9]+|br-[a-z0-9]{12})

If I go to
Configuration -> Templates -> Network Generic Devices by SNMP -> Macros -> Inherited and template macros

{$VFS.FS.FSNAME.NOT_MATCHES} = ^(XXX|/dev|/sys|/run|/proc|.+/shm$|/var/lib/docker/.+|/var/lib/kubelet/.+)
{$NET.IF.IFNAME.NOT_MATCHES} = (^Software Loopback Interface|^NULL[0-9.]*$|^[Ll]o[0-9.]*$|^[Ss]ystem$|^Nu[0-9.]*$|^veth[0-9a-z]+$|docker[0-9]+|br-[a-z0-9]{12})

So the Filesystem templates sets the {$NET.IF.IFNAME.NOT_MATCHES} to the global Macro, but not {$VFS.FS.FSNAME.NOT_MATCHES}.
The Network template sets {$VFS.FS.FSNAME.NOT_MATCHES} correctly, but not {$NET.IF.IFNAME.NOT_MATCHES}.

[tim.mooney]

Is this a bug in 6.2.7? If I set the global macro, it does not seem to override the variable for the correct template, only OTHER templates.

I can't find it right now, but I believe the macro inheritance order is set at the Global level -> set at the template level -> set at the host level

If I'm correct about that (and I might not be, so see if you can find it in the documentation), then a macro set at the Global level will never override a template that has a default setting for the same macro.

You either need to modify the default value for the macro at the template level OR clone the template to a new local copy of the template and then customize that.

[aspitzer]

I can't find it right now, but I believe the macro inheritance order is set at the Global level -> set at the template level -> set at the host level

If I'm correct about that (and I might not be, so see if you can find it in the documentation), then a macro set at the Global level will never override a template that has a default setting for the same macro.

You either need to modify the default value for the macro at the template level OR clone the template to a new local copy of the template and then customize that.

I think you are looking for: https://www.zabbix.com/documentation...os/user_macros
However I was interpenetrating it as the order of overloading, where the global is the last one to load/overwrite everyone else.

I thought the whole purpose of a the Global macro was to have a place to configure variables that are shared in a single place. The concrete example is the variable {$VFS.FS.FSNAME.NOT_MATCHES}. This is used an at least 4-5 templates.

I suppose the next question is: Am I supposed to REMOVE the template level Macro {$VFS.FS.FSNAME.NOT_MATCHES} in all the Linux SNMP/Zabbix templates so that it defaults to the Global? Maybe that is the correct procedure?

[aspitzer]

interpenetrating?!?!? Where did that come from? interpenetrating = interpreting... For some reason the wiki says that "editing my posts is forbidden!"

[tim.mooney]

I suspect that once you have more posts here, the ability to edit your existing posts will become available. I'm guessing the "can't edit posts" is an anti-spam measure for new accounts. These forums were getting a lot of bot spam.

[aspitzer]

Okay, I was able to solve this, but I dont understand the logic of having template vars not being able to be overwritten globally.
So I had to change VFS.FS.FSNAME.NOT_MATCHES in like 5 templates. I had to change NET.IF.IFNAME.NOT_MATCHES in like 5 templates. I then had to go to all the templates:
Template --> Discovery Rules and set "Update Interval" to 1h and "Keep lost resources" to 1h (to clean out all the old values) - the default is to keep them for 30d.

After about an hour or so, it looks like the filesystems and interfaces finally cleared out.

[cyber]

but I don't understand the logic of having template vars not being able to be overwritten globally.

That's just the way it works...

https://www.zabbix.com/documentation/current/en/manual/config/macros/user_macros

Zabbix resolves macros according to the following precedence:

1. host level macros (checked first)
2. macros defined for first level templates of the host (i.e., templates linked directly to the host), sorted by template ID
3. macros defined for second level templates of the host, sorted by template ID
4. macros defined for third level templates of the host, sorted by template ID, etc.
5. global macros (checked last)

In other words, if a macro does not exist for a host, Zabbix will try to find it in the host templates of increasing depth. If still not found, a global macro will be used, if exists.