Kubernetes node: how to avoid all these discovered overlay and tmpfs file systems?

[harri]

The Linux by Zabbix agent template provides these macros:

{
    "macro": "{$VFS.FS.FSTYPE.MATCHES}",
    "value": "^(btrfs|ext2|ext3|ext4|reiser|xfs|ffs|ufs|jfs|jfs2|vxfs|hfs|apfs|refs|ntfs|fat32|zfs)$",
    "description": "This macro is used in filesystems discovery. Can be overridden on the host or linked template level."
},
{
    "macro": "{$VFS.FS.FSTYPE.NOT_MATCHES}",
    "value": "^\\s$",
    "description": "This macro is used in filesystems discovery. Can be overridden on the host or linked template level."
},
​

AFAICT the host doesn't override these macros. There are no other templates involved messing up these macros, either. So how comes the overlay and tmpfs filesystems are not skipped in mounted filesystem discovery on my Kubernetes nodes? The nfs4 mounts are not discovered, for example.

[harri]

Hi Markku ,
sorry to say, but I think I got confused by the test returning the unfiltered list. The discovery still returns a bazillion of unwanted entries, but the unwanted items in Zabbix are all ext4. My bad.

I have added /var/lib/kubelet/ (the common directory of most unwanted mount points) to the blacklist now. caii.* is not discovered anymore, either. Both changes have been applied to the passive and active Linux templates.

        -
          macro: '{$NET.IF.IFNAME.NOT_MATCHES}'
          value: '(^Software Loopback Interface|^NULL[0-9.]*$|^[Ll]o[0-9.]*$|^[Ss]ystem$|^Nu[0-9.]*$|^veth[0-9A-z]+$|^cali[0-9A-z]+$|docker[0-9]+|br-[a-z0-9]{12})'
          description: 'Filter out loopbacks, nulls, docker veth links and docker0 bridge by default'
        -
          macro: '{$VFS.FS.FSNAME.NOT_MATCHES}'
          value: ^(/dev|/sys|/run|/proc|.+/shm$|/var/lib/kubelet/)
          description: 'This macro is used in filesystems discovery. Can be overridden on the host or linked template level'

Thank you very much for your help.

[markfree]

I'm not sure what you mean...
Your host must be linked to "Linux by Zabbix agent" template, therefore, it should find all FS that match that {$VFS.FS.FSTYPE.MATCHES} macro value (regex).

Considering your template uses the above macro values, and you did not change those macros at host level, when you link "Linux by Zabbix agent" template to your nodes, is "tmpfs" discovered?

[harri]

Yes, it is.

I would like to avoid all the persistent volumes generated by openebs on the fly, for example. And all these mounts to /var/lib/kubelet/pods.

[Markku]

Can you open the discovery rule on the host and use the Test button? Then you will see the actual discovery data in one of the fields. Do you see anything strange in that data? Are the FSTYPEs assigned correctly?

Other thing, are the LLD filter rules configured correctly (Type of calculation = And, all rules have correct macros)?

Markku

[harri]

Hi Markku
AFAICS the built in test returns the same as "zabbix_get -s srvl072a -k vfs.fs.discovery" (sorted using "jq -SM ."). The #FSTYPEs appear to be correct.

I am using the "Linux by Zabbix Agent" template (6.0) from git. The template defines

​{$VFS.FS.FSTYPE.MATCHES} ^(btrfs|ext2|ext3|ext4|reiser|xfs|ffs|ufs|jfs|jfs2 |vxfs|hfs|apfs|refs|ntfs|fat32|zfs)$
{$VFS.FS.FSTYPE.NOT_MATCHES} ^\s$

Please note that neither overlay nor tmpfs nor proc, autofs, cgroup, cgroup2 or others are in the "MATCHES" list, yet they are all shown by the built-in test. Adding "overlay" to the NOT_MATCHES list (just as an example) did not help.

Apparently this filter is ignored. Yet autofs, cgroup, cgroup2 and others are not shown in Zabbix. overlay and tmpfs are, so the conclusion is that there is yet another filter with a higher priority.

This Zabbix installation has been upgraded over time from 1.8 to 6.0. Maybe some old stuff? The host I am using for verification has been added to Zabbix just recently.

[Markku]

yet they are all shown by the built-in test

The Test function does not test the filters, it only fetches the discovery data.

You didn't answer my question about if the filters are configured correctly, so I take it you need help with that. Please share a screenshot about the filters page and all the user macro values for the macros that are used in the filters.

Markku

[harri]

Hi Markku, here are the screenshots:

[harri]

The Test function does not test the filters, it only fetches the discovery data.

Are you sure about this? I see separate buttons "get value" (ie fetch) and "get value and test" in my GUI. Not applying the filters in the 2nd step would be pretty much unexpected.

[Markku]

Thanks, I don't see any obvious problem in those but there are still some things to check.

Can you show the discovery data (the json)?

Also, are you sure the items are coming from this template and are still being discovered (and not just waiting for the deletion to happen in 30 days or whatever is configured in the discovery rule)?

Markku

[Markku]

Are you sure about this? I see separate buttons "get value" (ie fetch) and "get value and test" in my GUI. Not applying the filters in the 2nd step would be pretty much unexpected.

It surely looks here like the filters are not applied when testing. Documentation doesn't even show the Test button I see.

Markku

[harri]

Hi Markku, attached is the output of "zabbix_get -s srvl072a -k vfs.fs.discovery | jq -SM ."

I am sure these filesystems are discovered each day. Of course there are also old discovered file systems in the database. I have no indication that they are not dropped after 30 days. The point is, these overlay and tmpfs file system are generated with an artificial name again and again. The FS discovery on one of our major Kubernetes nodes returns 951 filesystems at the moment. And all Kubernetes and Docker nodes we run are affected by this problem, not to mention that Zabbix discovers a huge number of unwanted network interfaces as well. Similar problem, but lets focus on the file systems.

[Markku]

Ok, sorry, I don't have the possibility to open that file on this tablet so I'll see that later.

Markku

[harri]

Sorry about that. The web page didn't allow me to upload json, so I had to compress it.

[Markku]

Ok, so for example these were in the discovery data:

  {
    "{#FSNAME}": "/export/docker-data/overlay2/255451bb98338070aaeb42286ac1cbb217627ad9e7532fdf1038f620b3fbc1bc/merged",
    "{#FSTYPE}": "overlay"
  },

  {
    "{#FSNAME}": "/var/lib/kubelet/pods/79a209f8-59ae-44eb-8e5b-50da850d3f3e/volumes/kubernetes.io~projected/kube-api-access-6ptqk",
    "{#FSTYPE}": "tmpfs"
  },​

$VFS.FS.FSNAME.MATCHES: ok (filter matches everything)
$VFS.FS.FSNAME.NOT_MATCHES: ok (neither of these two matches the /dev, /sys, /run etc. filter value)
$VFS.FS.FSTYPE.MATCHES: does not match either of those (overlay and tmpfs is not present in the filter macro)
$VFS.FS.FSTYPE.NOT_MATCHES: ok (filter doesn't match anything)

Since the Type of calculation is And, it should not create items for either of these.

If you say the items are still created, either something else is creating the items, or they are remains of some older attempts.

If you delete the items manually, will they reappear? Is there anything interesting in the Audit log?

Markku

[harri]

Hi Markku ,
sorry to say, but I think I got confused by the test returning the unfiltered list. The discovery still returns a bazillion of unwanted entries, but the unwanted items in Zabbix are all ext4. My bad.

I have added /var/lib/kubelet/ (the common directory of most unwanted mount points) to the blacklist now. caii.* is not discovered anymore, either. Both changes have been applied to the passive and active Linux templates.

        -
          macro: '{$NET.IF.IFNAME.NOT_MATCHES}'
          value: '(^Software Loopback Interface|^NULL[0-9.]*$|^[Ll]o[0-9.]*$|^[Ss]ystem$|^Nu[0-9.]*$|^veth[0-9A-z]+$|^cali[0-9A-z]+$|docker[0-9]+|br-[a-z0-9]{12})'
          description: 'Filter out loopbacks, nulls, docker veth links and docker0 bridge by default'
        -
          macro: '{$VFS.FS.FSNAME.NOT_MATCHES}'
          value: ^(/dev|/sys|/run|/proc|.+/shm$|/var/lib/kubelet/)
          description: 'This macro is used in filesystems discovery. Can be overridden on the host or linked template level'

Thank you very much for your help.