Mounted filesystem not being autodiscovered on some nodes

[dstark]

I automate deployment of the Zabbix agents using Puppet.

All of my hosts have the default template applied: Template OS Linux, which has Mounted Filesystem Discovery enabled. However, the Filesystem Items only show up on some hosts, not others. I'm at a loss as to why.

I've verified it's enabled on all my hosts with default settings.

All other items/triggers are reporting fine.

I can perform a get on Filesystem keys successfully on all machines:
./zabbix_get -s <host_ip> -p 10050 -k "vfs.fs.size[/,free]"
8743227392

I don't see anything in the logs related to discovery.

There must be something I'm missing related to discovery. Thanks for your help!

[coreychristian]

Just wondering, have you verified the file system type of the mounts? For instance nfs mounts aren't discovered by default so it maybe a new fs type that's preventing them from being found.

You can find the list of file systems as a regex under 'Administration > General' and the regular expression drop down.

[dstark]

Hi Corey,

The filesystems I'm trying to monitor are ext4. Both the discovering and non-discovering hosts are ext4.

File systems for discovery:
1»^(btrfs|ext2|ext3|ext4|jfs|reiser|xfs|ffs|ufs|jf s|jfs2|vxfs|hfs|ntfs|fat32)$ [Result is TRUE]

Also, I am running Zabbix 2.2.3 server and agent.

[coreychristian]

That does seem odd, are you seeing any drives for the host that isn't working, or none at all (I would assume none based on checking root)

Can you post the output of the following?

./zabbix_get -s <host_ip> -p 10050 -k vfs.fs.discovery

[dstark]

Yes, I can see the drives fine and everything related to file system is healthy...other than it doesn't show up in Zabbix.

Here is the output for a host missing discovery and a different host with working discovery:

[root@Zabbix bin]# ./zabbix_get -s <MISSING_DISCOVERY_HOST_IP> -p 10050 -k vfs.fs.discovery
{
        "data":[
                {
                        "{#FSNAME}":"\/",
                        "{#FSTYPE}":"rootfs"},
                {
                        "{#FSNAME}":"\/proc",
                        "{#FSTYPE}":"proc"},
                {
                        "{#FSNAME}":"\/sys",
                        "{#FSTYPE}":"sysfs"},
                {
                        "{#FSNAME}":"\/dev",
                        "{#FSTYPE}":"devtmpfs"},
                {
                        "{#FSNAME}":"\/dev\/pts",
                        "{#FSTYPE}":"devpts"},
                {
                        "{#FSNAME}":"\/dev\/shm",
                        "{#FSTYPE}":"tmpfs"},
                {
                        "{#FSNAME}":"\/",
                        "{#FSTYPE}":"ext4"},
                {
                        "{#FSNAME}":"\/selinux",
                        "{#FSTYPE}":"selinuxfs"},
                {
                        "{#FSNAME}":"\/dev",
                        "{#FSTYPE}":"devtmpfs"},
                {
                        "{#FSNAME}":"\/proc\/bus\/usb",
                        "{#FSTYPE}":"usbfs"},
                {
                        "{#FSNAME}":"\/boot",
                        "{#FSTYPE}":"ext4"},
                {
                        "{#FSNAME}":"\/proc\/sys\/fs\/binfmt_misc",
                        "{#FSTYPE}":"binfmt_misc"},
                {
                        "{#FSNAME}":"\/var\/lib\/nfs\/rpc_pipefs",
                        "{#FSTYPE}":"rpc_pipefs"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user3",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user4",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user5",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user2",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user1",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/misc",
                        "{#FSTYPE}":"autofs"},
                {
                        "{#FSNAME}":"\/net",
                        "{#FSTYPE}":"autofs"}]}

[root@Zabbix bin]# ./zabbix_get -s <DISCOVERY_HOST_IP> -p 10050 -k vfs.fs.discovery
{
        "data":[
                {
                        "{#FSNAME}":"\/",
                        "{#FSTYPE}":"rootfs"},
                {
                        "{#FSNAME}":"\/proc",
                        "{#FSTYPE}":"proc"},
                {
                        "{#FSNAME}":"\/sys",
                        "{#FSTYPE}":"sysfs"},
                {
                        "{#FSNAME}":"\/dev",
                        "{#FSTYPE}":"devtmpfs"},
                {
                        "{#FSNAME}":"\/dev\/pts",
                        "{#FSTYPE}":"devpts"},
                {
                        "{#FSNAME}":"\/dev\/shm",
                        "{#FSTYPE}":"tmpfs"},
                {
                        "{#FSNAME}":"\/",
                        "{#FSTYPE}":"ext4"},
                {
                        "{#FSNAME}":"\/selinux",
                        "{#FSTYPE}":"selinuxfs"},
                {
                        "{#FSNAME}":"\/dev",
                        "{#FSTYPE}":"devtmpfs"},
                {
                        "{#FSNAME}":"\/proc\/bus\/usb",
                        "{#FSTYPE}":"usbfs"},
                {
                        "{#FSNAME}":"\/boot",
                        "{#FSTYPE}":"ext4"},
                {
                        "{#FSNAME}":"\/proc\/sys\/fs\/binfmt_misc",
                        "{#FSTYPE}":"binfmt_misc"},
                {
                        "{#FSNAME}":"\/var\/lib\/nfs\/rpc_pipefs",
                        "{#FSTYPE}":"rpc_pipefs"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user3",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user4",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user5",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user1",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/home\/guests\/exports\/user2",
                        "{#FSTYPE}":"nfs4"},
                {
                        "{#FSNAME}":"\/misc",
                        "{#FSTYPE}":"autofs"},
                {
                        "{#FSNAME}":"\/net",
                        "{#FSTYPE}":"autofs"}]}

They are identical.

[dstark]

Here is a snippet of errors I see in my zabbix_server.log. Certainly seems suspect.

 24503:20150211:160412.008 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "items_pkey"
DETAIL:  Key (itemid)=(100100000028942) already exists.
 [insert into items (itemid,name,key_,hostid,type,value_type,data_type,delay,delay_flex,history,trends,status,trapper_hosts,units,multiplier,delta,formula,logtimefmt,valuemapid,params,ipmi_sensor,snmp_community,snmp_oid,port,snmpv3_securityname,snmpv3_securitylevel,snmpv3_authprotocol,snmpv3_authpassphrase,snmpv3_privprotocol,snmpv3_privpassphrase,authtype,username,password,publickey,privatekey,description,interfaceid,flags,snmpv3_contextname) values (100100000028942,'Used disk space on $1','vfs.fs.size[/,used]',100100000010172,0,3,0,60,'',30,365,0,'','B',0,0,'1','',null,'','','','','','',0,0,'',0,'',0,'','','','','',100100000000100,4,''),(100100000028943,'Used disk space on $1','vfs.fs.size[/boot,used]',100100000010172,0,3,0,60,'',30,365,0,'','B',0,0,'1','',null,'','','','','','',0,0,'',0,'',0,'','','','','',100100000000100,4,'');
]
 24503:20150211:160412.020 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "triggers_pkey"
DETAIL:  Key (triggerid)=(100100000015958) already exists.
 [insert into triggers (triggerid,description,expression,priority,status,comments,url,type,value,state,flags) values (100100000015958,'Free inodes is less than 20% on volume /','{100100000016711}<20',2,0,'','',0,0,0,4),(100100000015959,'Free inodes is less than 20% on volume /boot','{100100000016712}<20',2,0,'','',0,0,0,4);
]
 24503:20150211:160412.031 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "triggers_pkey"
DETAIL:  Key (triggerid)=(100100000015958) already exists.
 [insert into triggers (triggerid,description,expression,priority,status,comments,url,type,value,state,flags) values (100100000015958,'Free disk space is less than 5% on volume /','{100100000016711}<5',3,0,'','',0,0,0,4),(100100000015959,'Free disk space is less than 5% on volume /boot','{100100000016712}<5',3,0,'','',0,0,0,4);
]
 24503:20150211:160412.041 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "triggers_pkey"
DETAIL:  Key (triggerid)=(100100000015958) already exists.
 [insert into triggers (triggerid,description,expression,priority,status,comments,url,type,value,state,flags) values (100100000015958,'Free disk space is less than 20% on volume /','{100100000016711}<20',2,0,'','',0,0,0,4),(100100000015959,'Free disk space is less than 20% on volume /boot','{100100000016712}<20',2,0,'','',0,0,0,4);
]
 24503:20150211:160412.048 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "graphs_pkey"
DETAIL:  Key (graphid)=(100100000001378) already exists.
 [insert into graphs (graphid,name,width,height,yaxismin,yaxismax,show_work_period,show_triggers,graphtype,show_legend,show_3d,percent_left,percent_right,ymin_type,ymin_itemid,ymax_type,ymax_itemid,flags) values (100100000001378,'Disk space usage /',600,340,0.000000,0.000000,0,0,2,1,1,0.000000,0.000000,0,null,0,null,4),(100100000001379,'Disk space usage /boot',600,340,0.000000,0.000000,0,0,2,1,1,0.000000,0.000000,0,null,0,null,4);
]

[coreychristian]

Based on the error it looks like the items already exist for that host.

You can try removing the host all together, otherwise you may need help some support to straighten out the database. Removing the host and having it added back in should give it a new host ID though which would likely clear up the issue.