vfs.fs.size works directly on agent but ZBX_NOTSUPPORTED when requested from server

[ILIV]

I'm trying to learn more about low-level discovery, and luckily I have a problem to tackle

This is a simple test configuration that I run on a single computer utilizing virtualbox VM instance where I run server, and agent is running directly on my personal laptop.

Server is running on Debian Squeeze

root@bitarena:~# dpkg -l |grep zabbix
ii zabbix-agent 1:2.0.2~cmd1-1 network monitoring solution - agent
ii zabbix-frontend-php 1:2.0.2~cmd1-1 network monitoring solution - PHP front-end
rc zabbix-proxy-mysql 1:2.0.2~cmd1-1 network monitoring solution - proxy (using MySQL)
ii zabbix-server-pgsql 1:2.0.2~cmd1-1 network monitoring solution - server (using PostgreSQL)

while Agent on Arch Linux

root@sega /home/iliv % pacman -Q |grep zabbix
zabbix-agent 2.0.2-1

Problem

root@bitarena:~# zabbix_get -s sega -k vfs.fs.size[/,free]
2815614976

root@bitarena:~# zabbix_get -s sega -k vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]
ZBX_NOTSUPPORTED

root@bitarena:~# zabbix_get -s sega -k vfs.fs.discovery
{
"data":[
{
"{#FSNAME}":"\/",
"{#FSTYPE}":"rootfs"},
{
"{#FSNAME}":"\/proc",
"{#FSTYPE}":"proc"},
{
"{#FSNAME}":"\/sys",
"{#FSTYPE}":"sysfs"},
{
"{#FSNAME}":"\/dev",
"{#FSTYPE}":"devtmpfs"},
{
"{#FSNAME}":"\/run",
"{#FSTYPE}":"tmpfs"},
{
"{#FSNAME}":"\/",
"{#FSTYPE}":"ext3"},
{
"{#FSNAME}":"\/dev\/pts",
"{#FSTYPE}":"devpts"},
{
"{#FSNAME}":"\/dev\/shm",
"{#FSTYPE}":"tmpfs"},
{
"{#FSNAME}":"\/proc\/sys\/fs\/binfmt_misc",
"{#FSTYPE}":"binfmt_misc"},
{
"{#FSNAME}":"\/tmp",
"{#FSTYPE}":"tmpfs"},
{
"{#FSNAME}":"\/home\/iliv\/.gvfs",
"{#FSTYPE}":"fuse.gvfs-fuse-daemon"},
{
"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf",
"{#FSTYPE}":"ext2"}]}

However, it works directly on Agent host:

iliv@sega ~ % zabbix_agent -t vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]
vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free] [u|409642881024]

iliv@sega ~ % zabbix_agent -t vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]
vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree] [d|41.606911]

Screenshots that show automagically LLD-created graphs:
i.imgur . com / EpRZu . png
i.imgur . com / EEAKp . png

Those not supported, of course, don't show up in graphs drop-down list at all.


I pumped up log debugging output to the max on Zabbix Server/Agent hosts, but it seems to be of little help:

Server

root@bitarena:~# tail -f /var/log/zabbix-server/zabbix_server.log |grep iliv
"{#FSNAME}":"\/home\/iliv\/.gvfs",
"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf",
13721:20121008:081248.519 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.522 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.527 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.528 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]'
13721:20121008:081248.529 query [txnlev:1] [select distinct i.itemid from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=23391 and i.key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]']
13721:20121008:081248.534 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]' and itemid<>23435]
update items set name='Free disk space on $1',key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]',type=0,value_type=3,data_type=0,delay=60,delay_f lex='',history=7,trends=365,trapper_hosts='',units ='B',multiplier=0,delta=0,formula='1',logtimefmt=' ',valuemapid=null,params='',ipmi_sensor='',snmp_co mmunity='',snmp_oid='',port='',snmpv3_securityname ='',snmpv3_securitylevel=0,snmpv3_authpassphrase=' ',snmpv3_privpassphrase='',authtype=0,username='', password='',publickey='',privatekey='',description ='',interfaceid=3,flags=4 where itemid=23435;
13721:20121008:081248.574 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.575 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.579 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.579 End of substitute_key_macros():SUCCEED data:'vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]'
13721:20121008:081248.581 query [txnlev:1] [select distinct i.itemid from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=23392 and i.key_='vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]']
13721:20121008:081248.583 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]' and itemid<>23437]
update items set name='Free inodes on $1 (percentage)',key_='vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]',type=0,value_type=0,data_type=0,delay=60,delay_f lex='',history=7,trends=365,trapper_hosts='',units ='%',multiplier=0,delta=0,formula='1',logtimefmt=' ',valuemapid=null,params='',ipmi_sensor='',snmp_co mmunity='',snmp_oid='',port='',snmpv3_securityname ='',snmpv3_securitylevel=0,snmpv3_authpassphrase=' ',snmpv3_privpassphrase='',authtype=0,username='', password='',publickey='',privatekey='',description ='',interfaceid=3,flags=4 where itemid=23437;
13721:20121008:081248.627 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.629 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.635 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.636 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]'
13721:20121008:081248.637 query [txnlev:1] [select distinct i.itemid from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=23393 and i.key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]']
13721:20121008:081248.645 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]' and itemid<>23439]
update items set name='Total disk space on $1',key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]',type=0,value_type=3,data_type=0,delay=3600,delay _flex='',history=7,trends=365,trapper_hosts='',uni ts='B',multiplier=0,delta=0,formula='1',logtimefmt ='',valuemapid=null,params='',ipmi_sensor='',snmp_ community='',snmp_oid='',port='',snmpv3_securityna me='',snmpv3_securitylevel=0,snmpv3_authpassphrase ='',snmpv3_privpassphrase='',authtype=0,username=' ',password='',publickey='',privatekey='',descripti on='',interfaceid=3,flags=4 where itemid=23439;
13721:20121008:081248.690 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.691 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.695 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.696 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,used]'
13721:20121008:081248.696 query [txnlev:1] [select distinct i.itemid from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=23394 and i.key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,used]']
13721:20121008:081248.699 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,used]' and itemid<>23441]
update items set name='Used disk space on $1',key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,used]',type=0,value_type=3,data_type=0,delay=60,delay_f lex='',history=7,trends=365,trapper_hosts='',units ='B',multiplier=0,delta=0,formula='1',logtimefmt=' ',valuemapid=null,params='',ipmi_sensor='',snmp_co mmunity='',snmp_oid='',port='',snmpv3_securityname ='',snmpv3_securitylevel=0,snmpv3_authpassphrase=' ',snmpv3_privpassphrase='',authtype=0,username='', password='',publickey='',privatekey='',description ='',interfaceid=3,flags=4 where itemid=23441;
13721:20121008:081248.741 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.743 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.748 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.749 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]'
13721:20121008:081248.749 query [txnlev:1] [select distinct i.itemid from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=23395 and i.key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]']
13721:20121008:081248.754 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]' and itemid<>23443]
update items set name='Free disk space on $1 (percentage)',key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]',type=0,value_type=0,data_type=0,delay=60,delay_f lex='',history=7,trends=365,trapper_hosts='',units ='%',multiplier=0,delta=0,formula='1',logtimefmt=' ',valuemapid=null,params='',ipmi_sensor='',snmp_co mmunity='',snmp_oid='',port='',snmpv3_securityname ='',snmpv3_securitylevel=0,snmpv3_authpassphrase=' ',snmpv3_privpassphrase='',authtype=0,username='', password='',publickey='',privatekey='',description ='',interfaceid=3,flags=4 where itemid=23443;
13721:20121008:081248.833 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.835 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.841 End of substitute_discovery_macros() data:'Free inodes is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.849 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.850 End of substitute_key_macros():SUCCEED data:'vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]'
13721:20121008:081248.851 End of DBlld_expand_trigger_expression() expr:'{sega:vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree].last(0)}<20'
13721:20121008:081248.853 query [txnlev:1] [select distinct t.triggerid,t.expression from triggers t,trigger_discovery td where t.triggerid=td.triggerid and td.parent_triggerid=13554 and t.description='Free inodes is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf']
13721:20121008:081248.861 End of DBlld_expand_trigger_expression() expr:'{sega:vfs.fs.inode[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree].last(0)}<20'
13721:20121008:081248.861 query [txnlev:1] [select distinct t.triggerid,t.expression from triggers t,functions f,items i where t.triggerid=f.triggerid and f.itemid=i.itemid and i.hostid=10086 and t.description='Free inodes is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf' and t.triggerid<>13556]
update triggers set description='Free inodes is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf',priority=2,comments='',url='',type=0 ,flags=4 where triggerid=13556;
13721:20121008:081248.914 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081248.915 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081248.923 End of substitute_discovery_macros() data:'Free disk space is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.930 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081248.931 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree]'
13721:20121008:081248.934 End of DBlld_expand_trigger_expression() expr:'{sega:vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree].last(0)}<20'
13721:20121008:081248.934 query [txnlev:1] [select distinct t.triggerid,t.expression from triggers t,trigger_discovery td where t.triggerid=td.triggerid and td.parent_triggerid=13553 and t.description='Free disk space is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf']
13721:20121008:081248.943 End of DBlld_expand_trigger_expression() expr:'{sega:vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,pfree].last(0)}<20'
13721:20121008:081248.945 query [txnlev:1] [select distinct t.triggerid,t.expression from triggers t,functions f,items i where t.triggerid=f.triggerid and f.itemid=i.itemid and i.hostid=10086 and t.description='Free disk space is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf' and t.triggerid<>13558]
update triggers set description='Free disk space is less than 20% on volume /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf',priority=2,comments='',url='',type=0 ,flags=4 where triggerid=13558;
13721:20121008:081249.019 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/home\/iliv\/.gvfs","{#FSTYPE}":"fuse.gvfs-fuse-daemon"}'
13721:20121008:081249.022 In DBlld_check_record() jp_row:'{"{#FSNAME}":"\/run\/media\/iliv\/9d244934-d9fa-4a8e-8dd7-5c595f5518cf","{#FSTYPE}":"ext2"}'
13721:20121008:081249.028 End of substitute_discovery_macros() data:'Disk space usage /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081249.029 query [txnlev:1] [select distinct g.graphid from graphs g,graph_discovery gd where g.graphid=gd.graphid and gd.parent_graphid=541 and g.name='Disk space usage /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf']
13721:20121008:081249.035 query [txnlev:1] [select distinct g.graphid from graphs g,graphs_items gi,items i where g.graphid=gi.graphid and gi.itemid=i.itemid and i.hostid=10086 and g.name='Disk space usage /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf' and g.graphid<>550]
13721:20121008:081249.044 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081249.044 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]'
13721:20121008:081249.044 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]']
13721:20121008:081249.049 End of substitute_discovery_macros() data:'/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf'
13721:20121008:081249.050 End of substitute_key_macros():SUCCEED data:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]'
13721:20121008:081249.050 query [txnlev:1] [select itemid from items where hostid=10086 and key_='vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]']
13721:20121008:081249.058 DBget_graphitems() [0] itemid:23435 key:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]'
13721:20121008:081249.058 DBget_graphitems() [1] itemid:23439 key:'vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,total]'
update graphs set name='Disk space usage /run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf',width=600,height=340,yaxismin=0.0000 00,yaxismax=0.000000,show_work_period=0,show_trigg ers=0,graphtype=2,show_legend=1,show_3d=1,percent_ left=0.000000,percent_right=0.000000,ymin_type=0,y max_type=0,ymin_itemid=null,ymax_itemid=null,flags =4 where graphid=550;

Agent

..
19266:20121008:182542.245 Requested [vfs.fs.size[/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf,free]]
19266:20121008:182542.245 Sending back [ZBX_NOTSUPPORTED]
..

What could be wrong there?

/run/media/iliv/9d244934-d9fa-4a8e-8dd7-5c595f5518cf is a mount point for an external USB drive with ext2 filesystem, nothing too special about it. Well, you've just seen yourself that I can get item values on Agent host without any problems. However, it is when the Server initiates this check things go wrong. Having seen that zabbix_agent process can actually obtain values for vfs.fs.size I'm a little baffled as to why the error says ZBX_NOTSUPPORTED.

Any ideas what's going on here or how further I could troubleshoot this problem?

[ILIV]

Answering my own question.

This is a permissions problem. You see, as I wrote yesterday it is an external USB drive connected to a workstation computer running GNOME 3.x, where external media is automatically mounted by D-BUS/udisksd and mount point is granted access permissions of the user that is currently running GNOME session and attempts to automatically mount a disk drive.

So, my disk is auto-mounted for system user iliv, and it appears udisksd relies on extended attributes to make sure this user can access the drive (I've overlooked that tiny trailing + symbol for so many times!) and zabbix_agentd runs as zabbix shell user

iliv@sega ~ % lsh /run/media/
total 0
drwxr-xr-x 4 root root 80 Oct 4 10:12 .
drwxr-xr-x 26 root root 840 Oct 9 08:16 ..
drwxr-x---+ 3 root root 60 Oct 9 12:06 iliv

iliv@sega ~ % getfacl /run/media/iliv
getfacl: Removing leading '/' from absolute path names
# file: run/media/iliv
# owner: root
# group: root
user::rwx
user:iliv:r-x
group::---
mask::r-x
other::---

zabbix being 'other' user of course fails to access this mount point.

If I manually remount drive as root user, zabbix_agentd can pull out numbers no matter what system user it's run as.