SNMP discovery missing some items (but not all)

[phemmer]

First off, the version: zabbix-server-pgsql-3.0.1-1.el7.x86_64

Now what is happening is that I've created a SNMP discovery rule for an F5 load balancer to discover the pool members (servers behind the load balancer). I then have an item prototype which uses the discovery's SNMPINDEX macro to pick up the status of those pool members (up, down, etc).

The problem is that only some of my pool members are being shown, while others are not.

Here's how things are configured:
Discovery rule:

Type: SNMPv2 agent
Key: ltmPoolMbrStatusNodeName
SNMP OID: discovery[{#LTM_POOL_MBR_STATUS_NODE_NAME},.1.3.6.1.4.1.3375.2.2.5.6.2.1.9]
Update interval (in sec): 30

Item prototype:

Name: Availability state for $1
Type: SNMPv2 agent
Key: ltmPoolMbrStatusAvailState[{#LTM_POOL_MBR_STATUS_NODE_NAME}]
SNMP OID: 1.3.6.1.4.1.3375.2.2.5.6.2.1.5.{#SNMPINDEX}
Type of information: Numeric (unsigned)
Data type: Decimal
Update interval (in sec): 30
Show value: ltmPoolMbrStatusAvailState (a mapping I've created)
Applications: Virtual Server

Here's what I see in "lastest data":


As you can see, there are 5 items in the list. Doing a `snmpwalk` I get 14 items, though some of them are duplicates (maybe this is causing the problem?), and 9 unique items. So there are 4 missing items.
This is the `snmpwalk` of the `.1.3.6.1.4.1.3375.2.2.5.6.2.1.9` OID (the discovery OID):

SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.11.47.67.111.109.109.111.110.47.100.109.122.18.47.67.111.109.109.111.110.47.101.99.111.109.45.100.101.118.45.49.80 = STRING: "/Common/ecom-dev-1"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.98.111.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.119.101.98.50.50.115.116.103.8080 = STRING: "/Common/fll2gweb22stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.514 = STRING: "/Common/fll2aspi01stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.514 = STRING: "/Common/fll2aspi02stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.54.48.49.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.601 = STRING: "/Common/fll2aspi01stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.54.48.49.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.601 = STRING: "/Common/fll2aspi02stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.117.100.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.514 = STRING: "/Common/fll2aspi01stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.117.100.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.514 = STRING: "/Common/fll2aspi02stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.35.47.67.111.109.109.111.110.47.99.109.115.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.99.109.115.48.49.115.116.103.80 = STRING: "/Common/fll2gcms01stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.35.47.67.111.109.109.111.110.47.119.119.119.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.18.47.67.111.109.109.111.110.47.101.99.111.109.45.100.101.118.45.49.8080 = STRING: "/Common/ecom-dev-1"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.38.47.67.111.109.109.111.110.47.115.101.97.114.99.104.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.115.114.99.53.50.115.116.103.8983 = STRING: "/Common/fll2gsrc52stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.38.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.115.48.49.115.116.103.80 = STRING: "/Common/fll2asps01stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.38.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.115.48.50.115.116.103.80 = STRING: "/Common/fll2asps02stg"
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.9.40.47.67.111.109.109.111.110.47.112.114.105.99.101.98.111.116.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.112.98.116.48.49.115.116.103.9000 = STRING: "/Common/fll2gpbt01stg"

And this is the `snmpwalk` of the `.1.3.6.1.4.1.3375.2.2.5.6.2.1.5` OID (the item prototype OID):

SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.11.47.67.111.109.109.111.110.47.100.109.122.18.47.67.111.109.109.111.110.47.101.99.111.109.45.100.101.118.45.49.80 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.98.111.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.119.101.98.50.50.115.116.103.8080 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.514 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.514 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.54.48.49.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.601 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.116.99.112.54.48.49.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.601 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.117.100.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.49.115.116.103.514 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.34.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.105.110.100.101.120.101.114.45.117.100.112.53.49.52.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.105.48.50.115.116.103.514 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.35.47.67.111.109.109.111.110.47.99.109.115.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.99.109.115.48.49.115.116.103.80 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.35.47.67.111.109.109.111.110.47.119.119.119.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.18.47.67.111.109.109.111.110.47.101.99.111.109.45.100.101.118.45.49.8080 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.38.47.67.111.109.109.111.110.47.115.101.97.114.99.104.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.115.114.99.53.50.115.116.103.8983 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.38.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.115.48.49.115.116.103.80 = INTEGER: 1
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.38.47.67.111.109.109.111.110.47.115.112.108.117.110.107.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.97.115.112.115.48.50.115.116.103.80 = INTEGER: 3
SNMPv2-SMI::enterprises.3375.2.2.5.6.2.1.5.40.47.67.111.109.109.111.110.47.112.114.105.99.101.98.111.116.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.112.98.116.48.49.115.116.103.9000 = INTEGER: 1

When I increase the log level to 4, I see the following items in the log, which would seem to indicate zabbix is aware of the missing OIDs: (only providing seemingly relevent snippets as the whole log is huge, but can provide the whole thing if needed).

 27890:20160406:002002.464 In zbx_snmp_set_result() type:4 value_type:1 data_type:0
 27890:20160406:002002.464 In zbx_snmp_get_octet_string()
 27890:20160406:002002.464 zbx_snmp_get_octet_string() full value:'STRING: "/Common/fll2asps01stg"' hint:'(null)'
 27890:20160406:002002.464 End of zbx_snmp_get_octet_string():'/Common/fll2asps01stg'
 27890:20160406:002002.464 End of zbx_snmp_set_result():SUCCEED

...

 27890:20160406:002002.480 In substitute_discovery_macros() data:'{#LTM_POOL_MBR_STATUS_NODE_NAME}'
 27891:20160406:002002.480 In zbx_snmp_set_result() type:129 value_type:3 data_type:0
 27890:20160406:002002.480 End of substitute_discovery_macros():SUCCEED data:'/Common/fll2asps02stg'
 27891:20160406:002002.480 End of zbx_snmp_set_result():NOTSUPPORTED
 27890:20160406:002002.480 End of substitute_key_macros():SUCCEED data:'ltmPoolMbrStatusAvailState[/Common/fll2asps02stg]'
 27891:20160406:002002.480 In zbx_snmp_set_result() type:129 value_type:3 data_type:0
 27890:20160406:002002.480 In substitute_key_macros() data:'ltmPoolMbrStatusAvailState[{#LTM_POOL_MBR_STATUS_NODE_NAME}]'
 27891:20160406:002002.480 End of zbx_snmp_set_result():NOTSUPPORTED
 27890:20160406:002002.480 In substitute_discovery_macros() data:'{#LTM_POOL_MBR_STATUS_NODE_NAME}'
 27891:20160406:002002.480 In zbx_snmp_set_result() type:70 value_type:3 data_type:0
 27890:20160406:002002.480 End of substitute_discovery_macros():SUCCEED data:'/Common/fll2asps01stg'
 27891:20160406:002002.480 End of zbx_snmp_set_result():SUCCEED
 27890:20160406:002002.480 End of substitute_key_macros():SUCCEED data:'ltmPoolMbrStatusAvailState[/Common/fll2asps01stg]'
 27891:20160406:002002.481 In zbx_snmp_set_result() type:66 value_type:3 data_type:0
 27890:20160406:002002.481 In substitute_key_macros() data:'ltmPoolMbrStatusAvailState[{#LTM_POOL_MBR_STATUS_NODE_NAME}]'
 27891:20160406:002002.481 End of zbx_snmp_set_result():SUCCEED
 27890:20160406:002002.481 In substitute_discovery_macros() data:'{#LTM_POOL_MBR_STATUS_NODE_NAME}'
 27891:20160406:002002.481 End of zbx_snmp_get_values():SUCCEED
 27890:20160406:002002.481 End of substitute_discovery_macros():SUCCEED data:'/Common/fll2gsrc52stg'
 27891:20160406:002002.481 End of zbx_snmp_process_standard():SUCCEED

...

 27890:20160406:002002.482 End of substitute_key_macros():SUCCEED data:'ltmPoolMbrStatusAvailState[/Common/fll2aspi02stg]'
 27890:20160406:002002.482 In substitute_key_macros() data:'ltmPoolMbrStatusAvailState[{#LTM_POOL_MBR_STATUS_NODE_NAME}]'
 27890:20160406:002002.482 In substitute_discovery_macros() data:'{#LTM_POOL_MBR_STATUS_NODE_NAME}'
 27890:20160406:002002.482 End of substitute_discovery_macros():SUCCEED data:'/Common/fll2asps01stg'
 27890:20160406:002002.482 End of substitute_key_macros():SUCCEED data:'ltmPoolMbrStatusAvailState[/Common/fll2asps01stg]'
 27890:20160406:002002.482 In substitute_key_macros() data:'ltmPoolMbrStatusAvailState[{#LTM_POOL_MBR_STATUS_NODE_NAME}]'
 27890:20160406:002002.482 In substitute_discovery_macros() data:'{#LTM_POOL_MBR_STATUS_NODE_NAME}'
 27890:20160406:002002.482 End of substitute_discovery_macros():SUCCEED data:'/Common/fll2gsrc52stg'

However one thing that stands out is the `zbx_snmp_set_result():NOTSUPPORTED` bit. But I don't know why it would be claiming unsupported.

Happy to provide more details upon request.

[phemmer]

I created a new discovery rule and item prototype to resolve the duplicates by adding some extra OIDs to the discovery rule, and using them in the item prototype's name. This did not help the issue. More items show up in the list, but it's just the former duplicates that are now not duplicated which show up. I'm still missing all the originally missing items.

[phemmer]

Finally figured this out. I was looking through the `items` table in the database and found entries with the `error` column set to values such as:

Cannot create item: value "1.3.6.1.4.1.3375.2.2.5.6.2.1.5.38.47.67.111.109.109.111.110.47.115.101.97.114.99.104.45.115.116.103.46.101.99.111.109.46.99.104.101.119.121.46.99.111.109.45.112.111.111.108.21.47.67.111.109.109.111.110.47.102.108.108.50.103.115.114.99.53.50.115.116.103.8983" is too long.

And there's an open zabbix bug on the issue, though it has been open for quite a while: https://support.zabbix.com/browse/ZBX-9309

Would be nice if there were a way to see such errors on the web ui. Other posts on the web say you're supposed to be able to list the template items with a filter of "State=Not supported", but this appears to only show static items, not ones created by discovery.