Ceph Module results in 'item type "18" cannot be used with traps' sending data

[KowMangler]

I cannot figure out what I need to do here. I'm suspecting the sender version needs to be downgraded (container so it is an easy enough to deploy with a new sender version) but I wanted to check what the heck "type 18" is.

zabbix_server version:

Zabbix 6.0.3

zabbix_proxy version:

zabbix_proxy (Zabbix) 6.0.3

zabbix_sender version:

zabbix_sender (Zabbix) 6.0.4

I set up debugging and see on the ceph mgr:

debug 2022-05-11T21:17:05.093+0000 7f379c280700 0 [zabbix DEBUG root] {'overall_status': 'HEALTH_OK', 'overall_status_int': 0, 'num_mon': 3, 'num_pools': 2, 'total_used_bytes': 12336111616, 'total_bytes': 161048690688, 'total_avail_bytes': 148712579072, '[device_health_metrics,rd_bytes]': 0, '[device_health_metrics,wr_bytes]': 0, '[device_health_metrics,rd_ops]': 0, '[device_health_metrics,wr_ops]': 0, '[device_health_metrics,bytes_used]': 0, '[device_health_metrics,stored_raw]': 0, '[device_health_metrics,percent_used]': 0.0, '[k8s-block,rd_bytes]': 914341888, '[k8s-block,wr_bytes]': 6354202624, '[k8s-block,rd_ops]': 6667, '[k8s-block,wr_ops]': 50517, '[k8s-block,bytes_used]': 12205612289, '[k8s-block,stored_raw]': 12205595648, '[k8s-block,percent_used]': 8.042498677968979, 'wr_ops': 50517, 'rd_ops': 6667, 'wr_bytes': 6354202624, 'rd_bytes': 914341888, 'num_osd': 3, 'osd_nearfull_ratio': 0.8500000238418579, 'osd_full_ratio': 0.949999988079071, 'osd_backfillfull_ratio': 0.8999999761581421, 'num_pg_temp': 0, '[osd.0,up]': 1, '[osd.0,in]': 1, '[osd.1,up]': 1, '[osd.1,in]': 1, '[osd.2,up]': 1, '[osd.2,in]': 1, 'num_osd_up': 3, 'num_osd_in': 3, '[osd.2,osd_fill]': 7.704398292835378, '[osd.2,num_pgs]': 45, '[osd.2,osd_latency_apply]': 0.0, '[osd.2,osd_latency_commit]': 0.0, '[osd.0,osd_fill]': 6.931182672962731, '[osd.0,num_pgs]': 39, '[osd.0,osd_latency_apply]': 4.0, '[osd.0,osd_latency_commit]': 4.0, '[osd.1,osd_fill]': 8.344012776877099, '[osd.1,num_pgs]': 47, '[osd.1,osd_latency_apply]': 3.0, '[osd.1,osd_latency_commit]': 3.0, 'osd_max_fill': 8.344012776877099, 'osd_min_fill': 6.931182672962731, 'osd_avg_fill': 7.659864580891735, 'osd_max_pgs': 47, 'osd_min_pgs': 39, 'osd_avg_pgs': 43.666666666666664, 'osd_latency_apply_max': 4.0, 'osd_latency_apply_min': 0.0, 'osd_latency_apply_avg': 2.3333333333333335, 'osd_latency_commit_max': 4.0, 'osd_latency_commit_min': 0.0, 'osd_latency_commit_avg': 2.3333333333333335, 'num_pg_active': 65, 'num_pg_peering': 0, 'num_pg_clean': 65, 'num_pg_scrubbing': 0, 'num_pg_undersized': 0, 'num_pg_backfilling': 0, 'num_pg_recovering': 0, 'num_pg_degraded': 0, 'num_pg_inconsistent': 0, 'num_pg_remapped': 0, 'num_pg_backfill_toofull': 0, 'num_pg_backfill_wait': 0, 'num_pg_recovery_wait': 0, 'num_pg': 65}
debug 2022-05-11T21:17:05.093+0000 7f379c280700 0 [zabbix DEBUG root] Executing: ['/usr/bin/zabbix_sender', '-z', 'zabbix-proxy', '-p', '31833', '-s', 'ceph-lab-test', '-vv', '-i', '-']
debug 2022-05-11T21:17:05.105+0000 7f37eeb7d700 15 mgr get_health_checks getting health checks for zabbix
debug 2022-05-11T21:17:05.173+0000 7f3817893700 15 mgr notify_all queuing notify to zabbix
debug 2022-05-11T21:17:05.201+0000 7f379c280700 0 [zabbix ERROR root] Failed to send.
File "/usr/share/ceph/mgr/zabbix/module.py", line 336, in send
zabbix.send(identifier, data)
File "/usr/share/ceph/mgr/zabbix/module.py", line 46, in send
RuntimeError: /usr/bin/zabbix_sender exited non-zero: b''
debug 2022-05-11T21:17:05.201+0000 7f379c280700 10 module zabbix health checks:
"message": "/usr/bin/zabbix_sender exited non-zero: b''"
debug 2022-05-11T21:17:05.201+0000 7f379c280700 10 module zabbix health checks:
debug 2022-05-11T21:17:05.201+0000 7f37eeb7d700 15 mgr get_health_checks getting health checks for zabbix
"message": "/usr/bin/zabbix_sender exited non-zero: b''"

Which results in messages on the proxy:

 153:20220511:212105.594 cannot process item "ceph.num_pg_temp" trap: item type "18" cannot be used with traps
153:20220511:212105.594 cannot process item "ceph.num_osd_up" trap: item type "18" cannot be used with traps
153:20220511:212105.594 cannot process item "ceph.num_osd_in" trap: item type "18" cannot be used with traps
153:20220511:212105.594 cannot process item "ceph.num_pg" trap: item type "18" cannot be used with traps
152:20220511:212205.685 cannot process item "ceph.overall_status" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.num_mon" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.num_pools" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.total_used_bytes" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.total_bytes" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.total_avail_bytes" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.num_osd" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.osd_nearfull_ratio" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.osd_full_ratio" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.osd_backfillfull_ratio" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.num_pg_temp" trap: item type "18" cannot be used with traps
152:20220511:212205.686 cannot process item "ceph.num_osd_up" trap: item type "18" cannot be used with traps
152:20220511:212205.687 cannot process item "ceph.num_osd_in" trap: item type "18" cannot be used with traps
152:20220511:212205.687 cannot process item "ceph.num_pg" trap: item type "18" cannot be used with traps

I can manually send the correct data from the sender and it shows the same message so communication is happening just proxy blows it with the data. I also tried not using a proxy and the server responds the same way.

[KowMangler]

Just tried it with the 6.0.3 sender, same result. Is there something else I need to configure somewhere?

[Hamardaban]

What is the type of item that accepts data (ceph.*)? It should be "zabbix trapper".

[KowMangler]

After looking at it with your question in mind I see where I went astray. The Ceph documentation has a built-in zabbix module for ceph manager to push the data via Zabbix Traps .The Zabbix documentation, on the other hand, has documentation that calls for using the Zabbix agent to execute queries (which is where I got the template) and uses Dependent items which is, I guess 'type "18"' . Totally my fault, but I wish the error messages were better. I see the template I should have used and will use that. Thanks for the help!

In the end it was my fault for making the assumption the approaches documented on each were collaborative instead of separate efforts with different angles of approach

[Hamardaban]

I use this template

https://docs.ceph.com/en/latest/mgr/zabbix/

A template. (XML) to be used on the Zabbix server can be found in the source directory of the module.

This template contains all items and a few triggers. You can customize the triggers afterwards to fit your needs.