404 Error from Zabbix server when trying to access DB credentials in Vault

[OsoPanda2072]

Hi

I setup a Zabbix in a all in one host (Web+Server+DB+Vault) following documentation from Zabbix and Vault
Vault is v1.11.4
Zabbix Server is V6.0.9

It uses cert/key signed with my own created CA cert. The system is not Internet facing and is to monitor a small number of services/infrastructure in the same network segment which lead to accept to have our own CA for Zavbbix server and pointing to vault via localhost (in the short term).

Zabbix is reporting the following error(I increased the verbosity of the logging on both Zabbix and Vault for troubleshooting)

zabbix_server.log:

3750:20221006:021800.449 using configuration file: /etc/zabbix/zabbix_server.conf
3750:20221006:021800.449 In zbx_load_modules()
3750:20221006:021800.449 End of zbx_load_modules():SUCCEED
3750:20221006:021800.449 In zbx_ipc_service_start() service:rtc
3750:20221006:021800.449 In zbx_ipc_socket_open()
3750:20221006:021800.449 End of zbx_ipc_socket_open():FAIL
3750:20221006:021800.449 End of zbx_ipc_service_start():SUCCEED
3750:20221006:021800.449 In zbx_http_get() URL 'https://localhost:8200/v1/secret/data/data/zabbix/database'
3750:20221006:021800.476 End of zbx_http_get():SUCCEED
3750:20221006:021800.477 cannot initialize database credentials from vault: unsuccessful response code "404"

Testing with the CLI and using Curl work as expected (return the username/password values)

Using CLI:

vault kv get secret/zabbix/database

======= Secret Path =======
secret/data/zabbix/database

<REMOVED>

====== Data ======
Key Value
--- -----
password <DBpassword>
username <DBusername>

Using curl:

curl --header "X-Vault-Token: <TOKEN>" https://localhost:8200/v1/secret/data/zabbix/database

{"request_id":"4de74fc6-5cf2-ca6a-caba-6c859f22c5be","lease_id":"","renewable":false,"lea se_duration":0,"data":{"data":{"password":"<DBpass word>,"username":"<DBusername>"},"metadata":{"crea ted_time":"2022-10-03T15:12:35.983165056Z","custom_metadata":null,"de letion_time":"","destroyed":false,"version":1}},"w rap_info":null,"warnings":null,"auth":null}

What I'm missing? the CLI and curl works ok, the log shows that is trying to acces the same url as the one with curl and double check the values on zaabix_server.conf to be the ones required

Relevant configuration files and values:

zabbix_server.conf:
VaultURL=https://localhost:8200
VaultToken=<TOKEN>
VaultDBPath=secret/data/zabbix/database

vault.hcl:
listener "tcp" {
address = "localhost:8200"
tls_disable = "false"
tls_cert_file = "/opt/vault/tls/tls.crt"
tls_key_file = "/opt/vault/tls/tls.key"
}
api_addr = "https://localhost:8200"
cluster_addr = "https://localhost:8201"
ui = false
log_level = "Debug"

Other information that might be relevant:
The engine in use is ver 2 for the mount point 'secret' as confirmed by vault

secret/ kv <REMOVED> map[version:2] <REMOVED>

Tried with FW disabled and same result and zabbix Web interface reports the same error of failing to obtain DB credentials
Any assistance will be much appreciated.

[Markku]

What's the deal with double data in the URL in the log?

3750:20221006:021800.449 In zbx_http_get() URL 'https://localhost:8200/v1/secret/data/data/zabbix/database'

Markku

[OsoPanda2072]

Thanks everybody for your responses! much appreciated
Markku I guess i missed that extra 'data' bit in the URL, guess I was already tired

Removed the 'data' on the URL on the zabbix conf files (server and front end) and rebooted and now it is working

It is interesting when using kv v2 that for the front end or the srvr you don't need to specify the mount point+'data' and can be done only with mount point + path but if you are doing a direct call using curlor setting up policies you need to use the mount point + 'data' + path

Cheers