Ad Widget

Collapse

Front-end warning "Unable to load database credentials from Vault.", no logged errors

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • schelcc
    Junior Member
    • Jun 2021
    • 3
    #1

    Front-end warning "Unable to load database credentials from Vault.", no logged errors

    Apologies if my formatting is off, this is my first post here.

    I have been working on installing & implementing Zabbix on a server, using Hashicorp Vault to protect the DB credentials. After getting it installed (or so I thought), I proceeded to start enabling the actual server. The server is enabled and running through systemd, and tail-ing the server log file shows a number of lines like "server #x started", no errors in sight. When I go to the UI through the browser however, I get the error in the title and can't proceed.

    When I perform these two tests (as described here https://www.zabbix.com/documentation...config/secrets),

    Code:
    # Test that secret is successfully added
$ vault kv get secret/zabbix/database

# Finally test with Curl, note that "data" need to be manually added after mount point and "/v1" before the mount point, also see --capath parameter
$ curl --header "X-Vault-Token: <Token specifically generated for zabbix>" https://127.0.0.1:8200/v1/secret/data/zabbix/database
    They both output the values as expected, working fine.

    After checking the token's capabilities, it has 'read' on both secret/database/zabbix and secret/data/zabbix/database

    If it helps, here is the policy:

    Code:
    path "secret/zabbix/database" {
    capabilities = ["read"]
}

path "secret/data/*" {
    capabilities = ["read"]
}
    ----
    Here's the vault config options (with certain values cleaned)

    Code:
    ### Option: VaultToken
# Vault authentication token that should have been generated exclusively for Zabbix server with read only permission
# to paths specified in Vault macros and read only permission to path specified in optional VaultDBPath
# configuration parameter.
# It is an error if VaultToken and VAULT_TOKEN environment variable are defined at the same time.
#
# Mandatory: no
# Default:
VaultToken=<previously mentioned token>

### Option: VaultURL
# Vault server HTTP[S] URL. System-wide CA certificates directory will be used if SSLCALocation is not specified.
#
# Mandatory: no
# Default:
VaultURL=http://127.0.0.1:8200

### Option: VaultDBPath
# Vault path from where credentials for database will be retrieved by keys password and username.
# Example: secret/zabbix/database
# This option can only be used if DBUser and DBPassword are not specified.
#
# Mandatory: no
# Default:
VaultDBPath=secret/zabbix/database
    Here's the server log:

    Code:
    131310:20210629:185915.036 Starting Zabbix Server. Zabbix 5.4.1 (revision ecb5b3378c).
131310:20210629:185915.036 ****** Enabled features ******
131310:20210629:185915.036 SNMP monitoring: YES
131310:20210629:185915.037 IPMI monitoring: YES
131310:20210629:185915.037 Web monitoring: YES
131310:20210629:185915.037 VMware monitoring: YES
131310:20210629:185915.037 SMTP authentication: YES
131310:20210629:185915.037 ODBC: YES
131310:20210629:185915.037 SSH support: YES
131310:20210629:185915.037 IPv6 support: YES
131310:20210629:185915.037 TLS support: YES
131310:20210629:185915.037 ******************************
131310:20210629:185915.037 using configuration file: /etc/zabbix/zabbix_server.conf
131310:20210629:185915.045 current database version (mandatory/optional): 05040000/05040000
131310:20210629:185915.046 required mandatory version: 05040000
131310:20210629:185915.052 server #0 started [main process]
131312:20210629:185915.052 server #1 started [configuration syncer #1]
131313:20210629:185915.153 server #2 started [housekeeper #1]
131314:20210629:185915.154 server #3 started [timer #1]
131315:20210629:185915.154 server #4 started [http poller #1]
131319:20210629:185915.156 server #8 started [history syncer #3]
131321:20210629:185915.157 server #10 started [escalator #1]
131322:20210629:185915.158 server #11 started [proxy poller #1]
131325:20210629:185915.159 server #14 started [poller #1]
131335:20210629:185915.169 server #24 started [trapper #5]
131328:20210629:185915.170 server #17 started [poller #4]
131337:20210629:185915.180 server #26 started [alert manager #1]
131329:20210629:185915.181 server #18 started [poller #5]
131317:20210629:185915.181 server #6 started [history syncer #1]
131316:20210629:185915.181 server #5 started [discoverer #1]
131330:20210629:185915.192 server #19 started [unreachable poller #1]
131323:20210629:185915.196 server #12 started [self-monitoring #1]
131324:20210629:185915.197 server #13 started [task manager #1]
131318:20210629:185915.197 server #7 started [history syncer #2]
131320:20210629:185915.197 server #9 started [history syncer #4]
131326:20210629:185915.198 server #15 started [poller #2]
131334:20210629:185915.199 server #23 started [trapper #4]
131338:20210629:185915.201 server #27 started [alerter #1]
131340:20210629:185915.201 server #29 started [alerter #3]
131345:20210629:185915.204 server #34 started [lld manager #1]
131349:20210629:185915.204 server #38 started [history poller #1]
131336:20210629:185915.205 server #25 started [icmp pinger #1]
131332:20210629:185915.206 server #21 started [trapper #2]
131327:20210629:185915.209 server #16 started [poller #3]
131347:20210629:185915.216 server #36 started [lld worker #2]
131351:20210629:185915.216 server #40 started [history poller #3]
131353:20210629:185915.218 server #42 started [history poller #5]
131333:20210629:185915.225 server #22 started [trapper #3]
131331:20210629:185915.235 server #20 started [trapper #1]
131339:20210629:185915.247 server #28 started [alerter #2]
131341:20210629:185915.247 server #30 started [preprocessing manager #1]
131343:20210629:185915.248 server #32 started [preprocessing worker #2]
131344:20210629:185915.248 server #33 started [preprocessing worker #3]
131346:20210629:185915.248 server #35 started [lld worker #1]
131348:20210629:185915.249 server #37 started [alert syncer #1]
131350:20210629:185915.249 server #39 started [history poller #2]
131352:20210629:185915.250 server #41 started [history poller #4]
131354:20210629:185915.251 server #43 started [availability manager #1]
131342:20210629:185915.306 server #31 started [preprocessing worker #1]

    I'm stumped as to what could be causing Zabbix to not be able to retrieve the DB credentials. If any more info is needed, please ask.

    Thanks!
    Tags: database, server, vault, zabbix
    • schelcc
      Junior Member
      • Jun 2021
      • 3
      #2
      After further inspection via TShark & Wireshark I've found it's using an old root token set for a previous vault which was input during the installation via the frontend UI. Is there a way to change this? It's not using the "VaultToken" value set in the config.

        Comment

        • schelcc
          Junior Member
          • Jun 2021
          • 3
          #3
          Finally got it, the token was set in the zabbix.conf.php . Unless I missed something, this was never mentioned in anything I could find. It would be very helpful if something about that file was mentioned somewhere.

            Comment

            • Miks.Kronkalns
              Junior Member
              • Mar 2017
              • 2
              #4
              schelcc, it's good to hear that you have solved the problem. After inspecting the documentation I have to agree that this is not described good enough. The link you have posted is the only reference to VAULT UI configuration and it talks only about consequences of using VAULT to store database passwords for UI. I will ask documentation team to describe all VAULT related configuration parameters.

              Here is the short description of what's missing:
              - VAULT - Vault service provider. Zabbix supports HashiCorp (since Zabbix 5.2) and CyberArk (since Zabbix 6.2) Vault services.
              - VAULT_URL - Vault API endpoint.
              - VAULT_DB_PATH - The path to Vault secret where database credentions are stored. Zabbix frontend expects that Vault secret contains object with keys "UserName" and "Content" for username and password.
              - VAULT_TOKEN - Vault authentication token for HashiCorp Vault.
              - VAULT_CERT_FILE and VAULT_KEY_FILE - SSL certificate file and SSL key file paths for CyberArk Vault.
              - VAULT_CACHE - switch to enable database credential caching in filesystem.

                Comment

                Previous template Next
                Working...