Ad Widget

Collapse

​Front-end warning "Unable to load database credentials from Vault."

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Yoji4ek
    Junior Member
    • May 2023
    • 4

    #1

    ​Front-end warning "Unable to load database credentials from Vault."

    ​Front-end warning "Unable to load database credentials from Vault."

    Hello, everyone!
    My request is look like the topic https://www.zabbix.com/forum/zabbix-...-logged-errors
    But configuring zabbix.conf.php didn't help.

    Manual requests from zabbix server to the vault server by vault utility or curl are successful. I receive all the data from the vault.

    Code:
    [root@srv1]# vault kv get devsecops/zabbix/srv1/database
    ================ Secret Path ================
    devsecops/data/zabbix/srv1/database
    
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2023-04-14T12:44:15.66262118Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
    
    ====== Data ======
    Key         Value
    ---         -----
    password    qwerty
    username    zabbix
    Code:
    [root@srv1]# curl --header "X-Vault-Token:  hvs.CAESIH32cjv0p5TdS6sx8QYOO5jzGP25qQWxqsIKfaUbxqnzGh4KHGh2cy5VTmZJVktqblVaOFRHWW93TUNMRFpaWEg" https://vaultserver:8200/v1/devsecops/data/zabbix/srv1/database
    {"request_id":"225f783c-e5ec-3d4a-0c0e-573ea2af3847","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"password":"qwerty","username":"zabbix"},"metadata":{"created_time":"2023-04-14T12:44:15.66262118Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}
    But configuring zabbix.conf.php didn't help

    Code:
    // Vault configuration. Used if database credentials are stored in Vault secrets manager.
    // Vault configuration. Used if database credentials are stored in Vault secrets manager.
    $DB['VAULT']                    = 'HashiCorp';
    $DB['VAULT_URL']                = 'https://vaultserver:8200';
    $DB['VAULT_DB_PATH']            = 'devsecops/zabbix/srv1/database';
    $DB['VAULT_TOKEN']              = 'hvs.CAESIH32cjv0p5TdS6sx8QYOO5jzGP25qQWxqsIKfaUbxqnzGh4KHGh2cy5VTmZJVktqblVaOFRHWW93TUNMRFpaWEg';
    $DB['VAULT_CERT_FILE']          = '';
    $DB['VAULT_KEY_FILE']           = '';
    // Uncomment to bypass local caching of credentials.
    // $DB['VAULT_CACHE']           = true;​
    On the frontend database error appears
    Click image for larger version

Name:	vault_error.png
Views:	798
Size:	7.1 KB
ID:	465140

    Configuring vault settings in the zabbix_server.conf will fail to start zabbix-server.service

    Code:
    [root@srv1 zabbix]# systemctl restart zabbix-server.service
    Job for zabbix-server.service failed because the service did not take the steps required by its unit configuration.
    See "systemctl status zabbix-server.service" and "journalctl -xeu zabbix-server.service" for details.
    [root@ldcvis-monsrv zabbix]# journalctl -xeu zabbix-server.service
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A start job for unit zabbix-server.service has finished with a failure.
    ░░
    ░░ The job identifier is 186308 and the job result is failed.
    May 30 17:11:13 srv1 systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 1.
    ░░ Subject: Automatic restarting of a unit has been scheduled
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ Automatic restarting of the unit zabbix-server.service has been scheduled, as the result for
    ░░ the configured Restart= setting for the unit.
    May 30 17:11:13 srv1 systemd[1]: Stopped Zabbix Server.
    ░░ Subject: A stop job for unit zabbix-server.service has finished
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A stop job for unit zabbix-server.service has finished.
    ░░
    ░░ The job identifier is 186401 and the job result is done.
    May 30 17:11:13 srv1 systemd[1]: Starting Zabbix Server...
    ░░ Subject: A start job for unit zabbix-server.service has begun execution
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A start job for unit zabbix-server.service has begun execution.
    ░░
    ░░ The job identifier is 186401.
    ...skipping...
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A start job for unit zabbix-server.service has finished with a failure.
    ░░
    ░░ The job identifier is 186308 and the job result is failed.
    May 30 17:11:13 srv1 systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 1.
    ░░ Subject: Automatic restarting of a unit has been scheduled
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ Automatic restarting of the unit zabbix-server.service has been scheduled, as the result for
    ░░ the configured Restart= setting for the unit.
    May 30 17:11:13 srv1 systemd[1]: Stopped Zabbix Server.
    ░░ Subject: A stop job for unit zabbix-server.service has finished
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A stop job for unit zabbix-server.service has finished.
    ░░
    ░░ The job identifier is 186401 and the job result is done.
    May 30 17:11:13 srv1 systemd[1]: Starting Zabbix Server...
    ░░ Subject: A start job for unit zabbix-server.service has begun execution
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A start job for unit zabbix-server.service has begun execution.
    ░░
    ░░ The job identifier is 186401.
    May 30 17:11:15 srv1 systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: Operation not permitted
    May 30 17:11:15 srv1 systemd[1]: zabbix-server.service: Failed with result 'protocol'.
    ░░ Subject: Unit failed
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ The unit zabbix-server.service has entered the 'failed' state with result 'protocol'.
    May 30 17:11:15 srv1 systemd[1]: Failed to start Zabbix Server.
    ░░ Subject: A start job for unit zabbix-server.service has failed
    ░░ Defined-By: systemd
    ░░ Support: https://access.redhat.com/support
    ░░
    ░░ A start job for unit zabbix-server.service has finished with a failure.
    ░░
    ░░ The job identifier is 186401 and the job result is failed.
    Tab stops: eu zabbix-server.service


    I have this situation on different versions of zabbix server (6.4.2 and 6.0.12)

    I checked network and noted that there no any traffic from zabbix server to the vault server while loading the frontend page, so my suggestion that there are no requests to the vault by zabbix server.
    But I don't have any assumptions what to do, to check or to configure.
    I will appreciate any advice or help.

    Thanks!​​
  • cyber
    Senior Member
    Zabbix Certified SpecialistZabbix Certified Professional
    • Dec 2006
    • 4811

    #2
    Never dealt with vault actually, but some general checks....
    As you tested everything as root, can do the same as user zabbix?
    Check for selinux errors...

    That pid file error... I saw someone asking for similar thing lately, but cannot remember the issue.. Is /run/zabbix present and writeable?

    Feels like its not your Vault issue, but issue with server startup ....

    Comment

    • Yoji4ek
      Junior Member
      • May 2023
      • 4

      #3
      Thank you for response.

      However, the problems begin only when I try to configure Vault settings.
      If I comment vault settings in the config files and use cleartext password for database (default configuration), zabbix starts and works fine.

      There are 2 config files:
      /etc/zabbix/zabbix_server.conf - contains database name, username, password which zabbix-server uses for DB connection
      /etc/zabbix/web/zabbix.conf.php - contains database name, username, password which zabbix-WEB-server uses for DB connection

      For example If I configure only zabbix.conf.php for Vault usage, zabbix-server starts and works, but WEB interface shows the error "Unable to load database credentials from Vault"

      So, I assume the reason is that Zabbix doesn't retrieve password from Vault. And according to network analysis Zabbix didn't even try to do it or try to do it in wrong way (locally or smth else).

      Comment

      • cyber
        Senior Member
        Zabbix Certified SpecialistZabbix Certified Professional
        • Dec 2006
        • 4811

        #4
        Is there anything in zabbix server logfile ? Usually /var/log/zabbix/zabbix_server.log...

        Comment

        • Yoji4ek
          Junior Member
          • May 2023
          • 4

          #5
          I have the success with config file /etc/zabbix/zabbix_server.conf
          Debug level of /var/log/zabbix/zabbix_server.log has been changed and I found there the reason:
          Code:
          2736614:20230531:172900.424 Zabbix Server stopped. Zabbix 6.4.3 (revision a6c7751b021).
          2736792:20230531:172900.455 Starting Zabbix Server. Zabbix 6.4.3 (revision a6c7751b021).
          2736792:20230531:172900.455 ****** Enabled features ******
          2736792:20230531:172900.455 SNMP monitoring: YES
          2736792:20230531:172900.455 IPMI monitoring: YES
          2736792:20230531:172900.456 Web monitoring: YES
          2736792:20230531:172900.456 VMware monitoring: YES
          2736792:20230531:172900.456 SMTP authentication: YES
          2736792:20230531:172900.456 ODBC: YES
          2736792:20230531:172900.457 SSH support: YES
          2736792:20230531:172900.457 IPv6 support: YES
          2736792:20230531:172900.457 TLS support: YES
          2736792:20230531:172900.458 ******************************
          2736792:20230531:172900.458 using configuration file: /etc/zabbix/zabbix_server.conf
          2736792:20230531:172900.458 cannot initialize database credentials from vault: "DBUser" configuration parameter cannot be used when "VaultDBPath" is defined
          2736807:20230531:172912.622 Starting Zabbix Server. Zabbix 6.4.3 (revision a6c7751b021).
          2736807:20230531:172912.623 ****** Enabled features ******
          2736807:20230531:172912.623 SNMP monitoring: YES
          ...
          So, I commented DBUser and DBPassword parameters in the /etc/zabbix/zabbix_server.conf config file and Zabbix-server has started:
          Code:
          2737108:20230531:173501.871 Starting Zabbix Server. Zabbix 6.4.3 (revision a6c7751b021).
          2737108:20230531:173501.872 ****** Enabled features ******
          2737108:20230531:173501.873 SNMP monitoring: YES
          2737108:20230531:173501.873 IPMI monitoring: YES
          2737108:20230531:173501.874 Web monitoring: YES
          2737108:20230531:173501.874 VMware monitoring: YES
          2737108:20230531:173501.874 SMTP authentication: YES
          2737108:20230531:173501.875 ODBC: YES
          2737108:20230531:173501.875 SSH support: YES
          2737108:20230531:173501.875 IPv6 support: YES
          2737108:20230531:173501.875 TLS support: YES
          2737108:20230531:173501.876 ******************************
          2737108:20230531:173501.876 using configuration file: /etc/zabbix/zabbix_server.conf
          2737108:20230531:173502.085 current database version (mandatory/optional): 06040000/06040000
          2737108:20230531:173502.085 required mandatory version: 06040000
          2737120:20230531:173502.212 starting HA manager
          2737120:20230531:173502.244 HA manager started in active mode
          2737108:20230531:173502.246 server #0 started [main process]
          2737122:20230531:173502.247 server [NODE="1"]Home[/NODE] started [service manager #1]
          2737123:20230531:173502.248 server [NODE="2"]Forum[/NODE] started [configuration syncer #1]
          2737128:20230531:173502.509 server #3 started [alert manager #1]
          2737133:20230531:173502.513 server [NODE="8"]Private Messages[/NODE] started [lld manager #1]
          2737134:20230531:173502.515 server [NODE="9"]Albums[/NODE] started [lld worker #1]
          2737137:20230531:173502.517 server [NODE="12"]CSS Examples[/NODE] started [timer #1]
          2737135:20230531:173502.519 server #10 started [lld worker #2]
          2737136:20230531:173502.527 server #11 started [housekeeper #1]
          2737139:20230531:173502.537 server #13 started [http poller #1]
          2737143:20230531:173502.544 server [NODE="16"]Zabbix Discussions and Feedback[/NODE] started [history syncer #2]
          2737142:20230531:173502.548 server [NODE="15"]General[/NODE] started [history syncer #1]
          2737132:20230531:173502.574 server #7 started [preprocessing manager #1]
          2737131:20230531:173502.583 server [NODE="6"]Special[/NODE] started [alerter #3]
          2737140:20230531:173502.586 server [NODE="14"]Zabbix Announcements[/NODE] started [discoverer #1]
          2737130:20230531:173502.589 server #5 started [alerter #2]
          2737151:20230531:173502.595 server [NODE="18"]Zabbix in Your Language[/NODE] started [history syncer #4]
          ...
          Also I saw the traffic from the zabbix server to the vault server while zabbix-server was starting.

          However the problem with /etc/zabbix/web/zabbix.conf.php remains
          No traffic to the vault and error "Unable to load database credentials from Vault" on the web ui is still there.
          I tried to comment parameters in the /etc/zabbix/web/zabbix.conf.php config file:
          Code:
          //$DB['USER'] = 'zabbix';
          //$DB['PASSWORD'] = 'qwerty';

          but had no success

          And there are not any usefull information in the zabbix_server.log.
          Maybe some another log file (http-server) should be investigated.​

          Comment

          • cyber
            Senior Member
            Zabbix Certified SpecialistZabbix Certified Professional
            • Dec 2006
            • 4811

            #6
            webserver probably has its own logs .. It does not write into server logs..

            Comment

            Working...