Ad Widget

Collapse

zabbix agent "zbx_tls_accept(): gnutls_handshake() failed:-31 Error in password file"

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • FrivolousDingo
    Junior Member
    • Dec 2018
    • 4

    #1

    zabbix agent "zbx_tls_accept(): gnutls_handshake() failed:-31 Error in password file"

    Hello,

    I'm running a Zabbix-Server 3.0.7 (1:3.0.7+dfsg-3) on Debian 9.
    I have this monitoring a number of remote VPS (and local hosts) all running Zabbix-Agent 3.0.7 (3.0.7+dfsg-3) also all on Debian 9.

    The agents are configured passively. I'm using TLS with PSK for each of the above for connections from the Zabbix Server. I have generated keys as follows:

    Code:
    # openssl rand -hex 32 > /etc/zabbix/zabbix_agent_psk
    # chmod 0400 /etc/zabbix/zabbix_agent_psk
    # chown zabbix:zabbix /etc/zabbix/zabbix_agent_psk
    This all works pretty much as you'd expect it too - 100% success rate. Until yesterday anyway...

    I installed zabbix-agent and zabbix-proxy-mysql (v3.0.7+dfsg-3 with prerequisites) on a Debian 9 box. I configured both the proxy and agent as passive, gave them unique PSK Identities and generated new PSK files as above.

    When the zabbix-server attempts to connect to the zabbix-agent running on the box, I see:

    Code:
    20194:20181213:150329.754 zbx_tls_accept() gnutls_handshake() returned: -31 Error in password file.
     20194:20181213:150329.755 failed to accept an incoming connection: from <SERVER_PUBLIC_IP>: zbx_tls_accept(): gnutls_handshake() failed: -31 Error in password file.
    On the zabbix-server:
    Code:
    4374:20181213:112609.840 Get value error: TCP successful, cannot establish TLS to [[zabbix_agent_host.somedomain.sometld]:10050]: zbx_tls_connect(): gnutls_handshake() failed: -110 The TLS connection was non-properly terminated.
    On seeing the client error message, my first thought is that I've chosen a duff PSK, PSK ID or not given the file the correct permissions...

    Code:
    root@pashley:/etc/zabbix# ls -altr
    total 52
    drwxr-xr-x  2 root   root    4096 Jun  4  2017 zabbix_proxy.conf.d
    drwxr-xr-x  2 root   root    4096 Dec 11 21:26 zabbix_agentd.conf.d
    -rw-r--r--  1 root   root       0 Dec 11 21:36 proxySchema.done
    drwxr-xr-x 95 root   root    4096 Dec 12 23:08 ..
    -r--r-----  1 zabbix zabbix    66 Dec 12 23:59 zabbix_proxy_psk
    -rw-r--r--  1 root   root   16012 Dec 13 11:08 zabbix_proxy.conf
    -r--------  1 zabbix zabbix    65 Dec 13 12:39 zabbix_agent_psk
    -rw-r--r--  1 root   root   10426 Dec 13 13:01 zabbix_agentd.conf
    drwxr-xr-x  4 root   root    4096 Dec 13 13:01 .
    Code:
    root@pashley:/etc/zabbix# cat zabbix_agent_psk
    6bebf5ca66a881dc2792b5d577c9fc6a4d80cc6d2496970a282400973400143a
    Code:
    root@pashley:/etc/zabbix# cat zabbix_proxy_psk
    7c22b25925cabaffeb24d5580714eaeb88f1d66db422e216d8e43506e36a4ca9
    I've tried different length keys (ranging from 32-bit to 2048-bit), I've tried copying keys from other boxes which I know work, I've tried adding and removing carriage returns (new line) from the end of the PSK file.

    Also tested with zabbix_get:

    Code:
    sudo zabbix_get -s zabbix_agent_host.somedomain.sometld -p 10050 -k "system.cpu.load[all,avg1]" --tls-connect=psk --tls-psk-identity="PSK_010" --tls-psk-file="/home/user/zabbixAgentTestPsk"
    zabbix_get [25159]: Get value error: TCP successful, cannot establish TLS to [[zabbix_agent_host.somedomain.sometld]:10050]: zbx_tls_connect(): gnutls_handshake() failed: -110 The TLS connection was non-properly terminated.
    I've also checked that the same version of libgnutls is installed on all of the debian hosts
    Code:
    root@pashley:/etc/zabbix# dpkg -l | grep gnutls
    ii  libcurl3-gnutls:amd64         7.52.1-5+deb9u8                amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
    ii  libgnutls30:amd64             3.5.8-5+deb9u4                 amd64        GNU TLS library - main runtime library
    I have the config, detailed logs (DebugLevel = 5) and packet captures, which, I will add them to the thread shortly.

    This is incredibly frustrating - hopefully I've missed something blindingly obvious and one of you kind people can point it out to me
  • FrivolousDingo
    Junior Member
    • Dec 2018
    • 4

    #2
    Agent configuration:

    Code:
    PidFile=/var/run/zabbix/zabbix_agentd.pid
    LogFile=/var/log/zabbix-agent/zabbix_agentd.log
    LogFileSize=0
    Server=zabbixserver.somedomain.sometld,<zabbixServerPublicIp>
    Hostname=brox
    Include=/etc/zabbix/zabbix_agentd.conf.d/*.conf
    UnsafeUserParameters=1
    TLSAccept=psk
    TLSPSKIdentity="PSK 010"
    TLSPSKFile=/etc/zabbix/zabbix_agent_psk
    Proxy configuration:

    Code:
    ProxyMode=1
    Server=127.0.0.1
    Hostname=pashley
    LogFile=/var/log/zabbix-proxy/zabbix_proxy.log
    PidFile=/var/run/zabbix/zabbix_proxy.pid
    DBName=zabbixProxy
    DBUser=zabbixProxy
    DBPassword=<DbPassword>
    DBSocket=/var/run/mysqld/mysqld.sock
    ProxyOfflineBuffer=24
    StartDiscoverers=2
    Timeout=10
    FpingLocation=/usr/bin/fping
    Fping6Location=/usr/bin/fping6
    LogSlowQueries=3000
    Include=/etc/zabbix/zabbix_proxy.conf.d/*.conf
    TLSAccept=psk
    TLSPSKIdentity="PSK PROXY 001"
    TLSPSKFile=/etc/zabbix/zabbix_proxy_psk
    Remember that the agent and proxy are on the same host and the zabbix server is having the same issue communicating to both of them.

    Configuration for the zabbix-server:

    Code:
    LogFile=/var/log/zabbix-server/zabbix_server.log
    PidFile=/var/run/zabbix/zabbix_server.pid
    DBHost=localhost
    DBName=zabbix
    DBUser=zabbix
    DBPassword=<DbPassword>
    DBSocket=/var/run/mysqld/mysqld.sock
    StartDiscoverers=2
    Timeout=4
    AlertScriptsPath=/etc/zabbix/alert.d/
    FpingLocation=/usr/bin/fping
    LogSlowQueries=3000
    Include=/etc/zabbix/zabbix_server.conf.d/*.conf

    Comment

    • FrivolousDingo
      Junior Member
      • Dec 2018
      • 4

      #3
      Except from zabbix-agent log

      Code:
       18567:20181213:105940.664 __zbx_zbx_setproctitle() title:'listener #3 [waiting for connection]'                                                                                                                                 
       18567:20181213:105940.860 In zbx_tls_accept()                                                                                                                                                                                   
       18567:20181213:105940.860 GnuTLS debug: level=3 "ASSERT: constate.c[_gnutls_epoch_get]:600"                                                                                                                                     
       18567:20181213:105940.860 GnuTLS debug: level=3 "ASSERT: buffers.c[get_last_packet]:1159"                                                                                                                                       
       18567:20181213:105940.860 GnuTLS debug: level=4 "HSK[0x55825fd59740]: CLIENT HELLO (1) was received. Length 108[108], frag offset 0, frag length: 108, sequence: 0"                                                             
       18567:20181213:105940.861 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Client's version: 3.3"                                                                                                                                    
       18567:20181213:105940.861 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Selected version TLS1.2"                                                                                                                                  
       18567:20181213:105940.861 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Parsing extension 'Encrypt-then-MAC/22' (0 bytes)"                                                                                                        
       18567:20181213:105940.861 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Parsing extension 'Safe Renegotiation/65281' (1 bytes)"                                                                                                   
       18567:20181213:105940.861 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported curves/10'"                                                                                                                    
       18567:20181213:105940.862 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported ECC Point Formats/11'"                                                                                                         
       18567:20181213:105940.862 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Signature Algorithms/13'"                                                                                                                
       18567:20181213:105940.862 GnuTLS debug: level=3 "ASSERT: db.c[_gnutls_server_restore_session]:272"                                                                                                                              
       18567:20181213:105940.862 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Encrypt-then-MAC/22'"                                                                                                                    
       18567:20181213:105940.862 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Safe Renegotiation/65281'"                                                                                                               
       18567:20181213:105940.862 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported curves/10'"                                                                                                                    
       18567:20181213:105940.863 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported ECC Point Formats/11'"                                                                                                         
       18567:20181213:105940.863 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Signature Algorithms/13'"                                                                                                                
       18567:20181213:105940.866 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Encrypt-then-MAC/22'"                                                                                                                    
       18567:20181213:105940.867 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Safe Renegotiation/65281'"                                                                                                               
       18567:20181213:105940.867 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Parsing extension 'Supported curves/10' (12 bytes)"                                                                                                       
       18567:20181213:105940.867 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Selected ECC curve SECP256R1 (2)"                                                                                                                         
       18567:20181213:105940.868 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Parsing extension 'Supported ECC Point Formats/11' (2 bytes)"                                                                                             
       18567:20181213:105940.868 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Parsing extension 'Signature Algorithms/13' (22 bytes)"                                                                                                   
       18567:20181213:105940.868 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (4.1) RSA-SHA256"                                                                                                                     
       18567:20181213:105940.868 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (4.3) ECDSA-SHA256"                                                                                                                   
       18567:20181213:105940.869 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (5.1) RSA-SHA384"                                                                                                                     
       18567:20181213:105940.869 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (5.3) ECDSA-SHA384"                                                                                                                   
       18567:20181213:105940.870 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (6.1) RSA-SHA512"                                                                                                                     
       18567:20181213:105940.871 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (6.3) ECDSA-SHA512"                                                                                                                   
       18567:20181213:105940.871 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (3.1) RSA-SHA224"                                                                                                                     
       18567:20181213:105940.871 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (3.3) ECDSA-SHA224"                                                                                                                   
       18567:20181213:105940.872 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (2.1) RSA-SHA1"                                                                                                                       
       18567:20181213:105940.872 GnuTLS debug: level=4 "EXT[0x55825fd59740]: rcvd signature algo (2.3) ECDSA-SHA1"                                                                                                                     
       18567:20181213:105940.872 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Keeping ciphersuite: GNUTLS_ECDHE_PSK_AES_128_CBC_SHA256 (C0.37)"                                                                                         
       18567:20181213:105940.872 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Keeping ciphersuite: GNUTLS_ECDHE_PSK_AES_128_CBC_SHA1 (C0.35)"                                                                                           
       18567:20181213:105940.872 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Keeping ciphersuite: GNUTLS_PSK_AES_128_GCM_SHA256 (00.A8)"                                                                                               
       18567:20181213:105940.872 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Keeping ciphersuite: GNUTLS_PSK_AES_128_CBC_SHA256 (00.AE)"
       18567:20181213:105940.872 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Keeping ciphersuite: GNUTLS_PSK_AES_128_CBC_SHA1 (00.8C)"                                                                                                 
       18567:20181213:105940.874 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Requested cipher suites[size: 10]: "                                                                                                                      
       18567:20181213:105940.874 GnuTLS debug: level=4 "      0xc0, 0x37 ECDHE_PSK_AES_128_CBC_SHA256"                                                                                                                                 
       18567:20181213:105940.874 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Selected cipher suite: ECDHE_PSK_AES_128_CBC_SHA256"                                                                                                      
       18567:20181213:105940.874 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Selected Compression Method: NULL"                                                                                                                        
       18567:20181213:105940.874 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Encrypt-then-MAC/22'"                                                                                                                    
       18567:20181213:105940.874 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Safe Renegotiation/65281'"                                                                                                               
       18567:20181213:105940.874 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported curves/10'"                                                                                                                    
       18567:20181213:105940.874 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Supported ECC Point Formats/11'"                                                                                                         
       18567:20181213:105940.875 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Found extension 'Signature Algorithms/13'"                                                                                                                
       18567:20181213:105940.875 GnuTLS debug: level=4 "HSK[0x55825fd59740]: Safe renegotiation succeeded"                                                                                                                             
       18567:20181213:105940.876 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Sending extension Encrypt-then-MAC (0 bytes)"                                                                                                             
       18567:20181213:105940.876 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Sending extension Safe Renegotiation (1 bytes)"                                                                                                           
       18567:20181213:105940.876 GnuTLS debug: level=4 "EXT[0x55825fd59740]: Sending extension Supported ECC Point Formats (2 bytes)"                                                                                                  
       18567:20181213:105940.876 GnuTLS debug: level=4 "HSK[0x55825fd59740]: SessionID: a302c810a90bdf441f9e9bfcad6a41cf4eb0e9a14468bab95426188a1d949ba2"                                                                              
       18567:20181213:105940.876 GnuTLS debug: level=4 "HSK[0x55825fd59740]: SERVER HELLO was queued [91 bytes]"                                                                                                                       
       18567:20181213:105940.877 GnuTLS debug: level=4 "HSK[0x55825fd59740]: SERVER KEY EXCHANGE was queued [75 bytes]"                                                                                                                
       18567:20181213:105940.878 GnuTLS debug: level=4 "HSK[0x55825fd59740]: SERVER HELLO DONE was queued [4 bytes]"                                                                                                                   
       18567:20181213:105940.878 GnuTLS debug: level=3 "ASSERT: buffers.c[get_last_packet]:1159"                                                                                                                                       
       18567:20181213:105940.908 GnuTLS debug: level=4 "HSK[0x55825fd59740]: CLIENT KEY EXCHANGE (16) was received. Length 75[75], frag offset 0, frag length: 75, sequence: 0"                                                        
       18567:20181213:105940.909 zbx_psk_cb() requested PSK identity "PSK 010"                                                                                                                                                         
       18567:20181213:105940.909 GnuTLS debug: level=3 "ASSERT: psk_passwd.c[_gnutls_psk_pwd_find_entry]:143"                                                                                                                          
       18567:20181213:105940.909 GnuTLS debug: level=3 "ASSERT: dhe_psk.c[proc_ecdhe_psk_client_kx]:423"                                                                                                                               
       18567:20181213:105940.909 GnuTLS debug: level=3 "ASSERT: handshake.c[handshake_server]:3308"                                                                                                                                    
       18567:20181213:105940.909 zbx_tls_accept() gnutls_handshake() returned: -31 Error in password file.                                                                                                                             
       18567:20181213:105940.909 End of zbx_tls_accept():FAIL error:'zbx_tls_accept(): gnutls_handshake() failed: -31 Error in password file.'                                                                                         
       18567:20181213:105940.909 failed to accept an incoming connection: from <ZabbixServerPublicIp>: zbx_tls_accept(): gnutls_handshake() failed: -31 Error in password file.                                                                 
       18568:20181213:105941.313 End of send_buffer():SUCCEED

      Comment

      • FrivolousDingo
        Junior Member
        • Dec 2018
        • 4

        #4
        Originally posted by FrivolousDingo
        Agent configuration:

        Code:
        TLSPSKIdentity="PSK 010"
        Proxy configuration:

        Code:
        TLSPSKIdentity="PSK PROXY 001"
        Pleased to report this is now solved. The issue was indeed a schoolboy error. TLSPSKIdentity value should never be quoted, as per above. A correct example would be:

        Code:
        TLSPSKIdentity=PSK 010

        Comment

        Working...