Zabbix server cannot connect to remote PostgreSQL database

[cgill-wh]

Greetings all!

I am going through the steps to compile/install the Zabbix server from the 2.4 documentation. I've already configured the installation with the correct options to support PostgreSQL:

./configure --enable-server --enable-agent --with-postgresql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2 --with-ldap --with-openipmi --with-ssh2 --with-jabber --with-unixodbc

Our DBA has also already configured a database for our Zabbix server to connect to, and I've verified it from the Zabbix server with psql:

zabbix@zabbix:/usr/local/etc$ psql -h <dbhost> -p <dbport> -U zabbix zabbix
password for user zabbix:
psql (9.4.5)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

zabbix=>\l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 zabbix    | zabbix   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
(4 rows)

So clearly psql client can connect with the zabbix db username/pw.

The required parts of the zabbix_server.conf are set up as such:

DBHost=<dbipaddress>
DBName=zabbix
DBSchema=public
DBUser=zabbix
DBPassword=<zabbixpassword>

As soon as I fire up zabbix_server from /usr/local/sbin/, in the log I see the following error messages:

83651:20160107:132034.628 Starting Zabbix Server. Zabbix 2.4.7 (revision 56694).
 83651:20160107:132034.628 ****** Enabled features ******
 83651:20160107:132034.629 SNMP monitoring:           YES
 83651:20160107:132034.629 IPMI monitoring:           YES
 83651:20160107:132034.629 WEB monitoring:            YES
 83651:20160107:132034.629 VMware monitoring:         YES
 83651:20160107:132034.629 Jabber notifications:      YES
 83651:20160107:132034.629 Ez Texting notifications:  YES
 83651:20160107:132034.629 ODBC:                      YES
 83651:20160107:132034.629 SSH2 support:              YES
 83651:20160107:132034.629 IPv6 support:              YES
 83651:20160107:132034.629 ******************************
 83651:20160107:132034.629 using configuration file: /usr/local/etc/zabbix_server.conf
 83651:20160107:132034.629 In load_modules()
 83651:20160107:132034.629 End of load_modules():SUCCEED
 83651:20160107:132034.629 In init_database_cache()
 83651:20160107:132034.629 In zbx_mem_required_size() size:0 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
 83651:20160107:132034.629 End of zbx_mem_required_size() size:543
 83651:20160107:132034.629 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8388583
 83651:20160107:132034.629 zbx_shmget() removing existing shm_id:1310728
 83651:20160107:132034.629 valid user addresses: [0x7f50a2ba0170, 0x7f50a339ffd8] total size: 8388200
 83651:20160107:132034.629 End of zbx_mem_create()
 83651:20160107:132034.629 In zbx_mem_required_size() size:0 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
 83651:20160107:132034.629 End of zbx_mem_required_size() size:435
 83651:20160107:132034.629 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777216
 83651:20160107:132034.629 zbx_shmget() removing existing shm_id:1343497
 83651:20160107:132034.629 valid user addresses: [0x7f50a1ba0178, 0x7f50a2b9fff0] total size: 16776824
 83651:20160107:132034.629 End of zbx_mem_create()
 83651:20160107:132034.629 In init_trend_cache()
 83651:20160107:132034.629 In zbx_mem_required_size() size:0 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
 83651:20160107:132034.629 End of zbx_mem_required_size() size:422
 83651:20160107:132034.629 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194304
 83651:20160107:132034.629 zbx_shmget() removing existing shm_id:1376266
 83651:20160107:132034.630 valid user addresses: [0x7f50a17a0170, 0x7f50a1b9fff0] total size: 4193920
 83651:20160107:132034.630 End of zbx_mem_create()
 83651:20160107:132034.630 End of init_trend_cache()
 83651:20160107:132034.630 End of init_database_cache()
 83651:20160107:132034.630 In init_configuration_cache() size:8388608
 83651:20160107:132034.630 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:7130317
 83651:20160107:132034.630 zbx_shmget() removing existing shm_id:1409035
 83651:20160107:132034.630 valid user addresses: [0x7f50a10d3170, 0x7f50a179fcc0] total size: 7129936
 83651:20160107:132034.630 End of zbx_mem_create()
 83651:20160107:132034.630 In zbx_strpool_create()
 83651:20160107:132034.630 In zbx_mem_create() descr:'string pool' param:'CacheSize' size:1258291
 83651:20160107:132034.630 zbx_shmget() removing existing shm_id:1441804
 83651:20160107:132034.630 valid user addresses: [0x7f50aa55f168, 0x7f50aa692328] total size: 1257920
 83651:20160107:132034.630 End of zbx_mem_create()
 83651:20160107:132034.630 End of zbx_strpool_create()
 83651:20160107:132034.630 End of init_configuration_cache()
 83651:20160107:132034.630 In init_selfmon_collector()
 83651:20160107:132034.630 init_selfmon_collector() size:7072
 83651:20160107:132034.630 zbx_shmget() removing existing shm_id:1474573
 83651:20160107:132034.630 End of init_selfmon_collector() collector:0x7f50aa6af000
 83651:20160107:132034.630 In zbx_vc_init()
 83651:20160107:132034.630 In zbx_mem_required_size() size:0 chunks_num:1 descr:'value cache size' param:'ValueCacheSize'
 83651:20160107:132034.630 End of zbx_mem_required_size() size:427
 83651:20160107:132034.631 In zbx_mem_create() descr:'value cache size' param:'ValueCacheSize' size:8388608
 83651:20160107:132034.631 zbx_shmget() removing existing shm_id:1507342
 83651:20160107:132034.631 valid user addresses: [0x7f50a08d3170, 0x7f50a10d2ff0] total size: 8388224
 83651:20160107:132034.631 End of zbx_mem_create()
 83651:20160107:132034.631 End of zbx_vc_init()
 83651:20160107:132034.631 In zbx_db_get_database_type()
 83651:20160107:132034.631 In DBconnect() flag:0
 83651:20160107:132034.641 [Z3001] connection to database 'zabbix' failed: [0] server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

 83651:20160107:132034.641 database is down: reconnecting in 10 seconds
 83651:20160107:132044.649 [Z3001] connection to database 'zabbix' failed: [0] server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

I saw the error messages "Server closed the connection unexpectedly..." when I was using the wrong options to connect with the psql client.

Has anyone seen these errors before? If not, how can I tell what command is being called to connect to the Zabbix db server via psql client?

[cgill-wh]

Believe it or not this was a password issue in the Zabbix server config :facepalm: