I started having an issue with Zabbix server yesterday after installing Ubuntu updates (apt update; apt upgrade; apt autoremove). The Zabbix agent is running fine; but the Zabbix server would not start (assorted errors as detailed below).
Last Zabbix Update 2021-02-23 (roughly a month ago):
2021-02-23 15:30:27 upgrade zabbix-server-mysql:amd64 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:30:27 status half-configured zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:27 status unpacked zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:27 status half-installed zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:28 status unpacked zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:17 upgrade zabbix-agent:amd64 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:31:17 status half-configured zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status half-installed zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:31:18 upgrade zabbix-frontend-php:all 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:31:18 status half-configured zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:18 status half-installed zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:19 status unpacked zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:31:28 configure zabbix-server-mysql:amd64 1:4.0.29-1+focal <none>
2021-02-23 15:31:28 status unpacked zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:28 status half-configured zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:29 status installed zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:33:40 configure zabbix-frontend-php:all 1:4.0.29-1+focal <none>
2021-02-23 15:33:40 status unpacked zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:40 status half-configured zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:41 status installed zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:41 configure zabbix-agent:amd64 1:4.0.29-1+focal <none>
2021-02-23 15:33:41 status unpacked zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:33:41 status half-configured zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:33:42 status installed zabbix-agent:amd64 1:4.0.29-1+focal
Currently Installed (has been running fine since installed 2021-02-23):
zabbix-agent/unknown,now 1:4.0.29-1+focal amd64 [installed]
zabbix-frontend-php/unknown,now 1:4.0.29-1+focal all [installed]
zabbix-release/unknown,now 1:4.0-3+focal all [installed]
zabbix-server-mysql/unknown,now 1:4.0.29-1+focal amd64 [installed]
Ubuntu Info:
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
Before yesterday's Ubuntu updates: Linux version 5.4.0-66-generic (buildd@lgw01-amd64-039) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 (Ubuntu 5.4.0-66.74-generic 5.4.86)
After yesterday's Ubuntu updates: Linux version 5.4.0-70-generic (buildd@lcy01-amd64-004) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 (Ubuntu 5.4.0-70.78-generic 5.4.94)
I reviewed the Zabbix server logs since 2021-02-23 and there have not been any errors until yesterday (started after the Ubuntu updates).
The first issue that occurred was this (syslog):
Mar 26 15:58:47 test07 zabbix_server[2120]: zabbix_server [2120]: ERROR: parameter "TLSCertFile" is defined, but "TLSCAFile" is not defined
I resolved this issue by uncommenting the TLSCAFile parameter in zabbix_server.conf and providing the correct path. Prior to the Ubuntu updates, Zabbix was working fine with just the TLSCertFile and TLSKeyFile parameters specified.
The next issue that started occurring was this (syslog):
Mar 27 09:04:01 test07 systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: Operation not permitted
The /run/zabbix directory exists and the Zabbix agent is able to create a PID file there, so I'm not sure why the Zabbix server is having issues.
ll /run/zabbix
total 4
-rw-rw-r-- 1 zabbix zabbix 3 Mar 27 11:57 zabbix_agentd.pid
srwx------ 1 zabbix zabbix 0 Mar 27 11:57 zabbix_server_alerter.sock=
srwx------ 1 zabbix zabbix 0 Mar 27 11:57 zabbix_server_preprocessing.sock=
I resolved this issue by commenting out the PidFile paramter in zabbix_server.conf so that is uses the default of /tmp/zabbix_server.pid:
ll /tmp
-rw-rw-r-- 1 zabbix zabbix 4 Mar 27 11:57 zabbix_server.pid
The next issue that started occurring was this (zabbix_server.log):
140450:20210327:090402.128 cannot load private key from file "/etc/ssl/private/(redacted).key":
file ../crypto/bio/bss_file.c line 288: error:0200100D:system library:fopen:Permission denied: fopen('/etc/ssl/private/(redacted).key','r')
file ../crypto/bio/bss_file.c line 290: error:20074002:BIO routines:file_ctrl:system lib
file ../ssl/ssl_rsa.c line 540: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
I have confirmed that this file exists, and it has been working fine since 2021-02-23:
-rw-r----- 1 root ssl-cert 1704 Jul 16 2020 (redacted).key
I resolved this issue by commenting out the TLSCertFile, TLSKeyFile, and TLSCAFile parameters in zabbix_server.conf.
At this point, Zabbix server is up and running, but without the TLS parameters and with the PID file going to /tmp instead of /run/zabbix like the Zabbix agent uses.
It seems like something permissions related (can't write to /run/zabbix and can't read from /etc/ssl/private, but Zabbix Agent seems to be working fine, and I haven't changed anything other than installing Ubuntu updates yesterday with apt update; apt upgrade; apt autoremove; reboot. As far as I can tell there weren't any updates to Zabbix yesterday as the log indicates the same version of Zabbix before and after the Ubuntu updates were installed.
As indicated, this is an existing installation, and it has been running fine for months. Any thoughts on what might have changed via the Ubuntu updates to cause this?
Last Zabbix Update 2021-02-23 (roughly a month ago):
2021-02-23 15:30:27 upgrade zabbix-server-mysql:amd64 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:30:27 status half-configured zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:27 status unpacked zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:27 status half-installed zabbix-server-mysql:amd64 1:4.0.27-1+focal
2021-02-23 15:30:28 status unpacked zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:17 upgrade zabbix-agent:amd64 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:31:17 status half-configured zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status half-installed zabbix-agent:amd64 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:31:18 upgrade zabbix-frontend-php:all 1:4.0.27-1+focal 1:4.0.29-1+focal
2021-02-23 15:31:18 status half-configured zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:18 status unpacked zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:18 status half-installed zabbix-frontend-php:all 1:4.0.27-1+focal
2021-02-23 15:31:19 status unpacked zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:31:28 configure zabbix-server-mysql:amd64 1:4.0.29-1+focal <none>
2021-02-23 15:31:28 status unpacked zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:28 status half-configured zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:31:29 status installed zabbix-server-mysql:amd64 1:4.0.29-1+focal
2021-02-23 15:33:40 configure zabbix-frontend-php:all 1:4.0.29-1+focal <none>
2021-02-23 15:33:40 status unpacked zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:40 status half-configured zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:41 status installed zabbix-frontend-php:all 1:4.0.29-1+focal
2021-02-23 15:33:41 configure zabbix-agent:amd64 1:4.0.29-1+focal <none>
2021-02-23 15:33:41 status unpacked zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:33:41 status half-configured zabbix-agent:amd64 1:4.0.29-1+focal
2021-02-23 15:33:42 status installed zabbix-agent:amd64 1:4.0.29-1+focal
Currently Installed (has been running fine since installed 2021-02-23):
zabbix-agent/unknown,now 1:4.0.29-1+focal amd64 [installed]
zabbix-frontend-php/unknown,now 1:4.0.29-1+focal all [installed]
zabbix-release/unknown,now 1:4.0-3+focal all [installed]
zabbix-server-mysql/unknown,now 1:4.0.29-1+focal amd64 [installed]
Ubuntu Info:
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
Before yesterday's Ubuntu updates: Linux version 5.4.0-66-generic (buildd@lgw01-amd64-039) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 (Ubuntu 5.4.0-66.74-generic 5.4.86)
After yesterday's Ubuntu updates: Linux version 5.4.0-70-generic (buildd@lcy01-amd64-004) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 (Ubuntu 5.4.0-70.78-generic 5.4.94)
I reviewed the Zabbix server logs since 2021-02-23 and there have not been any errors until yesterday (started after the Ubuntu updates).
The first issue that occurred was this (syslog):
Mar 26 15:58:47 test07 zabbix_server[2120]: zabbix_server [2120]: ERROR: parameter "TLSCertFile" is defined, but "TLSCAFile" is not defined
I resolved this issue by uncommenting the TLSCAFile parameter in zabbix_server.conf and providing the correct path. Prior to the Ubuntu updates, Zabbix was working fine with just the TLSCertFile and TLSKeyFile parameters specified.
The next issue that started occurring was this (syslog):
Mar 27 09:04:01 test07 systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: Operation not permitted
The /run/zabbix directory exists and the Zabbix agent is able to create a PID file there, so I'm not sure why the Zabbix server is having issues.
ll /run/zabbix
total 4
-rw-rw-r-- 1 zabbix zabbix 3 Mar 27 11:57 zabbix_agentd.pid
srwx------ 1 zabbix zabbix 0 Mar 27 11:57 zabbix_server_alerter.sock=
srwx------ 1 zabbix zabbix 0 Mar 27 11:57 zabbix_server_preprocessing.sock=
I resolved this issue by commenting out the PidFile paramter in zabbix_server.conf so that is uses the default of /tmp/zabbix_server.pid:
ll /tmp
-rw-rw-r-- 1 zabbix zabbix 4 Mar 27 11:57 zabbix_server.pid
The next issue that started occurring was this (zabbix_server.log):
140450:20210327:090402.128 cannot load private key from file "/etc/ssl/private/(redacted).key":
file ../crypto/bio/bss_file.c line 288: error:0200100D:system library:fopen:Permission denied: fopen('/etc/ssl/private/(redacted).key','r')
file ../crypto/bio/bss_file.c line 290: error:20074002:BIO routines:file_ctrl:system lib
file ../ssl/ssl_rsa.c line 540: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
I have confirmed that this file exists, and it has been working fine since 2021-02-23:
-rw-r----- 1 root ssl-cert 1704 Jul 16 2020 (redacted).key
I resolved this issue by commenting out the TLSCertFile, TLSKeyFile, and TLSCAFile parameters in zabbix_server.conf.
At this point, Zabbix server is up and running, but without the TLS parameters and with the PID file going to /tmp instead of /run/zabbix like the Zabbix agent uses.
It seems like something permissions related (can't write to /run/zabbix and can't read from /etc/ssl/private, but Zabbix Agent seems to be working fine, and I haven't changed anything other than installing Ubuntu updates yesterday with apt update; apt upgrade; apt autoremove; reboot. As far as I can tell there weren't any updates to Zabbix yesterday as the log indicates the same version of Zabbix before and after the Ubuntu updates were installed.
As indicated, this is an existing installation, and it has been running fine for months. Any thoughts on what might have changed via the Ubuntu updates to cause this?