Zabbix Server has Stopped

[railroad]

Installed Zabbix 5.03 on Ubuntu 20.04 this morning, both clean installs. Everything went smoothly, but after adding several hosts the zabbix-server stopped running. After some googling here is some information that might help.

Thanks in advance for the help.

sudo systemctl status zabbix-server

zabbix-server.service - Zabbix Server
Loaded: loaded (/lib/systemd/system/zabbix-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2020-08-31 14:24:19 UTC; 1s ago
Process: 1854 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Process: 1872 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Main PID: 1870 (code=exited, status=0/SUCCESS)

Aug 31 14:24:19 zabbix systemd[1]: zabbix-server.service: Control process exited, code=exited, status=1/FAILURE
Aug 31 14:24:19 zabbix systemd[1]: zabbix-server.service: Failed with result 'exit-code'.

cat /var/log/zabbix/zabbix_server.log

4687:20200831:144354.488 Starting Zabbix Server. Zabbix 5.0.3 (revision 146855bff3).
4687:20200831:144354.488 ****** Enabled features ******
4687:20200831:144354.488 SNMP monitoring: YES
4687:20200831:144354.488 IPMI monitoring: YES
4687:20200831:144354.488 Web monitoring: YES
4687:20200831:144354.488 VMware monitoring: YES
4687:20200831:144354.488 SMTP authentication: YES
4687:20200831:144354.488 ODBC: YES
4687:20200831:144354.488 SSH support: YES
4687:20200831:144354.488 IPv6 support: YES
4687:20200831:144354.488 TLS support: YES
4687:20200831:144354.488 ******************************
4687:20200831:144354.488 using configuration file: /etc/zabbix/zabbix_server.conf
4687:20200831:144354.497 current database version (mandatory/optional): 05000000/05000002
4687:20200831:144354.497 required mandatory version: 05000000
4687:20200831:144354.506 server #0 started [main process]
4690:20200831:144354.506 server #1 started [configuration syncer #1]
4690:20200831:144354.934 __mem_malloc: skipped 0 asked 136 skip_min 18446744073709551615 skip_max 0
4690:20200831:144354.934 [file:dbconfig.c,line:96] __zbx_mem_malloc(): out of memory (requested 136 bytes)
4690:20200831:144354.934 [file:dbconfig.c,line:96] __zbx_mem_malloc(): please increase CacheSize configuration parameter
4690:20200831:144354.934 === memory statistics for configuration cache ===
4690:20200831:144354.934 free chunks of size 24 bytes: 90
4690:20200831:144354.934 free chunks of size 40 bytes: 1
4690:20200831:144354.934 min chunk size: 24 bytes
4690:20200831:144354.934 max chunk size: 40 bytes
4690:20200831:144354.934 memory of total size 8388232 bytes fragmented into 73679 chunks
4690:20200831:144354.934 of those, 2200 bytes are in 91 free chunks
4690:20200831:144354.934 of those, 7207184 bytes are in 73588 used chunks
4690:20200831:144354.934 ================================
4690:20200831:144354.934 === Backtrace: ===
4690:20200831:144354.935 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x52) [0x55ee974a945d]
4690:20200831:144354.935 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_malloc+0x184) [0x55ee974a3dd6]
4690:20200831:144354.935 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1a9556) [0x55ee97467556]
4690:20200831:144354.935 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0x133) [0x55ee974afafb]
4690:20200831:144354.935 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x31) [0x55ee974af9c6]
4690:20200831:144354.935 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0xb4) [0x55ee97467fe5]
4690:20200831:144354.935 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1b2444) [0x55ee97470444]
4690:20200831:144354.935 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0xff4) [0x55ee9747512b]
4690:20200831:144354.935 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x11a) [0x55ee97316170]
4690:20200831:144354.935 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x3b) [0x55ee974b923c]
4690:20200831:144354.935 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x9e8) [0x55ee973071bc]
4690:20200831:144354.935 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x303) [0x55ee974a8f9f]
4690:20200831:144354.935 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x309) [0x55ee973067be]
4690:20200831:144354.935 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd2372d20b3]
4690:20200831:144354.935 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x55ee973057ae]
4687:20200831:144354.938 One child process died (PID:4690,exitcode/signal:1). Exiting ...
4687:20200831:144354.939 syncing trend data...
4687:20200831:144354.939 syncing trend data done
4687:20200831:144354.939 Zabbix Server stopped. Zabbix 5.0.3 (revision 146855bff3).

[railroad]

So I have the issue fixed, for future reference by someone...

4690:20200831:144354.934 [file:dbconfig.c,line:96] __zbx_mem_malloc(): out of memory (requested 136 bytes)
4690:20200831:144354.934 [file:dbconfig.c,line:96] __zbx_mem_malloc(): please increase CacheSize configuration parameter

In the file /var/zabbix/zabbix_server.conf I changed the setting to CacheSize=2048M. This allowed the server to start, but created too many connections for MySQL, shutting it down. I then had to edit /etc/mysql/my.cnf, and add:

[mysqld]
max_connections = 1000

Everything seems to be happy now, we'll see after a few hours.