HELP Zabbix Server stop running, unable to start Server

[browndawg74]

I just the zabbix appliance installed on my VM server. I have two Zabbix proxy servers installed and running. I was installing some hosts to the new server. The Zabbix server shut down and I can not get the server to start back up. Can someone help me, please?


Output of service zabbix-server status

● zabbix-server.service - Zabbix Server
Loaded: loaded (/lib/systemd/system/zabbix-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2019-01-23 20:44:51 EET; 3s ago
Process: 6229 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 6224 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 6227 (code=exited, status=1/FAILURE)
Jan 23 20:44:51 zabbix systemd[1]: zabbix-server.service: Unit entered failed state.
Jan 23 20:44:51 zabbix systemd[1]: zabbix-server.service: Failed with result 'exit-code'.

output from zabbix logs

258:20190123:200832.678 Starting Zabbix Server. Zabbix 4.0.3 (revision 87993).
4258:20190123:200832.678 ****** Enabled features ******
4258:20190123:200832.678 SNMP monitoring: YES
4258:20190123:200832.678 IPMI monitoring: YES
4258:20190123:200832.678 Web monitoring: YES
4258:20190123:200832.678 VMware monitoring: YES
4258:20190123:200832.678 SMTP authentication: YES
4258:20190123:200832.678 Jabber notifications: YES
4258:20190123:200832.678 Ez Texting notifications: YES
4258:20190123:200832.678 ODBC: YES
4258:20190123:200832.679 SSH2 support: YES
4258:20190123:200832.679 IPv6 support: YES
4258:20190123:200832.679 TLS support: YES
4258:20190123:200832.679 ******************************
4258:20190123:200832.679 using configuration file: /etc/zabbix/zabbix_server.conf
4258:20190123:200832.686 current database version (mandatory/optional): 04000000/04000003
4258:20190123:200832.686 required mandatory version: 04000000
4258:20190123:200833.522 __mem_malloc: skipped 0 asked 128 skip_min 18446744073709551615 skip_max 0
4258:20190123:200833.522 [file:dbconfig.c,line:94] zbx_mem_malloc(): out of memory (requested 128 bytes)
4258:20190123:200833.522 [file:dbconfig.c,line:94] zbx_mem_malloc(): please increase CacheSize configuration parameter
4258:20190123:200833.522 === memory statistics for configuration cache ===
4258:20190123:200833.522 free chunks of size 24 bytes: 38
4258:20190123:200833.522 free chunks of size 32 bytes: 2
4258:20190123:200833.522 min chunk size: 24 bytes
4258:20190123:200833.522 max chunk size: 32 bytes
4258:20190123:200833.522 memory of total size 8388232 bytes fragmented into 73677 chunks
4258:20190123:200833.522 of those, 976 bytes are in 40 free chunks
4258:20190123:200833.522 of those, 7208440 bytes are in 73637 used chunks
4258:20190123:200833.522 ================================
4258:20190123:200833.522 === Backtrace: ===
4258:20190123:200833.523 12: /usr/sbin/zabbix_server(zbx_backtrace+0x44) [0x55b77fd7eec6]
4258:20190123:200833.523 11: /usr/sbin/zabbix_server(__zbx_mem_malloc+0x163) [0x55b77fd7aca8]
4258:20190123:200833.523 10: /usr/sbin/zabbix_server(+0xc62fa) [0x55b77fd452fa]
4258:20190123:200833.523 09: /usr/sbin/zabbix_server(zbx_hashset_insert_ext+0x12f) [0x55b77fd843a9]
4258:20190123:200833.523 08: /usr/sbin/zabbix_server(zbx_hashset_insert+0x2d) [0x55b77fd84278]
4258:20190123:200833.523 07: /usr/sbin/zabbix_server(DCfind_id+0x9b) [0x55b77fd45d5d]
4258:20190123:200833.523 06: /usr/sbin/zabbix_server(+0xce2d0) [0x55b77fd4d2d0]
4258:20190123:200833.523 05: /usr/sbin/zabbix_server(DCsync_configuration+0xe41) [0x55b77fd518c8]
4258:20190123:200833.523 04: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x6b6) [0x55b77fcb744b]
4258:20190123:200833.523 03: /usr/sbin/zabbix_server(daemon_start+0x315) [0x55b77fd7e655]
4258:20190123:200833.523 02: /usr/sbin/zabbix_server(main+0x305) [0x55b77fcb6d7f]
4258:20190123:200833.523 01: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f5b74bdd830]
4258:20190123:200833.523 00: /usr/sbin/zabbix_server(_start+0x29) [0x55b77fcb5f49]

[LenR]

Do what this error says: : please increase CacheSize configuration parameter

[browndawg74]

I increase the CacheSize the to 4096m. I can not get the server to startup.

[LenR]

What changed from when it was working?

Check ulimits for Zabbix, maybe su - zabbix, then ulimit -a

Our production server runs with a 4G config cache and it's pretty big.

[browndawg74]

ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64014
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 64014
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[ingus.vilnis]

Paste exactly the lines of zabbix_server.conf file where you think you set the value.

[browndawg74]

### Option: CacheSize
# Size of configuration cache, in bytes.
# Shared memory size for storing host, item and trigger data.
#
# Mandatory: no
# Range: 128K-8G
# Default:
CacheSize=1024M

[ingus.vilnis]

According to the log file you have the default cache size of 8M allocated to server.

The code fragment shows correctly uncommented configuration line. If that is the file /etc/zabbix/zabbix_server.conf then you must find out why is the change ignored by Zabbix server.

How are you restarting the service to apply the changes?

[LenR]

Ensure that that is the only line for CacheSize= in the config

[browndawg74]

Thank you I got my server back up and running.