Zabbix Server Stopped after upgrade to 5.0

[DavidsonJr]

I upgrade to version 5.0 installed on Centos server 7.0. My Zabbix server dont start anymore, just start and stop. Output of my logs below.


12553:20200626:083336.505 Zabbix Server stopped. Zabbix 5.0.1 (revision c2a0b03480).
12579:20200626:083346.677 Starting Zabbix Server. Zabbix 5.0.1 (revision c2a0b03480).
12579:20200626:083346.677 ****** Enabled features ******
12579:20200626:083346.677 SNMP monitoring: YES
12579:20200626:083346.677 IPMI monitoring: YES
12579:20200626:083346.677 Web monitoring: YES
12579:20200626:083346.677 VMware monitoring: YES
12579:20200626:083346.677 SMTP authentication: YES
12579:20200626:083346.677 ODBC: YES
12579:20200626:083346.677 SSH support: YES
12579:20200626:083346.677 IPv6 support: YES
12579:20200626:083346.677 TLS support: YES
12579:20200626:083346.677 ******************************
12579:20200626:083346.677 using configuration file: /etc/zabbix/zabbix_server.conf
12579:20200626:083346.686 current database version (mandatory/optional): 05000000/05000000
12579:20200626:083346.686 required mandatory version: 05000000
12579:20200626:083346.710 database is not upgraded to use double precision values
12579:20200626:083346.712 server #0 started [main process]
12580:20200626:083346.713 server #1 started [configuration syncer #1]
12580:20200626:083347.536 __mem_malloc: skipped 9 asked 108424 skip_min 296 skip_max 24768
12580:20200626:083347.536 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 108424 bytes)
12580:20200626:083347.536 [file:dbconfig.c,line:95] __zbx_mem_realloc(): please increase CacheSize configuration parameter
12580:20200626:083347.536 === memory statistics for configuration cache ===
12580:20200626:083347.536 free chunks of size 24 bytes: 183
12580:20200626:083347.536 free chunks of size 48 bytes: 3
12580:20200626:083347.537 free chunks of size 56 bytes: 6
12580:20200626:083347.537 free chunks of size >= 256 bytes: 9
12580:20200626:083347.537 min chunk size: 24 bytes
12580:20200626:083347.537 max chunk size: 24768 bytes
12580:20200626:083347.537 memory of total size 8388232 bytes fragmented into 74732 chunks
12580:20200626:083347.537 of those, 71936 bytes are in 201 free chunks
12580:20200626:083347.537 of those, 7120600 bytes are in 74531 used chunks
12580:20200626:083347.537 ================================
12580:20200626:083347.537 === Backtrace: ===
12580:20200626:083347.539 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x42) [0x5570936ef698]
12580:20200626:083347.539 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_realloc+0x169) [0x5570936ea8e0]
12580:20200626:083347.539 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x17d300) [0x5570936b2300]
12580:20200626:083347.539 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_reserve+0xc1) [0x5570936f51e0]
12580:20200626:083347.539 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0xee) [0x5570936f5457]
12580:20200626:083347.539 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x2d) [0x5570936f5367]
12580:20200626:083347.539 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0x86) [0x5570936b2cb8]
12580:20200626:083347.539 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x185a57) [0x5570936baa57]
12580:20200626:083347.539 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0xd2c) [0x5570936be6df]
12580:20200626:083347.539 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x135) [0x557093589919]
12580:20200626:083347.539 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x5570936fe1bb]
12580:20200626:083347.539 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xa02) [0x55709357bf4f]
12580:20200626:083347.539 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x305) [0x5570936ef2e1]
12580:20200626:083347.540 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x312) [0x55709357b54b]
12580:20200626:083347.540 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f606171c555]
12580:20200626:083347.540 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x455c9) [0x55709357a5c9]
12579:20200626:083347.544 One child process died (PID:12580,exitcode/signal:1). Exiting ...
12579:20200626:083347.546 syncing trend data...
12579:20200626:083347.547 syncing trend data done
12579:20200626:083347.547 Zabbix Server stopped. Zabbix 5.0.1 (revision c2a0b03480).

[sfereday]

When you say the server doesn't start, do you mean the server webpage isn't accessible or the service itself is still in a dead stat after running systemctl start zabbix-server?

[DavidsonJr]

The webpage is accessible, but i receive on screen "zabbix server is not running"

I belive the serve just start and dead, .sequentially on start.



systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sex 2020-06-26 16:58:40 -03; 5s ago
Process: 2881 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 2876 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 2878 (code=exited, status=0/SUCCESS)

Jun 26 16:58:40 localhost.localdomain kill[2881]: -s, --signal <sig> send specified signal
Jun 26 16:58:40 localhost.localdomain kill[2881]: -q, --queue <sig> use sigqueue(2) rather than kill(2)
Jun 26 16:58:40 localhost.localdomain kill[2881]: -p, --pid print pids without signaling them
Jun 26 16:58:40 localhost.localdomain kill[2881]: -l, --list [=<signal>] list signal names, or convert one to a name
Jun 26 16:58:40 localhost.localdomain kill[2881]: -L, --table list signal names and numbers
Jun 26 16:58:40 localhost.localdomain kill[2881]: -h, --help display this help and exit
Jun 26 16:58:40 localhost.localdomain kill[2881]: -V, --version output version information and exit
Jun 26 16:58:40 localhost.localdomain kill[2881]: For more details see kill(1).
Jun 26 16:58:40 localhost.localdomain systemd[1]: Unit zabbix-server.service entered failed state.
Jun 26 16:58:40 localhost.localdomain systemd[1]: zabbix-server.service failed.

[tim.mooney]

If you examine the logs closely, there is this message:

12580:20200626:083347.536 __mem_malloc: skipped 9 asked 108424 skip_min 296 skip_max 24768
12580:20200626:083347.536 [file:dbconfig.c,line:95] __zbx_mem_realloc(): out of memory (requested 108424 bytes)
12580:20200626:083347.536 [file:dbconfig.c,line:95] __zbx_mem_realloc(): please increase CacheSize configuration parameter
12580:20200626:083347.536 === memory statistics for configuration cache ===
12580:20200626:083347.536 free chunks of size 24 bytes: 183
12580:20200626:083347.536 free chunks of size 48 bytes: 3
12580:20200626:083347.537 free chunks of size 56 bytes: 6
12580:20200626:083347.537 free chunks of size >= 256 bytes: 9
12580:20200626:083347.537 min chunk size: 24 bytes
12580:20200626:083347.537 max chunk size: 24768 bytes
12580:20200626:083347.537 memory of total size 8388232 bytes fragmented into 74732 chunks
12580:20200626:083347.537 of those, 71936 bytes are in 201 free chunks
12580:20200626:083347.537 of those, 7120600 bytes are in 74531 used chunks

Note the message about "please increase CacheSize configuration parameter". The Zabbix server stopped because of problems getting a large enough chunk of free memory.

You can try increasing CacheSize, as requested. I believe there are other people that have posted on the forums recently with the exact same issue. It's possible this is some kind of bug with Zabbix 5.x, or it could be "normal" but 5.x just requires more memory for CacheSize.

There are later versions of Zabbix out than 5.0.1, too, so it might be worth checking the release notes for any of them, to see if they mention fixes for memory or CacheSize issues, or memory fragmentation.

[DavidsonJr]

If you examine the logs closely, there is this message:



Note the message about "please increase CacheSize configuration parameter". The Zabbix server stopped because of problems getting a large enough chunk of free memory.

You can try increasing CacheSize, as requested. I believe there are other people that have posted on the forums recently with the exact same issue. It's possible this is some kind of bug with Zabbix 5.x, or it could be "normal" but 5.x just requires more memory for CacheSize.

There are later versions of Zabbix out than 5.0.1, too, so it might be worth checking the release notes for any of them, to see if they mention fixes for memory or CacheSize issues, or memory fragmentation.

THX,

I had read about cache and increase to 32MB, but don't work, i try get later version...