Zabbix Server Not Running

[akeccs]

I setup a Zabbix server as a VM in Vmware 6.7. After adding quite a of our switches to Zabbix the server no longer starts. Looking at the logs it states I need to increase my cache size limit, which I have done, now to 15GB, which seems excessive, however it still fails to start, I am not that knowledgeable on Linux so I am learning as I go.
I have attached the config to this post, if anyone can help it would be much appreciated.

2008:20221209:131504.601 ================================
  2008:20221209:131504.601 === Backtrace: ===
  2008:20221209:131504.602 16: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3f) [0x5614e750ae24]
  2008:20221209:131504.602 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_shmem_realloc+0xe8) [0x5614e75060d1]
  2008:20221209:131504.602 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1f7f46) [0x5614e7487f46]
  2008:20221209:131504.602 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_reserve+0xc1) [0x5614e750dfd0]
  2008:20221209:131504.602 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0xec) [0x5614e750e245]
  2008:20221209:131504.602 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x2d) [0x5614e750e157]
  2008:20221209:131504.602 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0x86) [0x5614e748880a]
  2008:20221209:131504.602 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x202c92) [0x5614e7492c92]
  2008:20221209:131504.602 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x12bc) [0x5614e7495a38]
  2008:20221209:131504.602 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x14f) [0x5614e7313872]
  2008:20221209:131504.602 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x5614e7517941]
  2008:20221209:131504.602 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x7197e) [0x5614e730197e]
  2008:20221209:131504.602 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x83e) [0x5614e7302b9a]
  2008:20221209:131504.602 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_daemon_start+0x3b1) [0x5614e750aa3a]
  2008:20221209:131504.602 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x36b) [0x5614e730121e]
  2008:20221209:131504.602 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7effeb49bcf3]
  2008:20221209:131504.602 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x5614e72ffd8e]
  2008:20221209:131504.602 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): out of memory (requested 366104 bytes)
  2008:20221209:131504.602 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): please increase CacheSize configuration parameter
  2005:20221209:131504.605 One child process died (PID:2008,exitcode/signal:1). Exiting ...
  2006:20221209:131505.094 HA manager has been paused
  2006:20221209:131505.098 HA manager has been stopped
  2005:20221209:131505.099 Zabbix Server stopped. Zabbix 6.2.3 (revision 98ee88fc19d).
  2014:20221209:131515.326 Starting Zabbix Server. Zabbix 6.2.3 (revision 98ee88fc19d).
  2014:20221209:131515.326 ****** Enabled features ******
  2014:20221209:131515.326 SNMP monitoring:           YES
  2014:20221209:131515.326 IPMI monitoring:           YES
  2014:20221209:131515.326 Web monitoring:            YES
  2014:20221209:131515.326 VMware monitoring:         YES
  2014:20221209:131515.326 SMTP authentication:       YES
  2014:20221209:131515.326 ODBC:                      YES
  2014:20221209:131515.326 SSH support:               YES
  2014:20221209:131515.326 IPv6 support:              YES
  2014:20221209:131515.326 TLS support:               YES
  2014:20221209:131515.326 ******************************
  2014:20221209:131515.326 using configuration file: /etc/zabbix/zabbix_server.conf
  2014:20221209:131515.336 current database version (mandatory/optional): 06020000/06020002
  2014:20221209:131515.336 required mandatory version: 06020000
  2015:20221209:131515.341 starting HA manager
  2015:20221209:131515.344 HA manager started in active mode
  2014:20221209:131515.344 server #0 started [main process]
  2016:20221209:131515.345 server #1 started [service manager #1]
  2017:20221209:131515.345 server #2 started [configuration syncer #1]
  2017:20221209:131515.842 __mem_malloc: skipped 10 asked 366104 skip_min 568 skip_max 246680
  2017:20221209:131515.842 === memory statistics for configuration cache ===
  2017:20221209:131515.842 free chunks of size >= 256 bytes:       10
  2017:20221209:131515.842 min chunk size:        568 bytes
  2017:20221209:131515.842 max chunk size:     246680 bytes
  2017:20221209:131515.842 memory of total size 29195520 bytes fragmented into 272409 chunks
  2017:20221209:131515.843 of those,     369896 bytes are in       10 free chunks
  2017:20221209:131515.843 of those,   28825624 bytes are in   272399 used chunks
  2017:20221209:131515.843 of those,    4358528 bytes are used by allocation overhead
  2017:20221209:131515.843 ================================
  2017:20221209:131515.843 === Backtrace: ===
  2017:20221209:131515.844 16: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3f) [0x556ca3769e24]
  2017:20221209:131515.844 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_shmem_realloc+0xe8) [0x556ca37650d1]
  2017:20221209:131515.844 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1f7f46) [0x556ca36e6f46]
  2017:20221209:131515.844 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_reserve+0xc1) [0x556ca376cfd0]
  2017:20221209:131515.844 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0xec) [0x556ca376d245]
  2017:20221209:131515.844 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x2d) [0x556ca376d157]
  2017:20221209:131515.844 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCfind_id+0x86) [0x556ca36e780a]
  2017:20221209:131515.844 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x202c92) [0x556ca36f1c92]
  2017:20221209:131515.844 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x12bc) [0x556ca36f4a38]
  2017:20221209:131515.844 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x14f) [0x556ca3572872]
  2017:20221209:131515.844 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x556ca3776941]
  2017:20221209:131515.844 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x7197e) [0x556ca356097e]
  2017:20221209:131515.844 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0x83e) [0x556ca3561b9a]
  2017:20221209:131515.844 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_daemon_start+0x3b1) [0x556ca3769a3a]
  2017:20221209:131515.844 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x36b) [0x556ca356021e]
  2017:20221209:131515.844 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f8d4f317cf3]
  2017:20221209:131515.844 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x556ca355ed8e]
  2017:20221209:131515.844 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): out of memory (requested 366104 bytes)
  2017:20221209:131515.844 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): please increase CacheSize configuration parameter
  2014:20221209:131515.847 One child process died (PID:2017,exitcode/signal:1). Exiting ...
  2015:20221209:131515.847 HA manager has been paused
  2015:20221209:131515.851 HA manager has been stopped
  2014:20221209:131515.852 Zabbix Server stopped. Zabbix 6.2.3 (revision 98ee88fc19d)​

[Markku]

You need to remove the comment character (#) from the CacheSize line:

# CacheSize=15G

and make it smaller again, like 128M or whatever is suitable.

Markku

[Markku]

Make sure that your Zabbix server host in the GUI has a Zabbix server template attached, then you will get graphs and alerts about the various cache utilizations in the long run.

Also, you may need to consider changing the database user password now that you sent it to the internet.

Markku

[akeccs]

I have removed the comment character and setting to 2G, still no go:

211113:20221212:093909.838 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x55a7ab2ecd8e]
211113:20221212:093909.838 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): out of memory (requested 366104 bytes)
211113:20221212:093909.838 [file:dbconfig.c,line:97] __zbx_shmem_realloc(): please increase CacheSize configuration parameter
211110:20221212:093909.843 One child process died (PID:211113,exitcode/signal:1). Exiting ...
211111:20221212:093909.843 HA manager has been paused
211111:20221212:093909.846 HA manager has been stopped
211110:20221212:093909.848 Zabbix Server stopped. Zabbix 6.2.3 (revision 98ee88fc19d).
211119:20221212:093920.076 Starting Zabbix Server. Zabbix 6.2.3 (revision 98ee88fc19d).
211119:20221212:093920.076 ****** Enabled features ******
211119:20221212:093920.076 SNMP monitoring:           YES
211119:20221212:093920.076 IPMI monitoring:           YES
211119:20221212:093920.076 Web monitoring:            YES
211119:20221212:093920.076 VMware monitoring:         YES
211119:20221212:093920.076 SMTP authentication:       YES
211119:20221212:093920.076 ODBC:                      YES
211119:20221212:093920.076 SSH support:               YES
211119:20221212:093920.076 IPv6 support:              YES
211119:20221212:093920.076 TLS support:               YES
211119:20221212:093920.076 ******************************
211119:20221212:093920.076 using configuration file: /etc/zabbix/zabbix_server.conf
211119:20221212:093920.086 current database version (mandatory/optional): 06020000/06020002
211119:20221212:093920.086 required mandatory version: 06020000
211120:20221212:093920.092 starting HA manager
211120:20221212:093920.094 HA manager started in active mode
211119:20221212:093920.094 server #0 started [main process]
211121:20221212:093920.095 server #1 started [service manager #1]
211122:20221212:093920.095 server #2 started [configuration syncer #1]
211122:20221212:093920.576 __mem_malloc: skipped 10 asked 366104 skip_min 568 skip_max 246680
211122:20221212:093920.577 === memory statistics for configuration cache ===
211122:20221212:093920.577 free chunks of size >= 256 bytes:       10
211122:20221212:093920.577 min chunk size:        568 bytes
211122:20221212:093920.577 max chunk size:     246680 bytes
211122:20221212:093920.577 memory of total size 29195520 bytes fragmented into 272409 chunks
211122:20221212:093920.577 of those,     369896 bytes are in       10 free chunks
211122:20221212:093920.577 of those,   28825624 bytes are in   272399 used chunks
211122:20221212:093920.577 of those,    4358528 bytes are used by allocation overhead
211122:20221212:093920.577 ================================

[cyber]

if 2G did not work, go up to 4. If that fails, go up to 8...

[akeccs]

Currently at 18G, guess I'll keep going up.

[akeccs]

Had to increase the size of the disk on the VM to go any higher, now getting a different error:

1489:20221212:124903.484 using configuration file: /etc/zabbix/zabbix_server.conf
  1489:20221212:124903.494 current database version (mandatory/optional): 06020000/06020002
  1489:20221212:124903.494 required mandatory version: 06020000
  1490:20221212:124903.500 starting HA manager
  1490:20221212:124903.503 HA manager started in active mode
  1489:20221212:124903.503 cannot initialize configuration cache: cannot get private shared memory of size 25769803776 for configuration cache: [12] Cannot allocate memory
  1490:20221212:124903.503 HA manager has been paused
  1490:20221212:124903.506 HA manager has been stopped
  1489:20221212:124903.507 Zabbix Server stopped. Zabbix 6.2.3 (revision 98ee88fc19d).
  1495:20221212:124915.733 Starting Zabbix Server. Zabbix 6.2.3 (revision 98ee88fc19d).
  1495:20221212:124915.733 ****** Enabled features ******
  1495:20221212:124915.733 SNMP monitoring:           YES
  1495:20221212:124915.734 IPMI monitoring:           YES
  1495:20221212:124915.734 Web monitoring:            YES
  1495:20221212:124915.734 VMware monitoring:         YES
  1495:20221212:124915.734 SMTP authentication:       YES
  1495:20221212:124915.734 ODBC:                      YES
  1495:20221212:124915.734 SSH support:               YES
  1495:20221212:124915.734 IPv6 support:              YES
  1495:20221212:124915.734 TLS support:               YES
  1495:20221212:124915.734 ******************************
​

Also tried changing this to 25769803776 didn't work, after reboot reverted:

[root@appliance zabbix]# sysctl -a |grep shmmax
kernel.shmmax = 18446744073692774399
​

[Markku]

211122:20221212:093920.577 memory of total size 29195520 bytes fragmented into 272409 chunks

That's not 2G, that's 32M (the default in 6.2) right there.

Please send these outputs:

sudo grep CacheSize /etc/zabbix/zabbix_server.conf
sudo grep Include /etc/zabbix/zabbix_server.conf

Markku

[akeccs]

[root@appliance zabbix]# sudo grep CacheSize /etc/zabbix/zabbix_server.conf
### Option: VMwareCacheSize
# VMwareCacheSize=1G
### Option: CacheSize
CacheSize=24G
### Option: HistoryCacheSize
# HistoryCacheSize=32M
### Option: HistoryIndexCacheSize
# HistoryIndexCacheSize=512M
### Option: TrendCacheSize
# TrendCacheSize=8M
### Option: TrendFunctionCacheSize
# TrendFunctionCacheSize=8M
### Option: ValueCacheSize
# ValueCacheSize=400M
​

[root@appliance zabbix]# sudo grep Include /etc/zabbix/zabbix_server.conf
### Option: Include
# Include=
# Include=/usr/local/etc/zabbix_server.general.conf
# Include=/usr/local/etc/zabbix_server.conf.d/
# Include=/usr/local/etc/zabbix_server.conf.d/*.conf
​

[Markku]

How about testing again with some sensible CacheSize value like 1G so that we eliminate huge memory allocation problems.

Then show the error log again.

In my tests the "memory of total size" output (that I quoted earlier) correlated directly with the CacheSize value.

root@appliance

If this is the Zabbix appliance, I don't know if it works somehow different. Anyone?

Markku

[Markku]

Also, you are not running the latest 6.2.x release so upgrading it would eliminate any issues that have been fixed after 6.2.3.

Markku

[akeccs]

Ok, I feel like I am loosing my mind, I now just set it back to 1G and rebooted the server, all seems to be working....

[Markku]

Great, just remember to apply the Zabbix server template (if it not assigned yet) so that you get metrics for these cache utilizations, they are very useful.

Markku

[akeccs]

Yes I have applied the template. Thank you for your time, much appreciated.