Zabbix Server zbx_mem_realloc Out of memory error - increasing CacheSize doesn't help

[honzasach]

Hello,
I am new here, but have been running Zabbix Server with a few Proxies for some time.
Zabbix version is 4.4.7. OS is CentOS 7 (centos-release-7-7.1908.0.el7.centos.x86_64) runing on a VM (Vmware).

Now I ran into a problem that Zabbix Server has shut down and is unable to start again due to the known "__zbx_mem_realloc(): out of memory" error.

vi /var/log/zabbix/zabbix_server.log

18482:20210212:124124.408 Starting Zabbix Server. Zabbix 4.4.7 (revision 77fb8c7ee0).
18482:20210212:124124.408 ****** Enabled features ******
18482:20210212:124124.408 SNMP monitoring: YES
18482:20210212:124124.408 IPMI monitoring: YES
18482:20210212:124124.408 Web monitoring: YES
18482:20210212:124124.408 VMware monitoring: YES
18482:20210212:124124.408 SMTP authentication: YES
18482:20210212:124124.408 ODBC: YES
18482:20210212:124124.408 SSH support: YES
18482:20210212:124124.408 IPv6 support: YES
18482:20210212:124124.408 TLS support: YES
18482:20210212:124124.408 ******************************
18482:20210212:124124.408 using configuration file: /etc/zabbix/zabbix_server.conf
18482:20210212:124124.411 current database version (mandatory/optional): 04040000/04040002
18482:20210212:124124.411 required mandatory version: 04040000
18482:20210212:124124.421 server #0 started [main process]
18492:20210212:124124.421 server #1 started [configuration syncer #1]
18492:20210212:124126.356 __mem_malloc: skipped 0 asked 24 skip_min 18446744073709551615 skip_max 0
18492:20210212:124126.356 [file:dbconfig.c,line:94] __zbx_mem_realloc(): out of memory (requested 16 bytes)
18492:20210212:124126.356 [file:dbconfig.c,line:94] __zbx_mem_realloc(): please increase CacheSize configuration parameter
18492:20210212:124126.356 === memory statistics for configuration cache ===
18492:20210212:124126.356 min chunk size: 18446744073709551615 bytes
18492:20210212:124126.356 max chunk size: 0 bytes
18492:20210212:124126.356 memory of total size 33554056 bytes fragmented into 311684 chunks
18492:20210212:124126.356 of those, 0 bytes are in 0 free chunks
18492:20210212:124126.356 of those, 28567128 bytes are in 311684 used chunks
18492:20210212:124126.356 ================================

I have tried to increase not only the CacheSize configuration parameter but also other "CacheSize" parameters.
These are the values now (probably higher than needed, I know):

grep CacheSize -A 1 /etc/zabbix/zabbix_server.conf

### Option: VMwareCacheSize
# Size of VMware cache, in bytes.
--
# VMwareCacheSize=8M
VMwareCacheSize=64MB

--
### Option: CacheSize
# Size of configuration cache, in bytes.
--
# CacheSize=8M
CacheSize=2048M

--
### Option: HistoryCacheSize
# Size of history cache, in bytes.
--
# HistoryCacheSize=16M
HistoryCacheSize=512M

### Option: HistoryIndexCacheSize
# Size of history index cache, in bytes.
--
# HistoryIndexCacheSize=4M
HistoryIndexCacheSize=32M

### Option: TrendCacheSize
# Size of trend cache, in bytes.
--
# TrendCacheSize=4M
TrendCacheSize=128M

### Option: ValueCacheSize
# Size of history value cache, in bytes.
--
# ValueCacheSize=8M
ValueCacheSize=256M

I have also read several posts about the shared memory topic in OS, but to be honest, I am not sure if this could be the case.
Here's the output of the related commands:

sysctl -a|grep shm

kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
vm.hugetlb_shm_group = 0

ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 0 zabbix 600 576 6 dest
0x00000000 32769 zabbix 600 219056 6 dest
0x591459cb 65538 root 600 1000 15

I ran through several forum posts, blog posts, internet websites but all of them are just recommend to increase the CacheSize configuration parameter and start the Zabbix Server again.
Apparently for many of those users, it helped, but not in my case. I am still unable to start the Zabbix server. Even after the whole server reboot.

Any kind of help, suggestion, idea very welcomed.

Thank you in advance.

[tim.mooney]

The part that caught my eye was the minchunksize and maxchunksize values. Min is an enormous value, which might imply an overflow of a signed quantity. Max is 0, which also looks suspicious.

It looks like you've previously modified many of your cache size settings. Were those increased "scientifically", or did someone just change them to larger values assuming that "bigger is better", and that that would help performance? I'm asking because I'm suspicious you've somehow arrived at a group of settings that are hurting your environment more than helping it. I don't know for certain, since I haven't looked into how the various settings interact, but that would be one of the first places I would investigate.

How big is your monitoring environment? What's your new values per second (NVPS)?