Hey all,
I recently added a large number of new agents to be monitored by my server. This was the ONLY change made. I was in the process of adding the new hosts to their groups when I got the message at the bottom of my screen "Zabbix Server is not running: the information displayed may not be current."
I quickly logged into my server and did a systemctl status on zabbix-server:
[user@server]# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2022-02-23 14:44:34 MST; 1s ago
Process: 29578 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 29565 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 29567 (code=exited, status=0/SUCCESS)
Feb 23 14:44:34 server kill[29578]: -q, --queue <sig> use sigqueue(2) rather than kill(2)
Feb 23 14:44:34 server kill[29578]: -p, --pid print pids without signaling them
Feb 23 14:44:34 server kill[29578]: -l, --list [=<signal>] list signal names, or convert one to a name
Feb 23 14:44:34 server kill[29578]: -L, --table list signal names and numbers
Feb 23 14:44:34 server kill[29578]: -h, --help display this help and exit
Feb 23 14:44:34 server kill[29578]: -V, --version output version information and exit
Feb 23 14:44:34 server kill[29578]: For more details see kill(1).
Feb 23 14:44:34 server systemd[1]: zabbix-server.service: control process exited, code=exited status=1
Feb 23 14:44:34 server systemd[1]: Unit zabbix-server.service entered failed state.
Feb 23 14:44:34 server systemd[1]: zabbix-server.service failed.
I checked var\log\zabbix\zabbix-server.log and found the following:
[user@server]# tail -n 30 /var/log/zabbix/zabbix_server.log
29882:20220223:144633.194 server #0 started [main process]
29889:20220223:144633.201 server #1 started [configuration syncer #1]
29889:20220223:144633.595 __mem_malloc: skipped 0 asked 24 skip_min 18446744073709551615 skip_max 0
29889:20220223:144633.602 [file:dbconfig.c,line:96] __zbx_mem_realloc(): out of memory (requested 16 bytes)
29889:20220223:144633.615 [file:dbconfig.c,line:96] __zbx_mem_realloc(): please increase CacheSize configuration parameter
29889:20220223:144633.622 === memory statistics for configuration cache ===
29889:20220223:144633.634 min chunk size: 18446744073709551615 bytes
29889:20220223:144633.640 max chunk size: 0 bytes
29889:20220223:144633.646 memory of total size 7200792 bytes fragmented into 74216 chunks
29889:20220223:144633.652 of those, 0 bytes are in 0 free chunks
29889:20220223:144633.664 of those, 7200792 bytes are in 74216 used chunks
29889:20220223:144633.670 of those, 1187440 bytes are used by allocation overhead
29889:20220223:144633.676 ================================
29889:20220223:144633.688 === Backtrace: ===
29889:20220223:144633.695 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x42) [0x55af35babe3e]
29889:20220223:144633.701 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_realloc+0x169) [0x55af35ba6cf6]
29889:20220223:144633.708 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x193146) [0x55af35b6c146]
29889:20220223:144633.719 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x19ea63) [0x55af35b77a63]
29889:20220223:144633.726 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x1367) [0x55af35b78fcd]
29889:20220223:144633.738 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x135) [0x55af35a33547]
29889:20220223:144633.744 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x55af35bbafe5]
29889:20220223:144633.751 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xa8c) [0x55af35a23768]
29889:20220223:144633.757 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x2fb) [0x55af35baba8a]
29889:20220223:144633.763 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x2b0) [0x55af35a22c92]
29889:20220223:144633.770 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcb11fe6555]
29889:20220223:144633.776 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x48d29) [0x55af35a21d29]
29882:20220223:144633.782 One child process died (PID:29889,exitcode/signal:1). Exiting ...
29882:20220223:144633.788 syncing trend data...
29882:20220223:144633.794 syncing trend data done
29882:20220223:144633.801 Zabbix Server stopped. Zabbix 5.0.20 (revision bf8d9f8abb).
Here is my current build:
RHEL7.9
Zabbix 5.0
Mariadb-Server 5.5.68
Httpd 2.4.6-97
Any advice on what to check? Based on the behavior, I think it might be a resource issue?
Im running on a VM through Vsphere
4 CPU
8GB RAM
45GB HDD
I recently added a large number of new agents to be monitored by my server. This was the ONLY change made. I was in the process of adding the new hosts to their groups when I got the message at the bottom of my screen "Zabbix Server is not running: the information displayed may not be current."
I quickly logged into my server and did a systemctl status on zabbix-server:
[user@server]# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2022-02-23 14:44:34 MST; 1s ago
Process: 29578 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
Process: 29565 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 29567 (code=exited, status=0/SUCCESS)
Feb 23 14:44:34 server kill[29578]: -q, --queue <sig> use sigqueue(2) rather than kill(2)
Feb 23 14:44:34 server kill[29578]: -p, --pid print pids without signaling them
Feb 23 14:44:34 server kill[29578]: -l, --list [=<signal>] list signal names, or convert one to a name
Feb 23 14:44:34 server kill[29578]: -L, --table list signal names and numbers
Feb 23 14:44:34 server kill[29578]: -h, --help display this help and exit
Feb 23 14:44:34 server kill[29578]: -V, --version output version information and exit
Feb 23 14:44:34 server kill[29578]: For more details see kill(1).
Feb 23 14:44:34 server systemd[1]: zabbix-server.service: control process exited, code=exited status=1
Feb 23 14:44:34 server systemd[1]: Unit zabbix-server.service entered failed state.
Feb 23 14:44:34 server systemd[1]: zabbix-server.service failed.
I checked var\log\zabbix\zabbix-server.log and found the following:
[user@server]# tail -n 30 /var/log/zabbix/zabbix_server.log
29882:20220223:144633.194 server #0 started [main process]
29889:20220223:144633.201 server #1 started [configuration syncer #1]
29889:20220223:144633.595 __mem_malloc: skipped 0 asked 24 skip_min 18446744073709551615 skip_max 0
29889:20220223:144633.602 [file:dbconfig.c,line:96] __zbx_mem_realloc(): out of memory (requested 16 bytes)
29889:20220223:144633.615 [file:dbconfig.c,line:96] __zbx_mem_realloc(): please increase CacheSize configuration parameter
29889:20220223:144633.622 === memory statistics for configuration cache ===
29889:20220223:144633.634 min chunk size: 18446744073709551615 bytes
29889:20220223:144633.640 max chunk size: 0 bytes
29889:20220223:144633.646 memory of total size 7200792 bytes fragmented into 74216 chunks
29889:20220223:144633.652 of those, 0 bytes are in 0 free chunks
29889:20220223:144633.664 of those, 7200792 bytes are in 74216 used chunks
29889:20220223:144633.670 of those, 1187440 bytes are used by allocation overhead
29889:20220223:144633.676 ================================
29889:20220223:144633.688 === Backtrace: ===
29889:20220223:144633.695 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x42) [0x55af35babe3e]
29889:20220223:144633.701 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_realloc+0x169) [0x55af35ba6cf6]
29889:20220223:144633.708 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x193146) [0x55af35b6c146]
29889:20220223:144633.719 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x19ea63) [0x55af35b77a63]
29889:20220223:144633.726 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x1367) [0x55af35b78fcd]
29889:20220223:144633.738 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x135) [0x55af35a33547]
29889:20220223:144633.744 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x55af35bbafe5]
29889:20220223:144633.751 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xa8c) [0x55af35a23768]
29889:20220223:144633.757 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x2fb) [0x55af35baba8a]
29889:20220223:144633.763 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x2b0) [0x55af35a22c92]
29889:20220223:144633.770 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcb11fe6555]
29889:20220223:144633.776 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x48d29) [0x55af35a21d29]
29882:20220223:144633.782 One child process died (PID:29889,exitcode/signal:1). Exiting ...
29882:20220223:144633.788 syncing trend data...
29882:20220223:144633.794 syncing trend data done
29882:20220223:144633.801 Zabbix Server stopped. Zabbix 5.0.20 (revision bf8d9f8abb).
Here is my current build:
RHEL7.9
Zabbix 5.0
Mariadb-Server 5.5.68
Httpd 2.4.6-97
Any advice on what to check? Based on the behavior, I think it might be a resource issue?
Im running on a VM through Vsphere
4 CPU
8GB RAM
45GB HDD
Comment