Zabbix server version: 5.0.11
Database: MySQL Ver 8.0.21
CentOS 8
RAM utilization doesn't go over 50%
CPU utilization doesn't go over 35%
ConfigurationCache and ValueCache don't go over 30%
I've been experiencing problems with random restarts of the service "zabbix_server".
There seems to be no correlation with anything particular. I've checked all sorts of values and data for cache, utilization, housekeeper etc.
From zabbix_server log:
3905088:20210505:143001.115 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3905088:20210505:143001.116 ====== Fatal information: ======
3905088:20210505:143001.116 Program counter: 0x7f25a50f2385
3905088:20210505:143001.116 === Registers: ===
3905088:20210505:143001.116 r8 = 7f2560b2cae0 = 139798512847584 = 139798512847584
3905088:20210505:143001.116 r9 = 0 = 0 = 0
3905088:20210505:143001.116 r10 = 0 = 0 = 0
3905088:20210505:143001.116 r11 = 7f2560b2c920 = 139798512847136 = 139798512847136
3905088:20210505:143001.116 r12 = 0 = 0 = 0
3905088:20210505:143001.116 r13 = 0 = 0 = 0
3905088:20210505:143001.116 r14 = 0 = 0 = 0
3905088:20210505:143001.116 r15 = 0 = 0 = 0
3905088:20210505:143001.116 rdi = 7f2560b2cae0 = 139798512847584 = 139798512847584
3905088:20210505:143001.116 rsi = a = 10 = 10
3905088:20210505:143001.116 rbp = 7f2560b2efc0 = 139798512857024 = 139798512857024
3905088:20210505:143001.116 rbx = 0 = 0 = 0
3905088:20210505:143001.116 rdx = 0 = 0 = 0
3905088:20210505:143001.116 rax = 7f2560b2cc20 = 139798512847904 = 139798512847904
3905088:20210505:143001.116 rcx = 7f2560b2ed30 = 139798512856368 = 139798512856368
3905088:20210505:143001.116 rsp = 7f2560b2c7c0 = 139798512846784 = 139798512846784
3905088:20210505:143001.116 rip = 7f25a50f2385 = 139799659750277 = 139799659750277
3905088:20210505:143001.116 efl = 10206 = 66054 = 66054
3905088:20210505:143001.116 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
3905088:20210505:143001.116 err = 4 = 4 = 4
3905088:20210505:143001.116 trapno = e = 14 = 14
3905088:20210505:143001.116 oldmask = fffffffe3e7bb207 = 18446744066167910919 = -7541640697
3905088:20210505:143001.116 cr2 = 30 = 48 = 48
3905088:20210505:143001.116 === Backtrace: ===
3905088:20210505:143001.116 13: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](zbx_backtrace+0x3f) [0x5570302ef4d0]
3905088:20210505:143001.116 12: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](zbx_log_fatal_info+0x141) [0x5570302ef72e]
3905088:20210505:143001.116 11: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](+0x1d3f07) [0x5570302eff07]
3905088:20210505:143001.116 10: /lib64/libpthread.so.0(+0x12b20) [0x7f25a50f3b20]
3905088:20210505:143001.116 9: /lib64/libpthread.so.0(+0x11385) [0x7f25a50f2385]
3905088:20210505:143001.116 8: /lib64/libgcc_s.so.1(+0x10bde) [0x7f259f846bde]
3905088:20210505:143001.116 7: /lib64/libgcc_s.so.1(_Unwind_ForcedUnwind+0x130) [0x7f259f847250]
3905088:20210505:143001.116 6: /lib64/libpthread.so.0(__pthread_unwind+0x46) [0x7f25a50f2456]
3905088:20210505:143001.116 5: /lib64/libpthread.so.0(+0x93db) [0x7f25a50ea3db]
3905088:20210505:143001.116 4: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(SltsqSigFunc+0x5a) [0x7f256a69556a]
3905088:20210505:143001.116 3: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(+0xacc09) [0x7f256a697c09]
3905088:20210505:143001.116 2: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(sslsshandler+0x5e) [0x7f256a6979de]
3905088:20210505:143001.116 1: /lib64/libpthread.so.0(+0x12b20) [0x7f25a50f3b20]
3905088:20210505:143001.116 0: /lib64/libc.so.6(clone+0x35) [0x7f25a334df15]
3905088:20210505:143001.116 === Memory map: ===
.
.
.
3905088:20210505:143001.118 ================================
3904880:20210505:143001.243 One child process died (PID:3905088,exitcode/signal:1). Exiting ...
zabbix_server [3904880]: Error waiting for process with PID 3905088: [10] No child processes
3904880:20210505:143001.376 syncing history data...
3904880:20210505:143001.426 syncing history data... 100.000000%
3904880:20210505:143001.426 syncing history data done
3904880:20210505:143001.426 syncing trend data...
3904880:20210505:143013.246 syncing trend data done
3904880:20210505:143013.264 Zabbix Server stopped. Zabbix 5.0.11 (revision 15ae5548ce).
3937542:20210505:143023.344 Starting Zabbix Server. Zabbix 5.0.11 (revision 15ae5548ce).
3937542:20210505:143023.344 ****** Enabled features ******
3937542:20210505:143023.344 SNMP monitoring: YES
3937542:20210505:143023.344 IPMI monitoring: YES
3937542:20210505:143023.344 Web monitoring: YES
3937542:20210505:143023.344 VMware monitoring: YES
3937542:20210505:143023.344 SMTP authentication: YES
3937542:20210505:143023.344 ODBC: YES
3937542:20210505:143023.344 SSH support: YES
3937542:20210505:143023.344 IPv6 support: YES
3937542:20210505:143023.344 TLS support: YES
3937542:20210505:143023.344 ******************************
3937542:20210505:143023.344 using configuration file: /etc/zabbix/zabbix_server.conf
3937542:20210505:143023.348 current database version (mandatory/optional): 05000000/05000002
3937542:20210505:143023.348 required mandatory version: 05000000
3937542:20210505:143023.359 server #0 started [main process]
3937543:20210505:143023.360 server #1 started [configuration syncer #1]
3937544:20210505:143026.964 server #2 started [ipmi manager #1]
3937545:20210505:143026.964 server #3 started [housekeeper #1]
3937546:20210505:143026.965 server #4 started [timer #1]
3937547:20210505:143026.965 server #5 started [http poller #1]
3937548:20210505:143026.965 server #6 started [http poller #2]
3937549:20210505:143026.966 server #7 started [discoverer #1]
3937550:20210505:143026.966 server #8 started [discoverer #2]
3937551:20210505:143026.966 server #9 started [discoverer #3]
3937552:20210505:143026.966 server #10 started [discoverer #4]
There is nothing more in the log file, no information about increasing values in configuration file or any problem in the configuration of zabbix, mysql etc.
It just starts collecting data after restarting and some time later restarts again.
Also there is nothing specific in the log file just before it crashes, simply it collects data.
2254607:20210504:100501.135 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2409940:20210504:120001.343 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2516060:20210504:194001.034 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2939131:20210504:201001.209 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2967219:20210505:001502.879 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3193766:20210505:001741.000 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3196614:20210505:001816.179 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3197052:20210505:002001.392 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3198524:20210505:002443.927 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3202245:20210505:002653.371 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3204212:20210505:002752.114 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3205397:20210505:020500.915 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3295792:20210505:035501.277 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3396508:20210505:040601.175 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3407131:20210505:045001.110 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3447592:20210505:075501.598 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3617964:20210505:080500.996 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3627872:20210505:082501.171 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3645622:20210505:083500.757 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3655326:20210505:090001.117 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3672719:20210505:111001.131 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3885421:20210505:135501.018 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3905088:20210505:143001.115 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
What could be the problem?
Please note that I am not an advanced Linux or Zabbix Server Administrator but I've tried to check all possible things and have not found any fix for this issue yet.
I'll be grateful for any suggestions. Thank you in advance,
Seri.
Database: MySQL Ver 8.0.21
CentOS 8
RAM utilization doesn't go over 50%
CPU utilization doesn't go over 35%
ConfigurationCache and ValueCache don't go over 30%
I've been experiencing problems with random restarts of the service "zabbix_server".
There seems to be no correlation with anything particular. I've checked all sorts of values and data for cache, utilization, housekeeper etc.
From zabbix_server log:
3905088:20210505:143001.115 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3905088:20210505:143001.116 ====== Fatal information: ======
3905088:20210505:143001.116 Program counter: 0x7f25a50f2385
3905088:20210505:143001.116 === Registers: ===
3905088:20210505:143001.116 r8 = 7f2560b2cae0 = 139798512847584 = 139798512847584
3905088:20210505:143001.116 r9 = 0 = 0 = 0
3905088:20210505:143001.116 r10 = 0 = 0 = 0
3905088:20210505:143001.116 r11 = 7f2560b2c920 = 139798512847136 = 139798512847136
3905088:20210505:143001.116 r12 = 0 = 0 = 0
3905088:20210505:143001.116 r13 = 0 = 0 = 0
3905088:20210505:143001.116 r14 = 0 = 0 = 0
3905088:20210505:143001.116 r15 = 0 = 0 = 0
3905088:20210505:143001.116 rdi = 7f2560b2cae0 = 139798512847584 = 139798512847584
3905088:20210505:143001.116 rsi = a = 10 = 10
3905088:20210505:143001.116 rbp = 7f2560b2efc0 = 139798512857024 = 139798512857024
3905088:20210505:143001.116 rbx = 0 = 0 = 0
3905088:20210505:143001.116 rdx = 0 = 0 = 0
3905088:20210505:143001.116 rax = 7f2560b2cc20 = 139798512847904 = 139798512847904
3905088:20210505:143001.116 rcx = 7f2560b2ed30 = 139798512856368 = 139798512856368
3905088:20210505:143001.116 rsp = 7f2560b2c7c0 = 139798512846784 = 139798512846784
3905088:20210505:143001.116 rip = 7f25a50f2385 = 139799659750277 = 139799659750277
3905088:20210505:143001.116 efl = 10206 = 66054 = 66054
3905088:20210505:143001.116 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
3905088:20210505:143001.116 err = 4 = 4 = 4
3905088:20210505:143001.116 trapno = e = 14 = 14
3905088:20210505:143001.116 oldmask = fffffffe3e7bb207 = 18446744066167910919 = -7541640697
3905088:20210505:143001.116 cr2 = 30 = 48 = 48
3905088:20210505:143001.116 === Backtrace: ===
3905088:20210505:143001.116 13: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](zbx_backtrace+0x3f) [0x5570302ef4d0]
3905088:20210505:143001.116 12: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](zbx_log_fatal_info+0x141) [0x5570302ef72e]
3905088:20210505:143001.116 11: /usr/sbin/zabbix_server: poller #156 [got 0 values in 0.004237 sec, getting values](+0x1d3f07) [0x5570302eff07]
3905088:20210505:143001.116 10: /lib64/libpthread.so.0(+0x12b20) [0x7f25a50f3b20]
3905088:20210505:143001.116 9: /lib64/libpthread.so.0(+0x11385) [0x7f25a50f2385]
3905088:20210505:143001.116 8: /lib64/libgcc_s.so.1(+0x10bde) [0x7f259f846bde]
3905088:20210505:143001.116 7: /lib64/libgcc_s.so.1(_Unwind_ForcedUnwind+0x130) [0x7f259f847250]
3905088:20210505:143001.116 6: /lib64/libpthread.so.0(__pthread_unwind+0x46) [0x7f25a50f2456]
3905088:20210505:143001.116 5: /lib64/libpthread.so.0(+0x93db) [0x7f25a50ea3db]
3905088:20210505:143001.116 4: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(SltsqSigFunc+0x5a) [0x7f256a69556a]
3905088:20210505:143001.116 3: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(+0xacc09) [0x7f256a697c09]
3905088:20210505:143001.116 2: /usr/lib/oracle/21/client64/lib/libclntshcore.so.21.1(sslsshandler+0x5e) [0x7f256a6979de]
3905088:20210505:143001.116 1: /lib64/libpthread.so.0(+0x12b20) [0x7f25a50f3b20]
3905088:20210505:143001.116 0: /lib64/libc.so.6(clone+0x35) [0x7f25a334df15]
3905088:20210505:143001.116 === Memory map: ===
.
.
.
3905088:20210505:143001.118 ================================
3904880:20210505:143001.243 One child process died (PID:3905088,exitcode/signal:1). Exiting ...
zabbix_server [3904880]: Error waiting for process with PID 3905088: [10] No child processes
3904880:20210505:143001.376 syncing history data...
3904880:20210505:143001.426 syncing history data... 100.000000%
3904880:20210505:143001.426 syncing history data done
3904880:20210505:143001.426 syncing trend data...
3904880:20210505:143013.246 syncing trend data done
3904880:20210505:143013.264 Zabbix Server stopped. Zabbix 5.0.11 (revision 15ae5548ce).
3937542:20210505:143023.344 Starting Zabbix Server. Zabbix 5.0.11 (revision 15ae5548ce).
3937542:20210505:143023.344 ****** Enabled features ******
3937542:20210505:143023.344 SNMP monitoring: YES
3937542:20210505:143023.344 IPMI monitoring: YES
3937542:20210505:143023.344 Web monitoring: YES
3937542:20210505:143023.344 VMware monitoring: YES
3937542:20210505:143023.344 SMTP authentication: YES
3937542:20210505:143023.344 ODBC: YES
3937542:20210505:143023.344 SSH support: YES
3937542:20210505:143023.344 IPv6 support: YES
3937542:20210505:143023.344 TLS support: YES
3937542:20210505:143023.344 ******************************
3937542:20210505:143023.344 using configuration file: /etc/zabbix/zabbix_server.conf
3937542:20210505:143023.348 current database version (mandatory/optional): 05000000/05000002
3937542:20210505:143023.348 required mandatory version: 05000000
3937542:20210505:143023.359 server #0 started [main process]
3937543:20210505:143023.360 server #1 started [configuration syncer #1]
3937544:20210505:143026.964 server #2 started [ipmi manager #1]
3937545:20210505:143026.964 server #3 started [housekeeper #1]
3937546:20210505:143026.965 server #4 started [timer #1]
3937547:20210505:143026.965 server #5 started [http poller #1]
3937548:20210505:143026.965 server #6 started [http poller #2]
3937549:20210505:143026.966 server #7 started [discoverer #1]
3937550:20210505:143026.966 server #8 started [discoverer #2]
3937551:20210505:143026.966 server #9 started [discoverer #3]
3937552:20210505:143026.966 server #10 started [discoverer #4]
There is nothing more in the log file, no information about increasing values in configuration file or any problem in the configuration of zabbix, mysql etc.
It just starts collecting data after restarting and some time later restarts again.
Also there is nothing specific in the log file just before it crashes, simply it collects data.
2254607:20210504:100501.135 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2409940:20210504:120001.343 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2516060:20210504:194001.034 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2939131:20210504:201001.209 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
2967219:20210505:001502.879 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3193766:20210505:001741.000 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3196614:20210505:001816.179 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3197052:20210505:002001.392 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3198524:20210505:002443.927 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3202245:20210505:002653.371 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3204212:20210505:002752.114 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3205397:20210505:020500.915 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3295792:20210505:035501.277 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3396508:20210505:040601.175 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3407131:20210505:045001.110 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3447592:20210505:075501.598 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3617964:20210505:080500.996 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3627872:20210505:082501.171 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3645622:20210505:083500.757 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3655326:20210505:090001.117 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3672719:20210505:111001.131 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3885421:20210505:135501.018 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
3905088:20210505:143001.115 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x30]. Crashing ...
What could be the problem?
Please note that I am not an advanced Linux or Zabbix Server Administrator but I've tried to check all possible things and have not found any fix for this issue yet.
I'll be grateful for any suggestions. Thank you in advance,
Seri.
Comment