Zabbix Server keeps crashing/restarting after update to 5.4

[Lgn]

Hi everyone,

After today's update of zabbix server from 5.2 to 5.4, server keeps crashing and restarting. It's running on CentOS Linux 8 and MySQL 8 with plenty of resources (192 cores and 512GB of memory).


Following messages are showing up in logs:

zabbix_server [1717091]: ERROR [file and function: <variant.c,variant_compare_dbl>, revision:5059e20a75, line:572] Something impossible has just happened.
1717091:20210519:170705.943 === Backtrace: ===
1717091:20210519:170705.944 18: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_backtrace+0x3f) [0x55555561130b]
1717091:20210519:170705.944 17: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x238183) [0x555555638183]
1717091:20210519:170705.944 16: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_variant_compare+0x14e) [0x555555638328]
1717091:20210519:170705.944 15: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1fb0ed) [0x5555555fb0ed]
1717091:20210519:170705.944 14: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1fb34c) [0x5555555fb34c]
1717091:20210519:170705.944 13: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x2011a3) [0x5555556011a3]
1717091:20210519:170705.944 12: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_eval_execute+0x4f) [0x55555560158c]
1717091:20210519:170705.944 11: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1e777a) [0x5555555e777a]
1717091:20210519:170705.944 10: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](evaluate_expressions+0x1dc) [0x5555555e7a58]
1717091:20210519:170705.944 9: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x190f33) [0x555555590f33]
1717091:20210519:170705.944 8: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x19454a) [0x55555559454a]
1717091:20210519:170705.944 7: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_sync_history_cache+0x87) [0x555555594f81]
1717091:20210519:170705.944 6: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](dbsyncer_thread+0x21f) [0x555555465ff2]
1717091:20210519:170705.944 5: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_thread_start+0x37) [0x55555562125c]
1717091:20210519:170705.944 4: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](MAIN_ZABBIX_ENTRY+0xdff) [0x555555454da3]
1717091:20210519:170705.944 3: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](daemon_start+0x2ff) [0x555555610f35]
1717091:20210519:170705.944 2: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](main+0x2f6) [0x555555453ed0]
1717091:20210519:170705.944 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7ffff47cc7b3]
1717091:20210519:170705.944 0: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](_start+0x2e) [0x555555452dce]
1717030:20210519:170705.950 One child process died (PID:1717091,exitcode/signal:1). Exiting ...
zabbix_server [1717030]: Error waiting for process with PID 1717091: [10] No child processes
1717030:20210519:170705.983 syncing history data...
zabbix_server [1717030]: ERROR [file and function: <variant.c,variant_compare_dbl>, revision:5059e20a75, line:572] Something impossible has just happened.
1717030:20210519:170705.999 === Backtrace: ===
1717030:20210519:170706.000 22: /usr/sbin/zabbix_server(zbx_backtrace+0x3f) [0x55555561130b]
1717030:20210519:170706.000 21: /usr/sbin/zabbix_server(+0x238183) [0x555555638183]
1717030:20210519:170706.000 20: /usr/sbin/zabbix_server(zbx_variant_compare+0x14e) [0x555555638328]
1717030:20210519:170706.000 19: /usr/sbin/zabbix_server(+0x1fb0ed) [0x5555555fb0ed]
1717030:20210519:170706.000 18: /usr/sbin/zabbix_server(+0x1fb34c) [0x5555555fb34c]
1717030:20210519:170706.000 17: /usr/sbin/zabbix_server(+0x2011a3) [0x5555556011a3]
1717030:20210519:170706.000 16: /usr/sbin/zabbix_server(zbx_eval_execute+0x4f) [0x55555560158c]
1717030:20210519:170706.000 15: /usr/sbin/zabbix_server(+0x1e777a) [0x5555555e777a]
1717030:20210519:170706.000 14: /usr/sbin/zabbix_server(evaluate_expressions+0x1dc) [0x5555555e7a58]
1717030:20210519:170706.000 13: /usr/sbin/zabbix_server(+0x190f33) [0x555555590f33]
1717030:20210519:170706.000 12: /usr/sbin/zabbix_server(+0x19454a) [0x55555559454a]
1717030:20210519:170706.000 11: /usr/sbin/zabbix_server(+0x194b9a) [0x555555594b9a]
1717030:20210519:170706.000 10: /usr/sbin/zabbix_server(+0x196ff6) [0x555555596ff6]
1717030:20210519:170706.000 9: /usr/sbin/zabbix_server(free_database_cache+0x31) [0x555555597060]
1717030:20210519:170706.000 8: /usr/sbin/zabbix_server(zbx_on_exit+0xcf) [0x5555554553af]
1717030:20210519:170706.000 7: /usr/sbin/zabbix_server(+0x212102) [0x555555612102]
1717030:20210519:170706.000 6: /lib64/libpthread.so.0(+0x12b20) [0x7ffff69b3b20]
1717030:20210519:170706.000 5: /lib64/libpthread.so.0(__wait+0x22) [0x7ffff69b33d2]
1717030:20210519:170706.000 4: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x12f5) [0x555555455299]
1717030:20210519:170706.000 3: /usr/sbin/zabbix_server(daemon_start+0x2ff) [0x555555610f35]
1717030:20210519:170706.000 2: /usr/sbin/zabbix_server(main+0x2f6) [0x555555453ed0]
1717030:20210519:170706.000 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7ffff47cc7b3]
1717030:20210519:170706.000 0: /usr/sbin/zabbix_server(_start+0x2e) [0x555555452dce]
1727931:20210519:170716.028 Starting Zabbix Server. Zabbix 5.4.0 (revision 5059e20a75).

Journalctl output:

May 19 17:02:06 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
May 19 17:02:06 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 6.
May 19 17:02:16 server-hostname systemd[1]: Stopped Zabbix Server.
May 19 17:02:16 server-hostname systemd[1]: Starting Zabbix Server...
May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1715814 which is not our child. We'll most likely not notice when it exits.
May 19 17:02:16 server-hostname systemd[1]: Started Zabbix Server.
May 19 17:02:41 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
May 19 17:02:41 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 7.
May 19 17:02:51 server-hostname systemd[1]: Stopped Zabbix Server.
May 19 17:02:51 server-hostname systemd[1]: Starting Zabbix Server...
May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Supervising process 1717030 which is not our child. We'll most likely not notice when it exits.
May 19 17:02:51 server-hostname systemd[1]: Started Zabbix Server.
May 19 17:07:06 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
May 19 17:07:06 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 8.
May 19 17:07:16 server-hostname systemd[1]: Stopped Zabbix Server.
May 19 17:07:16 server-hostname systemd[1]: Starting Zabbix Server...
May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1727931 which is not our child. We'll most likely not notice when it exits.
May 19 17:07:16 server-hostname systemd[1]: Started Zabbix Server.
May 19 17:07:42 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
May 19 17:07:42 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 9.
May 19 17:07:52 server-hostname systemd[1]: Stopped Zabbix Server.
May 19 17:07:52 server-hostname systemd[1]: Starting Zabbix Server...
May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Supervising process 1729240 which is not our child. We'll most likely not notice when it exits.
May 19 17:07:52 server-hostname systemd[1]: Started Zabbix Server.
May 19 17:12:05 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
May 19 17:12:05 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 10.
May 19 17:12:16 server-hostname systemd[1]: Stopped Zabbix Server.
May 19 17:12:16 server-hostname systemd[1]: Starting Zabbix Server...
May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1740019 which is not our child. We'll most likely not notice when it exits.
May 19 17:12:16 server-hostname systemd[1]: Started Zabbix Server.

[Lgn]

https://support.zabbix.com/browse/ZBX-19420