Hi All,
Recently we have updated our Zabbix dev server to 6.4.9 from 6.4.8. Its running on RHEL8, mariadb as the db and apache for the webserver.
Now when a Webbook is triggered, specifically our MS Teams integration it crashes the Zabbix-Server Service. The only way to get the zabbix-server service running correctly again is to re-boot the whole VM.
I am at a loss to workout what is causing this. It crashes the service as soon as a message is sent to teams, the message is getting sent to teams as I can see it in the teams channel. I can replicate the crash by using the "Test" option on the Teams Webhook in, media types.
The logs on the server aren't giving me a whole lot to go on. Even when changing the debugging to level 4. Below is a small output of when it crashes.
2718:20231208:134213.134 vmware_vm_snapshot_collect() empty list of fileKey
2693:20231208:134241.730 Got signal [signal:11(SIGSEGV),reason:1,refaddrnil)]. Crashing ...
2693:20231208:134241.730 ====== Fatal information: ======
2693:20231208:134241.730 Program counter: 0x558bd9f6b7f3
2693:20231208:134241.730 === Registers: ===
2693:20231208:134241.730 r8 = 558bdba3d300 = 94059173761792 = 94059173761792
2693:20231208:134241.730 r9 = 558bdb979010 = 94059172958224 = 94059172958224
2693:20231208:134241.730 r10 = 8 = 8 = 8
2693:20231208:134241.730 r11 = 0 = 0 = 0
2693:20231208:134241.730 r12 = 558bdb9d2788 = 94059173324680 = 94059173324680
2693:20231208:134241.730 r13 = 558bdb9d2780 = 94059173324672 = 94059173324672
2693:20231208:134241.730 r14 = 558bda20060e = 94059148346894 = 94059148346894
2693:20231208:134241.730 r15 = 558bdb9d8170 = 94059173347696 = 94059173347696
2693:20231208:134241.730 rdi = 0 = 0 = 0
2693:20231208:134241.730 rsi = 0 = 0 = 0
2693:20231208:134241.730 rbp = 558bdba3d300 = 94059173761792 = 94059173761792
2693:20231208:134241.730 rbx = 558bdb9d2770 = 94059173324656 = 94059173324656
2693:20231208:134241.730 rdx = 558bdb979010 = 94059172958224 = 94059172958224
2693:20231208:134241.730 rax = 0 = 0 = 0
2693:20231208:134241.730 rcx = 1 = 1 = 1
2693:20231208:134241.730 rsp = 7fffc629d688 = 140736518018696 = 140736518018696
2693:20231208:134241.730 rip = 558bd9f6b7f3 = 94059145639923 = 94059145639923
2693:20231208:134241.730 efl = 10202 = 66050 = 66050
2693:20231208:134241.730 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
2693:20231208:134241.730 err = 4 = 4 = 4
2693:20231208:134241.730 trapno = e = 14 = 14
2693:20231208:134241.731 oldmask = 0 = 0 = 0
2693:20231208:134241.731 cr2 = 0 = 0 = 0
2693:20231208:134241.731 === Backtrace: ===
2693:20231208:134241.732 22: /usr/sbin/zabbix_server: alerter #1 started(zbx_backtrace+0x41) [0x558bd9f8a4c
Does anyone have an ideas of how I can better troubleshoot this or has anyone else experienced this in their environment?
Currently running out prod version on 6.4.8 and the Teams webhook integration is working fine.
I also tried upgrading our dev instance to Alpha V7 and had exactly the same result.
Many thanks
Recently we have updated our Zabbix dev server to 6.4.9 from 6.4.8. Its running on RHEL8, mariadb as the db and apache for the webserver.
Now when a Webbook is triggered, specifically our MS Teams integration it crashes the Zabbix-Server Service. The only way to get the zabbix-server service running correctly again is to re-boot the whole VM.
I am at a loss to workout what is causing this. It crashes the service as soon as a message is sent to teams, the message is getting sent to teams as I can see it in the teams channel. I can replicate the crash by using the "Test" option on the Teams Webhook in, media types.
The logs on the server aren't giving me a whole lot to go on. Even when changing the debugging to level 4. Below is a small output of when it crashes.
2718:20231208:134213.134 vmware_vm_snapshot_collect() empty list of fileKey
2693:20231208:134241.730 Got signal [signal:11(SIGSEGV),reason:1,refaddrnil)]. Crashing ...
2693:20231208:134241.730 ====== Fatal information: ======
2693:20231208:134241.730 Program counter: 0x558bd9f6b7f3
2693:20231208:134241.730 === Registers: ===
2693:20231208:134241.730 r8 = 558bdba3d300 = 94059173761792 = 94059173761792
2693:20231208:134241.730 r9 = 558bdb979010 = 94059172958224 = 94059172958224
2693:20231208:134241.730 r10 = 8 = 8 = 8
2693:20231208:134241.730 r11 = 0 = 0 = 0
2693:20231208:134241.730 r12 = 558bdb9d2788 = 94059173324680 = 94059173324680
2693:20231208:134241.730 r13 = 558bdb9d2780 = 94059173324672 = 94059173324672
2693:20231208:134241.730 r14 = 558bda20060e = 94059148346894 = 94059148346894
2693:20231208:134241.730 r15 = 558bdb9d8170 = 94059173347696 = 94059173347696
2693:20231208:134241.730 rdi = 0 = 0 = 0
2693:20231208:134241.730 rsi = 0 = 0 = 0
2693:20231208:134241.730 rbp = 558bdba3d300 = 94059173761792 = 94059173761792
2693:20231208:134241.730 rbx = 558bdb9d2770 = 94059173324656 = 94059173324656
2693:20231208:134241.730 rdx = 558bdb979010 = 94059172958224 = 94059172958224
2693:20231208:134241.730 rax = 0 = 0 = 0
2693:20231208:134241.730 rcx = 1 = 1 = 1
2693:20231208:134241.730 rsp = 7fffc629d688 = 140736518018696 = 140736518018696
2693:20231208:134241.730 rip = 558bd9f6b7f3 = 94059145639923 = 94059145639923
2693:20231208:134241.730 efl = 10202 = 66050 = 66050
2693:20231208:134241.730 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
2693:20231208:134241.730 err = 4 = 4 = 4
2693:20231208:134241.730 trapno = e = 14 = 14
2693:20231208:134241.731 oldmask = 0 = 0 = 0
2693:20231208:134241.731 cr2 = 0 = 0 = 0
2693:20231208:134241.731 === Backtrace: ===
2693:20231208:134241.732 22: /usr/sbin/zabbix_server: alerter #1 started(zbx_backtrace+0x41) [0x558bd9f8a4c
Does anyone have an ideas of how I can better troubleshoot this or has anyone else experienced this in their environment?
Currently running out prod version on 6.4.8 and the Teams webhook integration is working fine.
I also tried upgrading our dev instance to Alpha V7 and had exactly the same result.
Many thanks