Trigger action triggering endlessly
Hello everyone,
I have a small problem. During an apt update command an event of a host was triggered. This event starts a webhook with a JSON payload as an action. The problem now is that this webhook is triggered permanently every 10s.
What I have already tried:
- Restarting the Zabbix server
- Deactivating the trigger on the host
- Deactivating the item on the host
- Deactivating the media type
- Deactivating the trigger action
But the webhook is still triggered. The only current solution is to stop the Zabbix server. As soon as it is restarted, the webhooks are triggered again in a continuous loop.
I have also already removed the corresponding entry in the "escalations" table in the database (inspired by this post: https://www.zabbix.com/forum/zabbix-...973#post194973)
Can anyone give me some help on how to fix the problem?
I am currently using version 6.4.9 with PostgreSQL.
Log excerpt from the start to the triggered webhook:
I have a small problem. During an apt update command an event of a host was triggered. This event starts a webhook with a JSON payload as an action. The problem now is that this webhook is triggered permanently every 10s.
What I have already tried:
- Restarting the Zabbix server
- Deactivating the trigger on the host
- Deactivating the item on the host
- Deactivating the media type
- Deactivating the trigger action
But the webhook is still triggered. The only current solution is to stop the Zabbix server. As soon as it is restarted, the webhooks are triggered again in a continuous loop.
I have also already removed the corresponding entry in the "escalations" table in the database (inspired by this post: https://www.zabbix.com/forum/zabbix-...973#post194973)
Can anyone give me some help on how to fix the problem?
I am currently using version 6.4.9 with PostgreSQL.
Log excerpt from the start to the triggered webhook:
Code:
6781:20231202:234014.121 Starting Zabbix Server. Zabbix 6.4.9 (revision 6186d441314).
6781:20231202:234014.121 ****** Enabled features ******
6781:20231202:234014.121 SNMP monitoring: YES
6781:20231202:234014.121 IPMI monitoring: YES
6781:20231202:234014.121 Web monitoring: YES
6781:20231202:234014.121 VMware monitoring: YES
6781:20231202:234014.121 SMTP authentication: YES
6781:20231202:234014.121 ODBC: YES
6781:20231202:234014.121 SSH support: YES
6781:20231202:234014.121 IPv6 support: YES
6781:20231202:234014.121 TLS support: YES
6781:20231202:234014.121 ******************************
6781:20231202:234014.121 using configuration file: /etc/zabbix/zabbix_server.conf
6781:20231202:234014.251 current database version (mandatory/optional): 06040000/06040026
6781:20231202:234014.251 required mandatory version: 06040000
6791:20231202:234014.363 starting HA manager
6791:20231202:234014.391 HA manager started in active mode
6781:20231202:234014.393 server #0 started [main process]
6793:20231202:234014.394 server #1 started [service manager #1]
6794:20231202:234014.395 server #2 started [configuration syncer #1]
6798:20231202:234014.531 server #3 started [alert manager #1]
6799:20231202:234014.531 server #4 started [alerter #1]
6800:20231202:234014.533 server #5 started [alerter #2]
6804:20231202:234014.535 server #8 started [lld manager #1]
6805:20231202:234014.536 server #9 started [lld worker #1]
6802:20231202:234014.536 server #6 started [alerter #3]
6806:20231202:234014.537 server #10 started [lld worker #2]
6803:20231202:234014.538 server #7 started [preprocessing manager #1]
6807:20231202:234014.539 server #11 started [housekeeper #1]
6809:20231202:234014.544 server #13 started [http poller #1]
6803:20231202:234014.549 [3] thread started [preprocessing worker #3]
6823:20231202:234014.556 server #20 started [proxy poller #1]
6815:20231202:234014.558 server #16 started [history syncer #2]
6803:20231202:234014.569 [1] thread started [preprocessing worker #1]
6808:20231202:234014.569 server #12 started [timer #1]
6812:20231202:234014.570 server #14 started [discoverer #1]
6834:20231202:234014.571 server #30 started [trapper #2]
6824:20231202:234014.580 server #21 started [self-monitoring #1]
6840:20231202:234014.580 server #32 started [trapper #4]
6821:20231202:234014.581 server #18 started [history syncer #4]
6826:20231202:234014.582 server #23 started [poller #1]
6827:20231202:234014.585 server #24 started [poller #2]
6838:20231202:234014.587 server #31 started [trapper #3]
6829:20231202:234014.591 server #26 started [poller #4]
6830:20231202:234014.591 server #27 started [poller #5]
6820:20231202:234014.595 server #17 started [history syncer #3]
6803:20231202:234014.596 [2] thread started [preprocessing worker #2]
6814:20231202:234014.596 server #15 started [history syncer #1]
6841:20231202:234014.598 server #33 started [trapper #5]
6832:20231202:234014.600 server #28 started [unreachable poller #1]
6822:20231202:234014.603 server #19 started [escalator #1]
6833:20231202:234014.606 server #29 started [trapper #1]
6825:20231202:234014.608 server #22 started [task manager #1]
6828:20231202:234014.610 server #25 started [poller #3]
6847:20231202:234014.617 server #37 started [history poller #2]
6854:20231202:234014.635 server #40 started [history poller #5]
6860:20231202:234014.676 server #42 started [trigger housekeeper #1]
6861:20231202:234014.677 server #43 started [odbc poller #1]
6844:20231202:234014.687 server #35 started [alert syncer #1]
6849:20231202:234014.688 server #38 started [history poller #3]
6859:20231202:234014.691 server #41 started [availability manager #1]
6850:20231202:234014.692 server #39 started [history poller #4]
6843:20231202:234014.693 server #34 started [icmp pinger #1]
6845:20231202:234014.694 server #36 started [history poller #1]
6799:20231202:234015.731 [ Webhook ] Executed with params: {"alert_message":"**Startzeit:** 22:33:09\r\n**Auswirkung:** Warning","alert_subject":"PI-AM03 - High CPU utilization (over 90% for 5m)","endpoint":"https://API_SERVER/779535285830418432/T3OHm5sl1cndkWrrYKU2ABkVn9gns18UJnr6Tb3GYhr2d13DJFPOD1dvAitmU0P2e6u3","event_value":"1","event_severity":"2"}
6799:20231202:234015.731 [ Webhook ] Severity: 2
6799:20231202:234015.731 [ Webhook ] Content: ⚠️ __**PI-AM03 - High CPU utilization (over 90% for 5m)**__ ⚠️
__**Beschreibung:**__
**Startzeit:** 22:33:09
**Auswirkung:** Warning
-------------------------
6799:20231202:234015.732 [ Webhook ] Request Body: {"content":" ⚠️ __**PI-AM03 - High CPU utilization (over 90% for 5m)**__ ⚠️ \n__**Beschreibung:**__\n**Startzeit:** 22:33:09\r\n**Auswirkung:** Warning\n-------------------------\n"}
6799:20231202:234016.041 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
6799:20231202:234016.041 ====== Fatal information: ======
6799:20231202:234016.041 Program counter: 0x556cf64f5cd4
6799:20231202:234016.041 === Registers: ===
6799:20231202:234016.041 r8 = 556cf7cfbc20 = 93926502415392 = 93926502415392
6799:20231202:234016.041 r9 = 556cf7cfbc20 = 93926502415392 = 93926502415392
6799:20231202:234016.041 r10 = 7f0b5bba43f0 = 139686760301552 = 139686760301552
6799:20231202:234016.041 r11 = c407ed6728def3a5 = 14125519783593964453 = -4321224290115587163
6799:20231202:234016.041 r12 = 556cf7cec818 = 93926502352920 = 93926502352920
6799:20231202:234016.041 r13 = 556cf7cec810 = 93926502352912 = 93926502352912
6799:20231202:234016.041 r14 = fffffffffffff640 = 18446744073709549120 = -2496
6799:20231202:234016.041 r15 = 556cf7c07500 = 93926501414144 = 93926501414144
6799:20231202:234016.041 rdi = 0 = 0 = 0
6799:20231202:234016.041 rsi = 556cf7bb3010 = 93926501068816 = 93926501068816
6799:20231202:234016.041 rbp = 556cf7cfbc20 = 93926502415392 = 93926502415392
6799:20231202:234016.041 rbx = 556cf7cec800 = 93926502352896 = 93926502352896
6799:20231202:234016.041 rdx = 556cf7cfb = 22931275003 = 22931275003
6799:20231202:234016.041 rax = 0 = 0 = 0
6799:20231202:234016.041 rcx = 0 = 0 = 0
6799:20231202:234016.041 rsp = 7ffff9e11138 = 140737385664824 = 140737385664824
6799:20231202:234016.041 rip = 556cf64f5cd4 = 93926477225172 = 93926477225172
6799:20231202:234016.041 efl = 10206 = 66054 = 66054
6799:20231202:234016.041 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
6799:20231202:234016.042 err = 4 = 4 = 4
6799:20231202:234016.042 trapno = e = 14 = 14
6799:20231202:234016.042 oldmask = 0 = 0 = 0
6799:20231202:234016.042 cr2 = 0 = 0 = 0
6799:20231202:234016.042 === Backtrace: ===
6799:20231202:234016.043 22: /usr/sbin/zabbix_server: alerter #1 started(zbx_backtrace+0x5d) [0x556cf64d689d]
6799:20231202:234016.043 21: /usr/sbin/zabbix_server: alerter #1 started(zbx_log_fatal_info+0x2d5) [0x556cf64d6c85]
6799:20231202:234016.043 20: /usr/sbin/zabbix_server: alerter #1 started(+0x31c2ba) [0x556cf64d72ba]
6799:20231202:234016.043 19: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0b5b9cc520]
6799:20231202:234016.043 18: /usr/sbin/zabbix_server: alerter #1 started(zbx_replace_invalid_utf8+0x4) [0x556cf64f5cd4]
6799:20231202:234016.043 17: /usr/sbin/zabbix_server: alerter #1 started(zbx_http_convert_to_utf8+0xf1) [0x556cf643b391]
6799:20231202:234016.043 16: /usr/sbin/zabbix_server: alerter #1 started(+0x12000b) [0x556cf62db00b]
6799:20231202:234016.043 15: /usr/sbin/zabbix_server: alerter #1 started(+0x129cc4) [0x556cf62e4cc4]
6799:20231202:234016.043 14: /usr/sbin/zabbix_server: alerter #1 started(+0x863e0) [0x556cf62413e0]
6799:20231202:234016.043 13: /usr/sbin/zabbix_server: alerter #1 started(+0x128e5b) [0x556cf62e3e5b]
6799:20231202:234016.043 12: /usr/sbin/zabbix_server: alerter #1 started(+0x129ead) [0x556cf62e4ead]
6799:20231202:234016.043 11: /usr/sbin/zabbix_server: alerter #1 started(+0x130318) [0x556cf62eb318]
6799:20231202:234016.044 10: /usr/sbin/zabbix_server: alerter #1 started(+0x12c06f) [0x556cf62e706f]
6799:20231202:234016.044 9: /usr/sbin/zabbix_server: alerter #1 started(duk_pcall+0x6f) [0x556cf62f308f]
6799:20231202:234016.044 8: /usr/sbin/zabbix_server: alerter #1 started(zbx_es_execute+0x1fa) [0x556cf62d99ba]
6799:20231202:234016.044 7: /usr/sbin/zabbix_server: alerter #1 started(zbx_alerter_thread+0x7a5) [0x556cf62555b5]
6799:20231202:234016.044 6: /usr/sbin/zabbix_server: alerter #1 started(zbx_thread_start+0x24) [0x556cf6412624]
6799:20231202:234016.044 5: /usr/sbin/zabbix_server: alerter #1 started(+0x903e2) [0x556cf624b3e2]
6799:20231202:234016.044 4: /usr/sbin/zabbix_server: alerter #1 started(MAIN_ZABBIX_ENTRY+0xef9) [0x556cf624c549]
6799:20231202:234016.044 3: /usr/sbin/zabbix_server: alerter #1 started(main+0x257) [0x556cf6240f57]
6799:20231202:234016.044 2: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f0b5b9b3d90]
6799:20231202:234016.044 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f0b5b9b3e40]
6799:20231202:234016.044 0: /usr/sbin/zabbix_server: alerter #1 started(_start+0x25) [0x556cf62482a5]
6799:20231202:234016.044 === Memory map: ===