Ad Widget

**cyber** · 07-08-2024, 08:43

You have other errors there also

Code:

45689:20240806:194627.863 HA manager started in active mode
45689:20240806:194627.863 cannot write to IPC socket: Broken pipe
....
46051:20240806:194847.963 server #244 started [availability manager #1]
...
45691:20240806:194937.075 cannot connect to availability manager service: Cannot connect to service "availability": [2] No such file or directory.

and after that config syncer exits... Is it always something with availability manager? I think config syncer dying is result not the reason. Maybe you have something about that availability manager in debug log?

**aschild** · 07-08-2024, 19:32

cannot connect to availability manager service: Cannot connect to service "availability": [2] No such file or directory.
Yes, seems to be the problem.
It does not create the zabbix_server_availability.sock file in /tmp for this (or it vanishes almost instantly)

Unfortunately the log does not show much more

158801:20240807:191718.142 server #229 started [availability manager #1]
158801:20240807:191718.142 zbx_setproctitle() title:'availability manager #1 [connecting to the database]'
158801:20240807:191718.143 In DBconnect() flag:0
and then
158541:20240807:191809.026 cannot connect to availability manager service: Cannot connect to service "availability": [2] No such file or directory.
158511:20240807:191809.035 One child process died (PID:158541,exitcode/signal:1). Exiting ...

Selinux is not enabled

**Markku** · 07-08-2024, 19:41

What does AppArmor's sudo aa-status say?

Markku

**aschild** · 08-08-2024, 09:49

Originally posted by Markku

What does AppArmor's sudo aa-status say?

Code:

 aa-status
apparmor module is loaded.
14 profiles are loaded.
14 profiles are in enforce mode.
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/ipsec/charon
   /usr/lib/ipsec/stroke
   /usr/sbin/ntpd
   /{,usr/}sbin/dhclient
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   tcpdump
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/lib/ipsec/charon (910)
   /usr/sbin/ntpd (788)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

**aschild** · 08-08-2024, 10:28

I have attached the DebugLevel=4 for the dying child process 197452

197452:20240808:095041.844 server #2 started [configuration syncer #1]
.
.
.
197452:20240808:095451.581 === memory statistics for configuration cache ===
197452:20240808:095451.581 free chunks of size 48 bytes: 1
197452:20240808:095451.581 free chunks of size >= 256 bytes: 9
197452:20240808:095451.581 min chunk size: 48 bytes
197452:20240808:095451.581 max chunk size: 2110185912 bytes
197452:20240808:095451.581 memory of total size 2142360448 bytes fragmented into 320177 chunks
197452:20240808:095451.581 of those, 2110211256 bytes are in 10 free chunks
197452:20240808:095451.581 of those, 32149192 bytes are in 320167 used chunks
197452:20240808:095451.581 of those, 5122816 bytes are used by allocation overhead
197452:20240808:095451.581 ================================
197452:20240808:095451.581 End of DCsync_configuration()
197452:20240808:095451.581 In DCsync_kvs_paths()
197452:20240808:095451.581 End of DCsync_kvs_paths()
197452:20240808:095451.581 In DCupdate_interfaces_availability()
197452:20240808:095451.581 In DCreset_interfaces_availability()
197452:20240808:095451.581 End of DCreset_interfaces_availability() interfaces:195
197452:20240808:095451.581 In zbx_ipc_socket_open()
197452:20240808:095552.091 End of zbx_ipc_socket_open():FAIL
197452:20240808:095552.091 cannot connect to availability manager service: Cannot connect to service "availability": [2] No such file or directory.
197446:20240808:095552.098 One child process died (PID:197452,exitcode/signal:1). Exiting ...
197446:20240808:095552.099 PROCESS EXIT: 197452
zabbix_server [197446]: Error waiting for process with PID 197452: [10] No child processes

The corresponding availability nanager is pid 197551 and only logs this and does not create the socket file in /tmp

197551:20240808:095502.185 server #69 started [availability manager #1]
197551:20240808:095502.185 zbx_setproctitle() title:'availability manager #1 [connecting to the database]'
197551:20240808:095502.190 In DBconnect() flag:0

zz0.zv08c17ta3zz

Attached Files

197452.zip (5.7 KB, 0 views)

**aschild** · 08-08-2024, 10:48

Ok, got that problem sorted out.

The cause was, that the MariaDB server was only accessible via ipv4 connections, but not ipv6.
Aparrently some parts of zabbix do fallback to ipv4 when using dns name for the dbserver, and others not (or take too long to fall back)

But now it fails very fast in the configuration syncher

Code:

199832:20240808:104553.959 using configuration file: /etc/zabbix/zabbix_server.conf
199832:20240808:104553.975 current database version (mandatory/optional): 06000000/06000045
199832:20240808:104553.975 required mandatory version: 06000000
199832:20240808:104553.977 database could be upgraded to use primary keys in history tables
199832:20240808:104553.978 database is not upgraded to use double precision values
199833:20240808:104553.980 starting HA manager
199833:20240808:104553.988 HA manager started in active mode
199832:20240808:104553.989 server #0 started [main process]
199834:20240808:104553.990 server #1 started [service manager #1]
199835:20240808:104553.991 server #2 started [configuration syncer #1]
199835:20240808:104554.077 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
199835:20240808:104554.077 ====== Fatal information: ======
199835:20240808:104554.077 Program counter: 0x7f6557d4fcb6
199835:20240808:104554.077 === Registers: ===
199835:20240808:104554.077 r8      =                0 =                    0 =                    0
199835:20240808:104554.077 r9      =         7fef4b78 =           2146388856 =           2146388856
199835:20240808:104554.077 r10     =         7fef4b10 =           2146388752 =           2146388752
199835:20240808:104554.077 r11     =     7f6557e233c0 =      140073242866624 =      140073242866624
199835:20240808:104554.077 r12     =     7f6492c45170 =      140069935796592 =      140069935796592
199835:20240808:104554.077 r13     =     7f6492cd1990 =      140069936372112 =      140069936372112
199835:20240808:104554.077 r14     =     7f6492d50488 =      140069936891016 =      140069936891016
199835:20240808:104554.077 r15     =                1 =                    1 =                    1
199835:20240808:104554.077 rdi     =                0 =                    0 =                    0
199835:20240808:104554.077 rsi     =     7f6492d504b0 =      140069936891056 =      140069936891056
199835:20240808:104554.077 rbp     =     7f6492d504b0 =      140069936891056 =      140069936891056
199835:20240808:104554.077 rbx     =                0 =                    0 =                    0
199835:20240808:104554.077 rdx     =                0 =                    0 =                    0
199835:20240808:104554.077 rax     =                0 =                    0 =                    0
199835:20240808:104554.077 rcx     =                0 =                    0 =                    0
199835:20240808:104554.077 rsp     =     7ffcacb04948 =      140723205720392 =      140723205720392
199835:20240808:104554.077 rip     =     7f6557d4fcb6 =      140073242000566 =      140073242000566
199835:20240808:104554.077 efl     =            10293 =                66195 =                66195
199835:20240808:104554.077 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
199835:20240808:104554.077 err     =                4 =                    4 =                    4
199835:20240808:104554.077 trapno  =                e =                   14 =                   14
199835:20240808:104554.078 oldmask =                0 =                    0 =                    0
199835:20240808:104554.078 cr2     =                0 =                    0 =                    0
199835:20240808:104554.078 === Backtrace: ===
199835:20240808:104554.079 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3c) [0x556d3bf1b37c]
199835:20240808:104554.079 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_log_fatal_info+0x28d) [0x556d3bf1b70d]
199835:20240808:104554.079 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1ead36) [0x556d3bf1bd36]
199835:20240808:104554.079 12: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7f6557ce2050]
199835:20240808:104554.079 11: /lib/x86_64-linux-gnu/libc.so.6(+0xa9cb6) [0x7f6557d4fcb6]
199835:20240808:104554.079 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCstrpool_replace+0x3d) [0x556d3bebb9dd]
199835:20240808:104554.079 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x192690) [0x556d3bec3690]
199835:20240808:104554.079 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x2f69) [0x556d3bec7039]
199835:20240808:104554.079 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0xe3) [0x556d3bdb5873]
199835:20240808:104554.079 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x20) [0x556d3bf1d800]
199835:20240808:104554.079 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x768fb) [0x556d3bda78fb]
199835:20240808:104554.079 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xaaf) [0x556d3bda8aef]
199835:20240808:104554.079 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x527) [0x556d3bd9f917]
199835:20240808:104554.079 2: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f6557ccd24a]
199835:20240808:104554.079 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f6557ccd305]
199835:20240808:104554.079 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x21) [0x556d3bda6bc1]
199835:20240808:104554.079 === Memory map: ===
199835:20240808:104554.079 556d3bd31000-556d3bd98000 r--p 00000000 fe:01 148488                     /usr/sbin/zabbix_server
199835:20240808:104554.079 556d3bd98000-556d3bff6000 r-xp 00067000 fe:01 148488                     /usr/sbin/zabbix_server
199835:20240808:104554.079 556d3bff6000-556d3c0cd000 r--p 002c5000 fe:01 148488                     /usr/sbin/zabbix_server
199835:20240808:104554.079 556d3c0cd000-556d3c0d1000 r--p 0039b000 fe:01 148488                     /usr/sbin/zabbix_server
199835:20240808:104554.079 556d3c0d1000-556d3c1a6000 rw-p 0039f000 fe:01 148488                     /usr/sbin/zabbix_server
199835:20240808:104554.079 556d3c1a6000-556d3c1b6000 rw-p 00000000 00:00 0
199835:20240808:104554.079 556d3dbf5000-556d3dc37000 rw-p 00000000 00:00 0                          [heap]
199835:20240808:104554.079 556d3dc37000-556d3dc58000 rw-p 00000000 00:00 0                          [heap]
199835:20240808:104554.079 556d3dc58000-556d3e409000 rw-p 00000000 00:00 0                          [heap]
199835:20240808:104554.079 7f6471adb000-7f6472845000 rw-p 00000000 00:00 0
199835:20240808:104554.079 7f6472845000-7f6472c45000 rw-s 00000000 00:01 1212449                    /SYSV00000000 (deleted)
199835:20240808:104554.079 7f6472c45000-7f6492c45000 rw-s 00000000 00:01 1212448                    /SYSV00000000 (deleted)
199835:20240808:104554.080 7f6492c45000-7f6512c45000 rw-s 00000000 00:01 1212446                    /SYSV00000000 (deleted)

**aschild** · 08-08-2024, 10:56

200296:20240808:105530.003 query [txnlev:0] [select t.timeperiodid,t.timeperiod_type,t.every,t.month,t .dayofweek,t.day,t.start_time,t.period,t.start_dat e,m.maintenanceid from maintenances_windows m,timeperiods t where t.timeperiodid=m.timeperiodid]
200296:20240808:105530.003 query [txnlev:0] [select maintenanceid,groupid from maintenances_groups order by maintenanceid]
200296:20240808:105530.004 query [txnlev:0] [select maintenanceid,hostid from maintenances_hosts order by maintenanceid]
200296:20240808:105530.004 In DCsync_hosts()
200296:20240808:105530.005 End of DCsync_hosts()
200296:20240808:105530.005 In DCsync_host_inventory()
200296:20240808:105530.006 End of DCsync_host_inventory()
200296:20240808:105530.006 In DCsync_hostgroups()
200296:20240808:105530.006 End of DCsync_hostgroups()
200296:20240808:105530.006 In DCsync_hostgroup_hosts()
200296:20240808:105530.006 End of DCsync_hostgroup_hosts()
200296:20240808:105530.006 In DCsync_maintenances()
200296:20240808:105530.006 End of DCsync_maintenances()
200296:20240808:105530.006 In DCsync_maintenance_tags()
200296:20240808:105530.006 End of DCsync_maintenance_tags()
200296:20240808:105530.006 In DCsync_maintenance_groups()
200296:20240808:105530.006 End of DCsync_maintenance_groups()
200296:20240808:105530.006 In DCsync_maintenance_hosts()
200296:20240808:105530.006 End of DCsync_maintenance_hosts()
200296:20240808:105530.006 In DCsync_maintenance_periods()
200296:20240808:105530.006 End of DCsync_maintenance_periods()
200296:20240808:105530.006 query [txnlev:0] [select i.interfaceid,i.hostid,i.type,i.main,i.useip,i.ip, i.dns,i.port,i.available,i.disable_until,i.error,i .errors_from,s.version,s.bulk,s.community,s.securi tyname,s.securitylevel,s.authpassphrase,s.privpass phrase,s.authprotocol,s.privprotocol,s.contextname from interface
200296:20240808:105530.007 query [txnlev:0] [select i.itemid,i.hostid,i.status,i.type,i.value_type,i.k ey_,i.snmp_oid,i.ipmi_sensor,i.delay,i.trapper_hos ts,i.logtimefmt,i.params,ir.state,i.authtype,i.use rname,i.password,i.publickey,i.privatekey,i.flags, i.interfaceid,ir.lastlogsize,ir.mtime,i.history,i. trends,i.invent
200296:20240808:105530.050 query [txnlev:0] [select i.itemid,i.hostid,i.templateid from items i inner join hosts h on i.hostid=h.hostid where h.status=3]
200296:20240808:105530.051 query [txnlev:0] [select i.itemid,i.hostid,i.templateid from items i where i.flags=2]
200296:20240808:105530.053 query [txnlev:0] [select itemid,parent_itemid from item_discovery]
200296:20240808:105530.057 query [txnlev:0] [select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.s tep,h.hostid,pp.error_handler,pp.error_handler_par ams from item_preproc pp,items i,hosts h where pp.itemid=i.itemid and i.hostid=h.hostid and (h.proxy_hostid is null or i.type in (5,15,18)) and h.status in (0,1) and
200296:20240808:105530.059 query [txnlev:0] [select p.item_parameterid,p.itemid,p.name,p.value,i.hosti d from item_parameter p,items i,hosts h where p.itemid=i.itemid and i.hostid=h.hostid and h.status in (0,1) and i.flags<>2 order by p.itemid]
200296:20240808:105530.060 In DCsync_interfaces()
200296:20240808:105530.061 Got signal [signal:11(SIGSEGV),reason:1,refaddr

nil)]. Crashing ...
200296:20240808:105530.061 ====== Fatal information: ======
200296:20240808:105530.061 Program counter: 0x7ff93673acb6
200296:20240808:105530.061 === Registers: ===
200296:20240808:105530.061 r8 = 0 = 0 = 0
200296:20240808:105530.061 r9 = 7fef4b78 = 2146388856 = 2146388856
200296:20240808:105530.061 r10 = 7fef4b10 = 2146388752 = 2146388752
200296:20240808:105530.061 r11 = 7ff93680e3c0 = 140708338000832 = 140708338000832
200296:20240808:105530.061 r12 = 7ff871632170 = 140705030938992 = 140705030938992
200296:20240808:105530.061 r13 = 7ff8716be990 = 140705031514512 = 140705031514512
200296:20240808:105530.061 r14 = 7ff87173d488 = 140705032033416 = 140705032033416
200296:20240808:105530.061 r15 = 1 = 1 = 1
200296:20240808:105530.061 rdi = 0 = 0 = 0
200296:20240808:105530.061 rsi = 7ff87173d4b0 = 140705032033456 = 140705032033456
200296:20240808:105530.061 rbp = 7ff87173d4b0 = 140705032033456 = 140705032033456
200296:20240808:105530.061 rbx = 0 = 0 = 0
200296:20240808:105530.061 rdx = 0 = 0 = 0
200296:20240808:105530.061 rax = 0 = 0 = 0
200296:20240808:105530.061 rcx = 0 = 0 = 0
200296:20240808:105530.061 rsp = 7fffbe2df838 = 140736384071736 = 140736384071736
200296:20240808:105530.061 rip = 7ff93673acb6 = 140708337134774 = 140708337134774
200296:20240808:105530.061 efl = 10293 = 66195 = 66195
200296:20240808:105530.061 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
200296:20240808:105530.061 err = 4 = 4 = 4
200296:20240808:105530.061 trapno = e = 14 = 14
200296:20240808:105530.061 oldmask = 0 = 0 = 0
200296:20240808:105530.061 cr2 = 0 = 0 = 0
200296:20240808:105530.062 === Backtrace: ===
200296:20240808:105530.063 15: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3c) [0x557c9bc3e37c]
200296:20240808:105530.063 14: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_log_fatal_info+0x28d) [0x557c9bc3e70d]
200296:20240808:105530.063 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1ead36) [0x557c9bc3ed36]
200296:20240808:105530.063 12: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7ff9366cd050]
200296:20240808:105530.063 11: /lib/x86_64-linux-gnu/libc.so.6(+0xa9cb6) [0x7ff93673acb6]

**aschild** · 08-08-2024, 16:26

I just did a upgrade to 6.4.17 and the same crash happens there

215524:20240808:162341.552 Got signal [signal:11(SIGSEGV),reason:1,refaddr

nil)]. Crashing ...
215524:20240808:162341.552 ====== Fatal information: ======
215524:20240808:162341.552 Program counter: 0x7f2309e8eea6
215524:20240808:162341.552 === Registers: ===
215524:20240808:162341.552 r8 = 7f2244e5c634 = 139785161524788 = 139785161524788
215524:20240808:162341.552 r9 = 0 = 0 = 0
215524:20240808:162341.552 r10 = 7f2309e8f380 = 139788466844544 = 139788466844544
215524:20240808:162341.552 r11 = 7f2309f643c0 = 139788467717056 = 139788467717056
215524:20240808:162341.553 r12 = 7f2244e5c634 = 139785161524788 = 139785161524788
215524:20240808:162341.553 r13 = 7f2244e5c638 = 139785161524792 = 139785161524792
215524:20240808:162341.553 r14 = 7f2244e5c628 = 139785161524776 = 139785161524776
215524:20240808:162341.553 r15 = 55db1aa5cb60 = 94399533271904 = 94399533271904
215524:20240808:162341.553 rdi = 0 = 0 = 0
215524:20240808:162341.553 rsi = 55db1aa5cb60 = 94399533271904 = 94399533271904
215524:20240808:162341.553 rbp = 55db1aa72eb8 = 94399533362872 = 94399533362872
215524:20240808:162341.553 rbx = 3 = 3 = 3
215524:20240808:162341.553 rdx = 7f2244e5c628 = 139785161524776 = 139785161524776
215524:20240808:162341.553 rax = 0 = 0 = 0
215524:20240808:162341.553 rcx = 20 = 32 = 32
215524:20240808:162341.553 rsp = 7ffd59196f58 = 140726098292568 = 140726098292568
215524:20240808:162341.553 rip = 7f2309e8eea6 = 139788466843302 = 139788466843302
215524:20240808:162341.553 efl = 10283 = 66179 = 66179
215524:20240808:162341.553 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
215524:20240808:162341.553 err = 4 = 4 = 4
215524:20240808:162341.553 trapno = e = 14 = 14
215524:20240808:162341.553 oldmask = 0 = 0 = 0
215524:20240808:162341.553 cr2 = 0 = 0 = 0
215524:20240808:162341.553 === Backtrace: ===
215524:20240808:162341.554 17: /usr/sbin/zabbix_server: trapper #1 [processing data](zbx_backtrace+0x46) [0x55db1996ac26]
215524:20240808:162341.554 16: /usr/sbin/zabbix_server: trapper #1 [processing data](zbx_log_fatal_info+0x295) [0x55db1996afa5]
215524:20240808:162341.554 15: /usr/sbin/zabbix_server: trapper #1 [processing data](+0x2f8556) [0x55db1996b556]
215524:20240808:162341.554 14: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7f2309e23050]
215524:20240808:162341.554 13: /lib/x86_64-linux-gnu/libc.so.6(+0xa7ea6) [0x7f2309e8eea6]
215524:20240808:162341.554 12: /usr/sbin/zabbix_server: trapper #1 [processing data](+0x1bbba3) [0x55db1982eba3]
215524:20240808:162341.554 11: /usr/sbin/zabbix_server: trapper #1 [processing data](DCset_interfaces_availability+0x61) [0x55db19836151]
215524:20240808:162341.555 10: /usr/sbin/zabbix_server: trapper #1 [processing data](zbx_process_proxy_data+0x2194) [0x55db19934984]
215524:20240808:162341.555 9: /usr/sbin/zabbix_server: trapper #1 [processing data](zbx_recv_proxy_data+0x38b) [0x55db1973addb]
215524:20240808:162341.555 8: /usr/sbin/zabbix_server: trapper #1 [processing data](+0xc2d64) [0x55db19735d64]
215524:20240808:162341.555 7: /usr/sbin/zabbix_server: trapper #1 [processing data](trapper_thread+0x320) [0x55db197374e0]
215524:20240808:162341.555 6: /usr/sbin/zabbix_server: trapper #1 [processing data](zbx_thread_start+0x20) [0x55db198b70b0]
215524:20240808:162341.555 5: /usr/sbin/zabbix_server: trapper #1 [processing data](+0x8e862) [0x55db19701862]
215524:20240808:162341.555 4: /usr/sbin/zabbix_server: trapper #1 [processing data](MAIN_ZABBIX_ENTRY+0xe61) [0x55db19702a91]
215524:20240808:162341.555 3: /usr/sbin/zabbix_server: trapper #1 [processing data](main+0x242) [0x55db196f77a2]
215524:20240808:162341.555 2: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f2309e0e24a]
215524:20240808:162341.555 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f2309e0e305]
215524:20240808:162341.555 0: /usr/sbin/zabbix_server: trapper #1 [processing data](_start+0x21) [0x55db196fe771]
215524:20240808:162341.555 === Memory map: ===
215524:20240808:162341.555 55db19673000-55db196f0000 r--p 00000000 fe:01 135664 /usr/sbin/zabbix_server
215524:20240808:162341.555 55db196f0000-55db1999a000 r-xp 0007d000 fe:01 135664 /usr/sbin/zabbix_server
215524:20240808:162341.555 55db1999a000-55db19a90000 r--p 00327000 fe:01 135664 /usr/sbin/zabbix_server
215524:20240808:162341.555 55db19a90000-55db19a94000 r--p 0041c000 fe:01 135664 /usr/sbin/zabbix_server
215524:20240808:162341.555 55db19a94000-55db19b63000 rw-p 00420000 fe:01 135664 /usr/sbin/zabbix_server
215524:20240808:162341.555 55db19b63000-55db19b74000 rw-p 00000000 00:00 0
215524:20240808:162341.555 55db1a9c0000-55db1aa02000 rw-p 00000000 00:00 0 [heap]
215524:20240808:162341.555 55db1aa02000-55db1aa24000 rw-p 00000000 00:00 0 [heap]
215524:20240808:162341.555 55db1aa24000-55db1b0a4000 rw-p 00000000 00:00 0 [heap]
215524:20240808:162341.555 7f2224938000-7f2224d38000 rw-s 00000000 00:01 6783015 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f2224d38000-7f2244d38000 rw-s 00000000 00:01 6783014 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f2244d38000-7f2244d4a000 rw-s 00000000 00:01 6783013 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f2244d4a000-7f22c4d4a000 rw-s 00000000 00:01 6783012 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f22c4d4a000-7f22c514a000 rw-s 00000000 00:01 6783009 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f22c514a000-7f22c714a000 rw-s 00000000 00:01 6783008 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f22c714a000-7f230714a000 rw-s 00000000 00:01 6783007 /SYSV00000000 (deleted)
215524:20240808:162341.555 7f230714a000-7f230714d000 r--p 00000000 fe:01 156713 /usr/lib/x86_64-linux-gnu/libcap.so.2.66
215524:20240808:162341.555 7f230714d000-7f2307152000 r-xp 00003000 fe:01 156713 /usr/lib/x86_64-linux-gnu/libcap.so.2.66
215524:20240808:162341.555 7f2307152000-7f2307154000 r--p 00008000 fe:01 156713 /usr/lib/x86_64-linux-gnu/libcap.so.2.66
215524:20240808:162341.555 7f2307154000-7f2307155000 r--p 0000a000 fe:01 156713 /usr/lib/x86_64-linux-gnu/libcap.so.2.66
215524:20240808:162341.555 7f2307155000-7f2307156000 rw-p 0000b000 fe:01 156713 /usr/lib/x86_64-linux-gnu/libcap.so.2.66

**Tester001** · 29-10-2024, 16:18

aschild, have you found a solution? I seem to be having a similar issue

**aschild** · 04-11-2024, 19:50

Originally posted by Tester001

aschild, have you found a solution? I seem to be having a similar issue

Nope.
I did invest a lot of time which db schema comparision, updates/data migrations.
But in the end I did not get a stable environment.

Each time a specific proxy and/or host did delivery data to the server, the server crashed.

So we we are currently setting up a second, completely separated, zabbix server and the servers is running stable. (This one running with Postgres and not MarriaDB/MySQL anymore)
We let the old zabbix running with disabled proxy+host for the ones we know trigger the crashing and are slowly moviing over config/templates to the new server.

Not ideal, but since our system was started with Zabbix 2.x and then done most of the upgrades, I think there is just too much incorrect/inconsistend data in the DB.

**Tester001** · 24-11-2024, 19:38

Thank you for responding, @aschild.

I didn't collect enough evidence to submit a bug, but have found the error be caused by a combination of my dataset in DB + int lld_update_items() function.

3135850:20241029:142331.036 15: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](zbx_backtrace+0x46) [0x565358b2ac76]
3135850:20241029:142331.036 14: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](zbx_log_fatal_info+0x285) [0x565358b2afe5]
3135850:20241029:142331.036 13: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](+0x2fd5c6) [0x565358b2b5c6]
3135850:20241029:142331.036 12: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f3db91bd140]
3135850:20241029:142331.036 11: /lib/x86_64-linux-gnu/libc.so.6(+0x171b58) [0x7f3db8a5fb58]
3135850:20241029:142331.036 10: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xf) [0x7f3db8977c1f]
3135850:20241029:142331.036 9: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](zbx_strdup2+0x33) [0x565358b511f3]
3135850:20241029:142331.036 8: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](lld_update_items+0xe4) [0x5653589a8f74]
3135850:20241029:142331.036 7: /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 47.484055 sec during 47.490763 sec](lld_process_discovery_rule+0xe4c)

I knew which LLD was running at the time so disabling/deleting it has stopped the crashes. Luckily I was able to isolate the crash to this particular function!

Ad Widget

Zabbix server crash in 6.0.32 on Debian 12

Zabbix server crash in 6.0.32 on Debian 12

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment