mariadb 10.11.3 LSIO container not coming up

[d0d0]

Zabbix server ran fine with mariadb LinuxServer.io (LSIO) container 10.6.13. Upgrading to the recently released LSIO container for mariadb 10.11.3 needed some work (https://info.linuxserver.io/issues/2023-05-29-mariadb/), but ran fine for a few days.

Today after stopping and starting the mariadb container it did not come up:

230601 11:44:09 mysqld_safe Logging to '/config/databases/f2ad4e5733ab.err'.
230601 11:44:10 mysqld_safe Starting mariadbd daemon with databases from /config/databases
Caught SIGTERM signal!
230601 11:44:12 mysqld_safe Logging to '/config/databases/f2ad4e5733ab.err'.
230601 11:44:12 mysqld_safe Starting mariadbd daemon with databases from /config/databases
Caught SIGTERM signal!

looping endlessly.

Has someone else experienced this problem? How to solve this (I am not a DB admin, but it is my DB instance)?

[d0d0]

This fixed it (as proposed by mariadb):

s6-setuidgid abc /usr/bin/mariadbd-safe --tc-heuristic-recover=ROLLBACK --datadir="${DATADIR}" --pid-file="/var/run/mysqld/mysqld.pid" --user="abc"

The culprit was a backup solution that stopped mariadb first, not zabbix-server...

[cyber]

Logging to '/config/databases/f2ad4e5733ab.err

Anything useful in that logfile? Or in any mariadb logfile?

[d0d0]

This is the last run in the error log:

2023-06-01 13:07:09 0 [Note] Starting MariaDB 10.11.3-MariaDB-log source revision 0bb31039f54bd6a0dc8f0fc7d40e6b58a51998b0 as process 3300
2023-06-01 13:07:09 0 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2023-06-01 13:07:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2023-06-01 13:07:09 0 [Note] InnoDB: Number of transaction pools: 1
2023-06-01 13:07:09 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-06-01 13:07:09 0 [Note] InnoDB: Using Linux native AIO
2023-06-01 13:07:09 0 [Note] InnoDB: Initializing buffer pool, total size = 256.000MiB, chunk size = 4.000MiB
2023-06-01 13:07:09 0 [Note] InnoDB: Completed initialization of buffer pool
2023-06-01 13:07:09 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
2023-06-01 13:07:10 0 [Note] InnoDB: Transaction 12054930 was in the XA prepared state.
2023-06-01 13:07:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
2023-06-01 13:07:10 0 [Note] InnoDB: Trx id counter is 12054933
2023-06-01 13:07:10 0 [Note] InnoDB: 128 rollback segments are active.
2023-06-01 13:07:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-06-01 13:07:10 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-06-01 13:07:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2023-06-01 13:07:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2023-06-01 13:07:10 0 [Note] InnoDB: log sequence number 24198883691; transaction id 12054934
2023-06-01 13:07:10 0 [Note] InnoDB: Loading buffer pool(s) from /config/databases/ib_buffer_pool
2023-06-01 13:07:10 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-06-01 13:07:10 0 [Note] InnoDB: Buffer pool(s) load completed at 230601 13:07:10
2023-06-01 13:07:10 0 [Note] InnoDB: Starting recovery for XA transactions...
2023-06-01 13:07:10 0 [Note] InnoDB: Transaction 12054930 in prepared state after recovery
2023-06-01 13:07:10 0 [Note] InnoDB: Transaction contains changes to 2 rows
2023-06-01 13:07:10 0 [Note] InnoDB: 1 transactions in prepared state after recovery
2023-06-01 13:07:10 0 [Note] Found 1 prepared transaction(s) in InnoDB
2023-06-01 13:07:10 0 [ERROR] Found 1 prepared transactions! It means that server was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start server with --tc-heuristic-recover switch to commit or rollback pending transactions.

[d0d0]

This fixed it (as proposed by mariadb):

s6-setuidgid abc /usr/bin/mariadbd-safe --tc-heuristic-recover=ROLLBACK --datadir="${DATADIR}" --pid-file="/var/run/mysqld/mysqld.pid" --user="abc"

The culprit was a backup solution that stopped mariadb first, not zabbix-server...