Problems with db after upgrade from 5.0.11 to 5.0.17

[sicko]

Hi all,

we had a regular upgrade via apt and after the upgrade mysql won't start anymore, here are the relevant logs:

Nov 16 11:04:46 xxxx systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 3066 ...
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Using Linux native AIO
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Uses event mutexes
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Number of pools: 1
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Using SSE2 crc32 instructions
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Completed initialization of buffer pool
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=10046619970600
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0 [Note] InnoDB: Starting final batch to recover 2187 pages from redo log.
Nov 16 11:04:46 xxxx mysqld[3067]: 2021-11-16 11:04:46 0x7f745954c700 InnoDB: Assertion failure in file /build/mariadb-10.3-Y84vY7/mariadb-10.3-10.3.31/storage/innobase/btr/btr0cur.cc line 340
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: Failing assertion: btr_page_get_prev(get_block->frame) == page_get_page_no(page)
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: We intentionally generate a memory trap.
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: If you get repeated assertion failures or crashes, even
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: immediately after the mysqld startup, there may be
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
Nov 16 11:04:46 xxxx mysqld[3067]: InnoDB: about forcing recovery.
Nov 16 11:04:46 xxxx mysqld[3067]: 211116 11:04:46 [ERROR] mysqld got signal 6 ;
Nov 16 11:04:46 xxxx mysqld[3067]: This could be because you hit a bug. It is also possible that this binary
Nov 16 11:04:46 xxxx mysqld[3067]: or one of the libraries it was linked against is corrupt, improperly built,
Nov 16 11:04:46 xxxx mysqld[3067]: or misconfigured. This error can also be caused by malfunctioning hardware.
Nov 16 11:04:46 xxxx mysqld[3067]:
Nov 16 11:04:46 xxxx mysqld[3067]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Nov 16 11:04:46 xxxx mysqld[3067]:
Nov 16 11:04:46 xxxx mysqld[3067]: We will try our best to scrape up some info that will hopefully help
Nov 16 11:04:46 xxxx mysqld[3067]: diagnose the problem, but since we have already crashed,
Nov 16 11:04:46 xxxx mysqld[3067]: something is definitely wrong and this may fail.
Nov 16 11:04:46 xxxx mysqld[3067]:
Nov 16 11:04:46 xxxx mysqld[3067]: Server version: 10.3.31-MariaDB-0+deb10u1
Nov 16 11:04:46 xxxx mysqld[3067]: key_buffer_size=134217728
Nov 16 11:04:46 xxxx mysqld[3067]: read_buffer_size=131072
Nov 16 11:04:46 xxxx mysqld[3067]: max_used_connections=0
Nov 16 11:04:46 xxxx mysqld[3067]: max_threads=153
Nov 16 11:04:46 xxxx mysqld[3067]: thread_count=0
Nov 16 11:04:46 xxxx mysqld[3067]: It is possible that mysqld could use up to
Nov 16 11:04:46 xxxx mysqld[3067]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467428 K bytes of memory
Nov 16 11:04:46 xxxx mysqld[3067]: Hope that's ok; if not, decrease some variables in the equation.
Nov 16 11:04:46 xxxx mysqld[3067]:
Nov 16 11:04:46 xxxx mysqld[3067]: Thread pointer: 0x0
Nov 16 11:04:46 xxxx mysqld[3067]: Attempting backtrace. You can use the following information to find out
Nov 16 11:04:46 xxxx mysqld[3067]: where mysqld died. If you see no messages after this, something went
Nov 16 11:04:46 xxxx mysqld[3067]: terribly wrong...
Nov 16 11:04:46 xxxx mysqld[3067]: stack_bottom = 0x0 thread_stack 0x49000
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5624c4c3ecee]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5624c4769dcd]
Nov 16 11:04:46 xxxx mysqld[3067]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f74809e1730]
Nov 16 11:04:46 xxxx mysqld[3067]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f74808457bb]
Nov 16 11:04:46 xxxx mysqld[3067]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f7480830535]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0x4e9cb3)[0x5624c44aacb3]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0x4ec5e5)[0x5624c44ad5e5]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0xa58d62)[0x5624c4a19d62]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0xa5dcf5)[0x5624c4a1ecf5]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0x953aa0)[0x5624c4914aa0]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0x953cb3)[0x5624c4914cb3]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0x959a99)[0x5624c491aa99]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0xa75577)[0x5624c4a36577]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0xac752a)[0x5624c4a8852a]
Nov 16 11:04:46 xxxx mysqld[3067]: /usr/sbin/mysqld(+0xa04a00)[0x5624c49c5a00]
Nov 16 11:04:46 xxxx mysqld[3067]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f74809d6fa3]
Nov 16 11:04:46 xxxx mysqld[3067]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f74809074cf]
Nov 16 11:04:46 xxxx mysqld[3067]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Nov 16 11:04:46 xxxx mysqld[3067]: information that should help you find out what is causing the crash.
Nov 16 11:04:46 xxxx mysqld[3067]: Writing a core file...
Nov 16 11:04:46 xxxx mysqld[3067]: Working directory at /var/lib/mysql
Nov 16 11:04:46 xxxx mysqld[3067]: Resource Limits:
Nov 16 11:04:46 xxxx mysqld[3067]: Fatal signal 11 while backtracing
Nov 16 11:05:16 xxxx mysql[2922]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
Nov 16 11:05:16 xxxx systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE
Nov 16 11:05:16 xxxx systemd[1]: mysql.service: Failed with result 'exit-code'.
Nov 16 11:05:16 xxxx systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.

Checking the db with mysqlcheck returns several tables corrupted:

zabbix.events
Warning : InnoDB: Index 'events_1' contains 126496 entries, should be 126502.
Warning : InnoDB: Index 'events_2' contains 126499 entries, should be 126502.
error : Corrupt

zabbix.history
Warning : InnoDB: Index 'history_1' contains 108438940 entries, should be 108448159.
error : Corrupt

zabbix.history_str
Warning : InnoDB: Index 'history_str_1' contains 39650413 entries, should be 39651387.
error : Corrupt

zabbix.history_text
Warning : InnoDB: Index 'history_text_1' contains 88108283 entries, should be 88111814.
error : Corrupt

I see it is a database issue, but has anyone had similar problems? We are running zabbix on debian buster, no problems so far.

Regards!

[sicko]

Solved with re-creating the database on a new mysql instance.

Regards!