[debian-mysql] Bug#1101340: mariadb-server-core: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.

Johnny johnny.strom at creamarketing.com
Wed Mar 26 07:15:14 GMT 2025 

Package: mariadb-server-core
Version: 1:10.11.11-0+deb12u1
Severity: critical
Tags: upstream
Justification: causes serious data loss
X-Debbugs-Cc: team at security.debian.org

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

   * What led up to the situation?
     Upgraded to mariadb (1:10.11.11-0+deb12u1) bookworm;
     after that so did these crashes start to happen.
     Previews version of mariadb in debian bookworm worked fine.

   * What exactly did you do (or not do) that was effective (or
     ineffective)?
     No workaround so far.
     This Mariadb crash happens on multiple servers with different hardware.
     Uppstream repport:
     https://jira.mariadb.org/projects/MDEV/issues/MDEV-35424?filter=allopenissues

     The maraidb settings for the servers in this report.

thread_stack           = 192K
thread_cache_size      = 8
max_connections        = 600
query_cache_size        = 0
query_cache_type        = 0
max_heap_table_size     = 128M
tmp_table_size          = 128M
read_buffer_size        = 512K
innodb_buffer_pool_size = 12G
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size  = 4M
innodb_flush_method     = O_DIRECT
innodb_log_file_size    = 1024M
innodb_log_files_in_group = 2
join_buffer_size          = 2M
table_definition_cache    = 10000 


   * What was the outcome of this action?
     Crash of mariadb on first web9 server.
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61628 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61462 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61447 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61602 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61351 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24  7:20:17 61342 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:40:12 web9 mariadbd[3888776]: 2025-03-24  7:40:12 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 24 07:40:12 web9 mariadbd[3888776]: 250324  7:40:12 [ERROR] /usr/sbin/mariadbd got signal 6 ;
Mar 24 07:40:12 web9 mariadbd[3888776]: Sorry, we probably made a mistake, and this is a bug.
Mar 24 07:40:12 web9 mariadbd[3888776]: Your assistance in bug reporting will enable us to fix this for the next release.
Mar 24 07:40:12 web9 mariadbd[3888776]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 24 07:40:12 web9 mariadbd[3888776]: a bug on https://jira.mariadb.org/.
Mar 24 07:40:12 web9 mariadbd[3888776]: Please include the information from the server start above, to the end of the
Mar 24 07:40:12 web9 mariadbd[3888776]: information below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Server version: 10.11.11-MariaDB-0+deb12u1 source revision: e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 24 07:40:12 web9 mariadbd[3888776]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 24 07:40:12 web9 mariadbd[3888776]: contains instructions to obtain a better version of the backtrace below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Following these instructions will help MariaDB developers provide a fix quicker.
Mar 24 07:40:12 web9 mariadbd[3888776]: Attempting backtrace. Include this in the bug report.
Mar 24 07:40:12 web9 mariadbd[3888776]: (note: Retrieving this information may fail)
Mar 24 07:40:12 web9 mariadbd[3888776]: Thread pointer: 0x0
Mar 24 07:40:12 web9 mariadbd[3888776]: stack_bottom = 0x0 thread_stack 0x30000
Mar 24 07:40:13 web9 mariadbd[3888776]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5576c94642ee]
Mar 24 07:40:13 web9 mariadbd[3888776]: /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x5576c8f42e29]
Mar 24 07:40:14 web9 mariadbd[3888776]: 2025-03-24  7:40:14 0 [Note] InnoDB: Memory pressure event freed 771915 pages
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f52b305b050]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x8aeec)[0x7f52b30a9eec]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f52b305afb2]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f52b3045472]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(+0x6c323b)[0x5576c8b5f23b]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(+0x6bb7fa)[0x5576c8b577fa]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x5576c93fa378]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x5576c93fac6f]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x4f)[0x5576c93f967f]
Mar 24 07:40:16 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f52b32d44a3]
Mar 24 07:40:17 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f52b30a81f5]
Mar 24 07:40:17 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x10989c)[0x7f52b312889c]
Mar 24 07:40:17 web9 mariadbd[3888776]: Writing a core file...
Mar 24 07:40:17 web9 mariadbd[3888776]: Working directory at /var/lib/mysql
Mar 24 07:40:17 web9 mariadbd[3888776]: Resource Limits (excludes unlimited resources):
Mar 24 07:40:17 web9 mariadbd[3888776]: Limit                     Soft Limit           Hard Limit           Units
Mar 24 07:40:17 web9 mariadbd[3888776]: Max stack size            8388608              unlimited            bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max core file size        0                    unlimited            bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max processes             256997               256997               processes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max open files            32768                32768                files
Mar 24 07:40:17 web9 mariadbd[3888776]: Max locked memory         524288               524288               bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max pending signals       256997               256997               signals
Mar 24 07:40:17 web9 mariadbd[3888776]: Max msgqueue size         819200               819200               bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max nice priority         0                    0
Mar 24 07:40:17 web9 mariadbd[3888776]: Max realtime priority     0                    0
Mar 24 07:40:17 web9 mariadbd[3888776]: Core pattern: core
Mar 24 07:40:17 web9 mariadbd[3888776]: Kernel version: Linux version 6.1.0-32-amd64 (debian-kernel at lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06)
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU time.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Mar 24 07:40:24 web9 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11 database server.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU time.

Crash of mariadb on second web8 server.
Mar 22 10:55:13 web8 mariadbd[3951656]: 2025-03-22 10:55:13 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:13 web8 mariadbd[3951656]: 250322 10:55:13 [ERROR] /usr/sbin/mariadbd got signal 6 ;
Mar 22 10:55:13 web8 mariadbd[3951656]: Sorry, we probably made a mistake, and this is a bug.
Mar 22 10:55:13 web8 mariadbd[3951656]: Your assistance in bug reporting will enable us to fix this for the next release.
Mar 22 10:55:13 web8 mariadbd[3951656]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 22 10:55:13 web8 mariadbd[3951656]: a bug on https://jira.mariadb.org/.
Mar 22 10:55:13 web8 mariadbd[3951656]: Please include the information from the server start above, to the end of the
Mar 22 10:55:13 web8 mariadbd[3951656]: information below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Server version: 10.11.11-MariaDB-0+deb12u1 source revision: e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 22 10:55:13 web8 mariadbd[3951656]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 22 10:55:13 web8 mariadbd[3951656]: contains instructions to obtain a better version of the backtrace below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Following these instructions will help MariaDB developers provide a fix quicker.
Mar 22 10:55:13 web8 mariadbd[3951656]: Attempting backtrace. Include this in the bug report.
Mar 22 10:55:13 web8 mariadbd[3951656]: (note: Retrieving this information may fail)
Mar 22 10:55:13 web8 mariadbd[3951656]: Thread pointer: 0x0
Mar 22 10:55:13 web8 mariadbd[3951656]: stack_bottom = 0x0 thread_stack 0x30000
Mar 22 10:55:16 web8 mariadbd[3951656]: 2025-03-22 10:55:16 0 [Note] InnoDB: Memory pressure event freed 387350 pages

Mar 22 10:55:21 web8 mariadbd[3951656]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ed836892ee]
Mar 22 10:55:21 web8 mariadbd[3951656]: /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55ed83167e29]
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 0 [Note] InnoDB: Memory pressure event freed 90 pages
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 418105 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s CPU time.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Mar 22 10:55:32 web8 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11 database server.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s CPU time.
Mar 22 10:55:32 web8 systemd[1]: Starting mariadb.service - MariaDB 10.11.11 database server...
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] Starting MariaDB 10.11.11-MariaDB-0+deb12u1 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid Xmmv8P9Fsl7fwijTPqiP1aMaxa8= as process 200075
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Number of transaction pools: 1
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Using liburing
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Initializing buffer pool, total size = 6.000GiB, chunk size = 96.000MiB
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Initialized memory pressure event listener
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2376864813498
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: End of log at LSN=2376872752547
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: To recover: 1923 pages
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: 128 rollback segments are active.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: log sequence number 2376872752547; transaction id 7237064055
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 22 10:55:34 web8 mariadbd[200075]: 2025-03-22 10:55:34 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 22 10:55:34 web8 mariadbd[200075]: Version: '10.11.11-MariaDB-0+deb12u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 12
Mar 22 10:55:34 web8 systemd[1]: Started mariadb.service - MariaDB 10.11.11 database server.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200167]: Upgrading MySQL tables if necessary.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200170]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200190]: Checking for insecure root accounts.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200196]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Mar 22 10:55:38 web8 mariadbd[200075]: 2025-03-22 10:55:38 0 [Note] InnoDB: Memory pressure event freed 62433 pages
Mar 22 10:55:39 web8 mariadbd[200075]: 2025-03-22 10:55:39 0 [Note] InnoDB: Buffer pool(s) load completed at 250322 10:55:39


   * What outcome did you expect instead?
That Mariadb did not crash.

*** End of the template - remove these template lines ***


-- System Information:
Debian Release: 12.10
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-32-amd64 (SMP w/64 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server-core depends on:
ii  libc6           2.36-9+deb12u10
ii  libcrypt1       1:4.4.33-2
ii  libgcc-s1       12.2.0-14
ii  libnuma1        2.0.16-1
ii  libpcre2-8-0    10.42-1
ii  libssl3         3.0.15-1~deb12u1
ii  libstdc++6      12.2.0-14
ii  libsystemd0     252.36-1~deb12u1
ii  liburing2       2.3-3
ii  mariadb-common  1:10.11.11-0+deb12u1
ii  zlib1g          1:1.2.13.dfsg-1

mariadb-server-core recommends no packages.

mariadb-server-core suggests no packages.

-- no debconf information

More information about the pkg-mysql-maint mailing list