[debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system

[Witold Baryluk]

Package: mariadb-server
Version: 1:10.11.3-1
Followup-For: Bug #1009290
X-Debbugs-Cc: witold.baryluk at gmail.com


Still broken in 10.11.3.

This is fresh installation, zero modifications.

Jun 24 19:14:24 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server...
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision  as process 2685
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Number of transaction pools: 1
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using liburing
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Completed initialization of buffer pool
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation
Jun 24 19:14:25 debian mariadbd[2685]: 230624 19:14:25 [ERROR] mysqld got signal 6 ;
Jun 24 19:14:25 debian mariadbd[2685]: This could be because you hit a bug. It is also possible that this binary
Jun 24 19:14:25 debian mariadbd[2685]: or one of the libraries it was linked against is corrupt, improperly built,
Jun 24 19:14:25 debian mariadbd[2685]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jun 24 19:14:25 debian mariadbd[2685]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jun 24 19:14:25 debian mariadbd[2685]: We will try our best to scrape up some info that will hopefully help
Jun 24 19:14:25 debian mariadbd[2685]: diagnose the problem, but since we have already crashed,
Jun 24 19:14:25 debian mariadbd[2685]: something is definitely wrong and this may fail.
Jun 24 19:14:25 debian mariadbd[2685]: Server version: 10.11.3-MariaDB-1 source revision:
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size=134217728
Jun 24 19:14:25 debian mariadbd[2685]: read_buffer_size=131072
Jun 24 19:14:25 debian mariadbd[2685]: max_used_connections=0
Jun 24 19:14:25 debian mariadbd[2685]: max_threads=153
Jun 24 19:14:25 debian mariadbd[2685]: thread_count=0
Jun 24 19:14:25 debian mariadbd[2685]: It is possible that mysqld could use up to
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K  bytes of memory
Jun 24 19:14:25 debian mariadbd[2685]: Hope that's ok; if not, decrease some variables in the equation.
Jun 24 19:14:25 debian mariadbd[2685]: Thread pointer: 0x0
Jun 24 19:14:25 debian mariadbd[2685]: Attempting backtrace. You can use the following information to find out
Jun 24 19:14:25 debian mariadbd[2685]: where mysqld died. If you see no messages after this, something went
Jun 24 19:14:25 debian mariadbd[2685]: terribly wrong...
Jun 24 19:14:25 debian mariadbd[2685]: stack_bottom = 0x0 thread_stack 0x49000
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559932b6ecfe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599326dc7c9]
Jun 24 19:14:25 debian mariadbd[2685]: libc_sigaction.c:0(__restore_rt)[0x7efc03891f90]
Jun 24 19:14:25 debian mariadbd[2685]: nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7efc038e0ccc]
Jun 24 19:14:25 debian mariadbd[2685]: posix/raise.c:27(__GI_raise)[0x7efc03891ef2]
Jun 24 19:14:25 debian mariadbd[2685]: stdlib/abort.c:81(__GI_abort)[0x7efc0387c472]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x67483c)[0x55993232883c]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xd2a515)[0x5599329de515]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe29078)[0x559932add078]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c5cf)[0x559932ae05cf]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c6bd)[0x559932ae06bd]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x685d85)[0x559932339d85]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xcd9051)[0x55993298d051]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599326dfafe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x80f257)[0x5599324c3257]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599324c421a]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x7008f7)[0x5599323b48f7]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599323ba31d]
Jun 24 19:14:25 debian mariadbd[2685]: x86/libc-start.c:74(__libc_start_call_main)[0x7efc0387d18a]
Jun 24 19:14:25 debian mariadbd[2685]: csu/libc-start.c:128(call_init)[0x7efc0387d245]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_start+0x21)[0x5599323af541]
Jun 24 19:14:25 debian mariadbd[2685]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Jun 24 19:14:25 debian mariadbd[2685]: information that should help you find out what is causing the crash.
Jun 24 19:14:25 debian mariadbd[2685]: Writing a core file...
Jun 24 19:14:25 debian mariadbd[2685]: Working directory at /var/lib/mysql
Jun 24 19:14:25 debian mariadbd[2685]: Resource Limits:
Jun 24 19:14:25 debian mariadbd[2685]: Limit                     Soft Limit           Hard Limit           Units
Jun 24 19:14:25 debian mariadbd[2685]: Max cpu time              unlimited            unlimited            seconds
Jun 24 19:14:25 debian mariadbd[2685]: Max file size             unlimited            unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max data size             unlimited            unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max stack size            8388608              unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max core file size        0                    unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max resident set          unlimited            unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max processes             1030035              1030035              processes
Jun 24 19:14:25 debian mariadbd[2685]: Max open files            32768                32768                files
Jun 24 19:14:25 debian mariadbd[2685]: Max locked memory         524288               524288               bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max address space         unlimited            unlimited            bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max file locks            unlimited            unlimited            locks
Jun 24 19:14:25 debian mariadbd[2685]: Max pending signals       1030035              1030035              signals
Jun 24 19:14:25 debian mariadbd[2685]: Max msgqueue size         819200               819200               bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max nice priority         0                    0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime priority     0                    0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime timeout      unlimited            unlimited            us
Jun 24 19:14:25 debian mariadbd[2685]: Core pattern: |/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h
Jun 24 19:14:25 debian mariadbd[2685]: Kernel version: Linux version 6.1.0-9-amd64 (debian-kernel at lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Failed with result 'core-dump'.
Jun 24 19:14:25 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server.


Maybe mariadb does not like it being on overlayfs.

Here are my mounts:

user at debian:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=131844608k,nr_inodes=32961152,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /run/live/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /run/live/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /run/live/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
overlay on / type overlay (rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work,redirect_dir=on)
tmpfs on /usr/lib/live/mount type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /usr/lib/live/mount/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /usr/lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /usr/lib/live/mount/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29782)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=26378640k,nr_inodes=6594660,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)


-- System Information:
Debian Release: 12.0
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'bookworm'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.1.0-9-amd64 (SMP w/32 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server depends on:
ii  adduser                3.134
ii  debconf [debconf-2.0]  1.5.82
ii  galera-4               26.4.13-1
ii  gawk                   1:5.2.1-2
ii  iproute2               6.1.0-3
ii  libc6                  2.36-9
ii  libdbi-perl            1.643-4
ii  libpam0g               1.5.2-6
ii  libssl3                3.0.9-1
ii  libstdc++6             12.2.0-14
ii  lsof                   4.95.0-1
ii  mariadb-client         1:10.11.3-1
ii  mariadb-common         1:10.11.3-1
ii  mariadb-server-core    1:10.11.3-1
ii  passwd                 1:4.13+dfsg1-1+b1
ii  perl                   5.36.0-7
ii  procps                 2:4.0.2-3
ii  psmisc                 23.6-1
ii  rsync                  3.2.7-1
ii  socat                  1.7.4.4-2
ii  zlib1g                 1:1.2.13.dfsg-1

Versions of packages mariadb-server recommends:
ii  libhtml-template-perl           2.97-2
ii  mariadb-plugin-provider-bzip2   1:10.11.3-1
ii  mariadb-plugin-provider-lz4     1:10.11.3-1
ii  mariadb-plugin-provider-lzma    1:10.11.3-1
ii  mariadb-plugin-provider-lzo     1:10.11.3-1
ii  mariadb-plugin-provider-snappy  1:10.11.3-1
ii  pv                              1.6.20-1

Versions of packages mariadb-server suggests:
ii  bsd-mailx [mailx]  8.1.2-0.20220412cvs-1
ii  mailutils [mailx]  1:3.15-4
pn  mariadb-test       <none>
ii  netcat-openbsd     1.219-1

-- debconf information:
  mariadb-server/nis_warning:
  mariadb-server/postrm_remove_databases: false
  mariadb-server/old_data_directory_saved: