[debian-mysql] Bug#943962: mariadb-server-10.3: mysqld crashes and hangs, no longer processing requests

[Richard van den Berg]

Package: mariadb-server-10.3
Version: 1:10.3.17-0+deb10u1
Severity: important

I run mysqldump (through automysqlbackup) daily. Several times per week
during this backup mysqld hangs. The process however stays running and still
accepts TCP and socket connections, however no SQL queries are ever
answered anymore. This is very serious because systemd does not catch this
and leaves the process running. Only a "pkill -9 mysqld" can resolve the
situation.

In the 15+ years I have been using mysql/mariadb I have never encoutered
this situation where my system is left hanging without a working database.
If mysqld crashed (this is bad enough) the process should end itself so
systemd can restart it.

The messages in error.log are:

corrupted size vs. prev_size
191101  7:23:41 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.17-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=5
max_threads=153
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467422
K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f2b600014b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2bc4814dd8 thread_stack 0x49000


-- System Information:
Debian Release: 10.1
  APT prefers stable
  APT policy: (990, 'stable'), (900, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server-10.3 depends on:
ii  adduser                   3.118
ii  debconf [debconf-2.0]     1.5.71
ii  galera-3                  25.3.25-2
ii  gawk                      1:4.2.1+dfsg-1
ii  iproute2                  4.20.0-2
ii  libc6                     2.28-10
ii  libdbi-perl               1.642-1+b1
ii  libgnutls30               3.6.7-4
ii  libpam0g                  1.3.1-5
ii  libstdc++6                8.3.0-6
ii  lsb-base                  10.2019051400
ii  lsof                      4.91+dfsg-1
ii  mariadb-client-10.3       1:10.3.17-0+deb10u1
ii  mariadb-common            1:10.3.17-0+deb10u1
ii  mariadb-server-core-10.3  1:10.3.17-0+deb10u1
ii  passwd                    1:4.5-1.1
ii  perl                      5.28.1-6
ii  psmisc                    23.2-1
ii  rsync                     3.1.3-6
ii  socat                     1.7.3.2-2
ii  zlib1g                    1:1.2.11.dfsg-1

Versions of packages mariadb-server-10.3 recommends:
ii  libhtml-template-perl  2.97-1

Versions of packages mariadb-server-10.3 suggests:
ii  bsd-mailx [mailx]  8.1.2-0.20180807cvs-1
ii  mailx              1:20071201-3
pn  mariadb-test       <none>
pn  netcat-openbsd     <none>
pn  tinyca             <none>

-- debconf information excluded