[debian-mysql] Bug#850682: mariadb-10.0: segmentation fault after killing a query blocked on disk space availability

[Apollon Oikonomopoulos]

Package: mariadb-server-10.0
Version: 10.0.27-0+deb8u1
Severity: grave
Control: tags -1 upstream

Dear Maintainer,

MariaDB 10.0 will segfault reproducibly in the following scenario:

 1. A query is blocked waiting for disk space to be freed
 2. The same query is killed (through the KILL command, or by issuing a 
    SIGINT to the DB shell).
 3. The internal 60-second sleep timeout in wait_for_free_space() has 
    elapsed.

Apparently the KILL command causes some of the query's resources to be 
immediately free()'d, which however will still be used by the sleeping 
thread in its next attempt to write data out to disk. The server's logs 
also seem to confirm this:

 ...
 Jan  9 12:17:18 marvin mysqld: 170109 12:17:18 [Warning] mysqld: Disk is full writing '/var/tmp/mysql/#sql_2525_0.MAD' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
 Jan  9 12:17:18 marvin mysqld: 170109 12:17:18 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
 Jan  9 12:18:18 marvin mysqld: 170109 12:18:18 [ERROR] mysqld got signal 11 ;
 ...
 Jan  9 12:18:18 marvin mysqld: Trying to get some variables.
 Jan  9 12:18:18 marvin mysqld: Some pointers may be invalid and cause the dump to abort.
 Jan  9 12:18:18 marvin mysqld: Query (0x7f1fdf08e020): is an invalid pointer
 Jan  9 12:18:18 marvin mysqld: Connection ID (thread ID): 31
 Jan  9 12:18:18 marvin mysqld: Status: KILL_QUERY
 ...

The problem can be trivially reproduced on an amd64 system using the attached
script that uses a small tmpfs and a crafted query to force the database to an
out-of-disk-space condition.

We actually hit this bug on a production system where a LEFT OUTER JOIN 
of GB-sized tables caused an 18GB on-disk temporary table to exhaust all 
temp space. Note that MariaDB Server 10.1 does not seem to be affected.

N.B: I know this is really a corner-case, however I'm setting the severity to
`grave' as I'm not sure how well MariaDB copes with the segmentation fault in
terms of data loss. If you feel that this is too aggressive, please downgrade
accordingly.

Regards,
Apollon

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'testing'), (500, 'stable'), (90, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.8.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=el_GR.UTF-8, LC_CTYPE=el_GR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mariadb-server-10.0 depends on:
ii  adduser                   3.115
ii  debconf [debconf-2.0]     1.5.59
ii  libaio1                   0.3.110-3
ii  libc6                     2.24-8
ii  libdbi-perl               1.636-1+b1
ii  libpam0g                  1.1.8-3.3
ii  libstdc++6                6.2.1-5
ii  lsb-base                  9.20161125
ii  mariadb-client-10.0       10.0.28-2
ii  mariadb-common            10.1.20-3
ii  mariadb-server-core-10.0  10.0.27-0+deb8u1
ii  passwd                    1:4.2-3.3
ii  perl                      5.24.1~rc4-1
ii  psmisc                    22.21-2.1+b1
ii  zlib1g                    1:1.2.8.dfsg-2+b3

Versions of packages mariadb-server-10.0 recommends:
ii  libhtml-template-perl  2.95-2

Versions of packages mariadb-server-10.0 suggests:
ii  bsd-mailx [mailx]  8.1.2-0.20160123cvs-3
ii  mailutils [mailx]  1:2.99.99-1.1+b1
pn  mariadb-test       <none>
pn  tinyca             <none>

-- debconf information excluded
-------------- next part --------------
#!/bin/sh

cat >/etc/mysql/mariadb.conf.d/99-local.cnf <<EOF
[mysqld]
tmpdir=/var/tmp/mysql
tmp_table_size=1024
EOF

mkdir /var/tmp/mysql
mount -o size=100k -t tmpfs tmpfs /var/tmp/mysql

systemctl restart mysql

sleep 5

mysql <<EOF
DROP DATABASE IF EXISTS oodtest;
CREATE DATABASE oodtest;
use oodtest;
CREATE TABLE test (id INT PRIMARY KEY NOT NULL AUTO_INCREMENT, data MEDIUMTEXT);
EOF

for i in $(seq 25); do
	echo "INSERT INTO test (data) VALUES (REPEAT('a', 16000));"
done | mysql oodtest

# This should block
mysql -e "SELECT * FROM test AS a LEFT OUTER JOIN test AS b ON a.data = b.data LEFT OUTER JOIN test AS c ON b.data = c.data ORDER BY a.data ASC" oodtest &

childpid=$!

sleep 5

# Send SIGINT to the client
echo "Killing the query"
kill -INT $childpid

# MySQL should die within a minute, disconnecting the client
echo "Waiting for MariaDB to die (in approx. 55s)"
wait
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-mysql-maint/attachments/20170109/c18fc20b/attachment.sig>