[debian-mysql] Bug#991839: mariadb-server-10.3: MariaDB intermittantly not starting on boot on AWS EC2 t2.medium instance

[Jeremy Davis]

Package: mariadb-server-10.3
Version: 1:10.3.29-0+deb10u1
Severity: important

Dear Maintainer,

I have hit a weird intermittant issue with MariaDB when running on an AWS EC2 t2.medium instance. Roughly 4 out of 5 reboots, MariaDB fails to start. After boot, the MariaDB service starts/restarts flawlessly. No other services appear to be effected.

I can't reproduce it on a t2.micro instance (i.e. MariaDB reliably starts on lower resource machines at boot). I'm guessing that it's a boot time race condition that occurs with higher/better resources?

I first noticed this with 1:10.3.27-0+deb10u1. I saw that 1:10.3.29-0+deb10u1 is now available but upgrade to that made no noticable difference.

When first noticed, I was also using an older (non-cloud) kernel (linux-image-amd64: 4.19+105+deb10u9). Updating to the latest (cloud) kernel (linux-image-cloud-amd64: 4.19+105+deb10u12) changed the message but not the intermittant bad behaviour:

With linux-image-amd64: 4.19+105+deb10u9
----------------------------------------

systemd[1]: Starting MariaDB 10.3.29 database server...
systemd[702]: mariadb.service: Failed to set up mount namespacing: No such file or directory
systemd[702]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: No such file or directory
systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.3.29 database server.

After update to linux-image-cloud-amd64: 4.19+105+deb10u12
----------------------------------------------------------

systemd[1]: Starting MariaDB 10.3.29 database server...
systemd[690]: mariadb.service: Failed to set up mount namespacing: Invalid argument
systemd[690]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: Invalid argument
systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.3.29 database server.

After some searching, it appears that removing/overriding some of the service security hardening measures may resolve the issue? But I didn't test (because I didn't want to reduce hardening).

However, adding 'ExecStartPre=/usr/bin/sleep 1' appears to reliably work around the issue. I.e.:

# cat /etc/systemd/system/mariadb.service.d/override.conf 
[Service]
ExecStartPre=/usr/bin/sleep 1

It strikes me as a bit dirty, but after adding that, it now appears to reliably start at boot.

If you want any further information, please let me know and I'll collect it ASAP.

Regards,
Jeremy

-- System Information:
Debian Release: 10.8
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-17-cloud-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=C (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C), LANGUAGE=en_US.UTF-8 (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mariadb-server-10.3 depends on:
ii  adduser                   3.118
ii  debconf [debconf-2.0]     1.5.71
ii  galera-3                  25.3.25-2
ii  gawk                      1:4.2.1+dfsg-1
ii  iproute2                  4.20.0-2+deb10u1
ii  libc6                     2.28-10
ii  libdbi-perl               1.642-1+deb10u2
ii  libgnutls30               3.6.7-4+deb10u6
ii  libpam0g                  1.3.1-5
ii  libstdc++6                8.3.0-6
ii  lsb-base                  10.2019051400
ii  lsof                      4.91+dfsg-1
ii  mariadb-client-10.3       1:10.3.29-0+deb10u1
ii  mariadb-common            1:10.3.29-0+deb10u1
ii  mariadb-server-core-10.3  1:10.3.29-0+deb10u1
ii  passwd                    1:4.5-1.1
ii  perl                      5.28.1-6+deb10u1
ii  psmisc                    23.2-1
ii  rsync                     3.1.3-6
ii  socat                     1.7.3.2-2
ii  zlib1g                    1:1.2.11.dfsg-1

Versions of packages mariadb-server-10.3 recommends:
pn  libhtml-template-perl  <none>

Versions of packages mariadb-server-10.3 suggests:
ii  bsd-mailx [mailx]  8.1.2-0.20180807cvs-1
pn  mariadb-test       <none>
pn  netcat-openbsd     <none>
pn  tinyca             <none>

-- debconf information:
  mariadb-server-10.3/old_data_directory_saved:
  mariadb-server-10.3/nis_warning:
  mariadb-server-10.3/postrm_remove_databases: false