[debian-mysql] Bug#991839: Bug#991839: mariadb-server-10.3: MariaDB intermittantly not starting on boot on AWS EC2 t2.medium instance

Tue Aug 3 06:13:40 BST 2021

Thanks for the report!

Could you do a favour and test and identical scenario with Debian
Bullseye and the MariaDB 10.5 in it?

Both MariaDB 10.3 and 10.5 inherit the upstream systemd service file as-is.

Sources:
10.3: https://salsa.debian.org/mariadb-team/mariadb-10.3/-/blob/buster/support-files/mariadb.service.in
10.5: https://salsa.debian.org/mariadb-team/mariadb-10.5/-/blob/master/support-files/mariadb.service.in

On Mon, Aug 2, 2021 at 8:12 PM Jeremy Davis <jeremy at turnkeylinux.org> wrote:
>
> Package: mariadb-server-10.3
> Version: 1:10.3.29-0+deb10u1
> Severity: important
>
> Dear Maintainer,
>
> I have hit a weird intermittant issue with MariaDB when running on an AWS EC2 t2.medium instance. Roughly 4 out of 5 reboots, MariaDB fails to start. After boot, the MariaDB service starts/restarts flawlessly. No other services appear to be effected.
>
> I can't reproduce it on a t2.micro instance (i.e. MariaDB reliably starts on lower resource machines at boot). I'm guessing that it's a boot time race condition that occurs with higher/better resources?
>
> I first noticed this with 1:10.3.27-0+deb10u1. I saw that 1:10.3.29-0+deb10u1 is now available but upgrade to that made no noticable difference.
>
> When first noticed, I was also using an older (non-cloud) kernel (linux-image-amd64: 4.19+105+deb10u9). Updating to the latest (cloud) kernel (linux-image-cloud-amd64: 4.19+105+deb10u12) changed the message but not the intermittant bad behaviour:
>
> With linux-image-amd64: 4.19+105+deb10u9
> ----------------------------------------
>
> systemd[1]: Starting MariaDB 10.3.29 database server...
> systemd[702]: mariadb.service: Failed to set up mount namespacing: No such file or directory
> systemd[702]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: No such file or directory
> systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
> systemd[1]: mariadb.service: Failed with result 'exit-code'.
> systemd[1]: Failed to start MariaDB 10.3.29 database server.
>
> After update to linux-image-cloud-amd64: 4.19+105+deb10u12
> ----------------------------------------------------------
>
> systemd[1]: Starting MariaDB 10.3.29 database server...
> systemd[690]: mariadb.service: Failed to set up mount namespacing: Invalid argument
> systemd[690]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: Invalid argument
> systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
> systemd[1]: mariadb.service: Failed with result 'exit-code'.
> systemd[1]: Failed to start MariaDB 10.3.29 database server.
>
> After some searching, it appears that removing/overriding some of the service security hardening measures may resolve the issue? But I didn't test (because I didn't want to reduce hardening).
>
> However, adding 'ExecStartPre=/usr/bin/sleep 1' appears to reliably work around the issue. I.e.:
>
> # cat /etc/systemd/system/mariadb.service.d/override.conf
> [Service]
> ExecStartPre=/usr/bin/sleep 1
>
> It strikes me as a bit dirty, but after adding that, it now appears to reliably start at boot.
>
> If you want any further information, please let me know and I'll collect it ASAP.
>
> Regards,
> Jeremy
>
> -- System Information:
> Debian Release: 10.8
>   APT prefers stable
>   APT policy: (500, 'stable')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 4.19.0-17-cloud-amd64 (SMP w/2 CPU cores)
> Locale: LANG=en_US.UTF-8, LC_CTYPE=C (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C), LANGUAGE=en_US.UTF-8 (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)
> Shell: /bin/sh linked to /usr/bin/dash
> Init: systemd (via /run/systemd/system)
>
> Versions of packages mariadb-server-10.3 depends on:
> ii  adduser                   3.118
> ii  debconf [debconf-2.0]     1.5.71
> ii  galera-3                  25.3.25-2
> ii  gawk                      1:4.2.1+dfsg-1
> ii  iproute2                  4.20.0-2+deb10u1
> ii  libc6                     2.28-10
> ii  libdbi-perl               1.642-1+deb10u2
> ii  libgnutls30               3.6.7-4+deb10u6
> ii  libpam0g                  1.3.1-5
> ii  libstdc++6                8.3.0-6
> ii  lsb-base                  10.2019051400
> ii  lsof                      4.91+dfsg-1
> ii  mariadb-client-10.3       1:10.3.29-0+deb10u1
> ii  mariadb-common            1:10.3.29-0+deb10u1
> ii  mariadb-server-core-10.3  1:10.3.29-0+deb10u1
> ii  passwd                    1:4.5-1.1
> ii  perl                      5.28.1-6+deb10u1
> ii  psmisc                    23.2-1
> ii  rsync                     3.1.3-6
> ii  socat                     1.7.3.2-2
> ii  zlib1g                    1:1.2.11.dfsg-1
>
> Versions of packages mariadb-server-10.3 recommends:
> pn  libhtml-template-perl  <none>
>
> Versions of packages mariadb-server-10.3 suggests:
> ii  bsd-mailx [mailx]  8.1.2-0.20180807cvs-1
> pn  mariadb-test       <none>
> pn  netcat-openbsd     <none>
> pn  tinyca             <none>
>
> -- debconf information:
>   mariadb-server-10.3/old_data_directory_saved:
>   mariadb-server-10.3/nis_warning:
>   mariadb-server-10.3/postrm_remove_databases: false
>
> _______________________________________________
> pkg-mysql-maint mailing list
> pkg-mysql-maint at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint