[debian-mysql] Bug#1151058: mariadb-server: socket activation restarts server mid-upgrade; stalled instance blocks service (SendSIGKILL=no)

[José Miguel Gonçalves]

Package: mariadb-server
Version: 1:10.11.19-0+deb12u1
Severity: important

Dear Maintainer,

This is related to #1141976: on this host, mariadb.socket was enabled
by the 1:10.11.18-0+deb12u1 upgrade (2026-10-08), not by me. However,
the problem below is independent of how the socket got enabled: the
maintainer scripts stop mariadb.service while leaving mariadb.socket
active, so any client can re-activate the server mid-upgrade. This
affects anyone who enables socket activation deliberately, too.

On Debian 12 (bookworm), upgrading mariadb-server from
1:10.11.18-0+deb12u1 to 1:10.11.19-0+deb12u1 (from bookworm-security)
left the database unavailable and ultimately led to a system-wide outage.
Since security updates are commonly applied by unattended-upgrades, this
can happen without an administrator present.

mariadb.socket listens on /run/mysqld/mysqld.sock, @mariadb and
0.0.0.0:3306. The host runs FreePBX/Asterisk with per-minute cron jobs
that connect to the database, and is a replication peer of another
server.

What happens:

1. The maintainer script stops mariadb.service; dpkg warns:
     Warning: Stopping mariadb.service, but it can still be activated by:
       mariadb.socket
2. The old 10.11.18 server shuts down cleanly in about one second.
3. In the same second, a client connection on mariadb.socket triggers
   socket activation, and systemd starts mariadb.service again, still
   with the old 10.11.18 binary, while the upgrade is in progress.
4. This new instance stalls during startup (after buffer pool load,
   before "ready for connections") and never becomes ready.
5. The maintainer script stops the service again. The instance ignores
   SIGTERM; because the unit has SendSIGKILL=no, systemd waits the full
   stop-sigterm and final-sigterm timeouts (2 x 15 min) and then leaves
   the process running.
6. All later start attempts fail ("Found left-over process ... Will not
   start SendSIGKILL=no service ... Device or resource busy"), and the
   postinst restart fails ("Could not execute systemctl: at
   /usr/bin/deb-systemd-invoke line 145").
7. Meanwhile mariadb.socket keeps accepting client connections that are
   never served, so clients hang indefinitely instead of failing. In my
   case cron jobs piled up overnight (~1700 cron, ~1400 php, ~250 mysql
   client processes) until the OOM killer fired and the machine became
   unreachable; only a power cycle recovered it.

Relevant journal (trimmed, times in WEST):

22:02:59 mariadbd[3904780]: /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
22:02:59 systemd[1]: Stopping mariadb.service - MariaDB 10.11.18 database server...
22:03:00 mariadbd[3904780]: InnoDB: Shutdown completed; log sequence number 83086382283
22:03:00 mariadbd[3904780]: /usr/sbin/mariadbd: Shutdown complete
22:03:00 systemd[1]: Stopped mariadb.service - MariaDB 10.11.18 database server.
22:03:00 systemd[1]: Starting mariadb.service - MariaDB 10.11.18 database server...
22:03:00 mariadbd[869784]: Starting MariaDB 10.11.18-MariaDB-0+deb12u1-log ... as process 869784
22:03:01 mariadbd[869784]: Using systemd activated socket host 0.0.0.0 port 3306
22:03:01 mariadbd[869784]: Using systemd activated unix socket /run/mysqld/mysqld.sock
22:03:01 mariadbd[869784]: Using systemd activated unix socket @mariadb
22:03:01 mariadbd[869784]: InnoDB: Buffer pool(s) load completed at 261015 22:03:01
   (no further output from 869784)
22:18:00 systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
22:33:00 systemd[1]: mariadb.service: State 'final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
22:33:00 systemd[1]: mariadb.service: Found left-over process 869784 (mariadbd) in control group while starting unit. Ignoring.
22:33:00 systemd[1]: mariadb.service: Will not start SendSIGKILL=no service of type KillMode=control-group or mixed while processes exist
22:33:00 systemd[1]: mariadb.service: Failed to run 'start-pre' task: Device or resource busy

I could not determine why the socket-activated instance stalled during
startup; no stack trace was captured before the power cycle. However,
the instance should never have been started in the first place.

I have only tested this on bookworm (10.11.x); I have not verified
whether the 11.8 packaging in trixie/sid still stops mariadb.service
without stopping mariadb.socket.

Expected behaviour:

The maintainer scripts should stop (or otherwise hold) mariadb.socket
together with mariadb.service before unpacking, and start both again
in postinst, e.g.

  deb-systemd-invoke stop mariadb.socket mariadb.service

so that no client can re-activate the server while the package is being
replaced.

Workaround:

Stop both units manually before upgrading:

  systemctl stop mariadb.socket mariadb.service
  apt upgrade
  systemctl start mariadb.socket mariadb.service

Regards,
José Gonçalves

-- System Information:
Debian Release: 12.15
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-53-amd64 (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mariadb-server depends on:
ii  adduser                3.134
ii  debconf [debconf-2.0]  1.5.82
ii  galera-4               26.4.23-0+deb12u1
ii  gawk                   1:5.2.1-2
ii  iproute2               6.1.0-3
ii  libc6                  2.36-9+deb12u14
ii  libdbi-perl            1.643-4+deb12u3
ii  libgcc-s1              12.2.0-14+deb12u1
ii  libpam0g               1.5.2-6+deb12u2
ii  libssl3                3.0.22-1~deb12u1
ii  libstdc++6             12.2.0-14+deb12u1
ii  lsof                   4.95.0-1
ii  mariadb-client         1:10.11.19-0+deb12u1
ii  mariadb-common         1:10.11.19-0+deb12u1
ii  mariadb-server-core    1:10.11.19-0+deb12u1
ii  passwd                 1:4.13+dfsg1-1+deb12u2
ii  perl                   5.36.0-7+deb12u4
ii  procps                 2:4.0.2-3
ii  psmisc                 23.6-1
ii  rsync                  3.2.7-1+deb12u6
ii  socat                  1.8.0.0-4~bpo12+1
ii  zlib1g                 1:1.2.13.dfsg-1

Versions of packages mariadb-server recommends:
ii  libhtml-template-perl           2.97-2
ii  mariadb-plugin-provider-bzip2   1:10.11.19-0+deb12u1
ii  mariadb-plugin-provider-lz4     1:10.11.19-0+deb12u1
ii  mariadb-plugin-provider-lzma    1:10.11.19-0+deb12u1
ii  mariadb-plugin-provider-lzo     1:10.11.19-0+deb12u1
ii  mariadb-plugin-provider-snappy  1:10.11.19-0+deb12u1
ii  pv                              1.6.20-1

Versions of packages mariadb-server suggests:
pn  mariadb-test    <none>
pn  netcat-openbsd  <none>

-- Configuration Files:
/etc/mysql/mariadb.conf.d/50-server.cnf changed [not included]

-- debconf information:
  mariadb-server/postrm_remove_databases: false
  mariadb-server/nis_warning:
  mariadb-server/old_data_directory_saved: