[debian-mysql] Bug#1121151: mariadb-server: Salsa CI "default-mysql-server and Bookworm upgrade" job fails due to provider plugin ABI mismatch

[aquilamacedo at riseup.net]

Package: mariadb-server
X-Debbugs-Cc: aquilamacedo at riseup.net
Version: 1:11.8.3-2
Severity: normal

Hi,

while running the MariaDB upgrade tests in Salsa CI, I noticed that the
bookworm -> sid upgrade path can leave the system in a state where
mariadbd fails to start due to an ABI mismatch between the server and
the provider plugins:
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8590614

AFAICS, this only hits bookworm-based systems. A similar bullseye-based
upgrade job does not hit this specific mismatch, because bullseye does
not ship the mariadb-plugin-provider-* packages.

During a partial upgrade from bookworm to sid, mariadbd 11.x aborts at
startup with:

  ERROR: mariadbd: Can't open shared library 'provider_bzip2.so'
  (errno: 8, API version for DAEMON plugin provider_bzip2 not supported
by this
  version of the server)

The reason is that the provider plugins are still at 10.11.x while the
server is now 11.x. At the same time, the default Debian configuration
asks MariaDB to load at least provider_lz4:

  plugin_load_add=provider_lz4
  provider_lz4=force_plus_permanent

With an ABI mismatch, this combination is fatal and the server never
comes up.

On bookworm, MariaDB 10.11 installs several provider plugins as
Recommends of mariadb-server.

In the failing scenario, the upgrade is only partial:

* mariadbd is upgraded to 11.x
* the provider plugins stay at 10.11.x, because upgrading
default-mysql-server alone does not force a upgrade of Recommends
* the configuration still tells mariadbd to load the provider plugins
* mariadbd 11.x finds 10.11.x DAEMON plugins and aborts with an API/ABI
mismatch

So we end up with:

  server: 11.x
  providers: 10.11.x

and the server is unusable.

In the MariaDB CI job "default-mysql-server and Bookworm upgrade", the
script does roughly:

  apt-get install default-mysql-server  # installs 10.11 + provider
plugins
  enable sid
  enable artifacts
  apt-get install default-mysql-server  # upgrades only the server
package

Because the provider plugins were already present from the first
install, they are not upgraded when only default-mysql-server is
requested. This reproduces the situation above and the job fails with
the DAEMON plugin API error

To make the CI job pass again, one possible workaround on the CI side
would be to explicitly install the provider plugins together with
default-mysql-server during the bookworm -> sid upgrade, so that apt
upgrades them in one go:

  apt-get install -y
  default-mysql-server
  mariadb-plugin-provider-bzip2
  mariadb-plugin-provider-lz4
  mariadb-plugin-provider-lzma
  mariadb-plugin-provider-lzo
  mariadb-plugin-provider-snappy

This would keep the server and provider plugins in sync and make the
upgrade path pass in CI.

Before changing the job, I wanted to check with you if you consider this
an acceptable CI-only workaround for this upgrade path, or if you would
prefer that this be addressed on the packaging side instead (for
instance by tightening dependencies or otherwise ensuring that provider
plugins cannot be left behind when upgrading the server).