[pkg-bacula-devel] Bug#1109499: [debian-mysql] Bug#1109499: bacula-director-sqlite3: fails to dist-upgrade from bookworm to trixie

[Otto Kekäläinen]

Hi!

> > > When there are failures, it's not bad when this is immediately visible
> > > to the administrator.
> >
> > If the server is not fully dedicated to just running Bacula, then
> > preventing dpkg/apt from doing anything is a bit overkill just for the
> > sake of notifying the user.
>
> It is the only reliable way to notify the user, and appropriate when
> something major failed.
>
> Abort with a good error message, and the user knows that something failed
> and has a good starting point for searching for searching for a solution.
>
> Really bad are silent failures where you only notice some time afterwards
> that something isn't working, and then have to start debugging what the
> problem might be.

You can for sure emit warnings there and they will be visible in the
shell session and dpkg log, but you should not stop the entire
dpkg/apt system from being used by having a maintainer script exit
with an error code and not recover.

There was an attempt to describe best practices related to this in the
developers' reference
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=801065) and that is
still to be done.


> > There are many ways to communicate things to the system admin, but if
> > they fail, the service would still restart the database schema
> > migration etc immediately after restart when the service file would
> > try to start Bacula and notice that upgrade wasn't completed.
..
> We have to really really really try hard to avoid rebooting while any
> kind of upgrade or migration is ongoing, sice this might create really
> nasty breakages.

Note "restart the service". I didn't suggest you reboot the host to
attempt to run Bacula.

> Can you 100% guarantee that it will never ever cause a problem when
> I unplug the power of the machine at any time during mariadb-upgrade?

No, but I cannot guarantee 100% that mariadb-upgrade passes in general
either. It is just a bunch of SQL commands.
I can however say that it is idempotent and can be run and attempted
multiple times. And this bug report is not about mariadb-upgrade, but
about the Bacula upgrade.

> > > > Doing an upgrade check and upgrading the
> > > > database as part of the startup sequence will also behave correctly if the
> > > > user is for example restoring an old database from backups and trying to
> > > > start it with a new version of Bacula.
> > >
> > > How do you restore anything from backups when the database of your
> > > backup server is broken?
> >
> > We are now talking about Bacula's own database schema, right? For
> > example, if the hardware was broken and the sysadmin is installing
> > Trixie on a new host and installs the latest Bacula but takes the
> > Bacula database from the old Bookworm server, the Bacula schema would
> > never migrate if it only happens when triggered by dpkg.
>
> Whatever the user does manually is outside the scope of what we
> have to support when upgrading.

Starting, stopping and in general interacting with the systemd/init
service we ship in Debian should not be "out of scope".

...
> And if connecting to the configured database fails, this is a serious
> error where aborting from the postinst is appropriate.
>
> There is also a certain irony that you want users to manually stop the
> MariaDB server before the upgrade due to fear of an unclean shutdown
> when running the old version, which guarantees that every MariaDB-using
> service that gets configured before mariadb-server has an unclean
> shutdown with the old version due to non-availability of the database.
>
> It would cause far less problems if the release notes would instead
> document that in this rare error situation of an unclean MariaDB
> shutdown it is required to downgrade to the bookworm mariadb-server
> for error recovery.

Unfortunately it is not that rare. Upgrades of Debian releases
includes upgrades of major MariaDB versions, and due to how upstream
code works and that they don't support crash recovery across arbitrary
major versions we have to continue to advice to users that they ensure
they had a clean shutdown.

We could though consider updating it to recommend either shutdown or
restart, as after a recent restart the buffers are small and a unclean
shutdowns are unlikely. But still if an unclean shutdown happens,
apt/dpkg does not offer any rollback features so re-installing the
binaries of the previous MariaDB version will require a lot of manual
work and can be a situation that is hard to recover from.

Anyway, this bug is about Bacula doing it's own schema upgrades from a
maint script and expecting that the database is always available.
However there is no such guarantee. It might be that dpkg is in the
process of upgrading the network stack or some network filesystems or
something, and it might be that the database isn't even local to the
host that is running Bacula. I think it is overkill for the Bacula
maint script to stop all of dpkg/apt from progressing in case of such
an error.

My strong recommendation would be for Bacula to do a health check and
potential upgrade when the service starts/restarts, and emit errors to
the system journal and stderr if the service fails to come online.