[Pkg-openssl-devel] Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so

[Helmut Grohne]

Package: upgrade-reports
Severity: important
X-Debbugs-Cc: debian-devel at lists.debian.org,debian-release.debian.org,pkg-systemd-maintainers at lists.alioth.debian.org,pkg-openssl-devel at alioth-lists.debian.net,deity at lists.debian.org

Hello,

while working on a bookworm -> trixie upgrade failure, I noticed a strange line 
showing up.

| Preparing to unpack .../openssh-server_1%3a10.0p1-5_amd64.deb ...
| systemctl: error while loading shared libraries: libcrypto.so.3: cannot open shared object file: No such file or directory

This is openssh-server.preinst failing to systemctl stop 
rescue-ssh.service. I talked to Colin and both of us agreed that this 
instance is probably practically irrelevant. However, I still think 
there is a problem. Due to the time64 transition, libssl3 was renamed to 
libssl3t64 and for some reason apt ends up removing libssl3 before 
unpacking libssl3t64. Given Breaks+Replaces, unpacking libssl3t64 after 
having deconfigured libssl3 before removing libssl3 should be fine, but 
apt does not like that solution. As a result, libcrypto.so.3 is 
temporarily removed.

deb-systemd-invoke is part of init-system-helpers and therefore 
essential. It calls out to systemctl, which is not essential but for all 
practical matters we really should be treating it as if it was and 
maintainer scripts expect it to work at all times. libssl3 or libssl3t64 
are pseudo-essential. Some part (apt or openssl) violates policy during 
the upgrade as being pseudo-essential requires it to work at all times 
even when unpacked.

In practice, this means that systemctl cannot be expected to work in 
maintainer scripts. This will mostly affect preinst scripts (not just 
openssh-server) trying to stop services. For instance, it is conceivable 
that we could fail to stop mariadb or postgresql due to this (but there 
is no practical evidence of this ever having happened). Failure to stop 
services violates assumptions placed by package maintainers and that may 
have all sorts of consequences. I have several reports of systemctl 
having failed during release upgrades without having failed the upgrade 
transaction as a whole.

It really is unclear whether this has practical consequences and whether 
there is a dataloss scenario something else that makes this problem 
practically relevant. We typically reboot after a dist upgrade (at least 
that's what release notes strongly recommend) and doing so tends to fix 
any failure to stop or start services. I have no evidence of this 
problem having caused a real issue (beyond that message).

If you have earlier upgraded from bookworm to trixie. You should be able 
to search in your /var/log/apt/term.log* for the earlier message to see 
whether you were affected.

In talking to Ivo and Paul, we agreed to report the problem to d-devel 
via upgrade-reports. At this stage we want to gauge the impact and 
better understand how serious this actually is, so following up on the 
bug report with evidence (dropping all lists if that's all you add) is 
highly appreciated.

The options for fixing this are dim. Reverting the t64 transition for 
openssl and going dual-ABI seems highly unlikely even though it would 
fix the problem at the root. Other options are dim, because we have no 
scripts that are guarantueed to run before apt chooses to remove 
bookworm's libssl3. We considered doing changes to bookworm to mitigate. 
Conceivably, a bookworm update could add a libssl3.preinst that diverts 
the library to keep it around until it is overwritten by libssl3t64.

I invite others to work on the problem as I have no capacity to do it. 
I'm still yak shaving another release upgrade problem and would like to 
enjoy DebConf. Thank you

Helmut