[Pkg-libvirt-maintainers] Bug#905772: Not Fixed by dh* in the meantime, actually got worse in experimental

[Christian Ehrhardt]

Hi,
I checked this issue for Ubuntu bug 1786179 as I wanted to drop the
related delta that we formerly had. That is the same topic as
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905772 that we
discuss here.

At first I thought the changes to dh_ resolved this sysV-vs-systemd
fight as I've seen it happen in other packages.

After initial install we have all sockets, but no service running => ok
But if services are running (later in the live-cycle) and we trigger a
reinstall/upgrade they are all restarted (which they are not supposed
to be done).

4     0  6927  6039  20   0  68836 55752 poll_s S+   ?          0:00
\_ apt install --reinstall libvirt-daemon-system
4     0  7045  6927  20   0  12924  4524 do_wai Ss+  pts/1      0:00
   \_ /usr/bin/dpkg --status-fd 25 --configure --pending
0     0  7046  7045  20   0  25560 16328 do_wai S+   pts/1      0:00
       \_ /usr/bin/perl -w /usr/share/debconf/frontend
/var/lib/dpkg/info/libvirt-daemon-system.postinst configure 5.4.
4     0  7055  7046  20   0   2572  1352 do_wai S+   pts/1      0:00
           \_ /bin/sh
/var/lib/dpkg/info/libvirt-daemon-system.postinst configure
5.4.0-0ubuntu1~ppa6
0     0  7492  7055  20   0  11028  2464 poll_s S+   pts/1      0:00
               \_ /bin/systemctl restart libvirt-guests.service
virtlockd-admin.socket virtlockd.service virtlockd.sock


This causes it to hang for 30-60 seconds on the upgrade which isn't
good but also not too bad.
But the logd/lockd services are restarted (I see new PIDs) which is
bad since those are all "--no-stop-on-upgrade".

I went back to (I know this is less of an option for Debian, but it
proved to be a great stop gap measure all too often) drop the sysV
scripts.

And back on that code not only does it look fine after install, now
also reinstall/upgrade work again.
The PIDs stay constant and no hang is perceived.
- install (sockets up, services down) = good
- reinstall (sockets up, services down) = good
- reinstall with services up
  - no hang perceived = good
  - PIDs still change = bad

So it is better without sysV, but not perfect (as in former versions) either.

Since there were discussions about reproducibility, the shortest test IMHO is:
$ apt install libvirt-daemon-system
$ systemctl start virtlogd.service
$ systemctl start virtlockd.service
$ systemctl status virtlogd.service virtlockd.service --no-pager
--lines 1 | grep PID
 Main PID: 14021 (virtlogd)
 Main PID: 14020 (virtlockd)
$ apt install --reinstall libvirt-daemon-system
# the PIDs will have changed, but they should not being "--no-stop-on-upgrade"

With compat 12 (even with sysV dropped as we did in Ubuntu) the
services will restart which is not what is wanted.
I verified e.g. Disco which has libvirt 5.0 and the sysV dropped, no
problem there.

For simplicity I compare/debug the versions with sysV dropped (less
interactions).
A lot of the postinst is the same, here "old = good" (Disco based on
libvirt 5.0) and "new = bad" (Eoan based on your 5.3 updated to
libvirt 5.4 from upstream).

old:
  dh_systemd_start/12ubuntu1 for libvirtd.service
  => deb-systemd-invoke $_dh_action 'libvirtd.service' (action => restart)
  dh_systemd_start/12ubuntu1 for "all others"
  deb-systemd-invoke start ... (all services and sockets)
  # on already running services that is a no-op the PIDs stay the same
new:
  dh_installsystemd/12.1.1ubuntu1 for all services
  deb-systemd-invoke $_dh_action (action => restart)
  # this is what breaks the current libvirt, calling restart on the service

I started a buster system to check if this is special to Ubuntu, or at
least only to what we have in -experimental.
Ok, as initially reported on the bug here in Buster this issue applies
to virtlockd but not virtlogd.
I installed 5.2.0-2 from experimental, and there as expected both PIDs
are recycled.
Which makes sense given that it seems a non libvirt specific issue in
the postinst as generated by dh_installsystemd.

The d/rules line for all affected services surely has --no-stop-on-upgrade:
  dh_installsystemd -p libvirt-daemon-system --no-stop-on-upgrade
$(LIBVIRT_SYSTEM_SERVICES)

Not sure what to do yet - I'm experimenting, but I wanted to keep you
in the loop as well.

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd