[Pkg-libvirt-maintainers] Bug#1111337: libvirt: After upgrading to trixie, some kvm guests never successfully boot

[Kenneth J. Pronovici]

Source: libvirt
Version: 11.3.0-3
Severity: important

I have a hypervisor (sol) with 4 guests (mercury, venus, mars, and
jupiter).  The hypervisor and first three guests run Debian.  The 4th
guest runs Home Assistant.  The Debian guests have been running in
roughly their current configuration since 2016 (wheezy) and have been
upgraded since then for each Debian release.

I have been working to upgrade all of the Debian guests to trixie. I
followed the same process as usual: I upgraded each guest one at a time,
rebooted, and tested.  Once all guests were working, I upgraded the
hypervisor.  After rebooting the hypervisor, none of the guests were
running, and neither was the virtual network.  I started the network
manually, but I haven't managed to get any of the guests working.

The initial problem with the guests was that their machine type
pc-i440fx-2.1 was no longer supported.  I adjusted this to
pc-i440fx-2.4, and virsh accepted that change. Now, I can start the
guests, and they report as running, but they never do anything useful. 

As far as I can tell, nothing is ever even written to the console.  The
guests definitely never come up far enough to respond to a ping or
accept an SSH connection.  If I try to cleanly shut down a guest, it
never stops.  I have to use destroy instead of shutdown.  When I attempt
to start a guest, I get a new kvm process that immediately uses 100% of
a CPU and stays like that for at least 30 minutes (which is as long as
I've been willing to wait).

Oddly, the jupiter guest (Home Assistant) boots without problems and
seems to be working fine.  So, I assume there must be something
wrong/different about the Debian guests that conflicts with the new
libvirt version in trixie, but can't figure out what that might be.

I compared the current state of each guest against my pre-upgrade backup
of /etc/libvirt, and none of the changes are surprising.  Besides
the change to the machine name, there are some minor changes to some XML
stanzas which libvirt seems to make on its own. (I removed them and they
came back.)  All of the other changes within /etc/livirt came with as 
part of the upgrade to trixie.

The main difference I see is that jupiter uses <os firmware='efi'>, and
its only disk is a .qcow2 file.  The other guests do not use EFI and
their disks are raw block devices instead.  

I hesitate to attach configuration here, because I'm not sure what will
be useful to you.  Let me me know what questions you have, and I can
attach configuration or run other diagnostics.

Thanks,

Ken

-- System Information:
Debian Release: 13.0
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.12.41+deb13-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)