[Pkg-libvirt-maintainers] Bug#1111337: Further Debugging

[Kenneth Pronovici]

I enabled debug logs. I don't see any AppArmor errors. However, I ran
across this in the logs when starting the mars guest:

   error : virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_ssd/lv_mars_root': Permission denied error :
   virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_spin/lv_mars_data': Permission denied error :
   virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_ssd/lv_mars_fast': Permission denied

To work around that, I edited /etc/libvirt/qemu.conf, adjusted the
security driver, and restarted the service:

   #security_driver = "selinux"
   security_driver = "none"

That resolved the permission denied error, but made no difference to the
behavior of the guest once it started.  I still see nothing in the
console, and it never becomes available for use.

I also followed your advice and tried importing the existing filesystem
into a new guest. I had saved off the original command used to create
this guest back in 2016, so I adjusted it to use --import and --osinfo
rather than one of install options.

   virt-install \
          --import \
          --osinfo debiantrixie \
          --check path_in_use=off \
          --check mac_in_use=off \
          --connect qemu:///system \
          --virt-type kvm \
          --name mars.2 \
          --memory 22528 \
          --vcpus 4 \
          --graphics none \
          --machine "pc-i440fx-10.0" \
          --disk /dev/vg_ssd/lv_mars_root \
          --disk /dev/vg_spin/lv_mars_data \
          --disk /dev/vg_ssd/lv_mars_fast \
          --network network=virtual,mac=52:54:00:00:00:03

This behaves the same way as the original mars guest.  It starts, but
nothing is ever written to the console and it never becomes available.
There are slight differences in the generated XML.  There are different
PCI ids assigned to the disks, and virt-install chooses host-passthrough
for the CPU instead of a specific CPU type as in the old guest.
However, there's nothing else that looks particularly significant.

Besides the original permission denied errors that I mentioned above,
there aren't any messages that contain 'error', 'exception', 'fail',
'failure', etc. in the libvirt debug logs, regardless of whether I'm
booting the original mars guest or the imported mars.2 guest.

Just as confirmation, I grabbed the contents of kern.log and syslog from
the filesystem using virt-cat:

   virt-cat -d mars /var/log/kern.log > kern.log

There has been nothing written to those logs since the last time the
guest was shut down (when rebooting the hypervisor after upgrading it),
which I think tends to support my theory that the guest isn't really
booting up, even though virsh reports it as started.

I've also experimented with:

- using only the root --disk
- using --cpu host-passthrough,secure=off
- using --boot hd,uefi=off
- using --boot bootmenu.enable=on,bios.useserial=on

None of these made any difference in the final behavior of the VM.  

However, the bootmenu change did yield some interesting results.  With
this option in use, the BIOS bootmenu flashes onto the screen about
every 10 seconds.  If I manage to hit ESC before it goes away, I can
capture this:

   SeaBIOS (version 1.16.3-debian-1.16.3-2)
   Machine UUID 4763922d-20fa-4efa-8261-621281bee979

   iPXE (https://ipxe.org) 00:02.0 C000 PCI2.10 PnP
   PMM+BEFC6C80+BEF06C80 C000

   Press ESC for boot menu.

   Select boot device:

   1. Virtio disk PCI:00:05.0
   2. Legacy option rom
   3. iPXE (PCI 00:02.0)

For option 1, "PCI:00:05.0" matches up with the slot assigned to the
root disk in its <address>, so at least that makes sense.

If I pick option 1, I get the following, which disappears in less than a
second:

   Booting from Hard Disk...
   GRUB loading..
   Welcome to GRUB!

After this is printed, the screen blanks, and I'm eventually sent back
to the BIOS boot menu again.  I suspect this tight loop explains the
100% CPU utilization I see in top, since utilization drops if I hit ESC
and sit at the BIOS menu without making a choice.

I haven't had any success in coaxing GRUB to tell me what's going on. I
manually changed the timeout in /boot/grub/grub.cfg, and that seems to
change the behavior of the BIOS menu loop. This makes me think that GRUB
renders the menu and waits for the timeout before selecting the default,
and then something crashes after that... but for some reason the menu
and whatever else happens after the crash aren't available to the
console.

Do you have any other suggestions about things that I should explore, or
things that might cause the old kernel or init to no longer work with
the new libvirt?

I haven't attached the libvirt debug logs, since they're fairly large.
Let me know if you want them.  

Thanks,

Ken