[Pkg-libvirt-maintainers] Bug#1009878: /usr/sbin/libvirtd: Live migration of guest VM fails with kernel oops error

[Ross Moutell]

Package: libvirt-daemon
Version: 7.0.0-3
Severity: important
File: /usr/sbin/libvirtd

Dear Maintainer,

   * What led up to the situation?

     Attempting live migration of VM between hosts, either from virt-manager on a seperate workstation or from the host itself via terminal.

     Example - virsh migrate --live web02 qemu+ssh://hypervisor01:64228/system 

   * What exactly did you do (or not do) that was effective (or ineffective)?

     Investigated the following logs.

     /var/log/syslog

     Apr 18 09:25:45 hypervisor04 libvirtd[542]: Cannot start job (query, none, none) for domain email01; current job is (none, none, migration in) owned by (0 <null>, 0 <null>, 0 remoteDispatchDomainMigratePrepare3Params (flags=0x19)) for (0s, 0s, 305s)

     /var/log/kern.log

     Apr 17 22:45:45 hypervisor05 kernel: [  804.114785] Internal error: Oops: 96000004 [#1] SMP
     Apr 17 22:57:49 hypervisor05 kernel: [  206.952482] Internal error: Oops: 96000004 [#1] SMP
     Apr 18 01:06:11 hypervisor05 kernel: [  463.133575] Internal error: Oops: 96000004 [#1] SMP
     Apr 18 11:12:39 hypervisor05 kernel: [36851.073954] Internal error: Oops: 96000004 [#2] SMP
     Apr 18 11:29:19 hypervisor05 kernel: [37850.896463] Internal error: Oops: 96000004 [#3] SMP

     error in dmesg

     [  324.673078] audit: type=1400 audit(1650240228.486:23): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-72799f47-939c-4415-92c3-73ec371425fd" pid=979 comm="apparmor_parser"
     [  324.768028] audit: type=1400 audit(1650240228.582:24): apparmor="DENIED" operation="capable" profile="libvirtd" pid=542 comm="rpc-worker" capability=39  capname="bpf"
     [  324.774241] audit: type=1400 audit(1650240228.586:25): apparmor="DENIED" operation="capable" profile="libvirtd" pid=542 comm="rpc-worker" capability=38  capname="perfmon"
     [  326.770324] audit: type=1400 audit(1650240230.582:26): apparmor="DENIED" operation="capable" profile="libvirtd" pid=542 comm="rpc-worker" capability=39  capname="bpf"

     I added

      capability bpf,
      capability perfmon,

     to /etc/apparmor.d/usr.sbin.libvirtd which resolved the DENIED errors but did not resolve the live migration failures.

   * What was the outcome of this action?

     The following errors were produced.

     kernel:\[37850.896463\] Internal error: Oops: 96000004 \[#3\] SMP

     Message from syslogd at hypervisor05 at Apr 18 11:29:19 ...

     kernel:\[37851.195226\] Code: 910003fd f9000bf3 2a0003f3 97ff7164 (b95ed801)

     The VM that was submitted for migration ends up hung in a paused state. The only way to recover it is force power off on the VM, then 'sudo systemctl restart libvirtd.service'. The VM can then be powered on again normally.

   * What outcome did you expect instead?

     Live migration to complete successfully which has been the case on eariler kernel versions. However at this time I do not know which kernel versions worked other than the one it shipped with which was as follows.

     linux-image-5.10.0-8-arm64  5.10.46-5  arm64  Linux 5.10 for 64-bit ARMv8 machines (signed)


-- System Information:
Debian Release: 11.3
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: arm64 (aarch64)

Kernel: Linux 5.10.0-13-arm64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_CRAP
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libvirt-daemon depends on:
ii  libblkid1                   2.36.1-8+deb11u1
ii  libc6                       2.31-13+deb11u3
ii  libdevmapper1.02.1          2:1.02.175-2.1
ii  libgcc-s1                   10.2.1-6
ii  libglib2.0-0                2.66.8-1
ii  libnetcf1                   1:0.2.8-1.1
ii  libparted2                  3.4-1
ii  libpcap0.8                  1.10.0-2
ii  libpciaccess0               0.16-1
ii  libselinux1                 3.1-3
ii  libudev1                    247.3-7
ii  libvirt-daemon-driver-qemu  7.0.0-3
ii  libvirt0                    7.0.0-3
ii  libxml2                     2.9.10+dfsg-6.7+deb11u1

Versions of packages libvirt-daemon recommends:
ii  libvirt-daemon-driver-lxc   7.0.0-3
pn  libvirt-daemon-driver-vbox  <none>
ii  libvirt-daemon-driver-xen   7.0.0-3
ii  libxml2-utils               2.9.10+dfsg-6.7+deb11u1
ii  netcat-openbsd              1.217-3
ii  qemu-system                 1:5.2+dfsg-11+deb11u1
ii  qemu-system-arm [qemu-kvm]  1:5.2+dfsg-11+deb11u1

Versions of packages libvirt-daemon suggests:
pn  libvirt-daemon-driver-storage-gluster       <none>
pn  libvirt-daemon-driver-storage-iscsi-direct  <none>
pn  libvirt-daemon-driver-storage-rbd           <none>
pn  libvirt-daemon-driver-storage-zfs           <none>
ii  libvirt-daemon-system                       7.0.0-3
pn  numad                                       <none>

-- no debconf information