[Pkg-libvirt-maintainers] Bug#993856: libvirt-daemon-system: vfio device passthrough fails with device pools due to apparmor profile

[Vasudev Kamath]

Package: libvirt-daemon-system
Version: 7.6.0-1
Severity: important

Dear Maintainer,

Possibly related bug [1]. Issue is similar to what is explained in this bug 
but is not addressed by the fix which is already present in src:libvirt 7.6 
version.

PS: Though I reporting from unstable machine actual test was done using libvirt 7.6 
from unstable built for Bullseye. 

I'm defining the network device pool which looks like below

<network>
  <name>passthrough</name>
  <uuid>f152e522-96d1-4a74-8aae-01f94244f8df</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='ens6np0'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x6'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x7'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x6'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x7'/>
  </forward>
</network>

And the network configuration in libvirt domain looks like below

    <interface type='network'>
      <mac address='52:54:00:e1:5b:95'/>
      <source network='passthrough'/>
      <teaming type='transient' persistent='ua-backup0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>

When I start the domain even though domain starts fine VF pass through does not happen and the following
message is seen in the dmesg output

[11236.601474] audit: type=1400 audit(1630925018.676:49): apparmor="DENIED" operation="open" profile="libvirt-e70e9c2c-110c-401c-982f-cb384d158471" name="/dev/vfio/315" pid=5929 comm=43505520382F4B564D requested_mask="wr" denied_mask="wr" fsuid=64055 ouid=64055

and passthrough does not happen.

Note that this does not happen if the device pool interface is not present during start of domain and hot
attached using below command

sudo virsh attach-device --live --config debian10 network-pool-debian10.xml

To get the above working here is what I did I edited the /etc/apparmor.d/libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471 to
add line /dev/vfio/vfio rw, and this is what the changed file looks like

iaas at 515-21020200100006:~$ sudo cat /etc/apparmor.d/libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471
#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile libvirt-e70e9c2c-110c-401c-982f-cb384d158471 flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  #include <libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471.files>
  #
  # for vfio hotplug on systems without static vfio (LP: #1775777)
  /dev/vfio/vfio rw,
}

Post the change I did following

sudo aa-teardown
sudo systemctl restart libvirtd
sudo systemctl restart apparmor

And on the next start device passthrough happens. I'm not sure if what I did is right but this seems to work
and I would be happy to see this done in the apparmor profile shipped by libvirt.

PS: I'm noob with apparmor all I did was bit of experiment to get the things working for my usecase.

If any other information is needed from my side please let me know.


[1] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1775777


Thanks and Regards,
Vasudev

-- System Information:
Debian Release: bookworm/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-8-amd64 (SMP w/8 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libvirt-daemon-system depends on:
ii  adduser                         3.118
ii  debconf [debconf-2.0]           1.5.77
ii  gettext-base                    0.21-4
ii  iptables                        1.8.7-1
ii  libvirt-clients                 7.6.0-1
ii  libvirt-daemon                  7.6.0-1
ii  libvirt-daemon-config-network   7.6.0-1
ii  libvirt-daemon-config-nwfilter  7.6.0-1
ii  libvirt-daemon-system-systemd   7.6.0-1
ii  logrotate                       3.18.1-2
ii  policykit-1                     0.105-31

Versions of packages libvirt-daemon-system recommends:
ii  dmidecode                    3.3-3
ii  dnsmasq-base [dnsmasq-base]  2.85-1
ii  iproute2                     5.13.0-2
ii  mdevctl                      0.81-1
ii  parted                       3.4-1

Versions of packages libvirt-daemon-system suggests:
ii  apparmor    2.13.6-10
pn  auditd      <none>
ii  nfs-common  1:1.3.4-6
ii  open-iscsi  2.1.3-5
pn  pm-utils    <none>
ii  radvd       1:2.18-3
ii  systemd     247.9-1
pn  systemtap   <none>
pn  zfsutils    <none>

-- Configuration Files:
/etc/libvirt/qemu.conf [Errno 13] Permission denied: '/etc/libvirt/qemu.conf'

-- debconf information:
  libvirt-daemon-system/id_warning: true