[Pkg-xen-devel] Bug#988333: libxenmisc4.16: libxl fails to grant necessary I/O memory access for gfx_passthru of Intel IGD

[Chuck Zmudzinski]

Detailed description of how I discovered the patch that fixes the bug 
and enables Intel IGD passthrough to Bullseye when using the traditional 
Qemu device model:

A long time ago, during the development of Xen 4.5 in 2014, two patches 
implemented a change to the way permission is granted for an 
unprivileged domain to access PCI/VGA-related I/O memory. Prior to this, 
AFAICT, permission was implicitly granted to access the memory the 
domain requested when a PCI device being passed to the domain was being 
configured. After the change, permission to access such memory is not 
granted without prior explicit permission being configured, and this is 
still the current behavior.

The relevant patches are:

1. 
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=abfb006f1ff4af1424e5b0d0589f0226377fda36

and

2. 
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=0561e1f01e87b777bcc47971e4ae1f420314f4a0

The first of these patches intended to implement explicit granting of 
permission to access the I/O memory that is needed to support the 
gfx_passthru feature in libxl_pci.c, in the 
libxl__grant_vga_iomem_permission function. The second patch implements 
the removal of implicit permission to access the PCI/VGA-related I/O 
memory and causes requests to access such memory by a domain to be 
denied unless prior explicit permission has been configured.

Specifically, the first patch adds 32 (0x20) units (I presume bytes) of 
data starting at memory address 0xa0000 >> XC_PAGE_SHIFT to the memory 
the domain is permitted to access. XC_PAGE_SHIFT is 12, so this memory 
range shows up in the logs when running Xen 4.5 as:

memory_map:add: dom1 gfn=a0 mfn=a0 nr=20

But my testing of these old patches with added custom logging shows that 
another two units (bytes?) are needed:

memory_map:access not permitted: dom1 gfn=fdffc mfn=cc490 nr=2

On versions of Xen prior to adding these two patches the mapping succeeds:

memory_map:add: dom1 gfn=fdffc mfn=cc490 nr=2

The failure to map this memory with gfx_passthru enabled is therefore a 
bug, a regression that was introduced with the two aforementioned 
upstream patches from the development of Xen 4.5 back in 2014.

This bug still exists in current supported versions of Xen because in 
these versions of Xen, passthrough of the Haswell Intel IGD to a Linux 
domain fails with a crash of the i915 Linux kernel module in the Linux 
unprivileged domain when the traditional Qemu device model is used in 
dom0, and the Linux domain is unable to handle IRQ's from the passed 
through IGD device when the upstream QEMU device model is used in dom0. 
Prior to the addition of the aforementioned patches, passthrough of the 
Haswell Intel IGD worked with Linux domains, but not after these patches 
were added to Xen.

When passing through the Haswell Intel IGD to a Windows domain, the 
Windows domain also tries to access the same memory (gfn=fdffc mfn=cc490 
nr=2) and is also denied access just as with a Linux domain, but the 
Windows domain is able to workaround this lack of access to that memory 
and the Intel IGD functions properly without a crash of the Windows 
Intel graphics driver in the Windows domain and, AFAICT, without any 
problems handling the IRQs. I think this is because of a difference in 
the way Windows handles IRQs from the passed through Intel IGD, in 
contrast to how Linux handles IRQs from the passed through Intel IGD.

With a Windows HVM, I see the following message in the log (Xen 4.5):

irq.c:380: Dom2 callback via changed to GSI 28

With a Linux HVM, instead I see the following message in the log (Xen 4.5):

irq.c:380: Dom1 callback via changed to Direct Vector 0xf3

AFAICT, this difference is what causes IGD passthrough to work with a 
Windows domain, but not with a Linux domain, in current supported 
versions of Xen.

This also might partially explain why there are problems configuring the 
IRQ for the Linux domain when using the upstream Qemu device model in 
dom0. It seems likely that when the callback for handling the IRQ is 
direct vector, rather than gsi, the domain needs to access the memory 
that is blocked by the aforementioned patches from way back in 2014. I 
suspect the blocked memory contains essential data about the IRQ that 
the Linux domain needs, but the Windows domain does not necessarily need 
it because it uses a different callback for the IRQ. But more 
investigation is needed and there is probably another bug that needs to 
be patched for the setup to work with the upstream Qemu device model 
instead of the traditional Qemu device model.

Nevertheless, this bug defined narrowly as a failure to grant the 
necessary I/O memory access to a Linux HVM domU for gfx_passthru of the 
Intel IGD is fixed by modifying the first patch discussed above from the 
Xen 4.5 branch committed back in 2014 so it also explicitly grants 
access to the memory defined by the variables mfn=0xcc490 nr=0x2

The patch for the Xen 4.16 package for Sid, version 
4.16.0+51-g0941d6cb-1 follows:

Patch to fix passthrough of Intel IGD to Linux domU

--- a/tools/libs/light/libxl_pci.c
+++ b/tools/libs/light/libxl_pci.c
@@ -2502,6 +2502,7 @@

      for (i = 0 ; i < d_config->num_pcidevs ; i++) {
          uint64_t vga_iomem_start = 0xa0000 >> XC_PAGE_SHIFT;
+        uint64_t vga_iomem2_start = 0xcc490; /* Probably IRQ data, nr = 
0x2 */
          uint32_t stubdom_domid;
          libxl_device_pci *pci = &d_config->pcidevs[i];
          unsigned long pci_device_class;
@@ -2531,6 +2532,25 @@
                    domid, vga_iomem_start, (vga_iomem_start + 0x20 - 1));
              return ret;
          }
+        ret = xc_domain_iomem_permission(CTX->xch, stubdom_domid,
+                                         vga_iomem2_start, 0x2, 1);
+        if (ret < 0) {
+            LOGED(ERROR, domid,
+                  "failed to give stubdom%d access to iomem range "
+                  "%"PRIx64"-%"PRIx64" for VGA passthru",
+                  stubdom_domid,
+                  vga_iomem2_start, (vga_iomem2_start + 0x2 - 1));
+            return ret;
+        }
+        ret = xc_domain_iomem_permission(CTX->xch, domid,
+                                         vga_iomem2_start, 0x2, 1);
+        if (ret < 0) {
+            LOGED(ERROR, domid,
+                  "failed to give dom%d access to iomem range "
+                  "%"PRIx64"-%"PRIx64" for VGA passthru",
+                  domid, vga_iomem2_start, (vga_iomem2_start + 0x2 - 1));
+            return ret;
+        }
          break;
      }