[Pkg-xen-devel] Bug#908453: xen-utils-common: README.comet seems to no longer apply to the current Xen packages

[Hans van Kranenburg]

Hi Keith,

Thanks for your elaborate report. I'll context-quote to try answer a few
things below.

Summary of the info below:
* PVH is not supposed to work with Xen 4.8.
* The documentation in README.comet is wrong for the recent Xen
4.8-stable that we ship now, we should fix that.
* For 4.8 and PV guests, XPTI is also available now (and enabled by
default).
* Silently ignoring invalid configuration is bad.

On 09/10/2018 05:51 AM, Keith Bare wrote:
> Package: xen-utils-common
> Version: 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9
> Severity: important
> 
> Dear Maintainer,
> 
>    * What led up to the situation?
> 
> I needed to build a new Xen DomU.  Since I had seen that Debian had picked
> up the 4.8 "comet" changes and pvshim, I wanted to experiment with Xen's PVH
> mode and have the DomU run the Linux 4.17 kernel in stretch-backports, which
> should be new enough to support PVH.

Parts of the PVH code have been backported. Running PVH guests yourself
is still not supported or recommended with Xen 4.8. If it was
temporarily, in the shim-branch, it was still not recommended.

> It seemed like this should work, given instructions in
> /usr/share/doc/xen-utils-common/README.comet
> -------- 
> * Converting a PV config to a PVH config
> 
> If you have a kernel capable of booting PVH, then PVH mode is both
> faster and more secure than PV or PVH-shim mode.
> 
> - Remove any reference to 'builder' (e.g., `builder="generic"`)
> - Add the following line:
>   type="pvh"
> --------

We just had a little conversation about this in #debian-xen on oftc irc,
and the latest info is that the documentation is (probably) wrong. The
README.comet is not present in upstream 4.8, but was copied from the
security advisory into our packaging.

XSA-254 mentions "PVH is supported in Xen 4.8 only with the 4.8 "Comet"
security release branch."

https://xenbits.xen.org/xsa/advisory-254.html

However...

>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
> 
> I ran xen-create-image (from the xen-tools package) with --dist=stretch to
> build the DomU and its configuration.  I also used --pygrub so Xen would
> boot the kernel installed in the DomU's filesystem.
> 
> I temporarily edited the DomU's Xen/xl configuration to use the pvshim, per
> the instructions in README.comet (type = 'pvh' / pvshim = 1), started the
> DomU, added the stretch-backports repository, and installed
> linux-image-amd64 from backports.  I then shut down the DomU, edited the
> DomU's configuration to remove the pvshim = 1 line, and re-started the DomU.
> 
> 
>    * What was the outcome of this action?
> 
> It appeared the DomU was running in PV mode, despite my having added type =
> 'pvh' to its Xen/xl configuration.
> 
> It was difficult to tell, but I believed the DomU was running in PV mode as
> its kernel printed:
> 
>  - [    0.000000] Hypervisor detected: Xen PV
>  - [    0.000000] Kernel/User page tables isolation: disabled on XEN PV.

Yes, that's PV.

> Some Internet searching also seemed to indicate that "xl list -l <id>"
> should mention "pvh" if the DomU is running in PVH mode.  However, I only
> saw:
> 
>  - "type": "pv",
> 
> There was no mention of "pvh" in the output.
> 
> 
>    * What outcome did you expect instead?
> 
> I expected the DomU to run in PVH mode.  I wasn't sure what this was
> supposed to look like, but I did some experiments (installed the old
> 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 packages and worked around
> bootloader being broken for PVH) and saw that in fact, with that version of
> the Xen packages, the DomU kernel prints:
> 
>  - [    0.000000] Hypervisor detected: Xen HVM
>  - [    0.000000] Booting paravirtualized kernel on Xen PVH
>  - [    0.000000] Kernel/User page tables isolation: enabled
> 
> And xl list -l <id> shows:
> 
>  - "type": "pvh",
> 
> 
>    * More thoughts/discussion.
> 
> It looks like the Debian packages lost support for booting DomUs in PVH mode
> with version 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u6.  Probably because:
> 
>     Update to new upstream version 4.8.3+xsa262+shim4.10.0+comet3.
>     (This is the upstream staging-4.8 branch, which is ahead of the
>     upstream CI-tested stable-4.8 branch by precisely the three
>     most recent XSA fixes.  We are switching away from the special
>     upstream 4.8 comet branch.)

...yes, that would be the missing puzzle piece which explains it. So in
that case, the README.comet which is still in our package is misleading
and incorrect.

> And maybe that's fine... if the mitigation comet and the pvshim provided is
> also effectively provided by XPTI changes that were present in the
> stable-4.8 branch, then I guess it isn't really necessary for anybody to use
> (and thus PVH-mode boot) the shim anymore. 

Yes, using XPTI for PV would be the least bad thing now for 4.8 I guess.
It's enabled by default by the way, so if you didn't see huge
performance problems in your dom0, you might as well try to run PV
guests just like before and see how they behave with XPTI.

> However, if that's the case,
> then it probably doesn't make sense to include README.comet and continue
> packaging the shim in the Debian packages anymore.

If someone is using the shim, we can't just go break that. :|

> The thing that's nefarious (and could be grounds for increasing the bug
> severity), is that anybody that followed the README.comet instructions and
> configured DomUs to boot a PVH-capable kernel without the shim is now,
> probably to their surprise, running their DomU in PV mode.  This means
> they've lost Linux's KPTI protections from Meltdown within their DomU.

No, 64-bit PV does not use KPTI because Xen provides an equivalent
mechanism for the guest (and has been doing that for a long time). Xen
XPTI will add protection to break out of the guest to the hypervisor and
back to the (or another) guest. So afaics now this does not cause
unwanted negative security problems.

> The
> underlying issue is that the xl command seems to silently ignore
> configuration directives it doesn't understand--which, without the 4.8
> comet2 changes, includes 'type'!

Silently ignoring unknown or unused configuration or silently ignoring
an invalid configuration combination and doing something else is bad
practice. I certainly agree.

This is a bug report for upstream. Given the size of my current
(Debian/Xen-related) todo-list, I can't help working on that now.

> This isn't a huge deal for me and my deployment (at the CMU Computer Club).
> Our Xen infrastructure was running jessie/Xen 4.4 at best when the
> Meltdown/Spectre news broke.  So our initial mitigation was to switch
> everything to run in HVM mode (and we've continued doing so since then).  I
> was interested in exploring PVH mode though, since it looked like it was
> more similar to PV mode in some ways that would make it work better with
> various tooling (e.g., xen-tools).  The fact it didn't work with the Debian
> packages the way I thought it would was surprising, and I figured it might
> be surprising for other people too.
> 
> [...]

Note: we're now working to get Xen 4.11 for Buster in shape, so at least
there's something to look forward to. :-)

Regards,
Hans van Kranenburg