Bug#776450: Xen PVH support for grub-xen in Buster

Sat Jan 5 21:51:13 GMT 2019

Hi Colin,

Thanks for the quick reply.

On 1/5/19 8:04 PM, Colin Watson wrote:
> On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg wrote:
>> In december, Xen PVH support has been committed in grub master:
>> https://www.mail-archive.com/grub-devel@gnu.org/msg28125.html
>>
>> The last pieces needed in the Linux kernel to boot PVH with grub2 landed
>> in Linux 4.20. I asked our kernel team to carry those patches on top of
>> the Linux 4.19 kernel that is going to ship in Buster, and that wish was
>> granted:
>>
>> https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795
>>
>> Please see the commit message in there for an explanation why having the
>> possibility to boot PVH+grub in Debian Buster out of the box would be great.
>>
>> So here's the question: Do you want to help completing the puzzle and
>> getting a PVH capable grub2 boot image in the grub-xen-* packages in Buster?
> 
> I think I'm OK with cherry-picking the relevant patch stack.  Presumably
> this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
> packages, as is usual for separate platform builds?

I was thinking about having an additional...

    /usr/lib/grub-xen/grub-i386-xen_pvh.bin

...in the grub-xen-host package, built with the same config file
(debian/grub-xen-host_grub.cfg in the packaging) and then...

    /usr/lib/grub/i386-xen_pvh/*.mod

...in grub-xen-bin, just as an additional third option besides i386-xen
and x86_64-xen? This would be the most convenient "upgrade path". If you
want to switch to PVH, just change kernel line in the xen guest config
file, and set type='pvh'.

>> Now, I create a grub.cfg file and then...
>>
>> ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
>> grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/
>> boot/grub/grub.cfg=grub.cfg
>>
>> ...I end up with something that I can use as kernel image for the Xen
>> PVH virtual machine.
> 
> Can you elaborate on what you mean by "use as kernel image for the Xen
> PVH virtual machine"?  In other words, what does the process of
> installing a boot loader in a PVH guest look like?

You don't. There is no BIOS, no boot loader, it is like booting a PV
guest with grub. The whole 'trick' about PVH is that it's HVM, but then
without the qemu etc part.

https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode

So, if I have this grub.cfg... (as example, not what we want for the
package)

---- >8 ----
root='(xen/xvda)'
insmod xzio
insmod gzio
insmod btrfs
insmod ext2
echo 'Loading Linux ...'
linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop
echo 'Loading initial ramdisk ...'
initrd /initrd.img
echo 'There we go! ...'
boot
---- >8 ----

...and then use grub-mkstandalone like above, and name my output file
flapsie.bin, and put it in / on the dom0, then I can do this in my xen
guest config file:

---- >8 ----
kernel = "/flapsie.bin"
type = "pvh"
---- >8 ----

Now if I start the Xen domU, it will load the kernel and initrd from the
filesystem inside the domU. No boot sector, no partition table, no boot
loader, no grub config file or anything grub related on the filesystem
of the virtual machine.

So this is what I'm used to [0], but the way in which the Debian
grub-xen packages work is a bit different of course, since it tries to
do most of the process inside the virtual machine, and has a very
generic config file that delegates as much as possible to additional
grub config with access to modules inside the virtual machine itself.

So the thought process I'm going through right now is how to add this
third option for the debian package.

> The current code in
> upstream grub-install essentially just does a grub-mkimage but doesn't
> have any platform-dependent install steps.
> 
> If you run upstream grub-install, are you then just supposed to write
> 'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your guest
> configuration file?  In PVH mode, is the file name referenced via
> "kernel" loaded from the dom0's filesystem or the domU's?

Ok, does the above make this more clear? It's the same way of working as
for PV.

> (Compare
> https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
> which we carry for PV guests, although I gather PVH is sufficiently
> different that the same approach probably doesn't work.)

Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
also don't see any of those files in the grub-xen-* packages?

I'll try to see if I can get things working when creating a grub image
using the grub-xen-host_grub.cfg like we have already, and then put
module stuff and a grub.cfg on the domU filesystem.

If that works, then it's a matter of figuring out how to get that
grub-i386-xen_pvh.bin and the modules built in the packaging?

Thanks,
Hans

[0] Background of the whole story how we ended up doing so for PV:
https://tech.mendix.com/linux/2014/09/06/pvgrub2/
https://tech.mendix.com/linux/2014/09/16/pvgrub2-live/