[Pkg-xen-devel] [Xen-devel] On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)

[Ian Campbell]

On Tue, 2015-09-08 at 15:03 +0000, Antti Kantee wrote:

> For unikernels, the rump kernel project provides Rumprun, which can 
> provide you with a near-full POSIX'y interface.

I'm not 100% clear: Does rumprun _build_ or _run_ the application? It sound
s like it builds but the name suggests otherwise.

>   Rumprun also provides 
> toolchain wrappers so that you can compile existing programs as Rumprun 
> unikernels.

Do these wrappers make a rump kernel build target look just like any other
ross build target? (I've just got to the end and found my answer, which was
yes. I've left this next section in since I think it's a nice summary of
why it matters that the answer is yes)

e.g. I have aarch64-linux-gnu-{gcc,as,ld,ar,etc} which I can use to build
aarch64 binaries on my x86_64 host, including picking up aarch64 libraries
and headers from the correct arch specific patch.

Do these rumprun-provided wrappers provide x86_64-rumpkernel
-{gcc,as,ld,ar,etc} ?

Appearing as a regular cross-compilation target is, I think, going to be
important to being able to create rumpkernel based versions of distro
packages.

I think that package maintainers ideally won't want to have to include a
bunch of rumpkernel specific code in their package, they just want to
leverage the existing cross-compilability of their package.

This may not be super critical for a standalone package with a tiny build
dependency chain. Consider building a QEMU rumpkernel to be shipped in a
Debian package. The existing QEMU source package in Debian has ~30 library
dependencies, many of which I expect have their own dependencies etc. For
another metric consider for the regular x86 binary:

$ ldd /usr/bin/qemu-system-x86_64  | wc -l
87
$

So without a normal looking cross env we are talking about asking maybe 60
-90 different library maintainers to do something special for rumpkernels.

(Surely some of those 60-80 are things you would omit from a rump build for
various reasons, but still...)

> If the above didn't explain the grand scheme of things clearly, have a 
> look at http://wiki.rumpkernel.org/Repo and especially the picture.  If 
> things are still not clear after that, please point out matters of 
> confusion and I will try to improve the explanations.

I think that wiki page is clear, but I think it's orthogonal to the issue
with distro packaging of rump kernels.

>   However, since a) nobody (else) ships applications as relocatable 
> static objects b) Rumprun does not support shared libraries, I don't 
> know how helpful the fact of ABI compatibility is.  IMO, adding shared 
> library support would be a backwards way to go: increasing runtime 
> processing and memory requirements to solve a build problem sounds plain 
> weird.  So, I don't think you can leverage anything existing.

This is an interesting point, since not building a shared library is
already therefore requiring packaging changes which are going to be at
least a little bit rumpkernel specific.

Is it at all possible (even theoretically) to take a shared library (which
is relocatable as required) and to do a compile time static linking pass on
it? i.e. use libfoo.so but still do static linking?

> We do have most of the Rumprun cross-toolchain figured out at this 
> point.  First, we don't ship any backend toolchain(s), but rather bolt 
> wrappers and specs on top of any toolchain (*) you provide.  That way we 
> don't have to figure out where to get a toolchain which produces binary 
> for every target that everyone might want.  Also, it makes bootstrapping 
> Rumprun convenient, since you just say "hey give me the components and 
> application wrappers for CC=foocc" and off you go.

> The produced wrappers look exactly like a normal cross-toolchain.  The 
> tuple is the same as what NetBSD uses, except with rumprun added in the 
> middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. 

Great, I think this solves a large problem (whether its a wrapper or a
"proper cross compiler" I think is of limited importance as long as it
behaves enough like the latter)

> I don't really have good solutions for the packaging problem.  Building 
> a "full distro" around rump kernels certainly sounds interesting,

FWIW I don't think we need a full distro, just sufficient build
dependencies for the actual useful things (maybe that converges on to a
full distro though).

>   Hard to say for now, I hadn't actually considered the case where a 
> distro might want to natively ship binary Rumprun images, as opposed to 
> just the toolchain and components.

One concrete example of what I want is a Debian package in the Debian
archive which the xen-tools.deb can depend on which provides a qemu
rumpkernel binary such that users can use stubdomain functionality for
their HVM guests.

Debian are (most likely) not going to accept a second copy of the QEMU
source in the archive and likewise they wouldn't want a big source package
which was "qemu + all its build dependencies" or anything like that,
especially when "all its build dependencies" is duplicating the source of
dozens of libraries already in Debian.

> If I were you folks, I'd start by getting qemu out of the door

That's certainly the highest priority, at the moment I don't think we
actually have a QEMU Xen dm based on rumpkernels which anyone could package
anyway, irrespective of how hard that might be.

> , and 
> worry about generalized solutions when someone wants to ship the second 
> unikernel (or third or any small N>1).

Unfortunately I think the N==1 case is tricky already from a distro
acceptance PoV. (At least for Binary distros, it's probably trivial in a
Source based distro like Gentoo)

Ian.