[Pkg-xen-devel] On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)

[Antti Kantee]

Hi,

Wei Liu hinted that I should "chime in and / or provide corrections" 
(his words).  I'll attempt to do exactly that by not really replying to 
anything specific.  For the record, when I say "we" in this mail, I mean 
"people who have contributed to the rump kernel project" (as also 
indicated by the email-hat).

First of all, there's a difference between a rump kernel (driver bundle 
built out of unmodified kernel components) and any unikernel you 
construct out of rump kernels ... sort of like how there's a difference 
between Linux and GNU/Linux.

For unikernels, the rump kernel project provides Rumprun, which can 
provide you with a near-full POSIX'y interface.  Rumprun also provides 
toolchain wrappers so that you can compile existing programs as Rumprun 
unikernels.  Rumprun also recently regrew the ability to run without the 
POSIX'y bits; some people found it important to be able to make a 
tradeoff between running POSIX'y applications and more compact "kernel 
plane" unikernels such as routers and firewalls.  But, for brevity and 
simplicity, I'll assume the POSIX'y mode for the rest of this email, 
since that's what the QEMU stubdom will no doubt use.

If the above didn't explain the grand scheme of things clearly, have a 
look at http://wiki.rumpkernel.org/Repo and especially the picture.  If 
things are still not clear after that, please point out matters of 
confusion and I will try to improve the explanations.

Also for simplicity, I'll be talking about rump kernels constructed from 
the NetBSD kernel, and the userspace environment of Rumprun being 
NetBSD-derived.  Conceptually, there's nothing stopping someone from 
plugging a GNU layer on top of NetBSD-derived rump kernels (a bit like 
Debian kXBSD?) or constructing rump kernels out of Linux.  But for now, 
let's talk about the only working implementation.

As far as I know, the API/ABI of the application environment provided by 
Rumprun is the same as the one provided by standard NetBSD.  Granted, I 
didn't perform the necessary experiments to verify that, so take the 
following with a few pinches of salt.  In theory, you could take 
application objects built for NetBSD and link them against Rumprun libs. 
  However, since a) nobody (else) ships applications as relocatable 
static objects b) Rumprun does not support shared libraries, I don't 
know how helpful the fact of ABI compatibility is.  IMO, adding shared 
library support would be a backwards way to go: increasing runtime 
processing and memory requirements to solve a build problem sounds plain 
weird.  So, I don't think you can leverage anything existing.

We do have most of the Rumprun cross-toolchain figured out at this 
point.  First, we don't ship any backend toolchain(s), but rather bolt 
wrappers and specs on top of any toolchain (*) you provide.  That way we 
don't have to figure out where to get a toolchain which produces binary 
for every target that everyone might want.  Also, it makes bootstrapping 
Rumprun convenient, since you just say "hey give me the components and 
application wrappers for CC=foocc" and off you go.

*) as long as it's gcc-derived, for now (IIRC gcc 4.8 - 5.1 are known to 
work well, older than that at least C++ won't work).  clang doesn't 
support specs files at least AFAIK, so someone would have to figure out 
how to move the contents of the specs into the wrappers, or whatever 
equivalent clang uses.  (patches welcome ;)

The produced wrappers look exactly like a normal cross-toolchain.  The 
tuple is the same as what NetBSD uses, except with rumprun added in the 
middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. 
That naming scheme means that most GNU-using software compiles nicely 
for Rumprun just by running configure as ./configure 
--host=x86_64-rumprun-netbsd followed by "make".  Sometimes you 
additionally need things like --disable-shared, but all in all 
everything works pretty well.  See 
http://repo.rumpkernel.org/rumprun-packages for a bunch of "case 
studies", not limited to just GNU autotools.

After "make", before launch we have an additional step called "bake", 
which links the specific kernel components onto the binary.  So for 
example, you can make the compiled binary run on Xen or KVM depending on 
which kernel components you bake onto it.  As a crude analogy, it's like 
scp'ing a binary to a Xen or KVM or bare metal system, but since the 
Rumprun unikernel is missing exec, we use the linker to perform "system 
selection".

So for shipping, one option is to ship the binary after "make", but then 
you also need to ship the toolchain.  The other option is to ship the 
baked binary, but then you lose some of your possibilities on how to 
target the binary.  I'm not sure either option is right for all cases.

We're still trying to figure out the exact form and figure of 
bake+launch.  In the original implementation we assumed that at 
launch-time we could cheaply control all of the details of the backend 
(a la xl.conf).  That assumption proved to be bad not only for example 
for embedded systems (which we should've foreseen), but also for cases 
like Amazon EC2 (where creating something launchable and launching 
something are seriously separate steps).  I'm not going to go into 
details in this thread, but just saying that we still have a few things 
to figure out in the full source-to-execution chain.  If someone wants 
to ship something, please please check with the rump kernel community 
first so that we know that you want to depend on some syntax/semantics 
not changing anymore.

I don't really have good solutions for the packaging problem.  Building 
a "full distro" around rump kernels certainly sounds interesting, and 
we're sort of trying to experiment with that with rumprun-packages. 
However, for packages sooner or later we need to assimilate a real 
packaging system which properly manages dependencies, licenses, 
vulnerabilities, etc.  I'm unsure we'd want to assimilate every existing 
packaging system -- Rumprun works the same on every platform, after all. 
  Hard to say for now, I hadn't actually considered the case where a 
distro might want to natively ship binary Rumprun images, as opposed to 
just the toolchain and components.

If I were you folks, I'd start by getting qemu out of the door, and 
worry about generalized solutions when someone wants to ship the second 
unikernel (or third or any small N>1).  If you can't sell to distros 
something that solves a problem, it's unlikely you'll be able to sell a 
framework for solving the same one problem (though, granted, it might be 
easier to sell a framework to computer folk -- "nevermind the solution, 
here's abstraction!").

If you haven't tried rumprun-packages, I recommend to pick something 
from there and see how it works.  Doing so might give some more 
perspective on how easy or difficult it would be to package QEMU.

Whoops, ended up being a bit longer than what I hoped for, but with any 
luck at least some new information was communicated.

   - antti