[Pkg-xen-devel] [Xen-devel] On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)

[Antti Kantee]

On 08/09/15 16:15, Ian Campbell wrote:
> On Tue, 2015-09-08 at 15:03 +0000, Antti Kantee wrote:
>
>> For unikernels, the rump kernel project provides Rumprun, which can
>> provide you with a near-full POSIX'y interface.
>
> I'm not 100% clear: Does rumprun _build_ or _run_ the application? It sound
> s like it builds but the name suggests otherwise.

For all practical purposes, Rumprun is an OS, except that you always 
cross-compile for it.  So, I'd say "yes", but it depends on how you want 
to interpret the situation.  We could spend days writing emails back and 
forth, but there's really no substitute for an hour of hands-on 
experimentation.

(nb. the launch tool for launching Rumprun instances is currently called 
rumprun.  It's on my todo list to propose changing the name of the tool 
to e.g. rumprunner or runrump or something which is distinct from the OS 
name, since similarity causes some confusion)

> Do these wrappers make a rump kernel build target look just like any other
> ross build target? (I've just got to the end and found my answer, which was
> yes. I've left this next section in since I think it's a nice summary of
> why it matters that the answer is yes)
>
> e.g. I have aarch64-linux-gnu-{gcc,as,ld,ar,etc} which I can use to build
> aarch64 binaries on my x86_64 host, including picking up aarch64 libraries
> and headers from the correct arch specific patch.
>
> Do these rumprun-provided wrappers provide x86_64-rumpkernel
> -{gcc,as,ld,ar,etc} ?

No, like I said and which you discovered later, 
x86_64-rumprun-netbsd-{gcc,as,ld,ar,etc}.  aarch64 would be 
aarch64-rumprun-netbsd-{...}.

> Appearing as a regular cross-compilation target is, I think, going to be
> important to being able to create rumpkernel based versions of distro
> packages.
>
> I think that package maintainers ideally won't want to have to include a
> bunch of rumpkernel specific code in their package, they just want to
> leverage the existing cross-compilability of their package.

Yes, that is critical.  We bled to achieve that goal.  It looks obvious 
now, but I can assure you it wasn't obvious a year ago.

> $ ldd /usr/bin/qemu-system-x86_64  | wc -l
> 87
> $

Heh, that's quite a lot.

>> If the above didn't explain the grand scheme of things clearly, have a
>> look at http://wiki.rumpkernel.org/Repo and especially the picture.  If
>> things are still not clear after that, please point out matters of
>> confusion and I will try to improve the explanations.
>
> I think that wiki page is clear, but I think it's orthogonal to the issue
> with distro packaging of rump kernels.

Sure, but I wanted to get the concepts right.  And they're still not 
right.  We're talking about packaging for *Rumprun*, not rump kernels in 
general.

>>    However, since a) nobody (else) ships applications as relocatable
>> static objects b) Rumprun does not support shared libraries, I don't
>> know how helpful the fact of ABI compatibility is.  IMO, adding shared
>> library support would be a backwards way to go: increasing runtime
>> processing and memory requirements to solve a build problem sounds plain
>> weird.  So, I don't think you can leverage anything existing.
>
> This is an interesting point, since not building a shared library is
> already therefore requiring packaging changes which are going to be at
> least a little bit rumpkernel specific.
>
> Is it at all possible (even theoretically) to take a shared library (which
> is relocatable as required) and to do a compile time static linking pass on
> it? i.e. use libfoo.so but still do static linking?

But shared libraries aren't "relocatable", that's the whole point of 
shared libraries! ;) ;)

I guess you could theoretically link shared libs with a different ld, 
and I don't think it would be very different from prelinking shared 
libs, but as Samuel demonstrated, it won't work at least with an 
out-of-the-box ld.

I think it's easier to blame Solaris for the world going bonkers with 
shared libs, bite the bullet, and start adding static linking back where 
it's been ripped out from.  Shared libs make zero sense for unikernels 
since you don't have anyone to share them with, so you're just paying 
extra for PIC for absolutely no return.  (dynamically loadable code is a 
separate issue, if you even want to go there ... I wouldn't)

>> I don't really have good solutions for the packaging problem.  Building
>> a "full distro" around rump kernels certainly sounds interesting,
>
> FWIW I don't think we need a full distro, just sufficient build
> dependencies for the actual useful things (maybe that converges on to a
> full distro though).

By "full distro" I meant "enough to get a majority of the useful 
services going".  Seems like once qemu works, we're 99% there ;)

> Debian are (most likely) not going to accept a second copy of the QEMU
> source in the archive and likewise they wouldn't want a big source package
> which was "qemu + all its build dependencies" or anything like that,
> especially when "all its build dependencies" is duplicating the source of
> dozens of libraries already in Debian.

Why do you need a second copy of the sources?  Or are sources always 
strictly associated with one package without any chances of pulling from 
a master package?  You are going to need two copies of the binaries 
anyway, so it doesn't seem like a particularly big deal to me, not that 
I'm questioning your statement.

>> If I were you folks, I'd start by getting qemu out of the door
>
> That's certainly the highest priority, at the moment I don't think we
> actually have a QEMU Xen dm based on rumpkernels which anyone could package
> anyway, irrespective of how hard that might be.
>
>> , and
>> worry about generalized solutions when someone wants to ship the second
>> unikernel (or third or any small N>1).
>
> Unfortunately I think the N==1 case is tricky already from a distro
> acceptance PoV. (At least for Binary distros, it's probably trivial in a
> Source based distro like Gentoo)

Ok.  I'll help where I can, but I don't think I can be the primus motor 
for solving the distro acceptance problem for Xen stubdomains.

If you can say to the packaging system "build with this cross toolchain 
but disable shared" you're already quite far along, and it seems like 
something that shouldn't be too difficult to get reasonable packaging 
systems to support.  But, details, details.  One major detail is that 
your target is quite wide, and not everyone along that target can be 
assumed to be reasonable :/