[Soc-coordination] Eucalyptus/vmbuilder integration project update week 12 and milestone/deliverable list overview

[Steffen Moeller]

Hi David,

many thanks for your extensive summary.

David Wendt JR. wrote:

> In this case, I've been working on code to auto-bundle kernels and ramdisks for Eucalyptus users.
> You see, while EC2 comes with it's own kernels and ramdisks which are the only ones you can use,
> Eucalyptus will allow you to bundle and upload your own kernels and ramdisks provided you have
> the proper permissions on the server. Which is quite useful considering that Eucalyptus comes
> without any kernels or ramdisks.

this neat little difference has indeed escaped me. With EC2 you indeed take existing
images and bundle around them. I had thought of it as a commodity workflow, though. There
is no checking on the sides of Amazon about what kernel you are actually using, is there?

> So I coded some extra functionality to bundle kernels and ramdisks into VMbuilder's
> ec2 plugin, with appropriate command line options: "--ec2-bundle-kernel",
> "--ec2-upload-kernel", and "--ec2-register-kernel".

Nice.

> And then I discovered that the Debian plugin doesn't install a kernel at all if you select a
> hypervisor environment that doesn't require a bootloader. That doesn't stop the Xen plugin,
> which queries the distro plugin for the location of the kernel using Distro.xen_kernel_path
> which actually just returns where the kernel -should be-. Then the Xen plugin blindly writes
> it to the image's xen.conf.

Uh.

> So, I wrote up a separate codepath for installing a kernel without Grub, fixed xen_kernel_path
> to actually look at the installed kernels for the proper Xen one, and wrote a regular
> kernel_path which works irregardless of if there's a Xen or regular kernel installed.
> It -theoretically- works right now (as in the state the git repo is in) but I think
> there's still some bugs involved in registering. It doesn't help that I have to
> spend 45 minutes debootstrapping in order to do a single test run.

Congratulations, this sounds all good. 45min is too long, indeed. This is some older
hardware, I presume...at pace with my OpenMoko FreeRunner.

> Now, as for the milestones... I reposted my original project proposal here: http://wiki.debian.org/DavidWendt/GSOCProposal a while back.
> In retrospect, my old milestones are a little vague and 'first drafty'. Admittedly I intended to revise or fix these with the help of one of the mentors during the submissions period, but I didn't get a chance to, so I must apologise if these feel inaccurate or badly written.

You should not care too much. Work on a description of your work, especially on the Wiki
pages so others can enjoy your advances - don't work on the milestones. Nobody really
cares, except for Arthur who needs to sell this back to Google, so he can also sell nice
Wiki pages. We are all on your (technical non-bureaucratic) side.

> "Design a configuration file format, or some other standard method of
> controlling the vmbuilder. I imagine that I would need to converse with
> the developers responsible for building Debian CD images to know what
> extra configuration options we need."
> This milestone is kinda badly worded, since VMbuilder itself already has reasonable defaults and can be controlled via normal command line options. So I guess this one was already done by the time I started.

Great!

> "Extend vmbuilder to handle last milestone's format and configuration changes."
> VMbuilder already has the ability to be configured via normal command-line options, so this milestone seems superflouous.

Yippee!

> "Create a command line script to automate the process of getting an
> official debian package of configuration files and generating a VM
> image."
> Well, VMBuilder already has reasonable defaults. But I did have to take an existing
> Debian plugin and do a ton of modifications to it in order to get it to work for
> building for a Eucalyptus/EC2 environment.

So this was some good Google-sponsored project-work, then. Great! Take the first two
milestones as the opportunity to educate yourself towards it.

> "Ensure vmbuilder can handle all the various forms of VM images.
> Currently, VMbuilder can already handle images for KVM, vmw6, vmserver,
> vbox, and qemu. We may want to support some grid computing system that
> uses a different image format"
> VMBuilder is plugin based, so most of the work in handling multiple image formats
> was already cut out for me. Nevertheless building on virtualbox, xen, and a few
> other environments was broken. Not to mention it didn't support building Debian
> on ec2. My mentor (Stephen Moeller) helped out and contributed a few patches of
> his own for building Debian lenny on virtualbox. I fixed up Xen support (mostly,
> Cedric Jeanneret helped fix a race in umounting devices on Debian lenny Xen),
> and EC2 is mostly working. (the auto bundling/uploading support is kinda wonky,
> should be fixed in a few days. There's also no default EC2 kernel IDs yet)

In very deep principle, Eucalyptus is completely agnostic to the means of virtualisation
done. Xen in my mind is one of the more difficult installations for Debian. I personally
found KVM or Virtualbox far easier to work with. In my communication with Eucalyptus
upstream about it, they will have a closer eye on alternatives to Xen for their upcoming
1.6 release. And because of other issues, I also think that we should really wait for that
new release (due end of August) before discussiong the choice of the virtualisation method
too deeply.

> "Make a nice graphical front-end for vmbuilder, to make it easier for others to
> build images and install them on their cluster." This milestone was added as a sort of
> 'optional last minute thing' in case we finished the other serious stuff early.
> That being said I'm currently mired in more important stuff (fixing kernel/image
> registration wonkiness) and I don't think this will be finished right away. Maybe in the future.

I agree. The workflow we sell is

 1. look up a wiki.debian.org pages and learn about VMbuilder and Eucalyptus
 2. copy a few lines from those pages to your local console
 3. run the image on the cloud

and if it is graphical or not, well, I don't see anybody using the graphics API, really.
And if so, then this would be via the Web, not on a local machine. And for some services
to be prepared for remote end users, we need to do some more work on many other fronts, still.

There is

 4. prepare the Eucalyptus cloud infrastructure locally.

Though this should truly wait for 1.6 in my mind. David may have another stance here.

> As for the deliverables...
> 
> "A process for generating trusted grid computing images that can be distributed like the CD images"
> It's not hard for the debian team to generate their own AMIs, they just need to decide on a
> few standard options. Making a 'trusted' image available on EC2 would require Debian to
> register an EC2 cert, bundle an image, and host/register it on S3, as far as I am aware.
> I don't know if the Debian foundation or team would want to go in that direction, it is quite possible.

I once manually did a Debian Med AMI (DMAMI). The challenge is to routinely update it, I
think. This is almost free of cost to prepare and upload. I would fund that from my
personal poor academic's pocket until we had time to evaluate the users' response to it.

> "An easy way for others to make their own images, or modify existing trusted ones"
> Again, making images with VMBuilder is pretty easy for laypersons, for example to make
> a debian Xen image, one types "vmbuilder xen debian --suite=lenny". The only caveat
> with VMBuilder is that it must be run as root, although the program checks for this
> and warns you. But it's much easier than the manual option.

What should be mentioned is that over David's work, communication with Eucalyptus upstream
has improved. We have submitted the euca2ools of theirs too the NEW QUEUE, as a free
replacement to the ec2-ami-tools of Amazon. Also the communication with the Ubuntu folks
working on Eucalyptus and associated tools has been very friendly and constructive and ...
just nice.

So, it is on my to thank David for his work, for him being self-paced and self-motivated
and self-thinking, so there is not much for me to mentor about. Use the opportunity to
communicate with the Eucalyptus folks more, but maybe this is just me overrating that.

Best,

Steffen