[Soc-coordination] Eucalyptus/vmbuilder integration project update week 12 and milestone/deliverable list overview

[David Wendt JR.]

Well, for the past week I've been sick, but I don't let a little fever of 103F stop me from working on the project!

In this case, I've been working on code to auto-bundle kernels and ramdisks for Eucalyptus users. You see, while EC2 comes with it's own kernels and ramdisks which are the only ones you can use, Eucalyptus will allow you to bundle and upload your own kernels and ramdisks provided you have the proper permissions on the server. Which is quite useful considering that Eucalyptus comes without any kernels or ramdisks.

So I coded some extra functionality to bundle kernels and ramdisks into VMbuilder's ec2 plugin, with appropriate command line options: "--ec2-bundle-kernel", "--ec2-upload-kernel", and "--ec2-register-kernel". And then I discovered that the Debian plugin doesn't install a kernel at all if you select a hypervisor environment that doesn't require a bootloader. That doesn't stop the Xen plugin, which queries the distro plugin for the location of the kernel using Distro.xen_kernel_path which actually just returns where the kernel -should be-. Then the Xen plugin blindly writes it to the image's xen.conf.

So, I wrote up a separate codepath for installing a kernel without Grub, fixed xen_kernel_path to actually look at the installed kernels for the proper Xen one, and wrote a regular kernel_path which works irregardless of if there's a Xen or regular kernel installed. It -theoretically- works right now (as in the state the git repo is in) but I think there's still some bugs involved in registering. It doesn't help that I have to spend 45 minutes debootstrapping in order to do a single test run.

Now, as for the milestones... I reposted my original project proposal here: http://wiki.debian.org/DavidWendt/GSOCProposal a while back.
In retrospect, my old milestones are a little vague and 'first drafty'. Admittedly I intended to revise or fix these with the help of one of the mentors during the submissions period, but I didn't get a chance to, so I must apologise if these feel inaccurate or badly written.

"Design a configuration file format, or some other standard method of
controlling the vmbuilder. I imagine that I would need to converse with
the developers responsible for building Debian CD images to know what
extra configuration options we need."
This milestone is kinda badly worded, since VMbuilder itself already has reasonable defaults and can be controlled via normal command line options. So I guess this one was already done by the time I started.

"Extend vmbuilder to handle last milestone's format and configuration changes."
VMbuilder already has the ability to be configured via normal command-line options, so this milestone seems superflouous.

"Create a command line script to automate the process of getting an
official debian package of configuration files and generating a VM
image."
Well, VMBuilder already has reasonable defaults. But I did have to take an existing Debian plugin and do a ton of modifications to it in order to get it to work for building for a Eucalyptus/EC2 environment.

"Ensure vmbuilder can handle all the various forms of VM images.
Currently, VMbuilder can already handle images for KVM, vmw6, vmserver,
vbox, and qemu. We may want to support some grid computing system that
uses a different image format"
VMBuilder is plugin based, so most of the work in handling multiple image formats was already cut out for me. Nevertheless building on virtualbox, xen, and a few other environments was broken. Not to mention it didn't support building Debian on ec2. My mentor (Stephen Moeller) helped out and contributed a few patches of his own for building Debian lenny on virtualbox. I fixed up Xen support (mostly, Cedric Jeanneret helped fix a race in umounting devices on Debian lenny Xen), and EC2 is mostly working. (the auto bundling/uploading support is kinda wonky, should be fixed in a few days. There's also no default EC2 kernel IDs yet)

"Make a nice graphical front-end for vmbuilder, to make it easier for others to build images and install them on their cluster."
This milestone was added as a sort of 'optional last minute thing' in case we finished the other serious stuff early. That being said I'm currently mired in more important stuff (fixing kernel/image registration wonkiness) and I don't think this will be finished right away. Maybe in the future.

As for the deliverables...

"A process for generating trusted grid computing images that can be distributed like the CD images"
It's not hard for the debian team to generate their own AMIs, they just need to decide on a few standard options. Making a 'trusted' image available on EC2 would require Debian to register an EC2 cert, bundle an image, and host/register it on S3, as far as I am aware. I don't know if the Debian foundation or team would want to go in that direction, it is quite possible.

"An easy way for others to make their own images, or modify existing trusted ones"
Again, making images with VMBuilder is pretty easy for laypersons, for example to make a debian Xen image, one types "vmbuilder xen debian --suite=lenny". The only caveat with VMBuilder is that it must be run as root, although the program checks for this and warns you. But it's much easier than the manual option.