[Soc-coordination] EC2/eucalyptus project update (week2)

[Eric Hammond]

Hi David,

I've been a bit out of the loop since this project was approved, but
seeing you're forging forward I thought I'd toss in some feedback in
case it helps.  I'm probably not going to be able to make the upcoming
status meeting today as it's in the middle of my normal sleep hours.

- You said you are in contact with an Ubuntu developer.  Can you share
which one?

- The Canonical folks who are primarily involved with vmbuilder and the
EC2 plugin are Soren Hansen and Chuck Short, and that group is managed
by Rick Clark.  You can often reach them on the #ubuntu-ec2 channel on
irc.freenode.net or I can introduce you if you'd like.

- I believe "ubuntu-vm-builder" is an old name.  As far as I know, it is
now simply called "vmbuilder".

- The vmbuilder project in Ubuntu is currently in a state of flux with
  bin/switch-branch trunrelation to building EC2 images.  The images
which are released by Canonical are being built using a developer branch
(not the trunk) along with some code that may be private.  There is an
outstanding EC2 patch merge request which is being reviewed for
inclusion into the trunk branch.  You may want to wait a bit until
things have settled with vmbuilder or work more closely with the
developers if they are able to help.

- Here's a somewhat old description of how images were built for EC2 by
Canonical.  It may be out of date.
  https://wiki.ubuntu.com/EC2Vmbuilder

- There was recent work done by Bernd Zeimetz on a development branch
for building Debian images with vmbuilder and a patch merge request was
submitted.  This wasn't specifically for EC2, but you'll want to be
aware of this in case it gets accepted into the vmbuilder trunk:
  https://code.launchpad.net/~bzed/vmbuilder/debian
No need to write competing code if somebody has already done this piece
of the work.

- There are two sets of EC2 tools: API and AMI, plus the API tools are
bundled by Amazon in several distinct sets including the base API tools,
CloudWatch, Elastic Load Balancer, and Auto Scaling.

- The AMI and API tools have been put under a license which allows them
to be distributed for use with Amazon EC2.  Ubuntu is packaging them for
this purpose.

- There has been a recent Twitter rumor that Amazon may relax the
license on the AMI and API tools, but it is such an unsubstantiated
rumor that I feel bad even spreading it here.  However, it might be good
to investigate this a bit before rewriting them as this would be
significant initial and ongoing effort.

- An important part of building Debian images for EC2 is going to be
picking which kernel should be used by default.  I have been building
Debian images using Amazon's 2.6.21fc8 kernel, but now that Canonical
has published a few Ubuntu kernels it might make more sense to use those
for some of the Debian releases.

- There's a Google Group for folks using Debian on EC2.  You may wish to
join and perhaps seek help from the friendly folks in the group.
  http://groups.google.com/group/ec2debian


Background: My expertise is in building EC2 images for Debian (and
Ubuntu) as published since 2007 on:
  http://alestic.com
The Debian images there are built using this software:
  http://ec2ubuntu-build-ami.notlong.com
Many of the concepts in that software were copied when building the EC2
plugin for vmbuilder.  You may want to look at what is done differently
for Debian in that code (very little).

I have also been working with Canonical's server team as they create the
EC2 plugin for vmbuilder and as they build new EC2 images.

Feel free to contact me at any time with questions or to bounce ideas
around.

Please do keep me updated with your progress.  I will be giving a
presentation about building images for EC2 at OSCON in July and would
love to include a mention of your work if it's getting to the point
where it's usable for building Debian AMIs.

--
Eric Hammond
ehammond at thinksome.com



David Wendt JR. wrote:
> Hello. David Wendt here, with a status update.
> 
> During "Community Bonding" aside from freaking out about finals (which I ended up failing) I had a chance to read up on the licensing for the Amazon-provided 'ec2-tools', which has an onerous usage restriction preventing us from packaging them. Work to write a replacement script is ongoing.
> 
> Eucalyptus has a bunch of Java dependancies that Debian/Ubuntu doesn't package yet; so in the interest of shipping on time Eucalyptus packed up a bunch of their Java dependancies into a giant eucalyptus-javadeps package. It's monolithic, and I have it on good authority that it can't build from source. So there's an ongoing effort to refactor the package into individual packages by the Ubuntu people as described here:
> 
> https://wiki.ubuntu.com/RefactorEucalyptusJavadepsSpec
> https://wiki.ubuntu.com/EucalyptusInMainSpec
> 
> I've gotten in contact with one of the Ubuntu developers and we are coordinating our work so that both Debian and Ubuntu get these dependancies sorted out properly.
> 
> Another piece of the project is the ubuntu-vm-builder; I've grabbed a copy from the upstream (which afaik is just launchpad: https://launchpad.net/vmbuilder ) and started a git repo at git://git.debian.org/git/pkg-escience/vmbuilder.git for all to peruse. There's not much to see right now, I'll strive for daily commits starting Monday.
> 
> The developers themselves seem to have planned to support multiple distros, there's a generic VMBuilder.distro class for representing the behaviors unique to individual Debianesques and versions. All you have to do to add support for a new distribution is to create a new plugin folder, and populate it with the prerequisite classes and folders. You can also make a new frontend ('debian-vm-builder', anyone?) if you like.
> 
> So I'll be spending the next week or two writing up VMBuilder classes for Debian releases, so vmbuilder can make Debian images in all supported formats. Such patches should be easily sent back to upstream for inclusion in Ubuntu as well. From there we can produce Debian AMIs and stuff.
> 
> I don't have any shiny .debs to hand out, sadly. You can follow my progress via the git archive as I write the Debian integration classes, as a consolation prize.