[Soc-coordination] Updates! (was Re: BoF at Debconf)

[Cameron Dale]

On 7/4/07, Steve McIntyre <steve at einval.com> wrote:
> Also - things have gone worryingly quiet of late. We're due to start
> on the mid-term reports soon, so we really need to see some activity
> reports folks!

My most recent status report was in the format of a blog post
available here (don't mind the date, apparently I'm living in the
past):

http://www.cs.sfu.ca/~camerond/personal/blog/posts/Jun-27-2006.html

I include it here in text form as well for your convenience:

Making DebTorrent Work With APT

It's been a while since I've given a status update for my Google
Summer of Code project to create a BitTorrent proxy for downloading
packages using APT, so here it is.

I've been working hard on integrating support for APT into the
DebTorrent program[1]. I've almost got it working perfectly, now
it's just a matter of testing to make sure all is well. The
functionality works like this:

 * DebTorrent listens on a port for HTTP requests
 * an http://localhost:port/mirror_name/debian/... entry is added to
   APT's sources.list file (similar to how apt-cacher works for
   proxying)
 * an apt-get update will then send HTTP requests to the DebTorrent
   program
   * DebTorrent proxies these requests, downloading files it doesn't
     have and saving them in a cache before passing them on to APT
   * DebTorrent recognizes requests for Packages files, and also
     uses them to start the torrents for those files
 * an apt-get install will send an HTTP request to DebTorrent for a
   package (.deb) file
   * rather than getting package files from the mirror, DebTorrent
     finds the file in one of its running torrents and enables it
     for download
   * the package file is downloaded (either from other peers, or
     using the backup HTTP downloader that gets it from the mirror
     if no peers can be found)
   * once the download is complete, DebTorrent passes the package
     file to APT

There are two things I really like about this. One of the best is
the backup HTTP downloader. It insures that if you're an early
adopter and there are no peers, or if the package you're requesting
is rare and can't be found in any connected peers, the download will
still occur in a reasonable amount of time (taking no more, or less,
mirror bandwidth than if you had just been using APT directly). The
other thing I like is that you get the BitTorrent-style peer-to-peer
downloading, with simple HTTP proxying thrown in for free. You can
run DebTorrent on a single computer on your network, and have the
others connect to it to initiate downloads and request packages.

I haven't done any serious time testing, but I estimate it currently
*only* takes twice as long to use as a regular APT update and
download from a mirror. Most of that slowdown is because it
currently only processes a single request at a time from APT, which
is not very efficient for BitTorrent systems where downloading from
multiple peers is how the highest download speeds are achieved. I
have been talking to the APT maintainer, Michael Vogt, about a
better way to do this[2], probably by adding a new APT transport
method for DebTorrent (i.e. debtorrent:// instead of http://). This
will not only speed up the downloads, but also hopefully provide
better feedback to the APT user, as currently it will seem nothing
is happening until the download comes in all at once at the end.

My work has unfortunately been slowed by other commitments, and
bugs. I have two papers to submit to a conference by Monday, but
after that I should be back full time on DebTorrent. I also spent a
long time tracking down and fixing a bug in the underlying
BitTornado[3] code (which lead to much rejoicing at 3am), only to
find it was fixed in upstream's CVS. (Doh!)

[1]: http://debtorrent.alioth.debian.org/
[2]: http://wiki.debian.org/DebTorrent/AptInteraction
[3]: http://www.bittornado.com