[Soc-coordination] Update: DebTorrent Release 0.1.4 (and apt-transport-debtorrent 0.1.0)

[Cameron Dale]

Hi,

Here is my final update for the Google Summer of Code (mostly taken
from my recent blog post).

The next release of DebTorrent is now available. This release includes
new functionality for communicating with APT using a new transport
method specifically designed for debtorrent, and many bug fixes.

The major changes in this release are all in the communication between
APT and DebTorrent. The HTTP server in DebTorrent that listens for APT
requests has been upgraded to support HTTP/1.1 persistent connections
and pipelining, which allows APT to have multiple outstanding requests
for files. This is useful as DebTorrent suffers from the typical
bittorrent slow start, so requesting multiple files at a time helps to
speed up the download considerably.

Though better, HTTP/1.1 is not ideal for DebTorrent however, as a
maximum of 10 outstanding requests is maintained by APT's http method,
and files must still be returned in the order they were requested
(which is not ideal for bittorrent-type downloading since downloads
occur randomly).

To further improve the APT communication I have modified APT's http
method to create a debtorrent method. This new debtorrent transport
for APT is packaged separately as apt-transport-debtorrent, and once
installed APT can be told to use it by replacing "http://" with
"debtorrent://" in your sources.list file. This method sends all
requests it receives immediately to DebTorrent, and will receive
responses from DebTorrent in any order. You can find this new method
on the Alioth project[1], or in my personal repository[2] (only amd64
and i386 versions are currently available).

[1]: https://alioth.debian.org/frs/?group_id=31109
[2]: http://debian.camrdale.org/

Unfortunately, the story doesn't end here. The APT code responsible
for sending requests to the method also limits the maximum number of
outstanding requests that it will send to the method to 10, which is
not really necessary since all existing methods limit the requests
they send out themselves. I have therefore patched the current APT
code to increase this limit to 1000 (a one line change), and released
this patched version as 0.7.6-0.1. You can find this patched version
in my personal repository[2] (again, only for i386 and amd64). I have
tested it with the other methods available and it causes no problems,
and I hope to get the change included in the regular APT code soon.

[2]: http://debian.camrdale.org/

To sum up:

 * new DebTorrent over HTTP = fast
 * new DebTorrent with new apt-transport-debtorrent = faster
 * new DebTorrent with new apt-transport-debtorrent and a patched APT = fastest

The last DebTorrent version (0.1.3.1) is currently in the NEW queue,
and judging by the length of it, will be there for about another week.
After DebTorrent is added to the archive, I will be upgrading it to
this new version. I also hope to get the new apt-transport-debtorrent
package into the NEW queue soon.

This brings to an end the Google Summer of Code, but development of
DebTorrent will of course continue (though probably a little slower ;)
). The next major change will be the addition of unique piece
numbers[3], which is almost complete but needs to be extensively
tested. I'd like to thank Anthony Towns, Steve McIntyre, and Michael
Vogt for their help over the last 4 months, and also the many others
who sent me encouraging emails or engaged in interesting discussions
about this project. It's the people who make a project like this a fun
and memorable thing to do.

[3]: http://wiki.debian.org/DebTorrent/UniquePieces

Here's the changelog for the new DebTorrent release:

 * APT communication supports HTTP/1.1 connections, including
   persistent connections and pipelining
 * Add support for the new debtorrent APT transport method
   (see the new apt-transport-debtorrent package)
 * Make the Packages decompression and torrent creation threaded
 * Improve the startup initialization of files
 * Add init and configuration files for the tracker
 * bug fixes:
   - restarts would fail when downloaded files have been modified
   - deleting old cached data would fail
   - small tracker bug causing exceptions
   - prevent enabling files before the initialization is complete
   - only connect to unique peers from the tracker that are not
     already connected
   - tracker would return all torrents' peers for every request

Thanks for everything,
Cameron