[Soc-coordination] Report 3 - Pluggable Acquire System for APT

[Bogdan Purcareata]

Codename: apt-fetcher
Mentors: Michael Vogt, David Kalnischkies
Project proposal page: [0]
Project design page: [1]

Here is a summary of weeks 5 and 6 of the coding period:

What I've done:
- finished implementation of the framework
- written unit tests for it
- written plugin for Contents
What problems I've run into:
- pkgAcquire doesn't fetch metadata just by creating pkgAcqIndex objects
- busy with bachelor project thesis
What I plan to do further:
- integrate with the acquire module and finish apt-get update default
functionality

Two weeks ago, at the last report's milestone, the design of the
pluggable acquire framework was complete and its implementation was
partial. As I've seen after research, the pkgAcquire object - the
object responsible for fetching the index files - makes use of
metaIndex objects, which fully describe a Debian Archive distribution
location and the metadata to download from it. On top of these
metaIndex objects, some metaIndexPlugins were built, which are
supposed to provide methods for handling a specific type of metaIndex
- since currently only Debian Archive locations are supported, a
debMetaIndexPlugin and its according debMetaIndex class were defined.
The metadata that a metaIndex object should download is handled by
acquireIndexPlugins - one acquireIndexPlugins per type of metadata
e.g. deb-packages, deb-sources, deb-translations, deb-contents,
deb-tags, etc. These acquireIndexPlugins are responsible for building
acquireIndex objects, which are further used to build specific
IndexTarget objects for the main fetcher class - the pkgAcquire. All
the acquireIndexPlugins and metaIndexPlugins, along with their
corresponding acquireIndex and metaIndex classes are handled by the
pkgAcquireFramework class.

As I've seen in the code, the APT's acquire module is generic enough
to support index file download, regardless of the contents. Moreover,
fetch techniques, such as compressed files or diff files, are already
handled. This way, the sole purpose of the pluggable acquire framework
becomes only to provide a set of IndexTarget objects a.k.a. specific
locations in the Debian Archive from which the index files should be
fetched. This is the pluggable component that will handle the parser's
sources and build fully specified metaIndex objects, which will then
be used to build acquireIndex objects for the corresponding remote
index files.

In the framework's header file [6] I've declared the methods for:
- the acquire framework
- the generic metaIndexPlugin
- the generic acquireIndexPlugin
- the generic acquireIndex

Furthermore, I've subclassed these into the metaIndexPlugin and
metaIndex classes for Debian - currently implemented in the
apt-pkg/deb/debmetaindex.cc - and implemeted acquireIndexPlugins for
SOURCES, PACKAGES, TRANSLATIONS and CONTENTS. Complete implemetation
is available in the framework implementation file [7], and some tests
are available in the framework tester file [8]. These tests parse the
contents of the sources.list file, with the public enhanced parser,
register plugins to the framework and then use it to build
metaIndexes. These metaIndexes are then used to generate IndexTarget
object, for specific locations in the Debian Archive. Currently, the
tester provides validation for these locations.

The next step is to actually fetch the index files. My initial thought
was that once pkgAcqIndex objects were registered in the pkgAcquire
fetcher, the process of downloading is taken care of. Once I got to
the point of actually downloading files, I got stuck. I imagine the
next step in development is to check out how this pkgAcquire object is
used but the apt-get command line tool. Once this is done, I will
implement the similar process in the tester, and check out how the
framework integrates with it. So far, I don't think further
functionality should be added to the framework, only refactoring.

For one of the two weeks since the last report, I've been busy with my
bachelor project thesis. The project itself was done by May 30th, but
I forgot about the additional paper. During this week, I wasn't able
to bring much contribution to apt-fetcher. For the next two weeks,
until the mid-term deadline, I will focus on intergrating the
pluggable framework with apt-get, and provide a complete metadata
download flow. I haven't discussed much with the mentors since the
last report, as I pretty much had in mind what I wanted to implement
and how I was going to do it. I'm expecting some discussions with them
in the near future, regarding the results so far, what can be improved
and what to be further done. Overall, I think I've covered designing
and implementing both the parser, and the pluggable acquire framework,
defining the generic models for the plugins and the format of the
sources.list. What's left is integration with apt-get, improvements to
the acquire logic and to the parser, exporting the functionality in
libapt. The most important step is to integrate the framework with
apt-get update.

My contributions to the APT package can be found in the repo [2]. The
implemented files are listed from [3] to [8].

Bogdan Purcareata

[0] http://wiki.debian.org/SummerOfCode2012/Projects#Pluggable_acquire-system_for_APT
[1] http://wiki.debian.org/BogdanPurcareata/PluggableAptBackend
[2] https://launchpad.net/apt-fetcher
[3] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.h
[4] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.cc
[5] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/test/libapt/parser_tester.cc
[6] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/framework.h
[7] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/framework.cc
[8] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/test/libapt/framework_tester.cc