[Soc-coordination] GSoC - APT Ordering - Report 2

Christopher Baines cbaines8 at gmail.com
Fri Jun 17 18:34:33 UTC 2011


Hello, 

Since my last report [1], not a lot has changed with respect to my
project, this is because all last week I have had exams and have not
been able to work. However due to the tendency of the mind to wonder
when attempting to revise colour changes in ligand substitution
reactions and the equations for the scalar product of vectors, I have
been thinking about my project and the apt code in general.

One thing that I have done since my last report is some further
investigation in to the ordering failure Michael showed me. I have
included some of the output from my testing (… are lines removed for
clarity). 

I will apologise to anyone in advance if the following attempt at code
annotations goes wrong. 

SmartConfigure openoffice.org-common ← SmartConfigure is a method
                                       used in APT to perform
                                       configuration on a package and
                                       all of its dependants. 
  DepAdd: openoffice.org-common ← DepAdd recursively (it calls itself)
                                  adds all the dependants of the package
                                  to the configuration list. You can see
                                  the iteration as it quite nicely
                                  indents the output. 
    DepAdd: openoffice.org-style-oxygen 
...
        DepAdd: libcurl3-gnutls ← Here is one of the packages I am
                                  interested in, its been added to the
                                  configure list, the DepAdd routine is
                                  now looking at its dependencies. 
          DepAdd: libgcrypt11 
...
          DepAdd: libldap-2.4-2 ← Its found the other package of
                                  interest, as you can see
                                  libcurl3-gnutls depends on
                                  libldap-2.4-2, this means that
                                  libldap-2.4-2 must be configured 
                                  before libcurl3-gnutls.
            DepAdd: libsasl2-2 
        DepAdd FAILS on: openoffice.org-core ← Something has gone wrong 
      DepAdd FAILS on: openoffice.org-style-oxygen 
    DepAdd: openoffice.org-style-human 
...
Conf libsysfs2 ... ← But even thought there has been a error, the
                     configuration of the list begins
...
Conf libcurl3-gnutls ... ← Now here is the problem, not only is
                           libcurl3-gnutls being configured before
                           libldap-2.4-2, libldap-2.4-2 is not even
                           configured with this group of packages!
Conf libxml2 (2.7.6.dfsg-1ubuntu1.1 Ubuntu:10.04/lucid-updates [amd64])
[] 
…

Something must have happened, or not happened in the Dep add routine to
cause the routine to ignore libldap-2.4-2. At the moment I have not had
the opportunity to discuss this  with my Mentor, this is something I
will hopefully learn more about and do further investigation in to the
code this coming week.

Currently I have written a small piece of code to intercept configure
calls from the package manager in APT and check for any unconfigured
dependencies (This is currently not in any repository). This currently
correctly detects the error above, but as the following example shows it
does not cope with dependency loops.

Here is the relevant output from APT:
Conf firefox (3.6.17+build3+nobinonly-0ubuntu0.10.04.1
Ubuntu:10.04/lucid-updates [amd64]) 
Conf firefox-branding (3.6.17+build3+nobinonly-0ubuntu0.10.04.1
Ubuntu:10.04/lucid-updates [amd64]) 
…
E: Package firefox should not be configured because package
firefox-branding is not configured

Now firefox depends on firefox-branding (seems sensible enough), but
firefox-branding also depends on firefox (this does not seem as
sensible). This results in a dependency loop. To configure these
packages, the loop needs to be broken, the Debian policy manual covers
the exact mechanics of how to do this [2], but as far as apt is
concerned, dpkg will handle this. 

The way this works is the Conf statements that appear in the output
above are grouped together by APT, and inserted in to one call to dpkg.
So the dpkg invocation will look like:  
dpkg --configure A B C
dpkg --unpack D
dpkg --configure D
Not, 
dpkg --configure A
dpkg --configure B
dpkg --configure C
dpkg --unpack D
dpkg --configure D

Thanks go to David Kalnischkies for the above example.

So the initial outcome of this is that my code is detecting these
breakages as errors where they are actually none. So I could add to my
check to see if there is a dependency loop between the two packages
concerned, but while this is simple enough for the example above there
could be any number of packages involved in this loop, so I am not sure
if this is the best approach.

There is also another possible problem though, as I said above APT
passes packages to dpkg in batches, now if there is a large run of
packages being configured (which does happen) this call will get split
in to many (because of limitations outside APT's control). 

Now imagine what would happen if for the above example the large list of
packages was split between firefox and firefox-branding, dpkg would not
be able configure the packages properly. Now this might or might not
impact firefox (depending on the way the package works), but for some
packages this could be problematic.

One solution I can think of for this problem, is before ordering
packages, split the list of packages in to batches (groups of packages).
Now this has two advantages:
 - By limiting the size of batches, you can prevent any splitting of
dpkg calls (without doing any complex analysis of dependencies for a
second time).
 - Batches could be processed in parallel, allowing packages to be
installed and fetched at the same time. 

There are some practical considerations about the process needed for
separating   packages in to batches e.g. criteria for separating
packages and how do deal with dependency loops and large batches, but
from thinking about the problems at hand, this seems to me to be a good
and flexible approach.

I am now just days from my final exam (on Tuesday), and from then I will
be free to work on APT full time (which I am very much looking forward
to). 

Also if anyone has any advice, comments, questions or corrections, feel
free to voice them. 

Chris

1.
http://lists.alioth.debian.org/pipermail/soc-coordination/2011-June/001005.html | http://lists.debian.org/deity/2011/06/msg00005.html
2.
http://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps
3. http://lists.debian.org/deity/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/soc-coordination/attachments/20110617/7c664a45/attachment.pgp>


More information about the Soc-coordination mailing list