[Soc-coordination] port bootstrap build-ordering tool report 5

[Johannes Schauer]

Hi,

The 3-5 lines summary:

Based on the dose3 library, I created a tool that allows to analyze and
help solving the build dependency cycle situation when bootstrapping
Debian for a new architecture. I extended dose3 to allow cross build
analysis and implemented an algorithm to enumerate elementary circuits.
I created a wiki page with the output of my algorithms.

Report:

I created the wiki page http://wiki.debian.org/DebianBootstrap/TODO
which lists the current output of my tools so that people do not need to
find out how to install and run my tools to benefit from their output.



As part of creating that wiki page, I added four more analysis options:

1) show vertex properties (Unfulfilled build dependencies, Binary
   packages depending on it being built, Installation set, Source
   packages that must be built for it but cannot, Source packages it is
   a build dependency of)

2) Find source packages that are only missing few build dependencies.
   Maybe these source package can be built without that dependency?

3) Find source packages that build-depend on many others but are only
   needed by few binary packages which are in turn only needed buy a few
   source packages. Maybe the source packages that need those few binary
   packages can be built without them?

4) Find binary packages that are only needed by few source package but
   need many other source packages to be built to satisfy their runtime
   dependencies. Maybe the source package that needs this binary package
   can be built without it?



I addded a tools directory containing apt_pkg based python scripts:

check-deps-apt.sh

   Run `apt-get --only-source --simulate -aarmel build-dep $source` on
   all source packages to verify deb-buildcheck functionality.

check-ma-same-versions.py

  checks if Multi-Arch: Same packages in two package files agree in
  their versions. Packages for armel/armhf build later than their
  counterparts on i386/amd64 leading to version mismatches in Debian
  Sid. This script allows to detect them before attempting to use the
  pair of package files.

filter-packages.py

  given a list of input binary and source packages, only output those
  that match a list of versioned binary and source package names given
  via stdin

  a list of binary and source packages is given via stdin, one package
  per line binary packages are encoded as: $pkgname (= $version) source
  packages are encoded as: src:$pkgname (= $version)
  
  commandline arguments are the two input files and the two output files
  
  after completion, the output sources and packages will only include
  those that match the name/version given in the list via stdin.

  This script is used by reduced_dist.ml

ma-diff.py

  checks for differences in the multiarch status of packages
  
  can be used to compare the multiarch status of two distributions and
  only compares packages that exist in both distributions

merge-packages.py

  merges two or more packages files, taking care that resulting package
  selection is unique in name, arch and version and outputs them in the
  package file given as the last commandline argument
  
  output is sorted by name, then by version, then by architecture

multiarchify.py

  add multiarch properies to a range of packages that are not yet
  multiarch.

  this allows to quickly test what would happen if a number of packages
  were actually multiarchified.

parse_dose_yaml.py

  gives a more concise output of the yaml that deb-buildcheck outputs

single-version.py

  filters a package description file so that only the newest version
  remains.

  It doesnt make sense to check something else than the latest version
  of source packages or to include binary packages of multiple versions
  in a dependency graph.

util.py

  filters a package description file so that only the newest version
  remains



Though I doubted their usefulness, Wookey insisted on a visualization of
the main scc that exists when compiling natively. I never managed to
render the dependency graph because the graph was always too large for
the cairo-renderer that graphviz seems to use to produce bitmaps.
Creating vector graphics (svg) works but I found no tool that was able
to parse them. When assuming that all source packages that have their
cross build dependencies satisfied can actually be cross built, the main
scc shrinks from 1000 nodes to 500 nodes. When then additionally not
printing the installation set of the contained binary packages, cairo is
able to render the dependency graph as a 32767 x 3923 bitmap. You can
download the image here:
http://mister-muffin.de/bootstrap/dep_graph_0.png



The makefile now patches downloaded Packages and Sources files using
diffs from Patrick. This allows to always work on the latest fixes her
produces without having to wait for them being included into Debian.
This is especially useful for additions that cannot be included into
Debian yet, for example :any, :native or :$arch modifiers that cannot be
included yet because of wanna-build.



I finished to include cross buildability into dose3 including more test
cases.  Once included by Pietro, deb-buildcheck will also be able to
check build dependency satisfaction for cross building. My results were
tested against apt-get.  An overview of those efforts can be found at
http://lists.mister-muffin.de/pipermail/debian-bootstrap/2012-August/000323.html
more details are available at http://bugs.debian.org/683786



There have been improvements and cleanups of the code and more
documentation, for example program descriptions when running with
--help.


cheers, josch