[Piuparts-devel] using piuparts master/slave for "sub-distributions"

[Andreas Beckmann]

Hi,

(this mail is a lot of brainstorming, looking for input)

Currently piuparts in master/slave mode works quite well for main. But
it would be nice to be able to run piuparts on contrib, non-free,
*-updates, */updates (aka security), *-backports too. And of course
nothing of this should be hardcoded, but fully configurable as other
distributions may have other naming schemes.

Following is a list where dependencies can be resolved in addition to
the current distribution/area:

squeeze/main: (no deps)
squeeze/contrib: squeeze/main squeeze/non-free
squeeze/non-free: squeeze/main squeeze/contrib
squeeze-updates/main: squeeze/main
squeeze-updates/contrib: squeeze/main squeeze/contrib squeeze/non-free
  (squeeze-updates/main squeeze-updates/non-free)
squeeze-updates/non-free: squeeze/main squeeze/contrib squeeze/non-free
  (squeeze-updates/main squeeze-updates/contrib)
squeeze/updates/main: squeeze/main
squeeze/updates/contrib: squeeze/main squeeze/contrib squeeze/non-free
  (squeeze/updates/main squeeze/updates/non-free)
squeeze/updates/non-free: squeeze/main squeeze/contrib squeeze/non-free
  (squeeze/updates/main squeeze/updates/contrib)
squeeze-backports/main: squeeze/main
squeeze-backports/contrib: squeeze/main squeeze/contrib
  squeeze/non-free squeeze-backports/main squeeze-backports/non-free
squeeze-backports/non-free: squeeze/main squeeze/contrib
  squeeze/non-free squeeze-backports/main squeeze-backports/contrib

All this can be performed with a single base system squeeze.tgz (ideally
only created from squeeze/main), custom scripts (post_setup_*) can be
used to update the sources.list for virtual distributions like
squeeze-security, squeeze-updates, squeeze-backports.

Urls to packages files are usually generic and can be computed from
$MIRROR, $DISTRO, $AREA, $ARCH, but for e.g. security and backports the
packages-url is "special" (at least the MIRROR part does not match the
MIRROR for the base system).

For piuparts-slave + piuparts not that many changes seem to be necessary.

More work needs to be done for piuparts-master (and piuparts-report) as
they will need to load multiple packages files and log databases (but
only consider a subset of the packages for "to-be-tested", the others
only for dependency resolution)

I tried sid/contrib and sid/non-free and most packages were
uninstallable due to missing dependencies:

12:05 Running section sid-contrib
12:05 successfully-tested: 19
12:05 dependency-cannot-be-tested: 8
12:05 dependency-does-not-exist: 132
12:05 no-dependency-from-alternatives-exists: 70

12:05 Running section sid-non-free
12:05 successfully-tested: 125
12:05 dependency-cannot-be-tested: 9
12:05 dependency-does-not-exist: 259
12:05 no-dependency-from-alternatives-exists: 88

On the other hand, there are not that many packages in contrib and
non-free (about 700 in total) so that it may be easier to just test them
along with main instead of separating them ...

Possible implementation fro the latter approach: in piuparts.conf add an
(optional) "area" key, defaulting to "main" and possibly multi-valued as
in "main contrib non-free" ("contrib" or "non-free" standalone would not
make that much sense).
When generating packages URLs, just generate one for each area value and
load them all together.

With this approach we can reduce the sub-distribution dependencies to

squeeze: (none)
squeeze-updates: squeeze
squeeze/updates: squeeze
squeeze-backports: squeeze

Andreas