[Piuparts-devel] Handling Piuparts Failures

[Andreas Beckmann]

Hi,

before we redesign "exceptions", I think we need to redesign "failures".

piuparts has grown to do install/upgrade/purge checks as well as some
sanity checks.

There are

*classic failures* like exploding maintainer scripts, they block
rdepends from being tested. Actually we could use a finer granularity:
if a package succeeds to install, we can continue with
  - purge
  - distupgrade
  - install rdepends
if a package fails to remove or purge, we can still
  - install rdepends (but we should skip purge as we will probably
      inherit a failure)
  - distupgrade

Actually we can just draw the dependency graph of a distro. Nodes are
connected by "install" edges. And we can draw the dependency graph of
distro+1 the same way. The two graphs can be connected with "upgrade"
edges (between the same packages). If a package no longer exists in
distro+1, we add a "upgrade" edge (to some $package:void node). The
package might misbehave by just being not removed. And the package might
return in distro+1+k. There is no use in having
package:void(distro-1) -> package(distro) edges.

A node that fails-to-install will block its rdepends subtree with
"dependency-failed-install" (alternate dependencies make this "interesting")
In our succeeds-to-install (succeeds-to-upgrade) subgraph, we can add
"remove" edges (parallel to the install edges). Some nodes will fail
here, some subgraphs will be blocked (dependency-failed-remove).
Same game with "purge" edges ... dependency-failed-purge

So a package goes through several stages:

	unknown
	installed
	removed
	purged

Each allows to run the next state as well as further tests. Let me add
one more: "deps-installed" - that will rarely fail, but may serve as a
reference snapshot:

	unknown
	deps-installed
	installed
	removed
	purged

In the context of --install-remove-install and --install-purge-install
we might have to distinguish between package-removed,
package-and-deps-removed (just "removed" above), package-purged, and
package-and-deps-purged (just "purged" above).

Next we get to the "sanity checks", they might be
* non-destructive - does not modify the chroot state
  e.g. copyright existence check
* mutating - slight modifications (+ restoring) of the chroot
  e.g. logrotate or cronfiles test
* destructive - heavy modifications of the chroot, even if restored
  e.g. --install-remove-install or --install-purge-install
  not neccessary a reliable baseline for further tests
These properties determine how often we need to run piuparts to get
reliable results for large number of "complicated" tests on a single
package.

The sanity checks might work
* local (looking only at a specific package or its files)
  e.g. copyright existence check
  e.g. local broken symlink check
* global (looking at all packages/files/...)
  e.g. debsums check
  (that could be run "local", but we want to know if $PACKAGE messed
  around somewhere else)
  e.g. global broken symlink check (the one we currently have and is
  ignored by everone - but it finds broken links created, not shipped)

the outcome of a sanity check can be
* pass
* fail (non-inheriting) - usually local checks
* fail (inheriting) - usually global checks
  failure shows up in all rdepends

So the questions that arise here are

When to fail? When to succeed? How to report?

* local checks should not fail, but report errors in a way to mark the
log appropriately. There can be any number of them, ordering is not
important. No exceptions are needed.
>From a testing POV (e.g. piuparts-master) this should be a "pass", from
a reporting POV this should be a "fail". So perhaps a
pass-with-errors

* global inheriting checks should have some severity ordering and would
probably need to fail (i.e. fail with the most severe, not the most
frequent error). rdepends could test up to the last successful test of
their dependecies, if this succeeds, the package is at least
"partially-checked" . These may need exceptions to improve the coverage.

For exceptions we need to distinguish
* direct exceptions - the big red error report pass-with-exceptions
* inherited exceptions - may have an informative report

Package health (match this to appropriate whether icons)

serious failure
important failure
untestable (pass with 0% tested, but not waiting to be tested)
partially tested (pass with some % inbetween, and parts skipped due
                  to dependency failures)
pass (with 100% tested)
unknown (waiting to be tested)

(define appropriate addition operator to combine the results from e.g.
all binary packages of a source package or all architectures

So piuparts needs to return (either as error code or strings in its output)
* continuing information (fail in stage X or pass all)
* health information (that can be serveral SERIOUS FAILURE, IMPORTANT
FAILURE, PARTIALLY CHECKED, EXCEPTION APPLIED, EXCEPTION INHERITED)

the "health information" is currently extracted by the known problems -
and the severity level (serious vs. important) should not neccessarily
be applied by piuparts itself but by the reporting tools in the framework

And we should be open to have a "pedantic" severity someday :-)


Andreas