[Piuparts-devel] Bug#696093: piuparts git repository branch, develop, updated. 0.52-37-g73cb9af

[Andreas Beckmann]

On 2013-05-30 17:55, Holger Levsen wrote:
> On Donnerstag, 30. Mai 2013, Holger Levsen wrote:
>>> sid_main.txt
>>> sid_main_pedantic.txt
> 
> Thinking about this further, I'm not sure I want this. I'd rather like the PTS 
> only display relevant piuparts issues, and no pedantic ones at all. 

So we need define what we consider "pedantic" errors.

I'd like to have a high-coverage sid test, i.e. one that finds a serious
installation/upgrade error in A even if its dependency B has a less
critical error (leftover file, broken symlink, missing copyright file,
..., i.e. anything that does not cause installation/upgrade/removal to
fail). Then we can have additional test suites that discover more of
these less critical errors (e.g. sid-nodoc, leftovers, broken-symlinks,
LC_ALL=foo_BAR, /bin/sh=trash, ...), but have less coverage since they
have to exclude the rdeps of failing packages.

> Or maybe, each distro specific piuparts issue and, once in smaller font, the 
> info that the package in one or more distros has pedantic piuparts issues.
> 
> But I really like it to stay this way, that piuparts issues are bad errors and 
> nothing pedantic. But maybe I miss an opportunity here.

For the PTS probably only piuparts errors in the following suites are
relevant:

unstable
testing
stable->testing
(testing->unstable)
experimental
(sid->experimental)

and it's probably up to us piuparts developers what we want to show
there :-)

IIRC the following information would be interesting for the PTS:
	package	status	URL
(and that probably per interesting suite), status \in { pass, fail,
unknown }

So while piuparts-report would generate such a list per test-suite, we
could merge them into a unified list per distribution, keeping an error
if there was any (usually keeping the most serious one and linking
there) and reporting pass only if all subtests are either "pass" or
"unknown".
That could also be useful for the case where the piuparts test itself
passes, but the log processing decides that there were errors (that's
what I like to call non-inheriting failures, adequate would be an
adequate example.) Turning these into hard failures would decrease
coverage significantly.

The PTS does not need to know details about what piuparts does as long
as we provide some data for it to use.

Maybe a format of
	package status severity URL
could be used to provide additional information, with possible severity
values e.g.
	* '-' (for non-failing packages)
	* serious
	* important
	* pedantic
but it wouldn't need to be used from the beginning, we could just
hardcode fail to serious and have it in the status file (until we decide
at some point that some issues may get less attention)

Andreas

PS: since we are talking about the PTS, I usually meant "source package"
where I wrote "package"