[Piuparts-devel] simplifying running piuparts (was: Re: Debian development and release: always releasable (essay))

[Andreas Beckmann]

@all maintainers: How would you like to run piuparts s.t. it easily
integrates into your workflow and allows improving Debian's quality?
This is something we could improve right now. Integrating piuparts into
the ftp-master/buildd side will take a much longer way.

On 2013-05-19 18:39, Ondřej Surý wrote:
[...]
> So apart from the more hands on some packages with high priority, it
> would really help me to have some automated tests which would be run
> before uploading to testing.
> 
> One thing which immediatelly comes to the mind is the install & upgrade test.
> 
> 1. install every built package
> 2. try upgrade from stable for every built package
> 3. try upgrade from testing for every built package
> 4. try upgrade from unstable for every built package

Let me see if we can find a way to simplify running piuparts for all
these ...

Since you are aiming to test transitions, this will need to cover
testing multiple source packages at the same time and a lot of
dependencies between them. May I assume you have a local repository of
these packages available? (You probably already have one to build
updated packages for the transition.) A file:// URL with some .debs and
a Packages file will be fine.
Or would you prefer to feed a bunch of .changes files to some script?

Once you run that yet-to-be-written script, you will get a bunch of
logfiles (for k .debs and 4 tests that would be 4*k logfiles), some
failed, some success. Since piuparts is very chatty, you probably want
to have a summary and a list of failures at the end ...
The failing logfiles will need to be analyzed manually.

Next question: Do you want to do incremental testing? Assume you
identified an incorrectly versioned dependency, fixed that, rebuilt the
package, updated the repository. Do you want to
* retest a specific failure
* retest all failures
* retest everything (you probably only want to do this at the end as it
may be very time consuming).
I would probably go for "test everything that does not have a success
log", so you could retest arbitrary subsets by just deleting the
corresponding (success-)logs.

The good thing is: by now piuparts should have all the needed options to
do this, it just needs a new driver script to simplify running it on a
bunch of local packages. (And running it on each package from a (set of)
.changes file(s) individually, not only installing all at the same time)

In http://bugs.debian.org/700849 I posted a script skeleton that
(requires manual adjustment to configure it and) allows to quickly run
one of the tests listed above for a local (or remote) repository.

This yet-to-be-written script should not be specific for uploads to sid,
but work for pu, opu or backports as well. Experimental may be a bit
more difficult as this is "frequently broken" (as in "requiring a very
careful selection of the install set mixing sid and experimental")


> Optionally:
> 5. do some testing with all r-deps (treeish?)

That may take some time to run ... I also don't know how to quickly
build the rdep tree. But if you already have the local repository and a
list of "interesting" packages (not in the local repo), it should be
quite easy to check how they behave (install/upgrade wise) in a
sid+prospective environment.

That's just my interpretation how this could work ...


Andreas