[Piuparts-devel] Bug#595119: Bug#595119: piuparts: modular way to add new tests?

[Holger Levsen]

Hi Timo, Scott,

On Mittwoch, 1. Juni 2011, Scott Schaefer wrote:
> >> this with the "custom scripting" interface
> >> (/usr/share/doc/piuparts/README.html or
> >> http://piuparts.debian.org/doc/README.html#_custom_scripts_with_piuparts
> >> ). While perhaps not ideal, this interface provides a 'modular way to
> >> add new tests'.
> > 
> > Looks good indeed but is this only for my private tests? How do I get my
> > tests to run on piuparts.debian.org?
> 
> Hm ... Well, that is a very good question.  And, I understand what you
> are trying to do is outside the scope of what can be expected of
> "private tests". 

I'd say the scripts dir is not really suited to add tests in a modular way, 
esp. once piuparts is able to log reasons for failure.

So I consider this bug report as valid :-)

IME the scripts dir is rather useful for working around buggy/otherwise 
untestable packages.

The scripts dir used on piuparts.d.o is available at 
http://svn.debian.org/viewsvn/piuparts/piatti/org/piuparts.debian.org/etc/scripts/

> Given what I have read, and reviewing list of wishlist bugs, I am
> reasonably certain a "framework for securely executing plugins" will be
> a big part of any future discussion re "piuparts 2.0".

I dont think so. I think running piuparts (for the whole archive) will always 
be "a security risk" and should only be done in throw-away environments.

> > I didn't notice this custom scripting interface at all at that time.

see http://piuparts.debian.org/doc/README.html#_custom_scripts_with_piuparts

> > 2) This test requires that emacs is installed

make it fail gracefully if emacs is not installed. (so it will work/not fail 
for all packages which dont depend on emacs)

> I am willing to "come back" to this bug at later date.  However, I don't
> expect that to happen until I or someone fixes the 3-4 important/normal
> bugs outstanding.

much appreciated! :-)

(I mostly replying to this one now as its a rather easy "target" :)


There are more things to consider: wheezy and squeeze are tested less strict 
than unstable, for two reasons: we really want to catch the important bugs in 
$testing (and stable) and we still want to report/find bugs (thus more tests 
in sid). And we also want as many packages as sensible possible tested in 
$testing (and less failures found means more dependent packages tested).

But I still am careful with adding new tests to sid: piuparts.d.o still lacks 
(wo)menpower to file all bugs found and IMO its more motivating to file bugs 
when the backlog is smaller. If we add too many pedantic tests, the backlog 
will be greater and the "piuparts test failed" status in the PTS will be less 
meaningful.

All this said, I still would like to run all those tests proposed as wishlist 
bugs against piuparts on piuparts.d.o! But, I wouldnt run them against the sid 
suite (nor $testing nor $stable) but against a new suite, sid-pedantic or 
such. I think this would give us a good compromise: sid is tested against the 
(more) meaningful tests "only" (thus allowing dependent packages to be more 
often tested), while we still expose packages against the "more exotic" tests.


cheers,
	Holger