Bug#922264: pkg-perl-autopkgtest: use "skippable" and "superficial" restrictions

[gregor herrmann]

On Fri, 15 Feb 2019 18:29:14 +0100, Xavier wrote:

> > For the skippable part:
> > - If I understand this correctly (from your text above and the spec
> >   [1]) then a skipped syntax.t and use.t would also lead to losing
> >   the benefit of faster migration? Do we want this?
> The benefit will be lost only if smoke test is skipped. I think it's a
> good thing (other tests are "superficial" <=> no benefit). Today if this
> test is skipped, it is considered by autopkgtest as "success"

Ah, I see. Ok, a lost benefit only for the skipped smoke tests
probably makes sense.
 
> > - As for the implementation in [0]:
> >   not sure if the "exit 0" in smoke is correct

This still confuses me.
Shouldn't it "exit $?" or just nothing (line 174)?

> > - What about the skipped tests within use.t and syntax.t? Should they
> >   or some of them also exit 77?
> runner do it for them. 

Well, only partially. First of all, runners can run more than one
test in a subdirectory (even if we currently only have 3 files in 4
subdirectories, with 3 times 1 and 1 time zero), second, there are
several places in syntax.t and use.t were all or parts of the tests
are skipped. -- But:

> I didn't modify them if all is skipped as it has
> no effect on a test marked as "superficial": 0 or 77 gives the same
> result: no benefit, no penalty

… this "no benefit, no penalty" makes it indeed kind of moot :)
 
> > In general I still don't have the full picture of what benefits and
> > penalties for testing migration will result from which combination of
> > the changes under which circumstances.
> 
> The only effect of this is that if smoke test is skipped, there is no
> benefit. And I think it's more clear to have the real result:
> 
> # EXAMPLE 1, SKIP use.t => benefit OK
> autopkgtest [18:23:17]: @@@@@@@@@@@@@@@@@@@@ summary
> command1: PASS
> command2: FAIL exit 77 (marked as skippable) # I don't remember the
>                                              # exact message
> command3: PASS (superficial)
> 
> # EXAMPLE 2, SKIP smoke => no benefit
> command1: FAIL exit 77 (marked as skippable)
> command2: PASS (superficial)
> command3: PASS (superficial)
> 
> # EXAMPLE 3, real failure in smoke => penalty
> command1: FAIL
> command2: PASS (superficial)
> command3: PASS (superficial)
> 
> # EXAMPLE 4, real failure in use.t => penalty
> command1: PASS
> command2: FAIL
> command3: PASS (superficial)

Thanks alot for those examples, they make it indeed easier for me to
understand the effects!

So, hm, yeah, I guess that all makes sense …


I hope someone else also has some minutes to think it through :)


(Random note, so we don't forget it: There are a few adjusted copies
of the autodep8 file in various packages as debian/tests/control
which should also be adjusted, at least in git for the next upload.)


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Josh With: Milk cow blues
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: Digital Signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-perl-maintainers/attachments/20190215/270aab65/attachment.sig>