Bug#922264: pkg-perl-autopkgtest: use "skippable" and "superficial" restrictions

[gregor herrmann]

On Wed, 13 Feb 2019 21:23:00 +0100, Xavier Guimard wrote:

> Some suggestions for pkg-js-autopkgtest based on pkg-js-autopkgtest
> discussion with autodep8 maintainers:
>  - tests skipped should return a 77 exit code and all tests marked as
>    "Restrictions: skippable". It avoids to consider that a test succeeds
>    if maintainer skipped it, but needs a merge request to autodep8. See
>    https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate
>    (changed by MR !11)
>  - runtime-deps* tests should be tagged as "Restrictions: superficial"
>    since these tests don't really test package features but just Perl
>    syntax
> 
> Then with this 2 changes, if "build-deps.d" is skipped, success won't
> give the benefit of 3-days-reduce.

Thanks for your work and the pull request [0]!

Some thoughts and more questions:

For the skippable part:
- If I understand this correctly (from your text above and the spec
  [1]) then a skipped syntax.t and use.t would also lead to losing
  the benefit of faster migration? Do we want this?
  Or does it just have no influence?
- As for the implementation in [0]:
  not sure if the "exit 0" in smoke is correct
- What about the skipped tests within use.t and syntax.t? Should they
  or some of them also exit 77?

For the superficial part:
Hm, yeah, use.t and syntax.t don't test that everything in the
package is fully functional; still, this "superficial" feels a bit
weird. But probably it's correct according to [1].

In general I still don't have the full picture of what benefits and
penalties for testing migration will result from which combination of
the changes under which circumstances.


Cheers,
gregor


[0] https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/merge_requests/2
[1] https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Cat Power: The Greatest
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: Digital Signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-perl-maintainers/attachments/20190215/00f9ed36/attachment.sig>