Bug#922264: pkg-perl-autopkgtest: use "skippable" and "superficial" restrictions

[Niko Tyni]

On Wed, Feb 13, 2019 at 09:23:00PM +0100, Xavier Guimard wrote:
> Package: pkg-perl-autopkgtest
> Version: 0.50
> Severity: wishlist

>  - tests skipped should return a 77 exit code and all tests marked as
>    "Restrictions: skippable". It avoids to consider that a test succeeds
>    if maintainer skipped it, but needs a merge request to autodep8. See
>    https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate
>    (changed by MR !11)

Makes sense to me.

>  - runtime-deps* tests should be tagged as "Restrictions: superficial"
>    since these tests don't really test package features but just Perl
>    syntax
> 
> Then with this 2 changes, if "build-deps.d" is skipped, success won't
> give the benefit of 3-days-reduce.

I agree with the intention here. The only non-superficial test of the
current ones is indeed the 'smoke' test.

However, I have an "architectural" concern.

The proposed implementation overloads the 'runtime-deps' vs 'build-deps'
separation. Theoretically, we might come up with a 'runtime-deps' test
that is not superficial even though all of them currently are, or a
'build-deps' test that is superficial unlike the current ones.

The separation precedes autodep8 and was designed to make it possible to
update the set of standard pkg-perl autopkgtests without having to update
every package separately. This was later solved in a more centralized
way by autodep8.

I think we should probably get rid of this now unnecessary layer of
indirection and just list smoke, use.t and syntax.t in autodep8, with
the proper restrictions. This seems buster+1 material.

Please don't consider this a blocker or veto for the proposed
implementation. It's a clear improvement and does not make
it any harder to simplify the architecture later.

Thanks for working on this,
-- 
Niko