Bug#922264: pkg-perl-autopkgtest: use "skippable" and "superficial" restrictions

[Xavier]

Le 15/02/2019 à 17:57, gregor herrmann a écrit :
> On Wed, 13 Feb 2019 21:23:00 +0100, Xavier Guimard wrote:
> 
>> Some suggestions for pkg-js-autopkgtest based on pkg-js-autopkgtest
>> discussion with autodep8 maintainers:
>>  - tests skipped should return a 77 exit code and all tests marked as
>>    "Restrictions: skippable". It avoids to consider that a test succeeds
>>    if maintainer skipped it, but needs a merge request to autodep8. See
>>    https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate
>>    (changed by MR !11)
>>  - runtime-deps* tests should be tagged as "Restrictions: superficial"
>>    since these tests don't really test package features but just Perl
>>    syntax
>>
>> Then with this 2 changes, if "build-deps.d" is skipped, success won't
>> give the benefit of 3-days-reduce.
> 
> Thanks for your work and the pull request [0]!
> 
> Some thoughts and more questions:
> 
> For the skippable part:
> - If I understand this correctly (from your text above and the spec
>   [1]) then a skipped syntax.t and use.t would also lead to losing
>   the benefit of faster migration? Do we want this?

The benefit will be lost only if smoke test is skipped. I think it's a
good thing (other tests are "superficial" <=> no benefit). Today if this
test is skipped, it is considered by autopkgtest as "success"

>   Or does it just have no influence?

skippable has no influence if test succeed, but don't consider that test
has failed if a 77 code is returned

> - As for the implementation in [0]:
>   not sure if the "exit 0" in smoke is correct
> - What about the skipped tests within use.t and syntax.t? Should they
>   or some of them also exit 77?

runner do it for them. I didn't modify them if all is skipped as it has
no effect on a test marked as "superficial": 0 or 77 gives the same
result: no benefit, no penalty

> For the superficial part:
> Hm, yeah, use.t and syntax.t don't test that everything in the
> package is fully functional; still, this "superficial" feels a bit
> weird. But probably it's correct according to [1].
> 
> In general I still don't have the full picture of what benefits and
> penalties for testing migration will result from which combination of
> the changes under which circumstances.

The only effect of this is that if smoke test is skipped, there is no
benefit. And I think it's more clear to have the real result:

# EXAMPLE 1, SKIP use.t => benefit OK
autopkgtest [18:23:17]: @@@@@@@@@@@@@@@@@@@@ summary
command1: PASS
command2: FAIL exit 77 (marked as skippable) # I don't remember the
                                             # exact message
command3: PASS (superficial)

# EXAMPLE 2, SKIP smoke => no benefit
command1: FAIL exit 77 (marked as skippable)
command2: PASS (superficial)
command3: PASS (superficial)

# EXAMPLE 3, real failure in smoke => penalty
command1: FAIL
command2: PASS (superficial)
command3: PASS (superficial)

# EXAMPLE 4, real failure in use.t => penalty
command1: PASS
command2: FAIL
command3: PASS (superficial)

Cheers,
Xavier