Bug#922264: pkg-perl-autopkgtest: use "skippable" and "superficial" restrictions

[Xavier]

Le 15/02/2019 à 18:29, Xavier a écrit :
> Le 15/02/2019 à 17:57, gregor herrmann a écrit :
>> On Wed, 13 Feb 2019 21:23:00 +0100, Xavier Guimard wrote:
>>
>>> Some suggestions for pkg-js-autopkgtest based on pkg-js-autopkgtest
>>> discussion with autodep8 maintainers:
>>>  - tests skipped should return a 77 exit code and all tests marked as
>>>    "Restrictions: skippable". It avoids to consider that a test succeeds
>>>    if maintainer skipped it, but needs a merge request to autodep8. See
>>>    https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate
>>>    (changed by MR !11)
>>>  - runtime-deps* tests should be tagged as "Restrictions: superficial"
>>>    since these tests don't really test package features but just Perl
>>>    syntax
>>>
>>> Then with this 2 changes, if "build-deps.d" is skipped, success won't
>>> give the benefit of 3-days-reduce.
>>
>> Thanks for your work and the pull request [0]!
>>
>> Some thoughts and more questions:
>>
>> For the skippable part:
>> - If I understand this correctly (from your text above and the spec
>>   [1]) then a skipped syntax.t and use.t would also lead to losing
>>   the benefit of faster migration? Do we want this?
> 
> The benefit will be lost only if smoke test is skipped. I think it's a
> good thing (other tests are "superficial" <=> no benefit). Today if this
> test is skipped, it is considered by autopkgtest as "success"
> 
>>   Or does it just have no influence?
> 
> skippable has no influence if test succeed, but don't consider that test
> has failed if a 77 code is returned
> 
>> - As for the implementation in [0]:
>>   not sure if the "exit 0" in smoke is correct
>> - What about the skipped tests within use.t and syntax.t? Should they
>>   or some of them also exit 77?
> 
> runner do it for them. I didn't modify them if all is skipped as it has
> no effect on a test marked as "superficial": 0 or 77 gives the same
> result: no benefit, no penalty
> 
>> For the superficial part:
>> Hm, yeah, use.t and syntax.t don't test that everything in the
>> package is fully functional; still, this "superficial" feels a bit
>> weird. But probably it's correct according to [1].
>>
>> In general I still don't have the full picture of what benefits and
>> penalties for testing migration will result from which combination of
>> the changes under which circumstances.
> 
> The only effect of this is that if smoke test is skipped, there is no
> benefit. And I think it's more clear to have the real result:

I found the exact message, it is very clear:

# EXAMPLE 1, SKIP use.t => benefit OK
autopkgtest [18:23:17]: @@@@@@@@@@@@@@@@@@@@ summary
command1: PASS
command2: SKIP exit status 77 and marked as skippable
command3: PASS (superficial)

# EXAMPLE 2, SKIP smoke => no benefit
command1: SKIP exit status 77 and marked as skippable
command2: PASS (superficial)
command3: PASS (superficial)

# EXAMPLE 3, real failure in smoke => penalty
command1: FAIL
command2: PASS (superficial)
command3: PASS (superficial)

# EXAMPLE 4, real failure in use.t => penalty
command1: PASS
command2: FAIL
command3: PASS (superficial)

Cheers,
Xavier