Bug#870334: pkg-perl-autopkgtest: revisiting smoke prove --recurse

[Niko Tyni]

On Tue, Aug 01, 2017 at 04:53:36PM -0400, gregor herrmann wrote:
> On Tue, 01 Aug 2017 19:58:55 +0300, Niko Tyni wrote:
> 
> > Hm. I agree the goal is to get the same list of tests that are run at
> > build time. 
> 
> I don't agree 100% I think.
> From what I've seen so far when looking at a dozen of the failures we
> have:
> - same tests in autopkgtest as during build, which fail because of
>   paths or missing files; so just the usual stuff but it appears only
>   now for t/**/*.t;

Yeah, this was expected.

> - more tests in autopkgtest than during build but perfectly
>   reasonable tests where it looks more like an issue that they are
>   not run during build (and the typical easy-to-fix-failures);

If the issue here is that they should be run during build too, the "right"
fix here as well seems to be unifying the list of build and runtime tests?

> - and then the group where something weird is in t/ which happens to
>   look like a test but is a data file or a (misplaced and not
>   environment-variable-protected) author test etc.
>   Which is somewhat a case for #870252

Sure, we should filter the more common tests of this kind out
of the runtime check list.

I still maintain that if we can automatically determine the list of tests
that get run during build time, we should base the list of runtime tests
on that rather than recursing blindly. Of course, if this determination
is too hard, it may be that recursing is good enough.

Do we agree on that?

> Hm, we could also "make" the distribution and rm blib/ before "make
> test". Doesn't feel very clean though ...

Indeed not :)

The 'make -n -p | grep TEST_FILES' thing seems most promising to me so far
(even if it's for EU::MM only.)

I just noticed that the Build files generated by Module::Build have a
'retest' option that looks promising:

   retest
       [version 0.2806]

       This is just like the "test" action, but doesn't actually build
       the distribution first, and doesn't add blib/ to the load path,
       and therefore will test against a previously installed version
       of the distribution.  This can be used to verify that a certain
       installed distribution still works, or to see whether newer
       versions of a distribution still pass the old regression tests,
       and so on.

Potentially this could be used instead of all the 'copy the tests
but not the source code' things. Not sure if it's worth it though.

Unfortunately I don't see a corresponding EU::MM feature, though
it seems it wouldn't be too hard to implement...
-- 
Niko