Bug#870334: pkg-perl-autopkgtest: revisiting smoke prove --recurse

[Niko Tyni]

On Wed, Aug 02, 2017 at 03:50:39PM -0400, gregor herrmann wrote:
> On Wed, 02 Aug 2017 11:10:25 +0300, Niko Tyni wrote:

> > Sure, we should filter the more common tests of this kind out
> > of the runtime check list.
> 
> Ack, and this should bring us a long way.
> (I saw quite a few t/author/ and t/00-compile/ today.)

This is now fixed in git fwiw (#870252)

> Yes, it makes total sense in theory; I just also happen to like the
> idea that we are currently running as many tests as possible.

I see your point.

> But yeah, if a realiable implementation of "find out build tests and
> run them during autopkgtest" is possible then it probably makes
> sense. And ...
  
> > The 'make -n -p | grep TEST_FILES' thing seems most promising to me so far
> > (even if it's for EU::MM only.)

I played with this a bit and while it seems to work, I think it
feels a bit too intrusive to me.  The second package I tried it with,
libbest-perl, generates tests when running Makefile.PL and then lists
them one by one in TEST_FILES. So we'd need to do this at least before
copying the tests to the tempdir, and the skipped tests might then become
a problem...

Also I'm a bit worried about other side effects that running Makefile.PL
might have.

For now, I've pushed what I have to the ntyni/autopkgtest-smoke-list
branch as a40524732a6967fc64030e51b00f4998b137208a .

> > I just noticed that the Build files generated by Module::Build have a
> > 'retest' option that looks promising:

Didn't get that far but this feels even more intrusive.
 
Any urgency this issue had is gone now that you did all the work of
fixing the --recurse regressions. Many thanks for that :)

I think I'm leaving this for now. We can see if it becomes a problem
later, and if not, just close this.
-- 
Niko