Bug#870334: pkg-perl-autopkgtest: revisiting smoke prove --recurse

[Niko Tyni]

On Tue, Aug 01, 2017 at 11:31:05AM -0400, Alex Muntada wrote:

> My idea is that we should smoke test the same list of tests that
> are actually run by "make test" or "./Build test" without having
> to parse those *.PL files to figure out that list. Therefore,
> the list should be obtained by running those same commands with
> some "--dry-run" and "--verbose" options, so we can get a list of
> the files being run as tests and put that list in the smoke setup
> files.
> 
> The issue here is how this "--dry-run" and "--verbose" mode can
> be run (e.g. "make -n test TEST_VERBOSE=1" comes to mind) and at
> which point of package building that should happen, without
> requiring human intervention if possible.

Hm. I agree the goal is to get the same list of tests that are run at
build time. I'm not very keen on parsing 'make -n' output.  Seems somewhat
doomed to fail. But yeah, I guess normally the output will end with
something like

 PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/*/*.t t/*/*/*.t

which could be parsed.

The TEST_FILES thing I was referring to earlier is in the generated
Makefile, sorry for my confusion. Parsing that one might work as the
syntax will supposedly not vary much across packages, but requires
running Makefile.PL first. That's probably not a huge problem as the
build dependencies are already installed.

Maybe the best of both worlds would be something like

 make -n -p | sed -n 's/^TEST_FILES = //p'

with a fallback to 't/*.t' if it ends up empty for some reason.

Alternatively, if we're running Makefile.PL anyway (I'm ignoring M::B
for the moment), this might all be solved if we could just force run
'make test' without actually building first. The tests would get run
with blib/lib etc. in @INC but if those were empty the installed files
would presumably get used anyway. Unfortunately make doesn't seem to be
too helpful with this...

One more wild idea I had is to patch the build machinery (probably EU::MM
itself) to record the list of tests it runs in a file. We could then
install that file in the binary package during the build, and look
there in the autopkgtest phase. This does feel somewhat intrusive,
just mentioning it for completeness :)
-- 
Niko