Bug#870334: pkg-perl-autopkgtest: revisiting smoke prove --recurse

Tue Aug 1 14:34:14 UTC 2017

On Tue, 01 Aug 2017 11:18:30 +0300, Niko Tyni wrote:

> Since 0.37, we're running 'prove --recurse' by default in the smoke
> test. This has resulted in ~50 regressions according to ci.d.n:
> there are packages ship .t files under t/ subdirectories (or starting
> with a dot) that are not run during builds and are either broken
> or not intended to be run at all.  Fixing these by adding t/*.t in
> debian/tests/pkg-perl/smoke-tests gets boring quite quickly, though it's
> certainly doable.

Hm, interesting finding ..

What triggered the change was that adding all the tests in
subdirectories also became boring. I had a look: Currently we have 24
debian/tests/pkg-perl/smoke-tests files with something like:

% cat libhtml-formfu-perl/debian/tests/pkg-perl/smoke-tests
t/*.t
t/*/*.t
t/*/*/*.t
t/*/*/*/*.t

> All these packages seem to be relying on the ExtUtils::MakeMaker default
> of TEST_FILES => t/*.t. I'm wondering if we should try to parse that
> out of Makefile.PL during autopkgtest instead of always recursing,
> though that does seem somewhat error prone.

libhtml-formfu-perl's Makefile.PL has

  "test" => {
    "TESTS" => "t/*.t t/bugs/*.t t/constraints/*.t t/deflators/*.t t/elements/*.t t/examples/*.t t/filters/*.t t/force_errors/*.t t/form/*.t t/i18n/*.t t/inflators/*.t …
  }

> FWIW, Module::Build seems to have a related 'recursive_test_files'
> option that is off by default (see Module::Build::API(3pm)).
> 
> It might help a bit to collect some statistics about the number of
> packages with *.t files in subdirectories, and whether those have
> TEST_FILES specified in Makefile.PL. Currently I have no idea if the
> failing packages are just a small minority of the affected ones, or if
> almost all of them are failing now.

Not sure if TEST_FILES is actually that helpful:

% grep TEST_FILES */Makefile.PL      
ack-grep/Makefile.PL:TEST_FILES=t/*.t t/lib/*.t
ack-grep/Makefile.PL:	$(FULLPERLRUN) t/runtests.pl 0 $(TEST_VERBOSE) "$(INST_LIB)" "$(INST_ARCHLIB)" $(TEST_FILES)
ack-grep/Makefile.PL:	$(FULLPERLRUN) t/runtests.pl 1 $(TEST_VERBOSE) "$(INST_LIB)" "$(INST_ARCHLIB)" $(TEST_FILES)
libanyevent-gearman-perl/Makefile.PL:                    .q{ $(TEST_FILES)} . "\n";
libapache2-mod-perl2/Makefile.PL:TEST_FILES =
libapache2-mod-perl2/Makefile.PL:	t/TEST -bugreport -verbose=$(TEST_VERBOSE) $(TEST_FILES)
libautovivification-perl/Makefile.PL:	PERL_DL_NONLAZY=1 PERLDB_OPTS="NonStop=1" $(FULLPERLRUN) -MTAP::Harness -e 'TAP::Harness->new({verbosity => q{$(VERBOSE)}, lib => [ q{$(INST_LIB)}, q{$(INST_ARCHLIB)} ], switches => [ q{-d} ]})->runtests(@ARGV)' $(TEST_FILES)
libcpanel-json-xs-perl/Makefile.PL:    s/TEST_FILES = /TEST_FILES = -Mcommon::sense /;
libdata-messagepack-perl/Makefile.PL:            .q{ $(TEST_FILES)};
libdata-peek-perl/Makefile.PL:		    $(TEST_FILES) 3>valgrind.log
libdata-peek-perl/Makefile.PL:	    '	- at env TEST_FILES="xt/*.t" make -e test_dynamic',
libdata-peek-perl/Makefile.PL:	q{	sandbox/leaktest $(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES)},
libdbd-csv-perl/Makefile.PL:	    '	- at env DBI_SQL_NANO=1 make -e test_dynamic TEST_FILES=t/[1-9]*.t',
libdbd-csv-perl/Makefile.PL:	    '	- at env TEST_FILES="xt/*.t" make -e test_dynamic',
libgeo-distance-xs-perl/Makefile.PL:\t\$(MAKE) -e \$(TEST_TYPE) TEST_FILES="xt/*.t"
libmemcached-libmemcached-perl/Makefile.PL:	HARNESS_PERL_SWITCHES='-MDevel::Cover=-coverage,branch,-coverage,condition,-coverage,pod,-coverage,statement,-coverage,subroutine' PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-MExtUtils::Command::MM" "-e" "test_harness($(TEST_VERBOSE), '$(INST_LIB)','$(INST_ARCHLIB)')" $(TEST_FILES)
libpoe-perl/Makefile.PL:  TEST_FILES
libpoe-perl/Makefile.PL:  test           => { TESTS => TEST_FILES  },
libppix-regexp-perl/Makefile.PL:	'$(FULLPERLRUN)', '$(AUTHORTEST_FILES)' );
libppix-regexp-perl/Makefile.PL:AUTHORTEST_FILES = t/*.t xt/author/*.t
libscope-upper-perl/Makefile.PL:	PERL_DL_NONLAZY=1 PERLDB_OPTS="NonStop=1" $(FULLPERLRUN) -MTAP::Harness -e 'TAP::Harness->new({verbosity => q{$(TEST_VERBOSE)}, lib => [ q{$(INST_LIB)}, q{$(INST_ARCHLIB)} ], switches => [ q{-d} ]})->runtests(@ARGV)' $(TEST_FILES)
libsoap-wsdl-perl/Makefile.PL:    test => { RECURSIVE_TEST_FILES => 1, },
libspreadsheet-read-perl/Makefile.PL:		    $(TEST_FILES) 3>valgrind.log
libspreadsheet-read-perl/Makefile.PL:	q{	sandbox/leaktest $(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES)},
libsystem-info-perl/Makefile.PL:		    $(TEST_FILES) 3>valgrind.log
libsystem-info-perl/Makefile.PL:	    '	- at env TEST_FILES="xt/*.t" make -e test_dynamic',
libsystem-info-perl/Makefile.PL:	q{	sandbox/leaktest $(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES)},
libtext-csv-xs-perl/Makefile.PL:		    $(TEST_FILES) 3>valgrind.log
libtext-csv-xs-perl/Makefile.PL:	    '	- at env TEST_FILES="xt/*.t" make -e test_dynamic',
libtext-csv-xs-perl/Makefile.PL:	q{	sandbox/leaktest  $(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES)},
libtext-csv-xs-perl/Makefile.PL:	q{	sandbox/leaktrace $(FULLPERLRUN) "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES)},

Another random grep:

% egrep 't/\*/\*\.t' */{Makefile,Build}.PL
libauthen-simple-ldap-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libauthen-simple-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libb-utils-perl/Makefile.PL:    test          => { TESTS => 't/*.t t/*/*.t' },
libclass-data-accessor-perl/Makefile.PL:tests "t/*.t t/*/*.t";
libcrypt-rijndael-perl/Makefile.PL:    'TESTS' => 't/*.t t/*/*.t'
libdata-clone-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libdevel-cover-perl/Makefile.PL:push @tests, grep !/e2e/, glob "t/*/*.t";
libdevel-cover-perl/Makefile.PL:                                         : "t/*/*.t"
libextutils-modulemaker-perl/Makefile.PL:    test        => { TESTS => join q{ }, map {glob} qw( t/*.t t/*/*.t ) },
libfile-zglob-perl/Makefile.PL:tests 't/*.t t/*/*.t t/*/*/*.t t/*/*/*/*.t';
libimager-perl/Makefile.PL:my $tests = 't/*.t t/*/*.t';
libio-file-withpath-perl/Makefile.PL:tests 't/*.t t/*/*.t t/*/*/*.t t/*/*/*/*.t';
libmakefile-dom-perl/Makefile.PL:tests('t/*.t t/mdom/*.t t/mdom/*/*.t'); # t/*/*.t t/*/*/*t t/*/*/*/*.t');
libmango-perl/Makefile.PL:  test      => {TESTS       => 't/*.t t/*/*.t'}
libminion-perl/Makefile.PL:  test      => {TESTS       => 't/*.t t/*/*.t'}
libmodule-cpants-analyse-perl/Makefile.PL:  test => {TESTS => 't/*.t t/*/*.t'},
libmojo-pg-perl/Makefile.PL:  test      => {TESTS       => 't/*.t t/*/*.t'}
libmojolicious-perl/Makefile.PL:  test => {TESTS => 't/*.t t/*/*.t'}
libmojomojo-perl/Makefile.PL:tests(join ' ', (glob('t/*.t'), glob('t/*/*.t')));
libmousex-types-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libplack-app-proxy-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libregexp-common-perl/Makefile.PL:my @tests = qw [t/*.t t/*/*.t];
libset-object-perl/Makefile.PL:  test => { TESTS => join(' ', glob('t/*/*.t')) },
libtest-without-module-perl/Makefile.PL:my @tests = map {glob $_ } 't/*.t', 't/*/*.t';
libtext-mecab-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libtext-mecab-perl/Makefile.PL:##             TESTS    => 't/*.t t/*/*.t'
libunicode-utf8-perl/Makefile.PL:tests 't/*.t t/*/*.t';
libwww-mechanize-shell-perl/Makefile.PL:my @tests = map { glob $_ } 't/*.t', 't/*/*.t';
libyaml-syck-perl/Makefile.PL:tests           't/*.t t/*/*.t';
libclass-dbi-sweet-perl/Build.PL:    test_files         => [ glob('t/*.t'), glob('t/*/*.t') ]
libgenome-perl/Build.PL:  #test_files => [qw|t/*.t t/*/*.t t/*/*/*.t t/*/*/*/*.t|],
libgetopt-complete-perl/Build.PL:  #test_files => [qw|t/*.t t/*/*.t t/*/*/*.t t/*/*/*/*.t|],

alexm has more idea, we'll add something hopefully more coherent later :)

Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at/ - Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: Digital Signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-perl-maintainers/attachments/20170801/70f00811/attachment.sig>