Bug#799444: pkg-perl-autopkgtest: prove --merge breaks some test suites

[Niko Tyni]

Package: pkg-perl-autopkgtest
Version: 0.25

In the 'smoke' tests, which run the upstream test suite against an
installed module, we currently use the 'prove' '--merge' option.
This interleaves STDERR and STDOUT, trying to preserve their relative
ordering. While wonderful when it works, this apparently breaks some
test suites: at least libterm-progressbar-perl (see #785045) and possibly
libcapture-tiny-perl (#785022).

Even prove(1) says '--merge' is risky:

       Caveat: this is a bit of a kludge. In particular note that if
       anything that appears on STDERR looks like a test result the test
       harness will get confused. Use this option only if you understand
       the consequences and can live with the risk.

I'm inclined to drop the option rather than make it configurable.
This will make the logs a bit harder to read, but should make the
tests more robust.

I note that we still need to redirect STDERR, otherwise output on that
will be considered a test failure. Alternatively we could change the test
definitions in the autodep8 package to add the restriction 'allow-stderr'.
Adding a 2>&1 to the 'prove' call in the 'smoke' script seems easier.

Thoughts welcome, I'll wait a bit before implementing this.
-- 
Niko Tyni   ntyni at debian.org