[Debian-med-packaging] Bug#806214: Status of build time test suites of tree-puzzle

[Shayan Doust]

Hello Andreas,

Some investigative work. I also compiled a clean copy directly from the
upstream website to prevent any sort of contamination and to rule out
possible upstream fault and then a packaging build attempt (debhelper 11).

> I have some gut feeling that the test files for comparison do not
> really fit the proper result.

This may be the case, but I am having a hard time getting the same
output as
https://buildd.debian.org/status/fetch.php?pkg=tree-puzzle&arch=all&ver=5.2-11&stamp=1539685923&raw=0

I have just ran qp-pure-prot.test as an example:

113,114c86,87
< WARNING: Result of chi-square test may not be valid because of
< small maximum likelihood frequencies and short sequence length!
---
> WARNING: Result of chi-square test may not be valid because of small
> maximum likelihood frequencies and short sequence length!

* Diffs like this indicate formatting faults which are easy to rectify,
or remove as a whole.

* The first line of the test output file contains TEST-PUZZLE and the
version number. This version number (5.3.rc16) is not visible in the
upstream expected output file, so diff will of course display this as a
delta.

* There is a higher verbosity of paragraphed description / output in the
test's newly-generated results file compared to what upstream has. Maybe
upstream is running an older / more obselete version of the program?

* From what I can see, the numbers seem to match for the most part, but
upstream has introduced chi2-value column which is not present in the
expected results file.

For instance, running qp-pure-bin.test. Here is a small slice of my output:

194c169
< (bipartition with sequences in input order : number of times seen (and
ratio))
---
> (bipartition with sequences in input order : number of times seen)
196,197c171,172
<  *..** :  1000 (1.000)
<  *...* :  1000 (1.000)
---
>  *..**  :  1000
>  *...*  :  1000
201c176
< (bipartition with sequences in input order : number of times seen (and
ratio))
---
> (bipartition with sequences in input order : number of times seen)

Ratios seem to have been added to the test. Scanning through all the
other test binaries and their outputs, my values seem to match up to
what is expected regardless of the issues above.

So next I decided to see when the tests directory was added. This
happened to be version 5.2. Quickly testing a couple of test binaries,
they run successfully. It seems like to me that the expected test
results file introduced in 5.2 were just never updated to 5.3~rc16. I am
not sure why your build log is different in terms of value. Maybe
upstream modified the current release candidate and version (some do
this). I will keep this updated with anything else that I spot or comes
to mind.

Kind regards,
Shayan Doust

On Mon, 8 Jul 2019 21:26:02 +0200 Andreas Tille <andreas at an3as.eu> wrote:
> Hi,
> 
> after switching tree-puzzle debhelper level to 9  I was cheating around
> the build time test suite via
> 
> override_dh_auto_test:
> 	# unfortunately most tests are failing for the moment
> 	# the issue is documented in
> 	#   debian/patches/patch_test_results.patch
> 	# and needs to be discussed with upstream
> 	dh_auto_test || true
> 
> The rationale was that just by switching the debhelper level the build
> time test suite was run at all.  Most probably it was failing all the
> time before and simply nobody realised this.  To sort this out we need
> to talk to upstream.  The issue is documented in bug #806214 (bug in
> CC).
> 
> I now bumped the upstream source in Git to the latest upstream release
> candidate.  Since this had not changed quite some time I assume upstream
> is not very rapidly pushing a final release.  However, this might be the
> right point in time to sort things out.
> 
> If you check the build log of 5.2-11 at
> 
>    https://buildd.debian.org/status/fetch.php?pkg=tree-puzzle&arch=all&ver=5.2-11&stamp=1539685923&raw=0
> 
> you can find
> 
> ...
> dh_auto_test || true
>  make -j1 check VERBOSE=1
> make[2]: Entering directory '/<<PKGBUILDDIR>>'
> Making check in src
> make[3]: Entering directory '/<<PKGBUILDDIR>>/src'
> make[3]: Leaving directory '/<<PKGBUILDDIR>>/src'
> Making check in doc
> make[3]: Entering directory '/<<PKGBUILDDIR>>/doc'
> make[4]: Entering directory '/<<PKGBUILDDIR>>/doc'
> make[4]: Nothing to be done for 'check-am'.
> make[4]: Leaving directory '/<<PKGBUILDDIR>>/doc'
> make[3]: Leaving directory '/<<PKGBUILDDIR>>/doc'
> Making check in data
> make[3]: Entering directory '/<<PKGBUILDDIR>>/data'
> make[3]: Nothing to be done for 'check'.
> make[3]: Leaving directory '/<<PKGBUILDDIR>>/data'
> Making check in tests
> make[3]: Entering directory '/<<PKGBUILDDIR>>/tests'
> make  check-TESTS
> make[4]: Entering directory '/<<PKGBUILDDIR>>/tests'
> make[5]: Entering directory '/<<PKGBUILDDIR>>/tests'
> SKIP: build-puzzle
> FAIL: qp-pure-bin.test
> FAIL: qp-pure-nucl.test
> FAIL: qp-tn-nucl.test
> FAIL: qp-hky-clock-nucl.test
> FAIL: qp-hky-rhet-nucl.test
> FAIL: qp-hky-rhet-clock-nucl.test
> FAIL: qp-pure-prot.test
> FAIL: qp-mtrev-prot.test

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-med-packaging/attachments/20190905/cbddb865/attachment.sig>