[Debian-med-packaging] Bug#806214: Status of build time test suites of tree-puzzle

[Andreas Tille]

Hi,

after switching tree-puzzle debhelper level to 9  I was cheating around
the build time test suite via

override_dh_auto_test:
	# unfortunately most tests are failing for the moment
	# the issue is documented in
	#   debian/patches/patch_test_results.patch
	# and needs to be discussed with upstream
	dh_auto_test || true

The rationale was that just by switching the debhelper level the build
time test suite was run at all.  Most probably it was failing all the
time before and simply nobody realised this.  To sort this out we need
to talk to upstream.  The issue is documented in bug #806214 (bug in
CC).

I now bumped the upstream source in Git to the latest upstream release
candidate.  Since this had not changed quite some time I assume upstream
is not very rapidly pushing a final release.  However, this might be the
right point in time to sort things out.

If you check the build log of 5.2-11 at

   https://buildd.debian.org/status/fetch.php?pkg=tree-puzzle&arch=all&ver=5.2-11&stamp=1539685923&raw=0

you can find

...
dh_auto_test || true
 make -j1 check VERBOSE=1
make[2]: Entering directory '/<<PKGBUILDDIR>>'
Making check in src
make[3]: Entering directory '/<<PKGBUILDDIR>>/src'
make[3]: Leaving directory '/<<PKGBUILDDIR>>/src'
Making check in doc
make[3]: Entering directory '/<<PKGBUILDDIR>>/doc'
make[4]: Entering directory '/<<PKGBUILDDIR>>/doc'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory '/<<PKGBUILDDIR>>/doc'
make[3]: Leaving directory '/<<PKGBUILDDIR>>/doc'
Making check in data
make[3]: Entering directory '/<<PKGBUILDDIR>>/data'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory '/<<PKGBUILDDIR>>/data'
Making check in tests
make[3]: Entering directory '/<<PKGBUILDDIR>>/tests'
make  check-TESTS
make[4]: Entering directory '/<<PKGBUILDDIR>>/tests'
make[5]: Entering directory '/<<PKGBUILDDIR>>/tests'
SKIP: build-puzzle
FAIL: qp-pure-bin.test
FAIL: qp-pure-nucl.test
FAIL: qp-tn-nucl.test
FAIL: qp-hky-clock-nucl.test
FAIL: qp-hky-rhet-nucl.test
FAIL: qp-hky-rhet-clock-nucl.test
FAIL: qp-pure-prot.test
FAIL: qp-mtrev-prot.test
FAIL: qp-vt-prot.test
FAIL: qp-wag-prot.test
FAIL: qp-clock.test
FAIL: qp-jtt-prot.test
FAIL: qp-jtt-rhet-prot.test
FAIL: qp-jtt-rhet-clock-prot.test
FAIL: lm-pure-prot.test
FAIL: ut-pure-prot.test
PASS: cons-pure-prot.test
SKIP: build-remark
============================
   : tests/test-suite.log
============================

# TOTAL: 19
# PASS:  1
# SKIP:  2
# XFAIL: 0
# FAIL:  16
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2
...


This situation has not changed much in 5.3~rc16 and we now have time for
one release cycle to investigate this issue.  I think the first step
should be to find out the reason for at least one test suite failure.  I
have some gut feeling that the test files for comparison do not really
fit the proper result.  However, I'd like to leave this issue for
somebody who is more familiar with thy phylogeny stuff.

If we have some better understanding about these failures we should
approach upstream (in the best case with a patch).  Once we will have
managed the build time issue (hopefully) we should be able to write
autopkgtests according to the build time tests.

Any comments, hints, suggestions?

Kind regards

       Andreas.

-- 
http://fam-tille.de