[Debian-med-packaging] Bug#775621: python-biopython: FTBFS in jessie: Tests failures

[Andreas Tille]

[Alexandros, please read below in how far raxml affects BioPython test
 suite and a proposal what to do at the very end]

Hi Lucas,

I just finished my tests and noticed the following.  After setting
up a fresh Jessie chroot I was able to reproduce the problem.  However,
some debugging showed that I forgot to

    sudo mount proc ${CHROTDIR}/proc -t proc

after this the package was building nicely.  Since I guess you have
setup your chroot properly I do not think that this would be a problem
on your side but I'm mentioning it anyway.

See more comments below.

On Sun, Jan 18, 2015 at 03:51:43PM +0100, Lucas Nussbaum wrote:
> On 18/01/15 at 23:50 +1100, Stuart Prescott wrote:
> > Control: tag -1 unreproducible
> > 
> > The actual failure is:
> > 
> > test_raxml_tool ... FAIL
> > 
> > […]
> > 
> > ======================================================================
> > ERROR: test_raxml (test_raxml_tool.AppTests)
> > Run RAxML using the wrapper.
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >   File "/«BUILDDIR»/python-
> > biopython-1.64+dfsg/.pybuild/pythonX.Y_2.7/build/Tests/test_raxml_tool.py", 
> > line 45, in test_raxml
> >     out, err = cmd()
> >   File "/«BUILDDIR»/python-
> > biopython-1.64+dfsg/.pybuild/pythonX.Y_2.7/build/Bio/Application/__init__.py", 
> > line 513, in __call__
> >     stdout_str, stderr_str)
> > ApplicationError: Non-zero return code 255 from 'raxmlHPC -m PROTCATWAG -n 
> > test -p 10000 -s Phylip/interlaced2.phy'
> > 
> > ----------------------------------------------------------------------
> > 
> > I cannot reproduce this failure in a jessie chroot. I'm unable to cause 
> > raxmlHPC to exit(-1) here when I try (although it certainly has plenty of 
> > places where it can do that in its code).
> 
> Trying to run raxml manually, I get:
> 
> (jessie-amd64-sbuild)user at ip-172-31-5-2:/tmp/python-biopython-1.64+dfsg/Tests$ raxmlHPC -m PROTCATWAG -n test -p 10000 -s Phylip/interlaced2.phy     
> Use raxml with SSE3 support (1 cpus)
> 
> The number of threads is currently set to 1
> Specify the number of threads to run via -T numberOfThreads
> NumberOfThreads must be set to an integer value greater than 1
> 
> (jessie-amd64-sbuild)user at ip-172-31-5-2:/tmp/python-biopython-1.64+dfsg/Tests$ echo $?
> 255
> 
> 
> However on my laptop, I get:
> 
> *** lucas at grr:/tmp/python-biopython-1.64+dfsg/Tests$ raxmlHPC -m PROTCATWAG -n test -p 10000 -s Phylip/interlaced2.phy
> Use raxml with AVX support (2 cpus)
> 
> This is the RAxML Master Pthread
> 
> This is RAxML Worker Pthread Number: 1
> 
> 
> So it seems that raxml tries to guess the fastest possible implementation based
> on CPU capabilities. /proc/cpuinfo on EC2 does not include AVX, so it fallbacks
> to a SSE3 implementation, that requires specifying the number of threads.
> (it works if I add -T 2)

You might like to have a look into /usr/bin/raxmlHPC[1] which tries to
detect the hardware in a quite primitive manner.  It was suggested and
tested in practice by a user.  The problematic thing is that raxmlHPC
either needs -T option for the "more powerful" architectures or does not
allow this option, which makes different executables not fully
compatible.  It might be that the wrapper script is rarely tested and
does not work on all architectures properly.  The only safe way to deal
with this would be to deactivate the test in the build process.
 
> Ideally, there would be a sane default for each RAxML implementation.

That's true and I will talk to raxml upstream about this - however,
that's to late for Jessie.

> It can easily be argued that this is not RC, given that it should be possible
> to build the package on a machine where SSE3 is not the default raxml
> implementation.

Yes, that's what I'd suggest.  Alternatively the fix would be to drop
the test,
 
> Another option could be to add '-T 2' to the raxml command-line, but I haven't
> checked what happens if -T is specified on an implementation that does not
> support threads.  Or just switch to using raxmlHPC-PTHREADS, but then it
> defeats the purpose of the test...

As I said. above:  If you force '-T 2' (at least) the last fallback
option will break.  Raxml upstream should rather make sure it will be
accepted in all executables and print a warning if it is simply ignored.

Lucas, as the reporter of this bug, would your agree that it is not RC?

Kind regards

       Andreas.

[1] http://anonscm.debian.org/viewvc/debian-med/trunk/packages/raxml/trunk/debian/bin/raxmlHPC?view=markup

-- 
http://fam-tille.de