[Debian-med-packaging] Bug#1079439: Bug#1079439: python-biopython: autopkgtest stuck on riscv64

[Reinhard Tartler]

On 2024-08-24 11:28, Étienne Mollier wrote:
> Hi Reinhard,
> 
> Reinhard Tartler, on 2024-08-24:
>> Actually, I've been able to compile pystack on the machine
>> (see https://github.com/bloomberg/pystack, and amazing tool)
>> and was able to get a trace of what the stuck process is doing:
> 
> This is a very handy tool and I need to recall about it when
> I'll next need it, probably too soon for my taste.  :)
> 
> […]
>>     (Python) File 
>> "/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Bio/Align/analysis.py",
>> line 484, in _yn00
>>         P = expm(Q * t)
>>     (Python) File
>> "/usr/lib/python3/dist-packages/scipy/linalg/_matfuncs.py", line 358, 
>> in
>> expm
>>         eAw = eAw @ eAw
>> 
>> I think that indicates that the problematic / looping test is indeed
>> Align_condonalign.py?
> 
> This is highly plausible.  It was a bold assumption from me to
> believe the standard output would be flushed to let us see
> whether the test started.

You know what, I tried something crazy. On the porterbox that I used to
reproduce it, I pressed Ctrl-C to interrupt the hanging process. This is 
the output:

======================================
(sid_riscv64-dchroot)siretart at ricci:~/python-biopython$ 
test_Align_codonalign ... Traceback (most recent call last):
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Tests/run_tests.py", 
line 343, in <module>
     errors = main(sys.argv[1:])
              ^^^^^^^^^^^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Tests/run_tests.py", 
line 187, in main
     return runner.run()
            ^^^^^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Tests/run_tests.py", 
line 324, in run
     ok = self.runTest(test)
          ^^^^^^^^^^^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Tests/run_tests.py", 
line 281, in runTest
     suite.run(result)
   File "/usr/lib/python3.12/unittest/suite.py", line 122, in run
     test(result)
   File "/usr/lib/python3.12/unittest/suite.py", line 84, in __call__
     return self.run(*args, **kwds)
            ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.12/unittest/suite.py", line 122, in run
     test(result)
   File "/usr/lib/python3.12/unittest/case.py", line 690, in __call__
     return self.run(*args, **kwds)
            ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.12/unittest/case.py", line 634, in run
     self._callTestMethod(testMethod)
   File "/usr/lib/python3.12/unittest/case.py", line 589, in 
_callTestMethod
     if method() is not None:
        ^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Tests/test_Align_codonalign.py", 
line 8242, in test_dn_ds
     dN, dS = calculate_dn_ds(pairwise_alignment, method="YN00")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Bio/Align/analysis.py", 
line 88, in calculate_dn_ds
     return _yn00(codons1, codons2, codon_table)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/home/siretart/python-biopython/.pybuild/cpython3_3.12/build/Bio/Align/analysis.py", 
line 484, in _yn00
     P = expm(Q * t)
         ^^^^^^^^^^^
   File "/usr/lib/python3/dist-packages/scipy/linalg/_matfuncs.py", line 
358, in expm
     eAw = eAw @ eAw
     ^^^
KeyboardInterrupt
================================

Note the 'test_Align_codonalign ... ' part at the beginning. That means 
that python's
unittest library indeed wrote that to stdout before running the test, 
but because
of the missing '\n' (newline), it didn't get printed because the buffer 
didn't get
flushed yet. That's probably not something that could be fixed in the 
biopython
package, and would probably affect other python software equally as 
well.

> The stack trace is even pointing to
> something off in scipy/linalg/_matfunc.py, even more so as scipy
> is only suggested by python3-biopython, so it is not available
> in the lightweight variant of the autopkgtest, which does run
> the test_Align_codonalign.py but doesn't hangup.  And indeed,
> here is what I see in test_Align_codonalign.py, starting line
> 8235:
> 
>         try:
>             import scipy
>         except ImportError:
>             # Silently skip the rest of the test
>             return
> 
> My current conclusion is: it seems to me that Biopython's test
> suite revealed something off in SciPy specifically on Risc-V.

I tend to agree. Looking at a previous testrun where this test passes,
https://ci.debian.net/packages/p/python-biopython/testing/riscv64/50850003/
we see this output:

744s test_Align_clustal ... ok
778s test_Align_codonalign ... ok
779s test_Align_emboss ... ok
781s test_Align_exonerate ... ok

Look at the timestamps. There is no way that test_Align_codonalign 
actually
passes in less than 2s, in particular not on riscv64. How did it pass so 
fast?
I believe the code that you showed above could have something to do with 
this.

Maybe we can disable test_Align_codonalign, at least on riscv64, and see 
if
that improves things? -- I'm also skeptical about the "silent skip" 
above.
Maybe that code above could be changed to print on stderr to provide 
better
diagnostics?

best,
-rt