[Python-modules-team] Bug#919929: python-scipy: autopkgtest fails (Re: bug#919929)

[Drew Parsons]

On 2019-03-10 03:51, Paul Gevers wrote:
> Hi Drew,
> 
> On 08-03-2019 03:08, Drew Parsons wrote:
>> On 2019-03-07 20:46, Paul Gevers wrote:
>>> If you upload now, your package will not migrate to testing before 
>>> the
>>> full freeze becomes effective so it would need an unblock. If you 
>>> want
>>> to fix this issue with the three lines I saw in the bug report, you 
>>> can
>>> go ahead. However, it is probably worth waiting for a resolution of 
>>> bug
>>> 915738 and combine it with that.
>>> 
>> 
>> Alas, the deprecation patch (in python-scipy 1.1.0-3) doesn't actually
>> prevent emission of the deprecation warnings, so they're still 
>> spamming
>> the debci log.
> 
> Do you have evidence they did anything at all? If not, please revert
> this if we get to a new upload.

It would seem it did not help.  In any case, the upstream patches 
supercede this patch, so it will be removed naturally.

>> To remove the deprecation warnings we'd need to deal with them at the
>> source. Upstream has patches
>> https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46
>> 
>> https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514
>> 
>> and for completeness
>> https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6fa
>> 
>> 
>> The deprecation problem (matrix API) appears in many places, but the 
>> fix
>> is straightfoward: replace np.matrix with matrix from from
>> scipy.sparse.sputils
>> 
>> Can you authorise an unblock to apply these 3 upstream patches to
>> python-scipy 1.1.0 ?
>> That won't necessarily fix the debci failure, but it will make it a 
>> lot
>> easier to see what's actually failing.
> 
> I am slightly unhappy with the second patch, as it seems to be doing
> more than you describe above in a few places. This may be correct but
> that is difficult to quickly judge.

The patches as they are don't apply cleanly to the 1.1.0 source, so I'll 
need to adapt them anyway.  I can retain only the ones relevant to 
updating the matrix API.


> Also, what is the general documented way that these wrappers can be
> used? scipy is sort of taking over the maintenanceship of these
> functions in this way if we're not careful.


It's a good question that the other scipy maintainers might have thought 
more about.  As far as I can tell, the scipy tests affected here involve 
sparse matrices.  The trouble arises from an "inadequacy" in the core 
numpy API, with numpy.matrix only being suitable for dense matrices.  
scipy could be described as "numpy+algorithms", with additional 
algorithms required to handle sparse matrices, provided in 
scipy.sparse.sputils.matrix.

numpy.matrix is documented at 
https://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html

The scipy sparse matrix API is at 
https://docs.scipy.org/doc/scipy/reference/sparse.html, but that's 
specifically for scipy.sparse.spmatrix

There is discussion of the distinction between numpy.matrix and 
numpy.ndarray (which is at the heart of the deeprecation warning) at 
https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#numpy-matrix-vs-2d-numpy-ndarray

The utility class scipy.sparse.sputils itself is apparently 
undocumented, by which I infer it's intended for internal use only, not 
a public API. I guess it's reasonable for a package to be testing it's 
own internal functions.  Strange thing is, scipy.sparse.sputils.matrix 
is not actually defined in scipy/sparse/sputils.py. Must be inherited or 
defined in some deep pythonfu that I haven't mastered yet.

I'll check that I can adapt those upstream patches to cleanly remove 
these deprecation warnings.

Drew