[Python-modules-team] Bug#643873: Bug#643873: Making Numpy transition less painful

[Jakub Wilk]

* Sandro Tosi <morph at debian.org>, 2011-10-31, 19:57:
>>Dependencies generated by dh_numpy
>>==================================
>>dh_numpy would generate dependency either on "python-numpy-abi$N" (by 
>>default) or on "python-numpy-api$N" (if a special option, say, 
>>--strict, is used).

I just realized that, in order to make packages using Cython happy, the 
first one will have to be:

"python-numpy-abi$N, python-numpy (>= $V)"

where $V is the first version with the current API. [0]

>>The second option would be needed for packages which are overzealous 
>>at checking ABI unless/until we fix them (we have at least one package 
>>in the archive which does that).

[1]

>do you think it's possible for the same source package to generate 2 
>binaries one needing python-numpy-abi$N and the other needing 
>python-numpy-api$N ?

It'd quite crazy, but in theory, yes, that could happen.

>how would you imagine to handle this? with "dh_numpy -pbin1 
>--strict\ndh_numpy -pbin2"

Exactly.

>>Avoiding strict dependencies for arch:all packages
>>==================================================
>>dh_numpy should not generate any dependencies in arch:all packages. 
>>Such packages cannot rely on Numpy ABI, and rebuilding them for 
>>transition purposes is causes lot of trouble. (We already had to do 
>>this in the past!) For the same reason, 
>>/usr/share/python/dist/python-numpy needs to go away.
>
>so just a python-numpy added to python:Depends?

I was rather thinking about not touching such packages at all. But 
adding unversioned python-numpy to ${python:Depends} would work, too.

>how does it fix with a source that generates an arch:all and an 
>arch:any (the latter using either -abi or -api)?

dh_numpy would generate different substitutions for each of them. I 
don't see any problem here.

>>Gracefully handling the upcoming transition
>>-------------------------------------------
>>A number of packages in the archive already has dependency on 
>>"python-numpy (<< 1:1.6)", which is not satisfiable by the package in 
>>experimental. To avoid this transition (or at least reduce its size) 
>>we could do one of two things:
>>1) Implement the proposed changes first in unstable, schedule binNMUs, 
>>then upload Numpy 1.6 to unstable.
>>2) Implement the proposed changes only for Numpy 1.6, but make 
>>dh_numpy generate alternative versioned dependencies:
>>"python-numpy (>= 1:1.5.1), python-numpy-abi$N | python-numpy (<< 1:1.6.2)".
>>Then upload Numpy 1.6, schedule binNMUs.
>
>Mh, my lazy instinct would say 2) that has also the advantage to bring
>1.6 into unstable (but I guess 1.7 is coming soon).

Yeah, that option sounded very appealing, but turned out not feasible 
after all. See my comment about Cython at the top of this mail.  
C_API_VERSION has been bumped from 4 to 6, so $V should be at least >= 
1:1.6.

>Please also note taht my perl foo is barely None, so I'd need some help 
>in extending dh_numpy :)

No problem, I can do the coding part once we agree how everything should 
work. :)


[0] Cython generates this code for importing C types:

|    if (!strict && ((PyTypeObject *)result)->tp_basicsize > (Py_ssize_t)size) {
|        PyOS_snprintf(warning, sizeof(warning),
|            "%s.%s size changed, may indicate binary incompatibility",
|            module_name, class_name);
|        #if PY_VERSION_HEX < 0x02050000
|        if (PyErr_Warn(NULL, warning) < 0) goto bad;
|        #else
|        if (PyErr_WarnEx(NULL, warning, 0) < 0) goto bad;
|        #endif
|    }
|    else if (((PyTypeObject *)result)->tp_basicsize != (Py_ssize_t)size) {
|        PyErr_Format(PyExc_ValueError,
|            "%s.%s has the wrong size, try recompiling",
|            module_name, class_name);
|        goto bad;
|    }

That is, import will fail if the size imported type is shorter at 
runtime that at build time.


[1] Pyrex (and very old versions of Cython) generates this code for 
importing C types:

|     if (((PyTypeObject *)result)->tp_basicsize != size) {
|         PyErr_Format(PyExc_ValueError,
|             "%s.%s does not appear to be the correct type object",
|             module_name, class_name);
|         goto bad;

So import will fail with exception if the size imported type is 
different at runtime that at build time.

-- 
Jakub Wilk