Bug#906222: gdal: undeterministically broken python packages

[Niko Tyni]

Source: gdal
Version: 2.3.1+dfsg-2
Severity: serious
User: reproducible-builds at lists.alioth.debian.org
Usertags: randomness
X-Debbugs-Cc: reproducible-bugs at lists.alioth.debian.org

Hi,

it looks like the build system of the gdal Python bindings will
undeterministically produce truncated or broken files.

I noticed this while debugging a python-stetl test rebuild failure
on Perl 5.28. The underlying reason was that our local rebuild of
gdal for Perl 5.28 was broken like this:

  # python -c 'import gdal'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python2.7/dist-packages/gdal.py", line 2, in <module>
      from osgeo.gdal import deprecation_warn
    File "/usr/lib/python2.7/dist-packages/osgeo/__init__.py", line 21, in <module>
      _gdal = swig_import_helper()
    File "/usr/lib/python2.7/dist-packages/osgeo/__init__.py", line 17, in swig_import_helper
      _mod = imp.load_module('_gdal', fp, pathname, description)
  ImportError: dynamic module does not define init function (init_gdal)

Looking into it further, I rebuilt gdal ten times on current sid/amd64
and got 

- seven almost empty Python 2.7 _gdal.x86_64-linux-gnu.so files
- five empty Python 2.7 ogr.py files
- two empty Python 2.7 gdal_array.py files
- three empty Python 2.7 gdal_array.py files

- four empty Python 3 ogr.py files
- three empty Python 3 gdal_array.py files (in different builds)
- four empty Python 3 gdal.py files

--rw-r--r-- root/root    525360 2018-07-31 13:37 ./usr/lib/python2.7/dist-packages/osgeo/_gdal.x86_64-linux-gnu.so
+-rw-r--r-- root/root     13952 2018-07-31 13:37 ./usr/lib/python2.7/dist-packages/osgeo/_gdal.x86_64-linux-gnu.so

and so forth.

It looks like at least the s390x build in sid is broken this way, with

-rw-r--r-- root/root         0 2018-07-31 12:37 ./usr/lib/python2.7/dist-packages/osgeo/gdal_array.py

The test history at

 https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/gdal.html

seems rather sad too.

I haven't located the cause yet; diffing build logs makes me wonder if
the 'generate' target of swig/python/GNUmakefile is getting erroneously
run in parallel with targets compiling the files that 'generate' modifies.
So maybe it's just a case of fragility on parallel builds?
-- 
Niko Tyni   ntyni at debian.org