Bug#944617: python3-h5py import performance severely degraded in 2.10.0 release (due to OpenMPI?)

[Jameson Graef Rollins]

Package: python3-h5py
Version: 2.10.0-2
Severity: normal

This was mentioned in a different report (#944192) but I want to give
it it's own report because I think it's serious enough.

I'm finding multiple problems with the latest 2.10.0 release in
unstable, which I think can all be traced to this new release being
compiled against the OpenMPI library.  I'm no longer sure it's good to
compile against this library by default, since it's degrading the
basic performance too much.

Other than the stderr output on import described in #944192 (and seen
below), the import performance is now x7 slower.  This is a pretty big
drag for any application using this library:

servo:~ 0$ python3 -c 'import h5py; print(h5py.__version__)'
--------------------------------------------------------------------------
[[4525,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: servo

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
2.10.0
servo:~ 0$ multitime -qq -n 10 python3 -c 'import h5py'
===> multitime results
1: -qq python3 -c "import h5py"
            Mean        Std.Dev.    Min         Median      Max
real        0.973       0.027       0.932       0.967       1.022
user        0.200       0.017       0.159       0.201       0.221
sys         0.068       0.014       0.034       0.069       0.093
servo:~ 0$


servo:~ 0$ python3 -c 'import h5py; print(h5py.__version__)'
2.8.0
servo:~ 0$ multitime -qq -n 10 python3 -c 'import h5py'
===> multitime results
1: -qq python3 -c "import h5py"
            Mean        Std.Dev.    Min         Median      Max
real        0.146       0.012       0.118       0.151       0.156
user        0.119       0.010       0.103       0.121       0.134
sys         0.027       0.008       0.016       0.027       0.044
servo:~ 0$ 

Most users of h5py are not using the OpenMPI functionality, so I don't
see how it's a win to degrade performance for most users in order to
increase performance for the few.

When I mentioned this issue upstream, their recommendation is that
debian should *not* be building against OpenMPI by default:

https://github.com/h5py/h5py/issues/1433#issuecomment-552640391

I agree with this recommendation.  I think there should be separate
packages providing OpenMPI support (e.g. python3-h5py-openmpi).

Thanks.

jamie.


-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (600, 'testing'), (500, 'stable'), (200, 'unstable'), (101, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.2.0-3-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages python3-h5py depends on:
ii  libc6                               2.29-3
ii  libhdf5-openmpi-103                 1.10.4+repack-10
ii  python3                             3.7.5-1
ii  python3-mpi4py                      3.0.2-13
ii  python3-numpy [python3-numpy-abi9]  1:1.16.5-1
ii  python3-six                         1.12.0-2

python3-h5py recommends no packages.

Versions of packages python3-h5py suggests:
pn  python-h5py-doc  <none>

-- no debconf information