Bug#944769: python3-h5py fails to import if offline due to apparent MPI failure

[Thibaut Paumard]

Dear Jamie,



Le 15/11/2019 à 09:48, Jameson Graef Rollins a écrit :

> 
> I'm sorry, but I strongly disagree.  This situation is not artificial in
> any way.  This bug was triggered by the default configuration on my
> laptop when my primary network interface was down, a very common
> situation.  Any application using h5py in this situation is rendered
> useless, so it effectively breaks every application that imports h5py.
> The severity should be set appropriately.  This package is not suitable
> for release in it's current state.

For what it's worth, this behavior of OpenMPI can also be controlled by
means of environment variables:

export OMPI_MCA_plm_rsh_agent=/bin/false
export OMPI_MCA_btl_tcp_if_include=lo

I do that in the test suite of my MPI code so it runs without touching
the network.

That said, if the end-user did not explicitly launch an MPI job, they
should not have to set MPI-related environment variables for having
their software run normally when unplugged. So either HDF5 manages to
handle MPI in a totally transparent manner, or it should not use MPI at
all when not running under "mpirun".

So thank you for having caught that, I would also see the issue as RC.

There are ways to detect whether a job is running under MPI (at least as
far as OpenMPI is concerned, I did not find the equivalent for MPICH).

What I do in my own code is checking for the environment variable
OMPI_COMM_WORLD_SIZE. If it exists, I know I'm running under OpenMPI and
I setup the MPI environment accordingly. If it does not, I proceed with
the serial version of my code.

This is elegant because it does not require initializing the MPI
infrastructure before making the check, so mpi4py could be imported only
if this environment variable does exist.

h5py could also have it's own environment variable or similar mechanism
to decide at run-time whether to use MPI.

This may be a way forward to keep the best of both worlds.

[...]

> If there is a need to provide OpenMPI support, then I humbly ask that we
> do so in a separate python3-h5py-openmpi package.

Not sure whether that would make sense. If python3-h5py-openmpi would
need to work as a drop-in replacement for python3-h5py, then the two
would have to conflict with each other and you wouldn't be able to use
both on the same machine.

Kind regards, Thibaut.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-science-maintainers/attachments/20191115/0b0bfddf/attachment.sig>