Bug#944769: python3-h5py fails to import if offline due to apparent MPI failure

[Drew Parsons]

On 2020-03-02 18:21, Thibaut Paumard wrote:
> Le 01/03/2020 à 20:26, Drew Parsons a écrit :
>> your h5py suggestion of [...]
>> works up to a point.
> 
> Hi Drew,
> 
> Do I read correctly in your other mail that you have found a 
> workaround?

That's right, I've been building up my python-fu.

> My first take would have been to apply my initial suggestion to each
> individual sub-module but that depends on the structure of the main 
> h5py
> module which I didn't check...

I seem to have managed it with just one formal import much as you 
suggested.  Then extra handling for the weak references and h5py.tests 
(which isn't accessible from h5py, must be imported as h5py.tests)


> I  can put some brain cells into this if this is still needed, let me 
> know.

The following seems to be working, but let me know if you spot any 
problems.

I've assumed h5py_serial and h5py_mpi are installed independently in the 
python path. An alternative would be to install them as subpackages 
under h5py (so activated with something like "import h5py.h5py_serial as 
_h5py" instead of "import h5py_serial as _h5py". Do you think it's 
better to keep the 2 builds as separate installed modules?


Here's my proposed h5py/__init__.py:

=============================================
# avoid running mpi with serial (1 process) jobs
# by checking whether mpi is actually being used over multiple processes
# Probe number of processes with OMPI_COMM_WORLD_SIZE (openmpi) and 
MPI_LOCALNRANKS (mpich)

from os import getenv as os_getenv
import sys, importlib

OPENMPI_MULTIPROC = os_getenv('OMPI_COMM_WORLD_SIZE') is not None
MPICH_MULTIPROC = os_getenv('MPI_LOCALNRANKS') is not None

MPI_ACTIVE = OPENMPI_MULTIPROC or MPICH_MULTIPROC

if MPI_ACTIVE:
     try:
         import h5py_mpi as _h5py
     except:
         import h5py_serial as _h5py
else:
     import h5py_serial as _h5py

# make generic h5py module behaviour the same as specific builds
# by importing public and weak internal symbols

public_api = [ k for k in _h5py.__dict__.keys() if not k.startswith('_') 
and not k.endswith('_') ]
weak_privates = [ k for k in _h5py.__dict__.keys() if k.startswith('_') 
and not k.endswith('_') ]

this_module=sys.modules[__name__]
for key in public_api+weak_privates:
     # "imports" symbols (makes them accessible)
     setattr(this_module,key,getattr(_h5py,key))
     # rename symbols as properties of toplevel h5py module
     sys.modules['h5py.{}'.format(key)] = getattr(_h5py,key)

del public_api
del weak_privates
del this_module


# tests are not loaded with h5py, but can be imported separately with
# import h5py.tests
tests = importlib.import_module('{}.tests'.format(_h5py.__name__))
sys.modules['h5py.tests'] = tests
del tests
=================================================