Bug#944769: python3-h5py fails to import if offline due to apparent MPI failure

[Thibaut Paumard]

Hi Drew,

I see, so you effectively copy all symbols from _h5py except those that
look like __foo__ which is Python internal stuff, *and* you give them an
entry under h5py in sys.modules. Nice, that should be pretty stable.

Actually, you don't treat public_api differently from weak_privates, and
you could be missing some symbols (if introduced later) of the form
_foo_, _foo__ or __foo_. You could simplify the code and cope with this
possibility with something like:

api=[ k for k in _h5py.__dict__.keys() if not (k.startswith('__') and
k.endswith('__')) ]

Like you, I would keep h5py_serial and h5py_mpi separate rather than
submodules of h5py. Mostly because the h5py folks could in the future
want to use those two names and do it in an incompatible manner.

Kind regards, Thibaut.

Le 02/03/2020 à 15:30, Drew Parsons a écrit :
> On 2020-03-02 18:21, Thibaut Paumard wrote:
>> Le 01/03/2020 à 20:26, Drew Parsons a écrit :
>>> your h5py suggestion of [...]
>>> works up to a point.
>>
>> Hi Drew,
>>
>> Do I read correctly in your other mail that you have found a workaround?
> 
> That's right, I've been building up my python-fu.
> 
>> My first take would have been to apply my initial suggestion to each
>> individual sub-module but that depends on the structure of the main h5py
>> module which I didn't check...
> 
> I seem to have managed it with just one formal import much as you
> suggested.  Then extra handling for the weak references and h5py.tests
> (which isn't accessible from h5py, must be imported as h5py.tests)
> 
> 
>> I  can put some brain cells into this if this is still needed, let me
>> know.
> 
> The following seems to be working, but let me know if you spot any
> problems.
> 
> I've assumed h5py_serial and h5py_mpi are installed independently in the
> python path. An alternative would be to install them as subpackages
> under h5py (so activated with something like "import h5py.h5py_serial as
> _h5py" instead of "import h5py_serial as _h5py". Do you think it's
> better to keep the 2 builds as separate installed modules?
> 
> 
> Here's my proposed h5py/__init__.py:
> 
> =============================================
> # avoid running mpi with serial (1 process) jobs
> # by checking whether mpi is actually being used over multiple processes
> # Probe number of processes with OMPI_COMM_WORLD_SIZE (openmpi) and
> MPI_LOCALNRANKS (mpich)
> 
> from os import getenv as os_getenv
> import sys, importlib
> 
> OPENMPI_MULTIPROC = os_getenv('OMPI_COMM_WORLD_SIZE') is not None
> MPICH_MULTIPROC = os_getenv('MPI_LOCALNRANKS') is not None
> 
> MPI_ACTIVE = OPENMPI_MULTIPROC or MPICH_MULTIPROC
> 
> if MPI_ACTIVE:
>     try:
>         import h5py_mpi as _h5py
>     except:
>         import h5py_serial as _h5py
> else:
>     import h5py_serial as _h5py
> 
> # make generic h5py module behaviour the same as specific builds
> # by importing public and weak internal symbols
> 
> public_api = [ k for k in _h5py.__dict__.keys() if not k.startswith('_')
> and not k.endswith('_') ]
> weak_privates = [ k for k in _h5py.__dict__.keys() if k.startswith('_')
> and not k.endswith('_') ]
> 
> this_module=sys.modules[__name__]
> for key in public_api+weak_privates:
>     # "imports" symbols (makes them accessible)
>     setattr(this_module,key,getattr(_h5py,key))
>     # rename symbols as properties of toplevel h5py module
>     sys.modules['h5py.{}'.format(key)] = getattr(_h5py,key)
> 
> del public_api
> del weak_privates
> del this_module
> 
> 
> # tests are not loaded with h5py, but can be imported separately with
> # import h5py.tests
> tests = importlib.import_module('{}.tests'.format(_h5py.__name__))
> sys.modules['h5py.tests'] = tests
> del tests
> =================================================
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-science-maintainers/attachments/20200302/758c2e44/attachment.sig>