Bug#946986: python3-h5py: h5py does not play nicely with subprocess and mpirun

[Rafael Laboissière]

* Drew Parsons <dparsons at debian.org> [2020-01-23 19:02]:

> On 2020-01-23 18:15, Thibaut Paumard wrote:
>> Hi,
>>
>> Le 22/01/2020 à 16:49, Drew Parsons a écrit :
>>> Might be worth bringing to the attention of the subprocess
>>> developers. I
>>> figure they'll either be interested to know about the problem or will
>>> have a simple explanation.
>>
>> I'm not sure subprocess is involved at all, it may be that MPI_Init() 
>> simply dies because h5py has already called it earlier.
>>
>> Hi have no time for testing right now, but a simple printf() before 
>> calling MPI_Init() should tell.
>
> That sounds plausible.

I am not sure this would be an explanation for the problem.

At any rate, I found a minimal example that exposes the bug.  Consider 
these two scripts:

 #############################################################
 ### File name: bugtest-without-h5py.py
 import subprocess
 simulationProc = subprocess.Popen("mpirun",
                                   shell=True,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE)
 (stdout, stderr) = simulationProc.communicate()
 returnCode = simulationProc.wait()
 print("stdout = ", stdout)
 print("stderr = ", stderr)
 print("return code = ", returnCode)
 #############################################################

 #############################################################
 ### File name: bugtest-with-h5py.py
 import subprocess
 import h5py
 simulationProc = subprocess.Popen("mpirun",
                                   shell=True,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE)
 (stdout, stderr) = simulationProc.communicate()
 returnCode = simulationProc.wait()
 print("stdout = ", stdout)
 print("stderr = ", stderr)
 print("return code = ", returnCode)
 #############################################################

The only difference between them is the line containing "import h5py" in 
the second.

Here is the result for the first script:

    $ python3 bugtest-without-h5py.py
    stdout =  b''
    stderr =  b'--------------------------------------------------------------------------\nmpirun could not find anything to do.\n\nIt is possible that you forgot to specify how many processes to run\nvia the "-np" argument.\n--------------------------------------------------------------------------\n'
    return code =  1

and here is the result for the second script:

    $ python3 bugtest-with-h5py.py
    stdout =  b''
    stderr =  b''
    return code =  1

It seems that, when h5py is imported, the mpirun command is not even 
launched by subprocess.Popen.  Notice that there is not even a program 
that is launched by mpirun that would call MPI_Init.

This is quite weird.

Best,

Rafael