Plea to fix the [ parallel | serial ] HDF5 problem

[Alastair McKinstry]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

This is a plea to revisit the parallel vs serial hdf5 / netcdf issue for
jessie.
Firstly, references to previous work:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657949
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=591346

In summary: HDF5 has two modes, parallel and serial. The design bug (*)
is that they
share the same API/ABI but are incompatible. In particular,
    * The parallel version can do parallel read /writes via MPI
but
    * The parallel version cannot use compression plugins, variable
length records, etc.
    * Parallel version has no C++ support, is not threadsafe.

So they conflict. This then means that packages like netcdf4, which can
be either serial or parallel,
cannot be built against both: it must be one or the other. Currently
netcdf in Debian is serial, and
cannot do parallel writes. This cannot be fixed (e.g. by providing both
serial and parallel netcdf
packages) because libhdf5-dev (serial) conflicts with libhdf5-mpi-dev.
(parallel).

This goes on, with packages choosing one version or the other, until you
end up with a package
that can't be built in Debian (e.g visit, which i'm working on), because
it depends on both
conflicting stacks.

The best solution I can think of is:
(1) make the packages install into
    /usr/lib/$MPIV/...
with $MPIV equalling either mpiversion (openmpi, mpich2, ...) or
'serial', or 'any'.
(2) Use alternatives to set a default.
(3) Mangle the libnames as follows:
        /usr/lib/openmpi/lib/libhdf5.so -->
/usr/lib/$(DEB_HOST_MULTIARCH)/libhdf5_openmpi.so.7.1.1
ie. adding _$(MPIV) to the libname.   
This results in a library that uses it being hard-coded to use that
particular version, but
(4) the version in /usr/lib/any/ does _not_ mangle the libname, but
points to a version with
libhdf5.so.7.1.1 with SONAME 'libhdf5.so.7', which is just a shell for
linking against, but which itself
is linked against a version (any, default serial) of the HDF5 libs.
Hence any application which does not
care of the given features (threadsafe, compression, parallelism) can
use this version.
Binaries built with it will resolve /usr/lib/*/libhdf5.so.7.1.1 pointing
to the default version on the system.

This will enable the building of a libnetcdf-$mpi version, for parallel
writes, and other libraries.

Any comments?

best regards
Alastair

- -- 
Alastair McKinstry, <alastair at sceal.ie>, <mckinstry at debian.org>,
https://diaspora.sceal.ie/u/amckinstry
A decent provision for the poor is the true test of civilization.
~Samuel Johnson, Boswell: Life of Johnson    
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTfKzHAAoJEN9LdrZRJ3QsPtMQAJXPDHUbyBkgxhd6tDq5NzM6
w18DyCNpzWz9nnNXmkENIUxN6kRER23f3NmeHvuSS+bJqMbi3rUwNWxKEc5IsgqB
V5Up/kBtVzD1+ThMNAKhzCfSLSzyvNwRr07hMvdwbEbMG9ed/B3+nbkzJuhCpLPN
TXZ+VaIePqtn2X8gqlR2ZHUTV+BH4j5A+I7N3Bcek1gky7fpmsKU3ZGk1o+DF+4o
LeyQMOEv0L2ZFm1fzcSJGZPNUEo2m5doGAvlD1lI5a7HZB6cT10jd7RcBlVqqmsI
+dd/yXtQdnrpUjABf7mjUcdb9e1xNiJseeNiiBBgiToDKSYvm3SysQlM6oWRm7cp
hGFmhsMFmu22wDLFpFMTPZPHUO9QzsT/iPY/CgfslkMb86+FIbrXKjhnm0IS4msc
Xn0Q5V70pWvQClxHCsrvZpg5CLjhdkQhIiUU2/m762M9hgvrUEv1bLRCWoW8dub9
p+pyU/d5hfVsEfBjV+v9JgX6+PjkrFjp7bPNhGfFH5/G9m3I7ZzMt6vtW1zxcnrm
AGltobIerdjM19N9UQN4MIXSklS7wBP+GT7t6ByQacjvpEfwWpQ+GLd/VN+bJFzd
AVerj9PwSxjk5Rdl2fubZCpI9ZnBZPBEzLjTJciRIREU3BImu03YV9bVlyotUNcM
KFp+n7VoZe6jmwAcBVO1
=K/xZ
-----END PGP SIGNATURE-----