Bug#882667: hypre: CUDA support: hacks impeded by -W flags

[Drew Parsons]

Source: hypre
Version: 2.13.0-1exp1
Severity: normal

With some hacking, we can start attempting to build hypre with CUDA
support (private build, or perhaps a separate hypre package in contrib).

Brief notes:

- The version of nvcc (nvidia-cuda-toolkit) in debian non-free is 8.0.61-3
nvcc does not yet support gcc 7, and nvcc 8 does not support gcc 6
To also use mpiCC (so mpi.h is known, etc), need OMPI_CXX
So need 
  CC=nvcc CUDACXX=mpiCC dh_auto_configure -- $(extra_flags) --enable-shared

- nvcc does not recognise dpkg-buildflags flags (-W flags in
particular), so need to set an empty LDFLAGS and friends

- must hack src/config/configure.in where nvcc flags are defined,
replace -fPIC with --compiler-options '-fPIC'

- build with g++-5  fails at src/utilities/gpuMem.c:
$nvcc -O --compiler-options '-fPIC' -DUSE_NVTX -DHYPRE_USE_GPU -DHYPRE_USE_MANAGED  -ccbin=mpiCC -expt-extended-lambda -arch compute_35 --std=c++11 -Xcompiler -fopenmp -Xcompiler   -DHYPRE_USE_CUDA  -DUSE_NVTX -DHYPRE_USE_GPU  -DHYPRE_USE_MANAGED -I /usr/local/cuda/include  -DHAVE_CONFIG_H -I.. -I./.. -I.    -c gpuMem.c
  gpuMem.c: In function ‘getstreamOlde’:
  gpuMem.c:335:23: error: storage size of ‘s’ isn’t constant
     static cudaStream_t s[MAXSTREAMS];
                         ^
  gpuMem.c: In function ‘getdomain’:
  gpuMem.c:351:31: error: storage size of ‘h’ isn’t constant
       static nvtxDomainHandle_t h[MAXDOMAINS];
                                 ^
  gpuMem.c: In function ‘getevent’:
  gpuMem.c:364:22: error: storage size of ‘s’ isn’t constant
     static cudaEvent_t s[MAXEVENTS];


This gpuMem.c does compile cleanly under OMPI_CXX=clang++-3.8
Pass through using export OMPI_CXX=clang++-3.8 in debian/rules.

Finally then, using 
  export OMPI_CXX=clang++-3.8
  CC=nvcc CUDACXX=mpiCC dh_auto_configure -- $(extra_flags) --with-cuda --enable-unified-memory --with-nvcc --enable-shared

the libHYPRE_utilities.so build proceeds but fails at link time
providing the soname, 
  nvcc -shared -o libHYPRE_utilities-2.13.0.so amg_linklist.o binsearch.o exchange_data.o fortran_matrix.o hypre_ap.o hypre_complex.o hypre_memory.o hypre_printf.o hypre_qsort.o memory_dmalloc.o mpistubs.o qsplit.o random.o threading.o timer.o timing.o umalloc_local.o hypre_mpi_comm_f2c.o hypre_error.o F90_HYPRE_error.o hypre_prefix_sum.o hypre_merge_sort.o hypre_hopscotch_hash.o gpuErrorCheck.o gpuMem.o -lmpi -lm -Wl,-soname,libHYPRE_utilities-2.13.0.so -Wl,-z,defs
  nvcc fatal   : Unknown option 'Wl,-soname,libHYPRE_utilities-2.13.0.so'

Again, the problem is that -W is not recognised by nvcc.  Would have
to hack the soname patch to encapsulate it inside --compiler-options

Is it worth it?




-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)