Bug#1082816: petsc: enable AMD GPU support

[Drew Parsons]

Source: petsc
Followup-For: Bug #1082816

If I'm reading the instructions correctly,
there are (at least) two pathways to GPU support in PETSc

1) direct, via KOKKOS
2) indirect, via hypre

I'll use fenicsx-performance-tests to help measure performance after
activating GPU support.

fenicsx (dolfinx) already suggests using hypre (via petsc) to get
large performance improvements. That's in MPI, before we get to GPU
usage. So I would expect strongest gains in the fenicsx benchmark will
be seen activating GPU in hypre.

For that reason I plan to first activate path 1, activating petsc
direct configuration with KOKKOS, on the grounds that that will make
it easier to measure direct (petsc+kokkos) performance gains, before
hypre is activated.
KOKKOS itself will also want GPU activation (Bug#1082813)

Once petsc is running with kokkos, I'll later activate GPU support in hypre
(-DHYPRE_ENABLE_HIP=ON will resolve Bug#882667)

Drew