Bug#953116: [petsc-maint] 32 bit vs 64 bit PETSc

[Drew Parsons]

On 2020-05-23 14:18, Jed Brown wrote:
> Drew Parsons <dparsons at debian.org> writes:
> 
>> Hi, the Debian project is discussing whether we should start providing 
>> a
>> 64 bit build of PETSc (which means we'd have to upgrade our entire
>> computational library stack, starting from BLAS and going through MPI,
>> MUMPS, etc).
> 
> You don't need to change BLAS or MPI.

I see, the PETSc API allows for PetscBLASInt and PetscMPIInt distinct 
from PetscInt. That gives us more flexibility. (In any case, the Debian 
BLAS maintainer is already providing blas64 packages. We've started 
discussions about MPI).

But what about MUMPS? Would MUMPS need to be built with 64 bit support 
to work with 64-bit PETSc?
(the MUMPS docs indicate that its 64 bit support needs 64-bit versions 
of BLAS, SCOTCH, METIS and MPI).



>> A default PETSc build uses 32 bit addressing to index vectors and
>> matrices.  64 bit addressing can be switched on by configuring with
>> --with-64-bit-indices=1, allowing much larger systems to be handled.
>> 
>> My question for petsc-maint is, is there a reason why 64 bit indexing 
>> is
>> not already activated by default on 64-bit systems?  Certainly C
>> pointers and type int would already be 64 bit on these systems.
> 
> Umm, x86-64 Linux is LP64, so int is 32-bit.  ILP64 is relatively 
> exotic
> these days.


oh ok. I had assumed int was 64 bit on x86-64. Thanks for the 
correction.


>> Is it a question of performance?  Is 32 bit indexing executed faster 
>> (in
>> the sense of 2 operations per clock cycle), such that 64-bit 
>> addressing
>> is accompanied with a drop in performance?
> 
> Sparse iterative solvers are entirely limited by memory bandwidth;
> sizeof(double) + sizeof(int64_t) = 16 incurs a performance hit relative
> to 12 for int32_t.  It has nothing to do with clock cycles for
> instructions, just memory bandwidth (and usage, but that is less often
> an issue).
> 
>> In that case we'd only want to use 64-bit PETSc if the system being
>> modelled is large enough to actually need it. Or is there a different
>> reason that 64 bit indexing is not switched on by default?
> 
> It's just about performance, as above.


Thanks Jed.  That's good justification for us to keep our current 32-bit 
built then, and provide a separate 64-bit build alongside it.


>  There are two situations in
> which 64-bit is needed.  Historically (supercomputing with thinner
> nodes), it has been that you're solving problems with more than 2B 
> dofs.
> In today's age of fat nodes, it also happens that a matrix on a single
> MPI rank has more than 2B nonzeros.  This is especially common when
> using direct solvers.  We'd like to address the latter case by only
> promoting the row offsets (thereby avoiding the memory hit of promoting
> column indices):
> 
> https://gitlab.com/petsc/petsc/-/issues/333

An interesting extra challenge.


> I wonder if you are aware of any static analysis tools that can
> flag implicit conversions of this sort:
> 
> int64_t n = ...;
> for (int32_t i=0; i<n; i++) {
>   ...
> }
> 
> There is -fsanitize=signed-integer-overflow (which generates a runtime
> error message), but that requires data to cause overflow at every
> possible location.

I'll ask the Debian gcc team and the Science team if they have ideas 
about this.

Drew