Bug#933968: ghc: FTBFS on hppa - virtual memory exhausted: Cannot allocate memory

[Laurence Parry]

In fact this seems to be happening on several platforms:
https://buildd.debian.org/status/package.php?p=ghc&suite=sid

I was looking particularly at the x32 build, which ran for over three
hours before falling over with a lack of virtual memory. It has never
successfully built 8.6.x, only 8.4.x:
https://buildd.debian.org/status/logs.php?pkg=ghc&arch=x32
Many of these platforms, including x32, are limited to 32-bit pointers.

I noticed that the file compiler/cmm/CmmOpt.hs was implicated as the
last compile:
https://gitlab.haskell.org/nh2/ghc/blob/master/compiler/cmm/CmmOpt.hs
(see build log at end)

This was one of several files which were permitted to exceed a default
GHC option, max-pmcheck-iterations, known to cause VM spikes:
https://gitlab.haskell.org/ghc/ghc/commit/d2e05c6bd0834421b0c48b3c4287fbe6ee889966

-- The default iteration limit is a bit too low for the definitions
-- in this module.
{-# OPTIONS_GHC -fmax-pmcheck-iterations=10000000 #-}

make also appears to be doing two jobs at once and this might not help.

The platforms impacted seem to mostly be "unregistereized":
https://gitlab.haskell.org/ghc/ghc/blob/master/configure.ac#L251
However, I don't know if that's a factor. It may simply be that such
platforms are less common and either more older or limited, or not
fixed as fast.

Possibly related: mips64el is dying instead due to a "relocation overflow":
[...]
-c ghc/./GHCi/UI/Monad.hs -o ghc/stage2/build/GHCi/UI/Monad.dyn_o
compiler/stage2/build/HsInstances.dyn_o:ghc_5.hc:function
ghc_HsInstances_zdfDataHsExpr6_entry: error: relocation overflow: 19
against 'stg_ap_0_fast'

I think this relates to having too much static code and data (2GB?)
for relative references to it to be used. This might be avoidable via
a compile option to change the memory model (but this might have other
impacts).

End of x32 build log:
[...]
-c compiler/codeGen/CgUtils.hs -o compiler/stage2/build/CgUtils.p_o
-dyno compiler/stage2/build/CgUtils.dyn_o
<<ghc: 525026844 bytes, 67 GCs, 16754416/43344944 avg/max bytes
residency (6 samples), 89M in use, 0.000 INIT (0.000 elapsed), 1.693
MUT (2.847 elapsed), 0.590 GC (0.611 elapsed) :ghc>>
"inplace/bin/ghc-stage1" -hisuf p_hi -osuf  p_o -hcsuf p_hc
-static -prof -eventlog  -H32m -O -lffi -optl-pthread -Wall
-Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header
-Iincludes/dist-ghcconstants/header   -this-unit-id ghc-8.6.5
-hide-all-packages -i -icompiler/backpack -icompiler/basicTypes
-icompiler/cmm -icompiler/codeGen -icompiler/coreSyn
-icompiler/deSugar -icompiler/ghci -icompiler/hsSyn -icompiler/iface
-icompiler/llvmGen -icompiler/main -icompiler/nativeGen
-icompiler/parser -icompiler/prelude -icompiler/profiling
-icompiler/rename -icompiler/simplCore -icompiler/simplStg
-icompiler/specialise -icompiler/stgSyn -icompiler/stranal
-icompiler/typecheck -icompiler/types -icompiler/utils
-icompiler/stage2/build -Icompiler/stage2/build
-icompiler/stage2/build/./autogen -Icompiler/stage2/build/./autogen
-Icompiler/. -Icompiler/parser -Icompiler/utils
-Icompiler/../rts/dist/build -Icompiler/stage2
-Icompiler/stage2/build/. -Icompiler/stage2/build/parser
-Icompiler/stage2/build/utils
-Icompiler/stage2/build/../rts/dist/build
-Icompiler/stage2/build/stage2   -optP-DGHCI -optP-include
-optPcompiler/stage2/build/./autogen/cabal_macros.h -package-id
array-0.5.3.0 -package-id base-4.12.0.0 -package-id binary-0.8.6.0
-package-id bytestring-0.10.8.2 -package-id containers-0.6.0.1
-package-id deepseq-1.4.4.0 -package-id directory-1.3.3.0 -package-id
filepath-1.4.2.1 -package-id ghc-boot-8.6.5 -package-id
ghc-boot-th-8.6.5 -package-id ghc-heap-8.6.5 -package-id ghci-8.6.5
-package-id hpc-0.6.0.3 -package-id process-1.6.5.0 -package-id
template-haskell-2.14.0.0 -package-id terminfo-0.4.1.2 -package-id
time-1.8.0.2 -package-id transformers-0.5.6.2 -package-id unix-2.7.2.2
-Wall -Wno-name-shadowing -Wnoncanonical-monad-instances
-Wnoncanonical-monadfail-instances -Wnoncanonical-monoid-instances
-this-unit-id ghc -XHaskell2010 -XNoImplicitPrelude -DNO_REGS -DNOSMP
-optc-DNOSMP -optc-DTHREADED_RTS -DSTAGE=2 -Rghc-timing  -Wcpp-undef
-no-user-package-db -rtsopts       -Wnoncanonical-monad-instances
-odir compiler/stage2/build -hidir compiler/stage2/build -stubdir
compiler/stage2/build    -c compiler/cmm/CmmOpt.hs -o
compiler/stage2/build/CmmOpt.p_o -dyno
compiler/stage2/build/CmmOpt.dyn_o
virtual memory exhausted: Operation not permitted
`x86_64-linux-gnux32-gcc' failed in phase `C Compiler'. (Exit code: 1)
<<ghc: 549770132 bytes, 1807 GCs, 154507149/358335820 avg/max bytes
residency (34 samples), 961M in use, 0.000 INIT (0.000 elapsed),
193.841 MUT (651.304 elapsed), 46.702 GC (46.946 elapsed) :ghc>>
make[3]: *** [compiler/ghc.mk:447:
compiler/stage2/build/HsInstances.p_o] Error 1
make[3]: *** Waiting for unfinished jobs....
<<ghc: 1220239748 bytes, 110 GCs, 40893852/133805660 avg/max bytes
residency (8 samples), 294M in use, 0.000 INIT (0.000 elapsed), 3.586
MUT (7.240 elapsed), 2.007 GC (2.019 elapsed) :ghc>>
make[2]: *** [Makefile:127: all] Error 2
make[2]: Leaving directory '/<<BUILDDIR>>/ghc-8.6.5+dfsg1'

Best regards,
-- 
Laurence "GreenReaper" Parry
https://www.greenreaper.co.uk/