Bug#910664: Acknowledgement (ghc: ghc package can no longer be cross-compiled)

Sat Dec 16 14:45:00 GMT 2023

Hi Helmut,

Sorry for the late reply. Comments inline.

On Mon, Dec 11, 2023 at 10:25PM, Helmut Grohne wrote:
> Regarding that --build flag, I now see that we actually run configure
> twice. Connecting what you said earlier, I guess that one of those
> invocations is for the stage1 compiler and the second one is for the
> stage2 compiler though I may be wrong. If that guess is correct, then
> yes, the stage1 compiler definitely needs --host=$(DEB_BUILD_GNU_TYPE),
> but doing it like that still passes host architecture compiler flags and
> you end up running e.g. aarch64-linux-gnu-gcc -fcf-protection or
> x86_64-linux-gnu-gcc -mbranch-protection=standard and fail either way.
> So the way to pass that --host flag is:
> 
>     dpkg-architecture -a$(DEB_BUILD_ARCH) -f -c dh_auto_configure --reload-all-buildenv-variables -- $(CONFIGURE_ARGS)
>     # CONFIGURE_ARGS contains neither --build nor --host, but it contains --target

Let me try to explain how GHC's build system works with the new Hadrian
tool. Then let's see how we can move forward and make it so that the
build fails at the right stage.

1. Run boot and configure. Here is where we decide if we want to
   build GHC for the same architecture or do cross compilation.

2. Run hadrian. The build artifacts will be under '_build/stageX'. With
   the Hadrian build system, the naming convention for these directories
   have changed. '_build/stage1' contains the binaries that were built
   using the stage1 compiler.

   If we are building a compiler for the same architecture,
   '_build/stage1/' will contain the stage2 compiler for this
   architecture (i.e., the binaries that we ship in our Debian package).

   If we are cross compiling, '_build/stage1/' will contain a *stage1
   cross-compiler*. As discussed in this issue [1], upstream is working
   towards making this a *stage2 cross-compiler*. Is is still a
   cross-compiler though, and not a cross-compiled GHC as it used to be.

   The final output of this step is a binary distribution of GHC, i.e.,
   a tarball that you can distribute to anyone that wants to install GHC.

3. We run configure and 'make install' to convert the binary
   distribution to a Debian binary package. This is why we run configure
   twice.

[1] https://gitlab.haskell.org/ghc/ghc/-/issues/19174

I hope the above explains why we cannot cross compile GHC (the build
system doesn't give us this option).

Given the above, I believe your patch needs a few adjustments.

> diff --minimal -Nru ghc-9.4.7/debian/rules ghc-9.4.7/debian/rules
> --- ghc-9.4.7/debian/rules	2023-10-18 21:49:38.000000000 +0200
> +++ ghc-9.4.7/debian/rules	2023-12-10 20:03:52.000000000 +0100
> @@ -47,8 +47,10 @@
>    EXTRA_HADRIAN_FLAGS += --flavour=quickest
>    # Do not build docs *at all* (avoid dependency on Sphinx)
>    EXTRA_HADRIAN_FLAGS += --docs=none
> +  STAGE1_TOOL = _build/stage1/bin/$(DEB_HOST_GNU_TYPE)-
>    BUILD_CROSS = YES
>  else
> +  STAGE1_TOOL = _build/stage1/bin/
>    BUILD_CROSS = NO
>  endif

As explained, this is not the stage1 tool, this is the tool built by the
stage1 compiler, and this directory contains vastly different tools
depending on whether we cross-compile or not.

> @@ -162,7 +164,7 @@
>  	# correct), but we use ghc-pkg from stage2 when we generate our control file.
>  	# Maybe we should consider using ghc-pkg from binary-dist-dir instead.
>  	# As a work-around for now, regenerate the stage2 package cache.
> -	_build/stage1/bin/ghc-pkg recache
> +	$(STAGE1_TOOL)ghc-pkg recache

In case we are cross-compiling, this is the cross-compiler ghc-pkg.
Calling 'ghc-pkg recache' here is wrong. I suppose we can skip this step
if we are cross-compiling (so we can reach the next failure).

>  # --------------------------------------------------------------------
> @@ -335,23 +337,23 @@
>  	rm -rf debian/testghc
>  	mkdir debian/testghc
>  	echo 'main = putStrLn "Foo"' > debian/testghc/foo.hs
> -	_build/stage1/bin/ghc debian/testghc/foo.hs -o debian/testghc/foo
> +	$(STAGE1_TOOL)ghc debian/testghc/foo.hs -o debian/testghc/foo
>  	[ "$$(debian/testghc/foo)" = "Foo" ]
>  	rm debian/testghc/*
>  	echo 'main = putStrLn "Foo"' > debian/testghc/foo.hs
> -	_build/stage1/bin/ghc debian/testghc/foo.hs -o debian/testghc/foo -O2
> +	$(STAGE1_TOOL)ghc debian/testghc/foo.hs -o debian/testghc/foo -O2
>  	[ "$$(debian/testghc/foo)" = "Foo" ]
>  	rm debian/testghc/*
>  	# Test runghc
>  	echo 'main = putStrLn "Foo"' > debian/testghc/foo.hs
> -	[ "$$(_build/stage1/bin/runghc debian/testghc/foo.hs)" = "Foo" ]
> +	[ "$$($(STAGE1_TOOL)runghc debian/testghc/foo.hs)" = "Foo" ]
>  	rm debian/testghc/*
>  	# Output information about GHC
>  	@printf "====BEGIN GHC INFO OUTPUT====\n"
> -	_build/stage1/bin/ghc --info
> +	$(STAGE1_TOOL)ghc --info
>  	@printf "====END GHC INFO OUTPUT====\n"
>  	@printf "====BEGIN GHC-PKG OUTPUT====\n"
> -	_build/stage1/bin/ghc-pkg list
> +	$(STAGE1_TOOL)ghc-pkg list
>  	@printf "====END GHC-PKG OUTPUT====\n"
>  endif

Running the tests when cross-compiling will not work. Either we are
calling the cross-compiler (which will produce binaries we cannot test)
or we are calling a cross-compiled GHC which will not work. I propose we
skip the tests as well, and let the build fail elsewhere.

> Given all of the above, I still think that my patch moves us forward,
> because it allows https://crossqa.debian.net/src/ghc display a real
> failure that we can use in communication with upstream and because
> calling stage1 tools in the right way pretty much is required to seeing
> the later failure from using Hadrian and being able to work on that
> aspect. This failure is not very useful to look at:
> https://crossqa.debian.net/build/ghc_9.4.7-1_amd64_20231029134737.log
> 
> Is this reasoning good enough to convince you that we do have options
> for improving the status quo?

I don't object in removing the error if this will help us move forward.
I can also apply your patch with the above changes (i.e., skip some
steps, instead of calling the cross-compiler). But I really believe we
should focus on having end-to-end instructions on how to cross-compile
GHC and then try to fix Debian's build rules. Fixing errors as we see
them doesn't always move us towards the right direction, if we don't
understand how the end-to-end process works.

-- 
Ilias