Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building

[Simon McVittie]

On Wed, 31 Jul 2024 at 17:17:29 +0200, Helmut Grohne wrote:
> Package: native-architecture
> Architecture: all
> Multi-Arch: no
> # empty package
> # do something about the multiarch hinter to make it stop suggest adding
> # M-A:foreign

Is the multiarch hinter a cross team thing, or someone else's? I find
it frustrating that it doesn't seem to have an overrides mechanism,
so when it has a recommendation that is well-intentioned but wrong,
the affected package can't silence it (like we can for e.g. Lintian or
blhc false positives).

> Package: cross-exe-wrapper
> Architecture: any
> Multi-Arch: same
> Depends: native-architecture | qemu-user, native-architecture | arch-test
> # wrapper below

I'd recommend making this include its own /bin/true implementation
(/usr/lib/*/gobject-introspection/deb-can-run-* in gobject-introspection
are an implementation of this), instead of making arch-test
load-bearing.

The way arch-test has been implemented (as an Architecture: all
package that builds binaries for an assortment of Debian and non-Debian
architectures) is very clever (perhaps *too* clever), but it requires
action to be taken by its (single) maintainer every time there is a new
architecture of interest, and breakage in any single architecture's
cross-toolchain (including -ports like ia64) will break its build.

(But if cross-exe-wrapper is Architecture: any, then it's already trivial
for it to build and install a small binary matching its host architecture,
so that should be an easy change.)

Ideally this would have a special case for the common i386-on-amd64 to
make it not require qemu-user, if that's possible (#1070773). Perhaps
something like this

Depends: native-architecture | dpkg-architecture-is-amd64 [i386] | qemu-user

where dpkg Provides: dpkg-architecture-${DEB_HOST_ARCH}? But that doesn't
have to be part of a minimum-viable-product implementation.

> Say we're satisfying a dependency on libglib2.0-dev for some
> architecture $host_arch. Doing so will cause apt to install
> cross-exe-wrapper:$host_arch (as M-A:same forwards the architecture
> constraint). apt will then attempt to install native-architecture for
> $host_arch. That is possible if $host_arch happens to be the native
> architecture (i.e. if dpkg happens to be $host_arch) and apt will favour
> this solution when available. When $host_arch is non-native, the first
> alternative will be unsatisfiable and apt will resort to the second one.

Concretely, let's say we're building on an amd64 system and we want to
compile a GLib-dependent package for amd64 and riscv64 hosts. We have:

libglib2.0-dev:amd64
    -> cross-exe-wrapper:amd64
        -> native-architecture:all (success)

libglib2.0-dev:riscv64
    -> cross-exe-wrapper:riscv64
        -> ~~native-architecture:all~~ (no, uninstallable)
        -> qemu-user (success)

It seems pretty subtle that the only thing making this work as intended
is that dpkg:amd64 will refuse to install native-architecture:all to
satisfy a dependency of a riscv64 package. That's a relatively obscure
corner of multiarch!

I do wonder whether it would be cleaner to have dpkg add a

    Provides: dpkg-architecture-is-${local:DEB-HOST-ARCH}

and then have cross-exe-wrapper depend on

    Depends: dpkg-architecture-is-${local:DEB-HOST-ARCH} | dpkg-architecture-is-amd64 [i386] | qemu-user

(where ${local:DEB-HOST-ARCH} is a substvar, in both cases set up in
the same way that ${local:DEB-HOST-ARCH-OS} works in src:glib2.0)

> The actual wrapper script's name should be architecture-dependent such
> as $host_triplet-cross-exe-wrapper.

That's ${DEB_HOST_GNU_TYPE}-cross-exe-wrapper (not the multiarch tuple),
in the same pattern we use for cross gcc and cross pkgconf and so on, right?

> It can roughly look like:
> 
> #!/bin/sh
> 
> if test "$(dpkg --print-architecture)" = #DEB_HOST_ARCH# || arch-test -n #DEB_HOST_ARCH# >/dev/null; then
>     exec "$@"
> else
>     exec qemu-#DEB_HOST_ARCH# -- "$@"
> fi

I'm not sure that qemu-user actually accepts "--" (which is why in the
straw-man specification I wrote, if I remember correctly I said something
like "the wrapped executable should be an absolute path and must not
start with -" to bypass that concern), but otherwise yes, that's more
or less what I had in mind.

Maybe it would be good to put a linux32 prefix on the command if targeting
a known 32-bit counterpart architecture (and not using qemu), like i386
on amd64? (exec linux32 "$@")

> Let me note that I have been arguing in favour of changing the semantics
> or Arch:all in ways that would break the aforementioned setup.

That's already the part of this that seems most fragile to me. If the dpkg
maintainer was willing to add the dpkg-architecture-is-* setup that I
suggested, I think that would give us a way to avoid needing to rely on
this behaviour?

> I recently encountered the
> following packages that would fail to install their Build-Depends due to
> choosing a host architecture Python interpreter and failing its
> postinst:
> [29 packages]

That's not actually all that many. I wonder whether it would be pragmatic
to add python3:native to their B-D and move on?

Or teach crossqa or sbuild to always --add-depends=python3:native when
cross-building? In the worst case scenario that's adding python3 when
it isn't actually needed, but increasingly many packages use Meson which
pulls in python3 anyway, so maybe that's not so bad?

Certainly if any of those are critical-path for unblocking whole subtrees
of currently-non-cross-compilable packages, or if they're important targets
for cross-compiling for any other reason, adding python3:native to their
B-D should be an easy workaround.

> There are two main downsides of going forward with barbarian as far as I
> can see. One is that we restrict the solution space and outright reject
> ever installing a host's instance of a M-A:foreign package in a cross
> build environment. I don't know why that would ever be necessary, but
> then we can no longer do it. The other is that we need to fix all the
> builders to make cross building unstable work. In particular, bookworm's
> sbuild will no longer be able to cross build unstable.

My instinct is to say that neither of these is particularly bad.

    smcv