[Python-modules-team] Bug#915856: sphinx: Failed cross building with Build-Depends on python3-sphinx

[Helmut Grohne]

Hi Dmitry,

Thank you for contacting me about this issue. It is a very relevant one
that I have neglected far too long, because I felt it was too difficult.
That shouldn't stop us from working on it however and I'm really happy
to help solve it. Beware that disruptive changes may be ahead though.

On Sun, Jan 20, 2019 at 12:45:36AM +0300, Dmitry Shachnev wrote:
> On Fri, Dec 07, 2018 at 11:42:09AM +0100, kaliko wrote:
> > here is a follow up of a discussion started on irc:debian-devel with
> > mitya57.
> >
> > I'm trying to cross build an "Architecture: any" package [0]
> > (commit:52326679) using the following:
> >
> >     sudo pbuilder create --host-arch armhf
> >     sudo pbuilder build --host-arch armhf ncmpc_0.33-1.dsc
> >
> > building failed on:
> >
> > "builddeps:/build/ncmpc_0.33-1.dsc:armhf : Depends: python3-sphinx:armhf but
> > it is not installable"
> 
> In other packages this is usually being fixed by splitting the documentation
> into a separate package that is Architecture: all.

This is actually the solution I have preferred thus far. For small
packages it is a poor trade-off though. At some point we'll be hitting
packages where this is undesirable.

> Then Sphinx is not needed when cross-building at all — there is no sense in
> cross-building Architecture: all packages, you can just build them natively.
> 
> To skip installing Sphinx on cross builds, you need to:
> 
> 1) Move python3-sphinx build-dependency to Build-Depends-Indep.
> 
> 2) After doing that, you can no longer use dh --with sphinxdoc, as dh will
> not find sphinxdoc.pm. So you will need something like this instead:

There actually is a way to use the addon.

DH_ADDONS=
%:
	dh $@ $(DH_ADDONS)
build binary %-indep: DH_ADDONS+=--with=sphinxdoc

> 3) Also maybe you will need to pass --binary-indep to pbuilder, as it does
> not automatically add that option for cross builds.

Since very recently, sbuild automatically adds --no-arch-all for cross
builds. Not sure about the situation of pbuilder.

> This was a solution that does not need changing or using Sphinx at all.

Thank you for not stopping here.

> Now let’s talk about Sphinx. There is bug #818115 which discusses how to
> make it possible to install Sphinx in cross-build environments.

Yes, please.

> Last discussion on that bug was in 2016, so I checked whether some
> things described there are still actual. Found this thing:
> 
> > So maybe we can say that architectures don't matter for sphinx by
> > marking it Multi-Arch: foreign? Unfortunately, no. python-sphinx
> > transitively depends on python-markupsafe (via python-jinja2) and thus
> > exposes it. Since python-markupsafe is an architecture dependent Python
> > extension, python-sphinx exposes the architecture. This is the gist of
> > the multiarch interpreter problem.

Please consider this just as one example. I looked through the
dependency stack and this was the first example found. There is likely
more underneath.

> Maybe this means that python3-sphinx can be marked Multi-Arch: foreign.
> 
> However, I am not sure this is a safe thing to do. One potential issue may be
> that Sphinx autodoc extension [1] imports arbitrary Python code, and that
> code may be a compiled (for the host architecture) extension that will fail to
> import with the build architecture interpreter.

I'm quite sure that by now marking python3-sphinx M-A:foreign is wrong
for precisely the reason you give: You don't know ahead of time whether
you'll have to import architecture-dependent code.

> Also, I wonder if just making src:sphinx binary packages Multi-Arch: foreign
> will be enough, or we should also mark all its dependencies Multi-Arch:
> foreign too.

These are two questions:

 * If a package is M-A:foreign, do all dependencies have to be
   M-A:foreign?

   No! I often use dependencies not being M-A:foreign as an argument,
   but that is only an approximation. What really counts is the
   "interface" of a package. Unfortunately, "interface" is pretty
   loosely defined in Debian so a conservative approach is to treat all
   dependencies as part of the "interface". So having all dependencies
   M-A:foreign is part of a sufficient, but not necessary argument.

 * Should there be a M-A:foreign sphinx package?

   Likely, that'd be a good idea. It must be done with care however. It
   must be made abundantly clear that combining sphinx (as opposed to
   python3-sphinx) with any sphinx extension is a serious bug. Using a
   sphinx with a sphinx extension is akin to a "missing dependency" and
   we usually call that an rc bug. (The missing dependency is
   python3-sphinx of course.) As a result, this approach is only useful
   for "simple" packages. It can be a partial solution and most cases
   can likely use either plain "sphinx" or split a -doc package.

A "weaker" alternative already employed by some other python extensions
is "Multi-Arch: allowed". It's essentially the same thing just renaming
"sphinx" to "python3-sphinx:any". It'll look more ugly in dependencies,
but it crucially is the same thing.

The heart of the problem is the multiarch interpreter problem though.
What you want to express here is that a number of dependencies must have
a common architecture, but can differ from other packages. In a sense,
we'd want group dependencies together and have the user select the
architecture for each group. We discussed this extensively at least in
Vaumarcus and concluded that the problem is too hard. Whenever you need
such a thing, you have to create a "dependency package" depending on all
packages in the group and making that dependency package Multi-Arch:
foreign. Now creating such dependency packages for every subset of
sphinx extensions is not gonna work obviously. I don't think this is
entirely solvable. Therefore I propose concentrating on practical,
individual, and relevant cases. Your suggestion to split -doc packages
is a very good one in that respect.

Helmut