[Python-modules-team] Bug#961206: improve sphinx usage for cross building

[Helmut Grohne]

Hi Dmitry,

On Tue, Jun 02, 2020 at 01:03:15PM +0300, Dmitry Shachnev wrote:
> [ Adding the bug back to Cc, I assume you did not intend to remove it. ]

Thank you. That was unintentional indeed. Fortunately, you quoted
everything.

> So what are our next steps? I will make a Sphinx 3.x upload with updated
> package description to experimental soon.

Unfortunately, I encountered another little roadblock. At least
sphinx-apidoc imports modules. Doing so certainly isn't M-A:foreign. I
don't know whether sphinx-autogen also does. In any case, this doesn't
work as planned.

Given that I've performed the analysis under the assumption that marking
sphinx M-A:foreign was ok, the rest of this email is going to assume
that even though this assumption is now known to be wrong.

We could do with a temporary, circular dependency. We could split the
sphinx package from python3-sphinx right now. Now sphinx must depend on
python3-sphinx forever and python3-sphinx must depend on sphinx
temporarily to avoid breaking rdeps. The latter dependency can be
dropped in like half a year maybe.

As soon as this split is in place, we can mark sphinx Multi-Arch:
foreign (assuming this was fine). Once that happens, cross builds can
immediately benefit from the split without requiring that everyone else
updates their Build-Depends.

> Do you want to prepare mass bug filing, or you prefer that we deal with the
> packages in Python team (DPMT/PAPT) first?

I strongly prefer reducing the number of bugs to file before filing
them.

There are also two parts of this transition:

 A) Fixing packages to work without the python3-sphinx -> sphinx
    dependency.
 B) Fixingg packages to use sphinx instead of python3-sphinx for cross
    building.

I think B is a long-tail transition and you basically don't have to care
about it at all. A is the transition you need to pay attention to.

So how about numbers?

Around 1200 packages (transivitively) build depend on python3-sphinx. I
built them with a path-exclude for the sphinx tools.  Around 500
packages FTBFS when /usr/bin/sphinx-* is removed. Around 450 clearly
fail at using one of the tools. The rest is roughly 25 filed FTBFS and
25 packages where I have no clue.

Here is the list of packages that definitely use a sphinx commandline
tool:

ahven aiohttp-wsgi aiomysql alot ansible astroplan astroquery
azure-cosmos-python azure-cosmos-table-python
azure-data-lake-store-python basemap bcolz beanbag bedops beets
bibtexparser binoculars blends breathe breezy bugwarrior buildbot casync
cdist ceph ceres-solver charliecloud cheetah chemps2 class.js cmake
compass-susy-plugin configobj configparser cpl-plugin-amber
cpl-plugin-fors cpl-plugin-giraf cpl-plugin-hawki cpl-plugin-muse
cpl-plugin-naco cpl-plugin-uves cpl-plugin-vimos cpl-plugin-visir
cpl-plugin-xshoo csvkit cvxopt cysignals cython dask dballe dcm2niix
deap debian-policy debomatic defcon developers-reference dex dhcpcanon
diceware django-axes django-guardian django-pipeline dms dogtag-pki doit
dolfin dolfinx dpdk dput-ng dune-common dynare elastalert epigrass
fabric fastchunking ffcx fityk flask-bcrypt flask-flatpages
flask-restful flask-wtf flent flycheck fmtlib fontparts fonttools fsspec
gcc-python-plugin gearmand ghc git-cola glom gnat-gps goiardi
goocalendar grapefruit graphite-api gromacs guidata gwcs h5py
hamradio-maintguide haproxy hdf-compass heat-cfntools hiera-py
highlight.js hy hyperkitty igdiscover isbg isc-kea isso itango jansson
jeepney jinja2 jsonpickle khard kitty knot knot-resolver krb5 lammps
lasagne libabigail libaws libcbor libcork libcorkipset libgnatcoll
libgpuarray libixion liblognorm liborcus libserial libtemplates-parser
libvigraimpex libxmlada llvm-toolchain-10 llvm-toolchain-8
llvm-toolchain-9 llvmlite logbook loggerhead logilab-common logzero
lua-argparse lua-unit luabind luacheck macsyfinder mailman3
mailmanclient mako mame mapproxy mash mathjax-docs
microsoft-authentication-library-for-python minieigen mlpy
mongo-c-driver morse-simulator mpd mpi4py-fft mpmath musicbrainzngs
mydumper mypy ndcube neo networkx nextcloud-desktop nfs-ganesha nghttp2
nitime nordugrid-arc-nagios-plugins notmuch ns3 ntopng numba numcodecs
numpy oar objgraph ocrmypdf onionbalance opencolorio openvswitch
opgpcard owncloud-client owslib pagure pam-python pandas parso
pathspider patroni patsy pdal pelican pgloader pgrouting photofilmstrip
php-mockery php-twig phpmyadmin pikepdf pillow pocl podcastparser
polybar postorius powerline pplpy praw psycopg2 py3c pybind11 pybindgen
pybtex pybtex-docutils pycares pycoast pycollada pydicom pydl pydocstyle
pyepr pyfftw pyfr pygccjit pygccxml pygments pygresql pyhamtools pyliblo
pylint pymongo pyopencl pyopenssl pyosmium pyparsing pypy pypy3 pyqi
pyqso pyregion pyresample pyrlp pyro4 pysdl2 pyserial-asyncio
pysoundfile pyswarms pytables pytest pytest-mpi pytest-qt python-acme
python-agate python-agate-dbf python-agate-excel python-agate-sql
python-aiohttp-security python-aiohttp-session python-aioice
python-aioresponses python-argon2 python-async-generator python-bioblend
python-biom-format python-bitstring python-bsddb3
python-cassandra-driver python-certbot python-certbot-dns-cloudflare
python-certbot-dns-digitalocean python-certbot-dns-dnsimple
python-certbot-dns-gehirn python-certbot-dns-google
python-certbot-dns-linode python-certbot-dns-ovh
python-certbot-dns-rfc2136 python-certbot-dns-route53
python-certbot-dns-sakuracloud python-cogent python-cymruwhois
python-darts.lib.utils.lru python-django python-django-bootstrap-form
python-django-celery-beat python-django-mptt python-django-registration
python-easygui python-escript python-fitbit python-gmpy2
python-gmusicapi python-gntp python-hpilo python-icalendar
python-ilorest python-imageio python-jenkins python-josepy
python-jsonrpc python-latexcodec python-leather python-libevdev
python-lockfile python-ly python-m2r python-matrix-nio python-midiutil
python-mongoengine python-mpd python-multipletau python-networkmanager
python-opentracing python-pathtools python-pbcore python-pgpy
python-plaster python-prctl python-prov python-ptk python-pyaudio
python-pyepsg python-pygraphviz python-pygtrie python-pykka
python-pyknon python-pyld python-pylibsrtp python-pymeasure
python-pymzml python-pyqtgraph python-pyramid-chameleon python-pyxattr
python-qtconsole python-qwt python-reportlab python-requests-cache
python-requests-oauthlib python-requests-toolbelt python-rpaths
python-rtree python-ruffus python-setuptools python-sfml python-simpy
python-simpy3 python-skbio python-srp python-stetl python-structlog
python-sunlight python-telegram-bot python-tinyrpc python-tornado4
python-treq python-trie python-twitter python-ulmo python-urlobject
python-validictory python-vispy python-werkzeug python-ws4py python-x2go
python-xarray python-xkcd python2.7 python3.8 pythondialog pytools pywps
pyx3 pyzo pyzor restic ripe-atlas-sagan ripe-atlas-tools rsyslog-doc ryu
salmon sasview sbws setuptools sfepy simplebayes six sleekxmpp slepc4py
snimpy sortedcollections sortedcontainers soundscaperenderer speedcrunch
sphinx-click sphinx-gallery sphinxcontrib-bibtex sqlalchemy-i18n
sqlparse squirrel3 statsmodels straight.plugin stressant subuser
supervisor sympy tap.py toil toolz trafficserver translate-toolkit
tryton-client tryton-server tsung twisted txtorcon
ubuntu-packaging-guide uncertainties varnish veusz vim-autopep8 virtaal
vitables waitress wand webassets webpy weresync why3 wslay wtforms x265
x2gobroker xmldiff yamllint yapsy yara zarr zathura zoneminder

I'm quite unsure about how to proceed from here. The split doesn't work
as planned. :-/

Helmut