Bug#992775: python3.9: stores wrong path to mkdir if built on unified-/usr system

[Simon McVittie]

Source: python3.9
Version: 3.9.2-1
Severity: important
Tags: patch bookworm sid
User: reproducible-builds at lists.alioth.debian.org
Usertags: usrmerge
X-Debbugs-Cc: reproducible-bugs at lists.alioth.debian.org

If python3.9 is built on a merged-/usr system (as created by new
installations of Debian >= 10, debootstrap --merged-usr, or installing the
usrmerge package into an existing installation) or on a system where /bin
and /usr/bin have been unified via a "symlink farm" in /bin, the path to
mkdir is recorded in installed Makefiles as "MKDIR_P = /usr/bin/mkdir -p"
instead of the canonical "MKDIR_P = /bin/mkdir -p".

This cannot currently be seen on the reproducible-builds.org infra, because
the python3.9 "build2" times out, but it can be reproduced by building
python3.9 locally.

If you have sbuild available, a way to reproduce this is:

- build twice, once with --add-depends-arch=usrmerge and once without
- both times, build with --build-path=/build/python or similar, to
  eliminate unrelated variation that involves the build path
- both times, build with DEB_BUILD_OPTIONS=noopt, to eliminate unrelated
  variation which appears to be caused by profile-guided optimization or
  by LTO
- compare built results with diffoscope
- ignore unrelated variation in usr/share/info/python3.9.info.gz

The problematic situation is if the package is *built* on a unified-/usr
system, but *used* on a non-unified-/usr system. In this situation,
/usr/bin/mkdir exists on the build system but not on the system where
python3.9 will be used, resulting in the installed Makefile being
unsuitable for the installed system.

Technical Committee resolution #978636 mandates heading towards a
transition to merged-/usr (as Ubuntu has already done), and this
will become a non-issue at the end of that transition; but variation
between merged-/usr and non-merged-/usr builds is a problem while that
transition is taking place, because it can lead to partial upgrades
behaving incorrectly. It is likely that this class of bugs will become
release-critical later in the bookworm development cycle.

Some developers advocate that instead of merged-/usr, /usr/bin and /bin
should be made equivalent via a "symlink farm" in /bin, but that would
have the same effect as merged-/usr for this particular package: the
transitional problem and the solution are the same.

The attached patch resolves this: with it applied, the package builds
more similarly with and without --add-depends-arch=usrmerge.

A side benefit of fixing this is that this change brings the package
a little closer to being reproducible (as recommended by Policy §4.15).

I suspect python3.10 has the same issue and would benefit from the same
solution, but I haven't tested that. If the attached patch seems valid,
please apply it to both.

    smcv
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-d-rules-Specify-canonical-path-to-mkdir.patch
Type: text/x-diff
Size: 1200 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-bugs/attachments/20210823/88fa1e5b/attachment-0001.patch>