Status of Required/Essential/Build-Essential in Debian

[Vagrant Cascadian]

Lately, I've been trying to get a handle on the status of the really
core packages in Debian, namely the essential, required and
build-essential package sets. The first two are present on nearly every
Debian system, and build-essential is the set of packages assumed to be
available whenever you build a package in Debian.

I will summarize below the outstanding issues for Debian with these
package sets.

I'd also be really curious to hear about the status of similar package
sets in other distros! I would also like to see if there is anything in
Debian or other distros that still needs to be pushed upstream, so we
can all benefit!


Essential:

  https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_essential.html

Almost done with essential, at 95% reproducible:

The only outlier is glibc, which currently doesn't build, but a version
that does build in debian experimental has a patch submitted specific to
Debian's packaging of glibc:

  different file permissions on ld.so.conf* and others
  https://bugs.debian.org/1010233


Required:

  https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_required.html

Also nearly there, at 88.9% reproducible (and one probably obsolete
package in the list, gcc-9):


apt has two remaining issues, one of which is trivial to fix:

 BuildId differences triggered by RPATH
 https://bugs.debian.org/1009796

The more difficult issue with apt is caused by toolchain bugs in
doxygen:

 https://tests.reproducible-builds.org/debian/issues/nondeterminstic_todo_identifiers_in_documentation_generated_by_doxygen_issue.html
 https://tests.reproducible-builds.org/debian/issues/nondeterministic_ordering_in_documentation_generated_by_doxygen_issue.html

There is a workaround patch for apt to disable building of documentation:

 support "nodoc" build profile
 https://bugs.debian.org/1009797


Build-Essential:

  https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_build-essential.html

Not bad at 87.1% reproducible.

linux has two issues, one unidentified issue relating to build paths,
and another documentation issue:

  https://tests.reproducible-builds.org/debian/issues/randomness_in_documentation_generated_by_sphinx_issue.html


libzstd has one remaining issue, where it embeds build paths in assembly
objects:

  https://tests.reproducible-builds.org/debian/issues/build_path_captured_in_assembly_objects_issue.html


gmp has one outstanding set of patches to fix build path issues:

  Embedded build paths in various files
  https://bugs.debian.org/1009931


binutils has several identified issues and probably some unidentified
issues:

  included log files introduce reproducibility issues (debian specific?)
  https://bugs.debian.org/950585
  https://tests.reproducible-builds.org/debian/issues/unstable/test_suite_logs_issue.html

  source tarball embeds build user and group (debian specific)
  https://bugs.debian.org/1010238


krb5 has one really perplexing issue related to build paths triggering
seemingly unrelated changes in the documentation, possibly toolchain
related (sphinx? doxygen?):

  differing build paths trigger different documentation
  https://bugs.debian.org/1000837


gcc-12 (and probably other gcc variants) also embeds test suite logs
very similar to bintuils described above. Probably many other issues,
especially related to build-time profile-guided-optimization and... who
knows! GCC also takes so long to build, it can be difficult for our test
infrastructure to actually build and/or run diffoscope without timing
out...


openssl contains a few unidentified issues relating to build paths, some
test suite failures in our test infrastructure, and a couple known build
path related issues:

  https://tests.reproducible-builds.org/debian/issues/build_path_captured_in_assembly_objects_issue.html

  Embeded compiler flags contain build paths
  https://bugs.debian.org/1009934


Build-Essential-Depends Bonus Round! (all the packages that
Build-Essential needs to build itself):

  https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_build-essential-depends.html

At 86.3% reproducible, it still doesn't look too bad, and there are a
lot of patches submitted and/or in progress. It is a much larger set of
packages, so I won't even try to summarize the status here.


Soooooooo.... A few closing thoughts...

A fair number of these are build paths issues, which we do not test in
Debian testing (currently bookworm), only in debian unstable and
experimental. So the numbers in general look a better for
testing/bookworm. Other distros by-and-large do not test build paths
variations, and while I'd like to fix those issues, they're a little
lower-priority.

Two other remaining issues are toolchain issues for documentation using
sphinx and doxygen, and are the last blockers for fixing apt and linux
(as well as numerous other packages). This seems like a high priority to
fix!

I have been chewing on the ideas of how to resolve the embedded test
suite log issues in binutils and gcc. There was some discussion of
possible ways to fix this that I (or anyone interested!) need to
follow-up on and start doing proof of concept changes in Debian:
  
  https://lists.debian.org/debian-devel/2022/02/msg00216.html  


Thanks for making it this far!

This is the hardest, lastest miles/kilometers, although we actually do
have a lot of pending fixes in progress... though a few seemingly
inscrutible challenges still to face!


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20220427/9cc90870/attachment.sig>