Updating dpkg-buildflags to enable reproducible=+fixfilepath by default

[Vagrant Cascadian]

The dpkg-buildflags feature reproducible=+fixfilepath was added to dpkg
in 2018.

## What does this feature do exactly?

From the dpkg-buildflags(1) manpage:

  fixfilepath

    This setting (disabled by default) adds
    -ffile-prefix-map=BUILDPATH=.  to CFLAGS, CXXFLAGS, OBJCFLAGS,                                                    
    OBJCXXFLAGS, GCJFLAGS, FFLAGS and FCFLAGS where BUILDPATH is set to
    the top-level directory of the package being built.  This has the                                                 
    effect of removing the build path from any generated file.                                                        

The result of enabling this feature by default will be to make several
hundred packages reproducible with varying build-path and reduce the
differences in many other packages, making it easier to identify other
more nuanced reproducibility issues.

It would be great to see the reproducible=+fixfilepath feature enabled
by default in dpkg-buildflags, and we would like to proceed forward with
this soon unless we hear any major concerns or other outstanding issues.


## Process regarding updating dpkg-buildflags defaults

Following the dpkg FAQ on how to add default build flags to
dpkg-buildflags:

  https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F

We do not expect any significant change in memory or build-times, or any
changes in run-time semantics (other than the issues noted below
regarding test suites). An archive-wide rebuild has been performed (more
below).


## Possible updates required to your packages

Minor updates to a small number of packages in the archive are needead,
although patches for most od them have already been sent. The simplest
workaround is a one-line change in debian/rules to disable the feature:

  DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath

Though, of course, identifying the exact reproducibility problem would
be preferable. One of the common issues is test suites relying on the
behavior of __FILE__ returning a full path to find fixtures or other
test data.

A small number of packages manually filter out
-fdebug-prefix-map=/build/dir when recording CFLAGS (etc.) into binary
packages, in order to make the build reproducible. Depending on the
implementation of this filter (specifically, whether it also filters out
-ffile-prefix-map as well), these packages may, ironically, actually
become unreproducible with reproducible=+fixfilepath -- they will not
catch this additional flag.  In these situations, please broaden the
regular expression (or similar) to make the build reproducible again or
avoid recording any CFLAGS whatsover depending on the circumstances.


## Background

We have been performing builds with DEB_BUILD_OPTIONS=reproducible=+all
(which includes +fixfilepath) at https://tests.reproducible-builds.org
since 2018. Currently we only enable this feature in sid and
experimental.


Lucas Nussbaum kindly performed an archive-wide rebuild and identified a
small number of packages that failed to build with this flag enabled:

  http://qa-logs.debian.net/2020/09/26.fixfilepath/00res.fixfilepath.only-failures.txt

Huge thanks to Lucas for that!


The failing packages have been marked in the reproducible builds
infrastructure:

  https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
  https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html

And I have filed patches for most of the affected packages:

  https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org&tag=fixfilepath

The few remaining packages FTBFS regardless of whether they use
reproducible=+fixfilepath, or are built with the default clang compiler,
version 9, which does not support this feature. I expect most of these
packages to build correctly once llvm-toolchain-defaults updates to 10
or 11, which is expected for bullseye:

  https://alioth-lists.debian.net/pipermail/pkg-llvm-team/2020-September/010784.html


We would like to move forward with this change soon, so please raise any
concerns or issues not covered already.


Thanks for reading this far!


live well,
  vagrant
  On behalf of the Reproducible Builds team
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20201027/fec4e8c6/attachment.sig>