Updating dpkg-buildflags to enable reproducible=+fixfilepath by default

Vagrant Cascadian vagrant at reproducible-builds.org
Sat Jan 9 21:52:21 GMT 2021


On 2021-01-09, Lisandro Damián Nicanor Pérez Meyer wrote:
> Note: in case we do not agree on this topic this will be the text I'll
> send to the
> tech-ctte.

Thanks for taking the time to draft some text. If we can come closer to
agreement on the proposed text, that would probably take a bit of the
load off of the tech-ctte. Hopefully some of my comments move in that
direction, although I also have added some counterpoint text as well...


> Let me start by noting that I have nothing against reproducibility. In fact
> quite the opposite: I love the idea... as long as it's properly implemented.

It is even "recommended" in Debian Policy policy to build reproducibly
with different build paths.


> The problem here is that __FILE__ is a public, well defined API with very valid
> use cases (more on this below), even if the current implementation of all the
> major compilers go against reproducibility. 

At least two major compilers, GCC and Clang provide the
-ffile-prefix-map feature to do exactly that(which is what dpkg is
enabling), so it seems a bit overstated to say that all major compilers
"go against" reproducibility. They do not enable reproducibility by
default.


> So if we want to mangle __FILE__
> (thus breaking API) this should be done by an opt-in, and **never** by opting
> out. Else we risk breaking a valid implementation, be it already on the archive
> any new package added afterwards.

While I understand that may be how you feel, it would be appreciated if
we could use something a little less loaded than "mangle". perhaps:

  So if we want to enable features using __FILE__ in a way which
  arguably breaks API


> Even more: we library maintainers continuously ask our upstreams to keep their
> API as stable as possible, and if not possible following some rules
> (SONAME change, for example). I will present an option to do the same ourselves
> without breaking API/ABI.
>
> # __FILE__ is a public, well defined API
>
> During the course of #876901 many reasons were used to both say that __FILE__
> is or not a well defined API. In fact one of the evidences used where the
> compiler's documentation. For example
> https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
> (enphasis mine):
>
>   This macro expands to the name of the current input file, in the form of a C
>   string constant. **This is the path by which the preprocessor opened
> the file**,
>   not the short name specified in ‘#include’ or as the input file name argument.
>   For example, "/usr/local/include/myheader.h" is a possible expansion of this
>   macro.
>
> This definition says that it's up to the preprocessor to define the path. So,
> what has been the behaviour of all major compilers during all these years? Using
> the full path. The proof of this is Qt itself. Check
> https://sources.debian.org/src/qtbase-opensource-src/5.15.2+dfsg-2/src/testlib/qtestcase.h/#L216
>
> This code has been working on **every** platform Qt works without any change.
> Qt is compiled in a myriad of OSs, using a myriad of compilers. They all do
> the exact same thing. So developers depend on a very stable definition
> of an API.
>
> And we all know that breaking API is bad, except if done carefully (read below).
>
> # Doing the right thing
>
> This is just an idea of "doing the right thing", like bumping SONAME
> on a library.
> It definitely doesn't have to be the only one.

I understand your position is fundamentally about "Doing the right
thing" but I would say that we are all trying to do the right thing.


> I understand that the reproducibility people do not want to consider
> providing the
> same build path. While this is arguable I do understand that
> reproducibility without
> depending on the build path is a good thing. So I've tried to come up with a
> path to achieve this.

> ## New macro and warning (if they do not exist already)
>
> This would be the first step. The idea is to provide a new macro that,
> by definition
> and documentation, it can be mangled with the help of the build
> system, much as you
> are currently doing with __FILE__ now.
>
> The second step in this is to make compilers create a reproducibility warning if
> some code uses __FILE__. This will have three effects effects:

We haven't proposed an alternate macro, which would surely take years at
best, possibly decades. That doesn't seem too realistic.


> - discouraging it's use
> - creating awareness on reproducibility.

We've discouraged the use of __FILE__ for years, have done plenty of
outreach on the subject of reproducibility, and gotten traction with
various upstream projects.


> - making other distros jump into reproducibility in a much easier way.

Arguably some distros (e.g. archlinux) are passing us by when it comes
to real-world reproducibility; I'm not sure Debian is the example by
which all other distros should be measured anymore. Which is good in
some ways, but somewhat disappointing to see Debian start something
great and then lag behind.


> ## Mangling by opting-in, or what to do in the meantime

Maybe drop the phrase "Mangling"? Perhaps

  Opt-in to the fixfilepath feature, ...

Or maybe:

  Keeping the opt-in status quo, ...

> Of course we want to make things reproducible as fast as possible too. This is
> why mangling __FILE__ was considered and implemented. But this is breaking API,
> no matter how you look at it.

Well, that's the crux of the argument, isn't it?

Is making an assumption about correct behavior by reading ambigous
documentation that can be interpreted either way, and then examining
default compiler behavior ... really an API?


> But that doesn't means it can not be used: if a
> package can be built mangling __FILE__ in order to get a reproducible way then
> the maintainer should really consider opting *in* for the change, but never
> ever should the maintainer be forced to opt *out* using a broken API.
>
> In fact I consider all the "fixes" of opting in a bug, and forcing this change
> a couple of weeks before freeze even worse. So I propose you to undo the change
> in dpkg and, after the release, starting to work on letting maintainers opting
> in.

It has been an option to opt-in to this feature for at least two years;
and some packages have done so. Adding boilerplate one-line changes to
hundreds or thousands of packages in Debian seems suboptimal to actually
effect change.

I don't know if size of the issue is relevent to tech-ctte, but it seems
worth noting that the archive-wide rebuild revealed less than 30
packages broken by this change, most of which have been patched to
opt-out of this feature. Most or all of the reamaining issues are due to
QFINDTESTDATA in test suites.

And for completeness, It was speculated that hundreds more packages
which currently disable test suites would need to also opt-out of this
feature in order to enable test suites, or come up with some other way
to remove these flags during tests.

It also seems worth asking what are the negative impacts of enabling
this, other than a handful of packages that need changes to
debian/rules in order to build?


Thanks for taking the time with all this everyone!


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20210109/b2ebcfcc/attachment.sig>


More information about the Reproducible-builds mailing list