[Reproducible-builds] [PATCH] Upstreaming a large amount of patches

[Reiner Herrmann]

Hi Bela and Christian,

[ adding reproducible-builds list to CC ]

On Tue, Mar 22, 2016 at 03:54:33PM -0700, Bela Lubkin wrote:
> Replying out of band off of the dietlibc list because this isn't about
> dietlibc...

> >
> > https://gist.githubusercontent.com/chris-se/e35381c75583a4c4bce2/raw/8bf327a048f396220c48fb4ffbd504d12b042007/features-01-reproducible-builds.patch
> >
> >       Debian is currently working on making the archive reproducible,
> >       see <https://reproducible-builds.org/> for details. dietlibc has
> >       one issue so far as I can tell, which is that the order of how
> >       the files are added to the library depends on the underlying
> >       filesystem, see <https://reproducible-builds.org/docs/stable-inputs/>.
> >
> >       The attached patch sorts the files during the build process to
> >       make the order deterministic. Basically, $(wildcard ...) is
> >       replaced by $(sort $(wildcard ...)) in the Makefiles.
> >
> >       (You still need a patched toolchain for the build to actually
> >       be reproducible.)
> >
> >       Patch author: Reiner Herrmann <reiner at reiner-h.de>
> 
> Looking at this patch, it seems like the real fundamental bug is in GNU
> Make for not only not-sorting wildcards, but having actually *changed*
> this between version 3.81 and 3.82.  As their release announcement says:
> 
> [https://lists.gnu.org/archive/html/info-gnu/2010-07/msg00023.html]
> 
> " * WARNING: Future backward-incompatibility!
> "   Wildcards are not documented as returning sorted values, but up to and
> "   including this release the results have been sorted and some makefiles are
> "   apparently depending on that.  In the next release of GNU make, for
> "   performance reasons, we may remove that sorting.  If your makefiles
> "   require sorted results from wildcard expansions, use the $(sort ...)
> "   function to request it explicitly.
> 
> (But the change actually went into 3.82, not a future release; as the
> ChangeLog in the source says:
> 
> " 2010-08-13  Paul Smith  <psmith at gnu.org>
> "
> "         * NEWS: Accidentally forgot to back out the sorted wildcard
> "         enhancement in 3.82, so update NEWS.
> 
> -- after it had already been posted to the mailing list as a threat of
> *future* change...)
> 
> "for performance reasons", they broke backwards compatibility and
> potentially packages throughout the universe of software development?
> Augh...

Yes, I also think that this is very unfortunate.
The performance "impact" is only visible in build processes and doesn't
affect end users, and even there it is probably negligible for almost all
packages.
But at the time of the change reproducible builds weren't considered as
that important yet and they probably didn't see any disadvantage in omitting
the sorting.

> BUT.  What is the point of my griping about this years later?
> 
> That (1) if they were going to do that, they should have retained a
> sorting-wildcard function (e.g. perhaps "$(sorted-wildcard ...)"); but
> mainly (2) that it is not too late to do that even today.
> 
> That is.  The reproducible-builds.org folks should be *adding* such a
> function to GNU Make.  Or perhaps reverting the change -- declaring that
> YES, GNU Make's "$(wildcard ...)" sorts -- and adding an
> "$(unsorted-wildcard ...)" which is (a teeny tiny shred) faster for
> people who really need such a thing.

Perhaps someone from the list is interested in adding that, or even
better, would be able to convince Make upstream to revert the change.

> Why do I think `$(sorted-wildcard ...)' is better than `$(sort $(wildcard
> ...))'?  Because it's syntactically less complex.  Because I expect it's
> easier and more reliable to programmatically transform Makefiles with
> just a word substitution, than to add an extra outer parenthetical
> expression around wildcard lookups which might themselves be full of
> complex syntax.

I think the best solution would be to change the default again to let
wildcard sort the lists (it will probably make a bunch of packages
reproducible without having to touch them). And if someone really
needs this bit of extra-performance, a fast-wildcard or unsorted-wildcard
could be added.


And thanks a lot Christian for submitting the patch to upstream! :)

Kind regards,
  Reiner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20160323/e780f9f7/attachment.sig>