[Reproducible-builds] Deterministic "ar" breaks build

[Christian Seiler]

(Cc'ing the reproducible-builds mailing list)

On 06/19/2016 10:23 AM, Ole Streicher wrote on debian-mentors@:
> since a while, the "ar" command is built with --enable-deterministic-archives,
> which basiacally resets (among others) the timestamp to null. This has
> the unfortune disadvantage, that Makefiles that use this timestamp do
> not work correctly anymore. For example, my wcslib package has
> 
> $(WCSLIB)(%.o) : %.c
> 	-@ echo ''
> 	   $(CC) $(CPPFLAGS) $(CFLAGS) -c $<
> 	   $(AR) Ur $(WCSLIB) $%
> 	-@ $(RM) $%
> 
> which results in a repeated compilation of all files every time when a
> dependency of the wcslib appears.
> 
> I could use the "U" flag or ar, but is there a way to get around this
> *and* keep determinism at the same time?

It will definitely be non-deterministic if you are using parallel
builds (-jN), regardless of file timestamps, because the order the
files are added is somewhat randomized.

For builds that are run after "make clean" you don't need U, it should
just work. (Well, I haven't used GNU make's special ar support
features myself yet, so maybe that also breaks with U.)

If you also want to support local builds that don't need to be
deterministic, but where you don't want to constantly rebuild all
object files, the only thing you can do here is to use U in the rule,
extract the archive at the end, and recreate it with the default
options. (Maybe using a different filename, so you have
temp_libsomething.a where ar operates on individually, and then you
create libsomething.a from that via an additional rule that does ar x
on temp_libsomething.a and then ar r on the resulting files. Ugly, but
if you don't want to rewrite your entire makefile, it's the easiest
solution. If you sort the list of extracted files (with LC_ALL=C.UTF-8
before sort), this should also take care of the problem when building
in parallel (if you support that at all).

I don't think there are other options, because for local builds where
you don't want rebuilds you _want_ to have timestamps in there, but
that is antithetical to deterministic builds.

Note though that dh_strip_nondeterminism is able to remove determinism
from ar files - but not related to the file order. So if you enforce
serial builds, and only care about Debian when it comes to keeping your
builds deterministic, then you can just rely on that instead of making
changes to your build system at all (other than adding U to ar), as
long as you don't link anything against those ar files in the same
package. OTOH, there are efforts to make other distributions
reproducible, and if the software is (or might be) also packaged there,
I would recommend fixing this in the upstream build system.

Regards,
Christian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20160619/0dae7504/attachment.sig>