Bug#900918: debian-installer: Please make the generated images reproducible

Tue Jan 22 06:08:44 GMT 2019

Hi Chris,

Chris Lamb <lamby at debian.org> (2019-01-21):
> > That's looking good but I'm seeing new warnings because of gzip's
> > being unhappy about the GZIP environment variable.
> 
> Interesting. However when you say "new" warnings I don't believe my
> patch set actually added/changed this; indeed, it has not changed
> since:
> 
>    https://salsa.debian.org/lamby/debian-installer/commit/28b863340cc5fd73fbaac85a3fb89e72e842b15c

I think I got stuff mixed up (hard to tell now as I deleted the 10+ GB
of build logs and artefacts I accumulated while testing various things
with your patch series and a couple of extra patches on top of it),
maybe because those warnings were moving around by a line or two in the
logs… Maybe that's why I thought they were new.

Putting the confusion aside, I pushed that to avoid the warnings:
  https://salsa.debian.org/installer-team/debian-installer/commit/0b025d7f485ecf7ed8969068f98e49d3141d77fd

> … so I'm just checking what you are requesting to be done here.

All in all, I'm not requesting you to do anything more on the d-i level
(there's been quite a lot of work already); just a mere suggestion to
maybe include an example of direct “gzip -n” specification for tar
invokations on the wiki, so that others don't have to look around how to
do that.

> > All gtk files have fontconfig-related cache/uuid changes…
> […]
> > FWIW, dropping all fontconfig-related bits (see attached patch)
> 
> This is #864082 in src:fontconfig — I've been playing whack-a-mole
> with fontconfig over the past 18 months or so and this was a fairly
> recent regression.
> 
> Are you planning on applying this patch to debian-installer.git?
> Naturally, I would prefer if #864082 was applied and in buster, or
> otherwise closed again.

I don't plan to apply that patch; it was only meaning to serve as a
basis to double check there were no other reproducibility issues in the
gtk images.

> > The mini.iso has apparently other changes… I'm attaching the
> > diffoscope output. Could this be because of missing tweaks to the
> > xorriso calls in build/config/x86.cfg?
> 
> Possibly. Let me try and reproduce and reload all of that into my
> brain and get back to you on this; IIRC there was some pushback
> against making it change behaviour only on SOURFE_DATE_EPOCH being
> present so — as you imply — a command-line change might be required.

OK. I think I'd prefer closing this very bug report whenever what's in
git gets uploaded (it's been a long run and thread already), and see
other issues discussed in a fresh bug report; would that work for you?

> > (Including lintian runtime, using pigz on a 8-way machine cuts real
> > time from 8m8s to 4m23s.)

Now less than 4m, after an extra commit:
  https://salsa.debian.org/installer-team/debian-installer/commit/234058c033ef05c9aab3ced7a7c8cd4917daff9b

> TIL pigz; thanks.
> 
> > Checking what happens [it] seems successive builds with pigz lead to
> > the same results. But those aren't the same as the results generated
> > with gzip. I don't suppose this is going to be a particularly huge
> > problem though?
> 
> Alas, it is a problem in thatfrom the outside nobody will know whether
> one built with pigz or gzip and thus it will be unclear how to
> reproduce the bit-for-bit identical binary. In other words, there is
> currently no ".buildinfo" equivalent here that specifies "I used
> Arch^Wpigz, btw" and got a SHA of $foo.

Ah right, pigz isn't in Build-Depends, so it's not included in the
.buildinfo file…

> One easy solution would be if that, if SOURCE_DATE_EPOCH is present,
> then we force the use of one tool. Although that would regrettably
> mean the lowest common denominator (ie. gzip)  which probably isn't
> the ideal due to the aforementioned performance gain for using pigz.

Right, for a development build (meaning without the generation of the
big debian-installer-images tarball), my gut feeling without specific
clocking data is that half the build time is spent waiting for gzip
to finish its work on a single core. Reproducibility is very nice but
I would definitely hate to lose this huge speed-up.

> Alternatively, we could make pigz a strict build requirement but
> that sounds a little antisocial.

Right.

> Perhaps we need to record the environment after all; again, I will
> reload all of this into my head anyway due to mini.iso (^) so this
> will be top-of-stack again.

Not having looked into the format or tool(s) generating it, would it
be reasonable to have an opt-in mechanism so that a package could
declare “please record the state of this package that might or might not
be installed”. So that pigz could be registered in some way, akin to:
“not-really-in-build-depends-but-oh-look-it-was-present-at-version-X”

Thanks again for your valuable feedback.

Cheers,
-- 
Cyril Brulebois (kibi at debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20190122/b7c252fd/attachment.sig>