[Reproducible-builds] A recap of DebConf14 for reproducible builds in Debian

Jérémy Bobbio lunar at debian.org
Tue Sep 2 03:44:12 UTC 2014 

Hi!

DebConf14 [1] is now over. It was a very productive time for
reproducible builds. Here's a tentative recap of what happened.

On August 26th, two events were scheduled:

 * a 45 minutes talk [2]: video [3], slides [4];
 * a BoF to discussion solutions to identified issues [5].

I must admit I was nervous before doing the talk. Not so many things had
seen progress since the talk at FOSDEM’14 in January, and I was unable
to complete a new rebuild experiment following Stéphane Glondu’s advices
with David Suárez. But the talk went well, and the reception was beyond
my hopes. There has been no push back on the suggested solution on
defining a canonical path where packages must be built. A couple of
members of the Debian technical committee who attended the talk showed
support and interest which is a good sign.

In the evening, the BoF was attended by 20 people. We decided not to
record it to avoid having to run a microphone around. The main objective
of the discussion was to define the framework in which we could consider
Debian package reproducible.

We agreed that the time of the latest entry in `debian/changelog` was a
sane source if binary packages needed to embed timestamps.

In the course of the discussion, we realized that using `.changes`
files [6] as the input of a rebuild process would be both impractical
and abusing their intended meaning. So we started sketching the content
of `.buildinfo` files which would be stored by the archive and would
contain everything needed to perform a rebuild.

We also discussed the choice of a canonical build path, the current set
of patches that were used for the previous rebuild experiments, a
post-processing addition to debhelper called “dh_strip_nondeterminism”,
and sbuild support.

What is pretty amazing is the amount of work that was done in the
following day leveraging on the discussion:

 * The `.buildinfo` format has been specified [7] on the wiki,
   and reviewed by a couple of people.
 * The patches to make the file order in control and data archives
   stable (#719845) have been rebased and submitted again to the
   BTS [8].
 * A new helper `dh_fixmtimes` was written and submitted to the BTS as
   #759886 [9]. This replaces the previous changes that were targetting
   `dpkg-deb` to achieve a similar result.
 * The patches which enables `dpkg-deb` to produce deterministic ar
   headers when used with `dpkg-buildpackage` has been rebased,
   updated, and submitted to the BTS as #759999 [10].
 * An addition to `dh_strip` to remove non-determistic data from
   static libraries was written and submitted to the BTS as
   #759895 [11]. This will be used instead of building binutils with
   `--enable-deterministic-archives` which has potential to make
   some build systems highly unhappy.
 * A patch for `dh-python` has been written to get a stable order
   in the generated control fields. Submitted to the BTS as
   #759231 [12].
 * `dh_genbuildinfo` [13] is a new helper that will use the output of
   `dpkg-genchanges` and `dh_buildinfo` [14] to generate almost correct
   `.buildinfo`. Good enough as a starting point for more tests.
 * A tool to remove non-determinism from Jar files, `sortjar` [15] is
   almost packaged. Some upstream clarifications are still missing.
 * `strip-nondetermism` [16] is meant to be called by debhelper and to
   remove non-deterministic data from various file formats. As it
   stands, it already supports gzip, zip and jar.
 * There has been a quick transmission on how to use the
   `cloud-scripts` [17] which make possible archive-wide rebuilds using
   EC2 VMs.
 * Discussions with Octave [18] and groff [19] upstreams have been
   started.
 * `pod2man` can now be made to have reproducible timestamps
   (#759405 [20]).
 * Discussions have happened on the base timestamp of files patched
   with dpkg 3.0 (quilt) format in #759404 [21].

That's a lot of great work (and I'm probably missing a thing or two),
so congrats to everybody involved!

We are close to be able to perform another archive-wide rebuild using
the new set of patches and assumptions. What is missing is an
(even half-working) `srebuild` script. To the best of my understanding,
Geoffrey Thomas will be working on it.

There's a new IRC channel, so feel free to join #debian-reproducible on
OFTC.

I have been trying to keep the TODO list [22] on the wiki page
up-to-date. If you want to enjoy this new rush of energy, come and help!

  [1]: https://debconf14.debian.org/
  [2]: https://summit.debconf.org/debconf14/meeting/78/reproducible-builds-for-debian/
  [3]: http://meetings-archive.debian.net/pub/debian-meetings/2014/debconf14/webm/Reproducible_Builds_for_Debian_a_year_later.webm
  [4]: http://reproducible.alioth.debian.org/presentations/2014-08-26-DebConf14.pdf
  [5]: https://summit.debconf.org/debconf14/meeting/79/reproducible-builds-for-debian-finding-solutions/
  [6]: https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debianchangesfiles
  [7]: https://wiki.debian.org/ReproducibleBuilds#Recording_the_environment
  [8]: https://bugs.debian.org/719845#61
  [9]: https://bugs.debian.org/759886
 [10]: https://bugs.debian.org/759999
 [11]: https://bugs.debian.org/759895
 [12]: https://bugs.debian.org/759231
 [13]: http://anonscm.debian.org/cgit/reproducible/debhelper.git/commit/?h=pu/reproducible_builds&id=a2a95893
 [14]: https://tracker.debian.org/dh-buildinfo
 [15]: http://anonscm.debian.org/cgit/reproducible/sortjar.git
 [16]: http://anonscm.debian.org/cgit/reproducible/strip-nondeterminism.git/
 [17]: http://anonscm.debian.org/cgit/collab-qa/cloud-scripts.git
 [18]: https://savannah.gnu.org/bugs/?43087
 [19]: https://lists.gnu.org/archive/html/groff/2014-08/msg00112.html
 [20]: https://bugs.debian.org/759405
 [21]: https://bugs.debian.org/759404
 [22]: https://wiki.debian.org/ReproducibleBuilds#Useful_things_you_.28yes.2C_you.21.29_can_do

-- 
Lunar                                .''`. 
lunar at debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20140902/811536b8/attachment.sig>

More information about the Reproducible-builds mailing list