[Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Thomas Schmitt]

Hi,

> Just to confirm, this is due to non-determinstic extent numbering due to
> sorting?

It is deterministic but depends on hard-to-influence and unexpected
parameters.

Daniel Kahn Gillmor detected it when comparing images on byte level.
First tail tip of the problem:
  http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00009.html
  "we have /BOOT/GRUB/GRUB.CFG;1 with extent 2316 in the first build,
   and extent 47 in the second." ["extent" = block address of data]
  "Note that the build-host filesystems varied"

My assessment:
  http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00012.html
  "The files are grafted into a red-black tree according to their inode
   and device numbers on hard disk. This is done to merge hardlinks.
   The tree is then serialized into an array which gets sorted according
   to the weight values. The extent addresses are then assigned according
   to the sorted sequence."

Daniel began to work around the problem by massive explicite definitions
of sort weight. But the "weight" concept is intended to move a few file
extents near to the start of the image, not to define each file's rank.
So i began to think Red-Black tree apart from extent addresses:
  http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00015.html
  "I am currently thinking towards keeping the red-black tree
   for hardlink detection but to not take the array from
   that tree. Instead the array would be produced by traversing
   the ISO 9660 filesystem tree and skipping the files of which
   the IsoFileSrc object was already put into the array by a
   previous node in the tree."

I tested this heavily ... until the radiator of my southbridge popped off.
Testing continued on a new machine, but local bzr needed some teaching.
So my local sandbox user's id shows up as author of
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/revision/1208
  "Sorted the data file content extents by ECMA-119 tree rather than
   by the red-black tree which shall consolidate files with identical
   source object. Discovered and repaired a flaw in transitivity of
   iso_stream_cmp_ino()."
Ouch, in hindsight.

The radiator mishap made me a Debian user with a well usable qemu. So i had
no more excuse to let the libburnia packages rot on Debian. Since september
2015, i prepare their ./debian directories and propose them to Dominique
Dumont for upload. Thus "testing" is up to date.


> In my experience of hunting
> down reproducible issues, it *can* turn into a bit of a rabbit hole..

I was quite surprised when the extent address problem popped up.
The red-black tree and the array is by my predecessor Vreixo Formoso.
But the transitivity issue was caused by me when piling up features.
So the red-black tree did not even reliably collect all identical file
content sources.

At least in my discussion with Daniel, all riddles were solved. No unexpected
changes happened between builds any more. But we did not test isohybrid.

------------------------------------------------------------------------

> > I do not deem the overriding of file timestamps a normal goal of
> > reproducibility.

> Well, if you are using any tool that generates files (for example,
> the tool that's actually building the files to put on your ISO!), then
> these files will have varying timestamps between builds, no?

It obviously depends on which stage of input processing you want differences
to actually make a difference in the resulting ISO.

In my view as xorriso programmer, the stage would be the transition from
input files on hard disk to files in the ISO. I want to keep this narrow
idea of reproducibility-demanding input.
But to our luck, the "manipulator" aspect of xorriso offers an opportunity
to express your idea of equivalent input.

I have to put the plight on you to trigger this by an extra option.


> > "filesystem manipulator" comes in handy
> Can you briefly elaborate/confirm what you mean by that? Post-process the
> ISO with the manipulators rather than do this during build?
 
  $ xorriso
  GNU xorriso 1.4.4 : RockRidge filesystem manipulator, libburnia project.

Note the words "filesystem manipulator" in its self-description.

Although most people use it by its mkisofs emulation, this is only a
part of its capabilities. It does not just build an ISO. It can change
names and attributes, remove files, apply data filters, extract files
from the ISO (even before it was written), ...

In your case i propose to use the existing functionality to change the
files in the ISO model before the image gets produced.

In other cases one loads the ISO, manipulates its files, and adds the
manipulation result as further session to the end of the image or medium.
Aka multi-session with session history and opportunity to mount older
image states.
Or one uses an existing ISO as base input for a new ISO which then contains
the manipulation result.


> Further to this, someone once did a survey of all the Debian-area ISO/chroot
> build tools and found there were at least ten.

I regularly shed my mockery over the Debian ISO producing community
because they still did not manage to get all mini.iso and LiveCD to
the boot capabilities of debian-cd.


> All of these would need separate patching to achieve reproducible ISOs,

According to my assessment they all would need --modification-date=
in order to prevent automatic timestamps. Many need a xorriso with the
isohybrid MBR id bug fixed (valgrind is watching it now while it repacks
my ISO collection).

For a bootable installation ISO it might be a safe guess that file
timestamps do not matter and that ownership or permissions don't either.
So one would probably make sure to apply -r and --set_all_file_dates.

------------------------------------------------------------------------

valgrind has ended its work.
But rebuilding a Debian mini.iso twice still yields a difference in the
timestamps of the Boot Catalog file. Caused by
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/view/head:/libisofs/eltorito.c#L300
not even by a target->now, but plain
    now = time(NULL);

Did you omit option -c in your test run ?
Else it would riddle me, why this did not hit you.
This node is created so late in the production process, that no manipulation
would reach it. I will have to use image modification time.

debian-8.4.0-amd64-netinst.iso shows even more differences on repacking:

- In GPT Header block at offset 512 bytes
    16 -  19 | head_crc   | CRC-32 of this header while head_crc is 0
    56 -  71 | guid       | Disk GUID, Random
    88 -  91 | p_arr_crc  | CRC-32 of the partition array

- In GPT array at offset 6144 bytes and 6272
    16 -  31 | part_guid  | Unique partition GUID, Random

- And the whole thing again in the GPT backup.

GPT is a consequence of Matthew Garrett's EFI isohybrid layout.
Actually the GPT is unused, because the MBR is not "protective".

Oh well ...


Have a nice day :)

Thomas