[Reproducible-builds] generating reproducible ISOs with xorriso

[Daniel Kahn Gillmor]

On Thu 2015-06-04 09:53:11 -0400, Thomas Schmitt wrote:
> I saw some sin-list page about packages shortly after
> xorriso-1.4.0 was released.
> It complained about libburn's doc/doxygen.conf.in which i hope to
> have fixed by
>   http://libburnia-project.org/changeset/5446

Thanks!  We don't think it's a sin, but you're absolved anyway :)

> Afaik, GRUB2 uses the ISO 9660 Modification Date as UUID of the ISO.
> There are several ISO 9660 timestamps. One would have to fixate
> them by explicit xorriso commands.

We can certainly give patches to grub to do that explicitly (and we
might come ask questions here if we can't figure out the right
commands).

If we can identify the specific commands (beyond what you point out
below), would there a general interest upstream in something like a
--reproducible=TARGETDATE, which would make these decisions explicitly
within xorriso, rather than every invocation having to learn the full
suite of variations?  It's probably easier to ask grub upstream (and
other xorriso users) to add a single flag than to add a complex set.

> The dull ISO 9660 names get sorted and their data content
> is stored in the sequence of that sorting. So if the tree of
> paths stays unaltered, the sequence of data extents should
> stay unaltered too.
> The mapping from Rock Ridge names to ISO 9660 names is supposed
> to be deterministic in this case. (It is complicated, though.)

Hm, this *doesn't* seem to be what's happening with the grub ISO:

https://reproducible.debian.net/dbd/unstable/amd64/grub2_2.02~beta2-23.debbindiff.html

for example, if i'm reading that right, then in
/usr/lib/grub-rescue/grub-rescue-cdrom.iso, we have
/BOOT/GRUB/GRUB.CFG;1 with extent 2316 in the first build, and extent 47
in the second.

Am i reading that right?  I think the contents of the files in the ISO
should already be stable across builds, and it's just the extents that
are changing.

Note that the build-host filesystems varied, and the generated files may
have been placed in the build-host filesystems at different times on
each build.  Maybe it's still sorting by dirent order instead of by
ISO9660 names?

> The second number after "Bootoff" is the block address of the
> data extent of an El torito boot image data file. Thus covered
> by the sorting if all file sizes are unaltered.

OK, that's good -- we've seen this pattern before, where fixing some
known variations resolves some more complex variations (GNU buildid is a
classic example).  I'm happy to ignore that until we get the extent
business sorted, and then maybe it will go way on its own :)

> I riddle what the first number might be.
> debian-7.7.0-amd64-netinst.iso has "Bootoff 350 848".
> The first El Torito boot image begins at block 848 decimal.
> Maybe just a disguised hex ? 0x350 = 848 

yes, i think that's correct.  You can also see those figures in hex in
/boot.catalog

> Well possible.
> After the usual xorriso -as mkisofs options of debian-cd
> one could add native xorriso commands. Use a timestamp
> format that is not prone to time zone computation. E.g.
> the native format of ISO 9660:
>
>   timestamp="2015060415363300" 
>   xorriso -as mkisofs ...all.the.debian-cd.options... \
>   -- \
>   -volume_date "c" "$timestamp" \
>   -volume_date "m" "$timestamp" \
>   -volume_date "x" "default" \
>   -volume_date "f" "default" \
>   -alter_date "b" "$timestamp" / -- \
>   -alter_date "c" "$timestamp" / -- \

thanks, this is very useful!

> Filesystem times "x" = Expire and "f" = Valid Since are best set
> to the default value 000...000. Just in case anybody cares to
> obey them.

i had no idea that these flags even existed, scary...

> One will have to make experiments with the ISO production
> scripts which shall be used. Identify the deviating spots
> in the ISO, their meaning, and their origin.
>
> I am willing to help with advise and/or necessary new features.

Thanks for your help already, Thomas, and for your willingness to help
in the future.  Any thoughts about the extent issue described above?

Regards,

   --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 948 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20150604/444889c1/attachment.sig>