[Pkg-libburnia-devel] Reproducible patches for libisoburn and libisofs

[Thomas Schmitt]

Hi,

meanwhile i remember that xorriso ISO production already went through a
Debian Reproducible Builds discussion:
  http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00006.html

This discussion yielded that your xorriso-1.3.2 with SOURCE_DATE_EPOCH
will not work reproducibly if you present the same input file tree on
different hard disks or after copying it to another location in the
global tree.
Moving the tree root should be ok, though.

If you need reliable effect of SOURCE_DATE_EPOCH right now, then you have
to port your changeset to xorriso-1.4.2 or 1.4.4.

1.4.4 is in Debian testing or available as standalone tarball at

  https://www.gnu.org/software/xorriso/#download

This tarball contains the libraries which will get linked statically
to form a xorriso binary that depends only on vanilla system libraries.
No interference with installed libburn.so, libisofs.so, libisoburn.so.

-------------------------------------------------------------------------

i wrote:
> >      --set_all_file_dates "$timestring"

Chris Lamb wrote:
> Assuming it also sets iso_write_opts_set_always_gmt, yes.

It is default of xorriso

  $ xorriso -no_rc -status -compliance
  ...
  -compliance clear:...:always_gmt:...

because timezones are a swamp on their own.
But yes, --set_all_file_dates would include xorriso command
  -compliance always_gmt


> I'm not 100% convinced though. I'm a little loathed to drop the
> SOURCE_DATE_EPOCH handling in general

It is a quick and global solution. But it does not fit into the libraries'
traditions. xorriso is controlled by commands, the libraries by API.

The reason for my approach is that xorriso-1.4.4 can already do nearly
everything you want to achieve.


> zooming out a bit from these suggested new switches,

One single suggested switch, to be exacting. :))
And only if really file timestamps shall not matter.


> it does make it somewhat difficult for the fly-by developer to build
> a reproducible ISO

I do not deem the overriding of file timestamps a normal goal of
reproducibility. In general the POSIX attributes of a file are as significant
as is the file content or the softlink target string.

It's rather the automatically generated timestamps of an ISO which i would
call an obstacle to reproducibility of ISO production. (Or random sequences
of file content extents. See below.)

Of course, if your use case makes it desirable to override the file timestamps
then "filesystem manipulator" comes in handy. So i currently strive to put
your use case on that track.
 

> developer [...] will need to hunt down and learn [...] all the
> various command-line arguments

A valid point, indeed.
Currently there is the need to use command -volume_date "uuid" or -as mkisofs
option --modification-date=, with the bug-like problem that isohybrid gets
an MBR id from current time.
(I plan to derive the isohybrid MBR id from the --modification-date=
 setting if present.)


> As I understand the code in Xorriso_make_iso_write_opts,
> isoburn_igopt_set_scdbackup_tag appears to be called unconditionally

It copies to libisoburn the scdbackup_tag_name, which is empty by default.
This causes libisoburn not to call iso_write_opts_set_scdbackup_tag().
Else you'd get a message at the end of the run:
  xorriso : NOTE : scdbackup tag written : x_y B60805.122358 3111051 397e84253510b0dc3d8f17178585e226

The documentation indeed does not state that empty name means no tag.
I changed this now.


This reminded me of the Debian Reproducible Builds discussion:
  http://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00006.html
where the scdbackup tag would have been noticed during image comparison.

That comparison yielded that xorriso-1.3.2 from Debian "stable" is quite
hopeless in respect to reproducibility, anyways:
A change in sequence of directory traversal on hard disk could cause
a change in the sequence of data file content extents. (See mentioning of
Red-Black tree in the discussion.)

Hopefully fixed by release 1.4.2. No need to manipulate file sorting weights
as discussed in the thread.


> Sure, but you would you really be backing up with SOURCE_DATE_EPOCH defined?

You did not convince me of SOURCE_DATE_EPOCH yet. (Might change if
i stumble over a target->now problem which --modification-date= does
not solve.)

Anyways, if i want a scdbackup tag, then one with real date.
On the other hand without special option no scdbackup tag emerges.

So i think your use case and mine are wide enough apart already.


> Would it matter if their MBR ID clashed? ;-)

I really don't know. hpa was not overly verbous wbout the meaning of
this field in his MBR producer isohybrid.pl.
So i decided to be cowardish and scramble it.

----------------------------------------------------------------------

I am now testing a MBR id solution based on --modification-date=.

Then i finish my assessment of target->now  whether it can endanger
reproducibility despite --modification-date=.


Have a nice day :)

Thomas