package uploaded to our repo

[Mattia Rizzolo]

They told me it was not totally clear what happens here, why I did this
upload, what triggered the chanegs I did, and why last night.

On Tue, Sep 20, 2016 at 10:55:55PM +0000, Mattia Rizzolo wrote:
> dpkg_1.18.10.0~reproducible1.dsc has just been uploaded to https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

The changelog against 1.18.10.0~reproducible0 (i.e. what we have been
running in the last months) is this:
  * quite a lot cleanup of the .buildinfo implementation by guillem
  * rewrite of the "deterministic modes in control.tar" by guillem
  * temporary backward compatibility for --buildinfo-id after guillem
    renamed the option so we can transition over our tools
  * "rewrote" (within quotes as it's really silly) the "disable
    Environment field from .buildinfo" thing, after the big modification
    happened to dpkg-genbuildinfo.

The trigger:
yesterday evening a bystander I don't know about came in the IRC
channel, noticing how the diffoscope report of newly built packages in
the last days was totally incomplete, only showing the diff between the
.buildinfo files instead of unpacking over into the .debs.
After some minutes of debugging I discovered this was caused by a change
in pbuilder, which started to behave sanely¹.
The buildinfo spec says² that a .dsc must be included in
Checksums-Sha256 if that one is present.  Regardless of whether this is
a good choice, this is broken in multiple ways:
 * on the idea:
   + a .dsc could not be present for a binary build
   + until recently source builds were not reproducible, so an already
     present .dsc would be overwritten during a full build, and the
     .buildinfo would record the new one instead of the original
 * lexically: that field contains a list of built artifacts that have
   been built and distributed, putting a .dsc in it goes against this
   definition.
The problem is that we do a binary-only build (-b), so the changes file
does not contain the .dsc, but that one was referenced in .buildinfo
nonetheless.  Until pbuilder 0.226 the .dsc would have been copied over
even if not referenced in .changes; starting with 0.226 it's not copied
anymore.
This broke diffoscope, as it considers a .buildinfo referencing a
non-existing file as an invalid DotBuildinfoFile, and therefor falling
back to TextFile.

The fix:
I went to the our dpkg sources and see why it does that, and
individuated the interesting part.  Then I remembered that guillem did a
bunch of work on it too, and thought about looking whether some of the
stuff could be merged in our tree (mostly for wider testing).  Turned
out that he found such thing weird too, and so guarded that part of code
with an `if` that would get executed only in the case of a source build.
Hence I got in touch with him, had his patches rebased, fixed a couple
of glitches, and incorporated.  I didn't want to squash the commits, as
I would still like to keep some history of the evolution of the thing
still.

What I did:
* rolled back the history like we do with every new dpkg release
* substituted the patches for the "deterministic modes in control.tar"
  with one wrote by guillem
* appended the patches to the .buildinfo implementation from guillem
* "rewrote" (within quotes as it's really silly) the "disable
  Environment field from .buildinfo" thing, after the big modification
  happened to dpkg-genbuildinfo (the original one was from ntyni)
* added back a temporary --buildinfo-identifier flag in
  dpkg-buildpackage after guillem renamed id.
* built, tested, rebuilt, uploaded to our repo.


Next steps:
it would be great if somebody could figure what's the real gain of
having .dsc in Checksums-Sha256.  Also consider that within the context
of a single sane archive (as in: files once landed don't change) like
Debian's such trick is not needed, as a source package can already be
identified by other information already stored in .buildinfo like Source
and Version.



Thanks for reading this far,
Mattia (hoping to don't have to write any such long email for at least
some hours…)

¹ IOW: https://bugs.debian.org/492312
  https://anonscm.debian.org/git/pbuilder/pbuilder.git/commit/?id=806db12
² https://wiki.debian.org/ReproducibleBuilds/BuildinfoSpecification#buildinfo_field_descriptions

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20160921/31b6d3f0/attachment.sig>