Bug#882511: dpkg-buildpackage: should allow caller to force inclusion of source in buildinfo

[Guillem Jover]

Control: forcemerge 882511 1068483

Hi!

After replying to Adrian's report, I recalled there being a previous one
that was similar, and then recalled that I had an even older branch that
implemented a potential solution for this. See below.

On Thu, 2017-11-23 at 16:23:29 +0100, Ximin Luo wrote:
> Package: dpkg-dev
> Version: 1.19.0.4
> Severity: wishlist
> Tags: patch

> dpkg-buildpackage currently does not automatically list the source .dsc nor
> its hash in the call to dpkg-genbuildinfo when doing a binary-only build. This
> is understandable because in a binary-only build, dpkg-buildpackage does not
> have any concept of a source package and therefore does not know (and cannot
> verify) if the working tree was actually generated from any .dsc or not.
> 
> However, the caller knows this information, and it is useful for reproducible
> builds to track exactly which (i.e. hash-wise) source code generates which
> binary packages. So it should be possible for the caller to tell
> dpkg-buildpackage, "yes please do include the .dsc hash in the buildinfo, I am
> telling you it is correct, you can assume this safely".
> 
> Tools like sbuild/pbuilder could then do this, as well as users or rebuilders.
> 
> The attached patch implements this in the simplest way possible. It allows the
> caller to run something like:
> 
>   $ dpkg-buildpackage --no-sign -b --buildinfo-option=--build=full
> 
> The resulting $pkg_$ver_$arch.buildinfo then contains the .dsc and its hash.
> 
> However this requires the caller to know which option to pass, which would either be
> 
>   --buildinfo-option=--build=full
>   --buildinfo-option=--build=any,source
>   --buildinfo-option=--build=all,source
> 
> depending on whether the original build request (to dpkg-buildpackage) was a -b, -B, or -A.
> 
> For this reason, it may be better (more usable) to add a --force-source-in-buildinfo
> flag (or similar name) and when this is switched on, do this instead:
> 
> -push @buildinfo_opts, "--build=$build_types" if build_has_none(BUILD_DEFAULT);
> +push @buildinfo_opts, "--build=$build_types,source" if build_has_none(BUILD_DEFAULT);
> 
> Let me know if you like this idea and I'll be happy to implement that instead of
> the attached patch.

The problem with this solution is that it is prone do accidental use,
as it is very easy for a user to unknowingly have recreated the sources
from a locally extracted tree (be that modified or not).

On Sat, 2024-04-06 at 02:57:40 +0200, Guillem Jover wrote:
> On Sat, 2024-04-06 at 02:56:02 +0300, Adrian Bunk wrote:
> > Package: dpkg-dev
> > Version: 1.22.6
> > Severity: normal
> > X-Debbugs-Cc: reproducible-builds at lists.alioth.debian.org
> 
> > A thought I already wrote in a recent debian-devel discussion:
> > 
> > In theory source package filenames should be eternally and globally
> > unique, but in practice there are cornercases where this assumption
> > might break like for example:
> > - *stable-security does not currently have a copy of the sources
> >   in the main archive, one always have to upload the source archive
> >   there and this might accidentally be a different orig.tar
> > - dak does not keep an eternal history of everything it ever knew,
> >   e.g. RM and later re-NEW of a source version might have a different
> >   source .orig.tar or even different sources for a Debian revision
> > - Debian and Ubuntu might have different orig.tar for the same version,
> >   if Ubuntu updated a package before Debian did, or with packages
> >   were development is completely independent in Debian and Ubuntu
> >   (e.g. OpenStack, KDE)
> > 
> > The reason for different files might be as trivial as "git archive"
> > not always producing the same output when running in different
> > environments, e.g. the autogenerated tarball for a git tag on Github
> > might have different checksums depending on whether it is downloaded
> > today or next year despite identical contents due to slightly
> > different gzip compression.
> > 
> > Should buildinfo files contain the hashes of the source package,
> > to clearly define what sources have been used?
> 
> Ideally? Yes, and I think we considered that at the time when we
> introduced the .buildinfo files. Although a ref to the .dsc does get
> included if the build is also creating the source package.
> 
> The problem is that when dpkg-buildpackage is not building the source
> package, there is no guarantee the source package is going to be
> present, or that if it is present it matches what is currently being
> built from the working directory.

I've now finished the change I had in that branch, which implements
support so that dpkg-buildpackage can be passed a .dsc or a source-dir,
and in the former will first extract it, and for both then it will
change directory to the source tree. If it got passed a .dsc then it
will instruct dpkg-genbuildinfo to include a ref to it.

Which I think accomplishes the requested behavior in a safe way? I've
attached what I've got, which I'm planning on merging for 1.22.7. I'll
probably split that into two commits though before merging.

Thanks,
Guillem
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-dpkg-buildpackage-Add-support-for-building-from-a-.d.patch
Type: text/x-diff
Size: 5440 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20240409/eb55bc43/attachment.patch>