[Reproducible-builds] Bug#769844: linux: please make linux build reproducibly

Tue May 12 20:00:53 UTC 2015

On Wed, 2015-01-07 at 20:49 +0100, Jérémy Bobbio wrote:
[...]
> With the attached patch, linux can be built reproducibly with the
> current experimental framework. The following changes have been
> introduced since the previous patch:
> 
>  * Files in the source tarball are added in a stable order.
>  * kernel-doc is made deterministic when parsing `@foo()` in
>    descriptions.
>  * The last set of manpages when generating manpages from files
>    referenced in multiple sections of the documentation is
>    deterministic.
> 
> Some `@foo()` constructs in the kernel should probably be fixed to be
> just `foo()`. It's unclear what should be used for functions passed as
> parameters.

Hardly anyone seems to use the output of kernel-doc so the sources are
of very variable quality.  Function pointers aren't functions so I think
they should be @foo.

Moving on to your patch:

Most of the added patches should go upstream and therefore belong under
bugfix/all/ not debian/.  Please can you also add a sign-off to these
(after reading what Documentation/SubmittingPatches says about that).

> diff -Nru linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
> --- linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch       1970-01-01 01:00:00.000000000 +0100
> +++ linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch       2015-01-07 16:34:00.000000000 +0100
> @@ -0,0 +1,55 @@
> +Description: generate manpages in deterministic order
> + When header files are referenced multiple times in different
> + documentation parts, manpages for the various functions will also
> + be generated multiple times.
[...]

I don't yet understand this, so I'll come back to it later.

> diff -Nru linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
> --- linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch   1970-01-01 01:00:00.000000000 +0100
> +++ linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch   2015-01-07 16:43:31.000000000 +0100
> @@ -0,0 +1,21 @@
> +Description: parse kernel-doc deterministically
> + Regular expressions for highlights in kernel-doc are stored in a Perl
> + hash. These hashes are ordered differently for each Perl run. This will
> + prevent kernel-doc to behave deterministically when parsing
> + “@foo()” as in some runs it will be interpreted as a parameter and
> + in the others it will be interpreted as a function.
> + .
> + We now sort the %highlights hash to get the same behavior on every run.
> +Author: Jérémy Bobbio <lunar at debian.org>
> +
> +--- linux-3.16.7-ckt2.orig/scripts/kernel-doc
> ++++ linux-3.16.7-ckt2/scripts/kernel-doc
> +@@ -2586,7 +2586,7 @@ $kernelversion = get_kernel_version();
> + 
> + # generate a sequence of code that will splice in highlighting information
> + # using the s// operator.
> +-foreach my $pattern (keys %highlights) {
> ++foreach my $pattern (sort keys %highlights) {
> + #   print STDERR "scanning pattern:$pattern, highlight:($highlights{$pattern})\n";
> +     $dohighlight .=  "\$contents =~ s:$pattern:$highlights{$pattern}:gs;\n";
> + }

OK, applied.

> diff -Nru linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
> --- linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch  1970-01-01 01:00:00.000000000 +0100
> +++ linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch  2015-01-05 14:50:58.000000000 +0100
> @@ -0,0 +1,26 @@
> +Description: parse Debian KBUILD_BUILD_TIMESTAMP in gen_initramfs_list.sh
> + gen_initramfs_list.sh assumes that KBUILD_BUILD_TIMESTAMP only contains a
> + date. In Debian, it's a longer string looking like:
> + “Debian 3.16.7-2 (2014-11-06)”. Here we adapt the script to parse Debian
> + format.
> + .
> + This change is necessary to have reproducible builds as the date will
> + then be used as mtime for files in initramfs archive instead of the
> + current time.
> +Author: Jérémy Bobbio <lunar at debian.org>
> +Bug-Debian: https://bugs.debian.org/769844
> +Forwarded: not-needed
> +
> +--- linux-3.16.7-ckt2.orig/scripts/gen_initramfs_list.sh
> ++++ linux-3.16.7-ckt2/scripts/gen_initramfs_list.sh
> +@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
> +       if [ -z ${cpio_file} ]; then
> +               timestamp=
> +               if test -n "$KBUILD_BUILD_TIMESTAMP"; then
> +-                      timestamp="$(date -d"$KBUILD_BUILD_TIMESTAMP" +%s || :)"
> ++                      source_date=$(echo "$KBUILD_BUILD_TIMESTAMP" |
> ++                                      sed -e 's/.*(\([0-9-]\+\)).*/\1/')
> ++                      timestamp="$(date -d"$source_date" +%s || :)"
> +                       if test -n "$timestamp"; then
> +                               timestamp="-t $timestamp"
> +                       fi

I did something more like what Ian suggested instead of this.

[...]
> --- linux-3.16.7-ckt2/debian/rules.real 2014-11-04 05:41:34.000000000 +0100
> +++ linux-3.16.7-ckt2/debian/rules.real 2015-01-07 14:37:52.000000000 +0100
> @@ -87,7 +87,9 @@
>         rm -rf '$@' '$(DIR)'
>         $(call copy_source,$(DIR))
>         chmod -R u+rw,go=rX '$(DIR)'
> -       cd '$(BUILD_DIR)'; tar -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 'linux-source-$(UPSTREAMVERSION)'
> +       cd '$(BUILD_DIR)'; find 'linux-source-$(UPSTREAMVERSION)' -type f -print0 | \
> +               LC_ALL=C sort -z | tar --mtime='$(SOURCE_DATE)' --null -T - \
> +                       -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 
>         rm -rf '$(DIR)'
>  
>  $(BUILD_DIR)/linux-patch-$(UPSTREAMVERSION)-%.patch.xz: $(STAMPS_DIR)/source_none $(STAMPS_DIR)/source_%

The '-type f' is wrong because there are symlinks that need to be
included.  And why exclude directories?

> @@ -202,7 +204,7 @@
>         cd $(DIR)/Documentation/DocBook; \
>         find * -name '*.html' -print | \
>         cpio -pd --preserve-modification-time '$(CURDIR)/$(OUT_DIR)/html'
> -       gzip -9qfr $(OUT_DIR)/Documentation
> +       gzip -9nqfr $(OUT_DIR)/Documentation
>         +$(MAKE_SELF) install-base
>  
>  install-manual: PACKAGE_NAME = linux-manual-$(VERSION)

Applied this bit.

Ben.

-- 
Ben Hutchings
For every action, there is an equal and opposite criticism. - Harrison

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 811 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20150512/e9630684/attachment.sig>