[Reproducible-builds] Bug#805321: debian-installer: builds unreproducible netboot images

Cyril Brulebois kibi at debian.org
Tue Nov 17 00:36:27 UTC 2015 

Hi,

Steven Chamberlain <steven at pyro.eu.org> (2015-11-17):
> Attached is my jessie-kfreebsd implementation.  As I said, it should be
> much cleaner to implement this in sid with newer GNU tar.
> 
> Regards,
> -- 
> Steven Chamberlain
> steven at pyro.eu.org

> diff --git a/build/Makefile b/build/Makefile
> index ec5a084..6261a4d 100644
> --- a/build/Makefile
> +++ b/build/Makefile
> @@ -56,7 +56,7 @@
>  # Add to PATH so dpkg will always work, and so local programs will be found.
>  PATH := util:$(PATH):/usr/sbin:/sbin
>  EATMYDATA = $(shell which eatmydata 2>/dev/null)
> -GZIP = $(shell which pigz gzip | head -1)
> +GZIP = $(shell which pigz gzip | head -1) -n

I think I already added -n to a bunch of calls. Not sure whether adding
it here once and for all would be better than adding it where it's
missing though. Anyway, not my biggest question/comment/concern here.

>  # We don't want this to be run each time we re-enter.
>  ifndef DEB_HOST_ARCH
> @@ -149,7 +149,7 @@ MFSROOT_LIMIT := 68m
>  endif
>  
>  define mkfs.ufs1
> -  sh -c 'makefs -t ffs -s $(MFSROOT_LIMIT) -f 3000 -o minfree=0,version=1 $$0 ${TREE}'
> +  sh -c 'makefs -t ffs -T $(SOURCE_DATE_EPOCH) -s $(MFSROOT_LIMIT) -f 3000 -o minfree=0,version=1 $$0 ${TREE}'

Straightforward enough.

>  define e2fsck
> @@ -803,7 +803,14 @@ $(TEMP_MINIISO): $(TEMP_BOOT_SCREENS) arch_miniiso
>  
>  # various kinds of information, for use on debian-cd isos
>  $(DEBIAN_CD_INFO): $(TEMP_BOOT_SCREENS) $(TEMP_CD_INFO_DIR)
> -	(cd $(TEMP_CD_INFO_DIR); tar czf - .) > $@
> +	# Clamp timestamps to be no newer than last changelog entry, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
> +	find $(TEMP_CD_INFO_DIR) -newermt "@$(SOURCE_DATE_EPOCH)" -print0 | xargs -0r touch --no-dereference --date="@$(SOURCE_DATE_EPOCH)"
> +	# Create tarball with files sorted in a stable order, see
> +	# https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs
> +	# and without timestamp in the gzip header, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
> +	( cd $(TEMP_CD_INFO_DIR) && find . -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -czf -) > $@
>  	update-manifest $@ $(MANIFEST-DEBIAN_CD_INFO)

Once.

>  # a directory full of files for netbooting
> @@ -822,7 +829,14 @@ $(NETBOOT_TAR): $(TEMP_NETBOOT_DIR)
>  	# Create an version info file.
>  	echo 'Debian version:  $(DEBIAN_VERSION)' > $(TEMP_NETBOOT_DIR)/version.info
>  	echo 'Installer build: $(BUILD_DATE)' >> $(TEMP_NETBOOT_DIR)/version.info
> -	(cd $(TEMP_NETBOOT_DIR); tar czf - .) > $@
> +	# Clamp timestamps to be no newer than last changelog entry, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
> +	find $(TEMP_NETBOOT_DIR) -newermt "@$(SOURCE_DATE_EPOCH)" -print0 | xargs -0r touch --no-dereference --date="@$(SOURCE_DATE_EPOCH)"
> +	# Create tarball with files sorted in a stable order, see
> +	# https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs
> +	# and without timestamp in the gzip header, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
> +	( cd $(TEMP_NETBOOT_DIR) && find . -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -czf -) > $@

Twice.

>  	update-manifest $@ $(MANIFEST-NETBOOT_TAR) $(UDEB_LISTS)
>  
>  $(TEMP_BOOT_SCREENS): arch_boot_screens
> diff --git a/build/config/x86.cfg b/build/config/x86.cfg
> index 3caadd2..b0fc9a2 100644
> --- a/build/config/x86.cfg
> +++ b/build/config/x86.cfg
> @@ -332,6 +332,11 @@ arch_miniiso: x86_syslinux x86_grub_efi
>  			| todos > $(TEMP_CD_TREE)/win32-loader.ini; \
>  	fi
>  
> +	# Clamp timestamps to be no newer than last changelog entry, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
> +	find $(TEMP_CD_TREE) -newermt "$(SOURCE_DATE)" -print0 \
> +	 | xargs -0r touch --no-dereference --date="$(SOURCE_DATE)"
> +

Refraining from writing “almost thrice”. [XXX]

>  	if [ "$(GRUB_EFI)" = y ]; then \
>  		xorriso -as mkisofs -r -J -b isolinux.bin -c boot.cat \
>  			-no-emul-boot -boot-load-size 4 -boot-info-table \
> diff --git a/debian/changelog b/debian/changelog
> index 42aed37..09c8a02 100644
> --- a/debian/changelog
> +++ b/debian/changelog
> @@ -1,3 +1,21 @@
> +debian-installer (20150422+kbsd8u2) jessie-kfreebsd; urgency=medium
> +
> +  * Improve reproducibility of debian-installer netboot images:
> +    (Closes: #805321)
> +    - clamp timestamps in the d-i ramdisk to be no later than
> +      the most recent debian/changelog entry of this package
> +      - raise makefs dependency on >= 20100306-5+kbsd8u1
> +    - clamp timestamps in the mini.iso similarly
> +    - clamp timestamps in the netboot tarball;  store files in a
> +      stable order
> +    - clamp timestamps in the cd info tarball;  store files in a
> +      stable order
> +    - clamp timestamps in the output debian-installer-images tarball;
> +      store files in a stable order
> +    - disable timestamps in gzip output (e.g. initrd.gz and tarballs)
> +
> + -- Steven Chamberlain <steven at pyro.eu.org>  Tue, 10 Nov 2015 21:38:46 +0000
> +
>  debian-installer (20150422+kbsd8u1) jessie-kfreebsd; urgency=medium
>  
>    * Rebuild using udebs from the jessie-kfreebsd suite, also using
> diff --git a/debian/control b/debian/control
> index 100ca5a..6f4df5b 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -162,7 +162,7 @@ Build-Depends:
>  #		architectures if SSL_CERTS has been set locally.
>  	win32-loader (>= 0.7.2) [i386 amd64 kfreebsd-i386 kfreebsd-amd64 hurd-i386],
>  #		Alternative boot method for win32 platforms.
> -	makefs [kfreebsd-any],
> +	makefs (>= 20100306-5+kbsd8u1~) [kfreebsd-any],

ACK.

>  #		Used to create an UFS1 filesystem from a directory tree.
>  	grub-pc (>= 2.02~beta2~) [kfreebsd-i386 kfreebsd-amd64 hurd-i386],
>  #		Used as the CD-ROM's bootloader
> diff --git a/debian/rules b/debian/rules
> index 1f1512a..9b76973 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -3,6 +3,7 @@
>  ARCH=$(shell dpkg-architecture -qDEB_BUILD_ARCH)
>  VERSION=$(shell LC_ALL=C dpkg-parsechangelog | grep ^Version: | cut -d ' ' -f 2)
>  DATE=$(shell echo $(VERSION) | cut -d '.' -f 1)
> +SOURCE_DATE_EPOCH = $(shell date -d "$$(dpkg-parsechangelog -SDate)" +%s)
>  SUITE=$(shell LC_ALL=C dpkg-parsechangelog | grep ^Distribution: | cut -d ' ' -f 2)
>  ifeq (${SUITE},UNRELEASED)
>  USE_UDEBS_FROM=unstable
> @@ -31,7 +32,7 @@ clean:
>  # Must run as root, so is not run as part of regular build.
>  build-images:
>  	$(MAKE) -C build all_build stats release \
> -		USE_UDEBS_FROM=$(USE_UDEBS_FROM) BUILD_DATE=$(DATE) \
> +		USE_UDEBS_FROM=$(USE_UDEBS_FROM) BUILD_DATE=$(DATE) SOURCE_DATE_EPOCH="$(SOURCE_DATE_EPOCH)" \

Not fully ACK, see last part of my mail.

>  		USE_PROPOSED_UPDATES=$(USE_PROPOSED_UPDATES) \
>  		TRANSSTATUS=$(TRANSSTATUS) BOOTMENU_BEEP=$(BOOTMENU_BEEP)
>  
> @@ -71,7 +72,14 @@ binary-arch: install
>  	dh_gencontrol
>  	dh_md5sums
>  	dh_builddeb
> -	cd debian/tmp && tar czvf ../../../$(TARNAME) .
> +	# Clamp timestamps to be no newer than last changelog entry, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
> +	find debian/tmp -newermt "@$(SOURCE_DATE_EPOCH)" -print0 | xargs -0r touch --no-dereference --date="@$(SOURCE_DATE_EPOCH)"
> +	# Create tarball with files sorted in a stable order, see
> +	# https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs
> +	# and without timestamp in the gzip header, see
> +	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
> +	cd debian/tmp && find . -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -czvf ../../../$(TARNAME)
>  	dpkg-distaddfile $(TARNAME) raw-installer -

Thrice.

I think those 3(.5) occurrences really should be factorized, especially
given the logic is the same: replacing "cd foo && tar bar ." with more
code. Somewhere under build/util would probably be suitable.

Also, the [XXX] part above is using SOURCE_DATE, which is undefined as
far as I can tell since SOURCE_DATE_EPOCH is what's getting defined.
Maybe it should call the same new util (with different parameters since
we only need touch here, and no tar call)?



Finally, not everything is built using debian/rules targets (with or
without dpkg-buildpackage). One should still be able to just run e.g.:
“make -C build build_netboot-gtk USE_UDEBS_FROM=sid”

See BUILD_DATE handling, for example. We end up with a default setting
through:
|    build/config/common:BUILD_DATE ?= $(shell date -u '+%Y%m%d-%H:%M')

[ Please note that calling $(shell dpkg-parsechangelog -SDate) to set
SOURCE_DATE_EPOCH there would only work when building from the toplevel
directory, and not from the build/ subdirectory for example. ]

Mraw,
KiBi.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20151117/940268f6/attachment.sig>

More information about the Reproducible-builds mailing list