[Reproducible-builds] Notes on OpenWRT Reproducibility

[bnewbold at robocracy.org]

Hi there! This is a long rambling email, sorry.

I ran across the OpenWRT reproducible build page[0] and diffoscope outputs 
this morning and thought they were super rad. I spent a bit of time to see 
if I could push things forward. Presumably this will mostly involve 
interaction with the OpenWRT developers, but I wanted to check in here 
first for feedback and to share what I learned.

1. There's a statement that diffoscope "cannot detect .bin files as 
squashfs". I don't think that's accurate, I could rename .squashfs files 
to .bin and diffoscope found the magic and worked fine. I think the issue 
is the huge number of hardware-specific ".bin" and ".img" formats OpenWRT 
generates[1]. Many of these are in uImage format (detectable by magic), 
others are just {kernel, bootloader, fs} at fixed offsets. For the former 
it would be great if diffoscope could pick up on the format in any way. I 
found a random script[2] that tries to extract uImage binaries into the 
original files, but it would be a good start to just present the metadata 
and offsets.

2. A few small changes to reproducible_openwrt.sh would be helpful. I 
would enable .tar.gz rootfs output for all targets 
(CONFIG_TARGET_ROOTFS_TARGZ=y) and grab some extra "image" files. In 
addition to .bin and .squashfs, {*.img, *.ext4, *.rootfs.tar.gz, *.img.gz, 
*.ubi, *.ubifs, *.itb} are full image suffixes. {*-uImage, *-zImage, 
*-vmlinuz, *.dtb, *.elf} are suffixes of kernel files which are sometimes 
distributed separately.
Also, perhaps add CONFIG_CLEAN_IPKG=y (see below).
Also, perhaps an ARM build target; A20-OLinuXino-Lime? Something with 
device tree, otherwise doesn't really matter.
I can submit a patch for the above if that would help, but I wouldn't have 
an easy way to test it.

In particular, being able to compare the rootfs images and kernel 
images separately might be sufficient instead of trying to untangle all 
the many random combined binary image formats.

3. I was sort of surprised to see linux-latest passing reproducibility in 
Debian on armhf. Was anything special necessary for this, eg changes to 
mkimage to generate uImage without a timestamp? Any advice or HOWTO out 
there on reproducible kernel builds? I see Ben Hutchings's kbuild patch 
for SOURCE_DATE_EPOCH support; is that sufficient?

4. busybox uses kconfig (or something very similar), which generates a 
AUTOCONF_TIMESTAMP macro in a header file (passed through as a "banner" 
variable capturing build time). I assumed this had to do with autotools at 
first, but it seems to be an old kconfig thing. I don't know how many 
other projects use these tools, but it might be worth making an 
identifier. I worked around the problem by checking for SOURCE_DATE_EPOCH 
in the Makefile and overriding the macro, but it might be cleaner to check 
in kconfig itself and generate an appropriate macro variable instead, as a 
reusable pattern.

5. diffoscope has squashfs support, but is there actually a way to 
generate squashfs files reproducibly? Perhaps there is a way to strip the 
metadata after the fact?

6. The opkg package manger leaves a log of .ipkg installs which end up on 
the disk and include a (UNIX) Installed-Time field. This situation is very 
similar to /var/cache with Debian. The CONFIG_CLEAN_IPKG=y flag is 
sufficient to just delete this log file, or opkg could add an argument to 
override the time, or a sed script could "fix" all the timestamps. A 
question for the OpenWRT developers.


I haven't gotten to any complete reproducible image files yet; I've got 
some patches with partial progress here: 
https://github.com/bnewbold/openwrt-repro/compare/master...repro

These are mostly just passing an existing TIMESTAMP variable (from version 
control) through to tar, having gzip not make timestamps, sorting a text 
file, etc.

The next things to tackle would be kernel images and a few more packages 
that capture timestamps. Also, with squashfs (vs tar), device files like 
/dev/console are problematic because they can't be "touched" to change 
mtime for mksquashfs.

Thanks for not only pushing through so much progress within Debian but 
also setting up this example infrastructure for OpenWRT, BSDs, etc. 
Inspiring!

--bryan

[0]: https://reproducible.debian.net/openwrt/openwrt.html
[1]: http://wiki.openwrt.org/doc/howto/obtain.firmware.generate
[2]: http://buffalo.nas-central.org/wiki/How_to_Extract_an_uImage