Reproducible builds in January 2024

Chris Lamb chris at reproducible-builds.org
Thu Feb 8 21:38:46 GMT 2024 

--------------------------------------------------------------------
        o
      ⬋   ⬊      January 2024 in Reproducible Builds
     o     o
      ⬊   ⬋      https://reproducible-builds.org/reports/2024-01/
        o
--------------------------------------------------------------------

Welcome to the January 2024 report from the Reproducible Builds
project. In these reports we outline the most important things that we
have been up to over the past month. If you are interested in
contributing to the project, please visit our 'Contribute' [1] page on
our website.

 [1] https://reproducible-builds.org/contribute/

                                    §


"How we executed a critical supply chain attack on PyTorch"
-----------------------------------------------------------

John Stawinski [2] and Adnan Khan [3] published a lengthy blog post
detailing how they executed a supply-chain attack [4] against PyTorch
[5], a popular machine learning platform "used by titans like Google,
Meta, Boeing, and Lockheed Martin":

> Our exploit path resulted in the ability to upload malicious PyTorch
> releases to GitHub, upload releases to [Amazon Web Services],
> potentially add code to the main repository branch, backdoor PyTorch
> dependencies – the list goes on. In short, it was bad. Quite bad.

The attack pivoted on PyTorch's use of "self-hosted runners [7]" as well
as submitting a pull request to address a trivial typo in the project's
README file to gain access to repository secrets and API keys that
could subsequently be used for malicious purposes.

 [2] https://johnstawinski.com/
 [3] https://adnanthekhan.com/
 [4] https://johnstawinski.com/2024/01/11/playing-with-fire-how-we-executed-a-critical-supply-chain-attack-on-pytorch/
 [5] https://pytorch.org/
 [7] https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners

                                    §


New Arch Linux forensic filesystem tool
---------------------------------------

On our mailing list [8] this month, long-time Reproducible Builds
developer kpcyrd announced a new tool [9] designed to forensically
analyse Arch Linux [10] filesystem images.

Called archlinux-userland-fs-cmp [11], the tool is "supposed to be
used from a rescue image (any Linux) with an Arch install mounted to,
[for example], /mnt." Crucially, however, "at no point is any file
from the mounted filesystem eval'd or otherwise executed. Parsers are
written in a memory safe language."

More information about the tool can be found on their announcement
message [12], as well as on the tool's homepage [13]. A GIF of the tool
in action [14] is also available.

 [8] https://lists.reproducible-builds.org/pipermail/rb-general/
 [9] https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003232.html
 [10] https://archlinux.org/
 [11] https://github.com/kpcyrd/archlinux-userland-fs-cmp
 [12] https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003232.html
 [13] https://github.com/kpcyrd/archlinux-userland-fs-cmp
 [14] https://asciinema.org/a/MFefYEdvU2O5LlIzseQnyBky5

                                    §


Issues with our SOURCE_DATE_EPOCH code?
---------------------------------------

Chris Lamb started a thread on our mailing list [15] summarising some
potential problems with the source code snippet the Reproducible Builds
project has been using to parse the SOURCE_DATE_EPOCH [16]
environment variable:

> I'm not 100% sure who originally wrote this code, but it was probably
> sometime in the ~2015 era, and it must be in a huge number of codebases
> by now.
>
> Anyway, Alejandro Colomar was working on the shadow security tool and
> pinged me regarding some potential issues with the code. You can see
> this conversation here: [17].

Chris ended his message with a request that those with intimate or low-
level knowledge of time_t, C types, overflows and the various parsing
libraries in the C standard library (etc.) contribute with further info.

 [15] https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003225.html
 [16] https://reproducible-builds.org/docs/source-date-epoch/
 [17] https://github.com/shadow-maint/shadow/commit/cb610d54b47ea2fc3da5a1b7c5a71274ada91371#r136407772

                                    §


Distribution updates
--------------------

In Debian this month, Roland Clobus posted another detailed update of
the status of reproducible ISO images [18] on our mailing list. In
particular, Roland helpfully summarised that "all major desktops build
reproducibly with bullseye, bookworm, trixie and sid provided
they are built for a second time within the same DAK run (i.e. [within]
6 hours)". Additionally 7 of the 8 bookworm images from the official
download link [19] build reproducibly at any later time.

In addition to this, three reviews of Debian packages were added, 17
were updated and 15 were removed this month adding to our knowledge
about identified issues [20].

Elsewhere, Bernhard posted another monthly update [21] for his work
elsewhere in openSUSE.

 [18] https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003217.html
 [19] https://get.debian.org/cdimage/release/current-live/amd64/iso-hybrid/
 [20] https://tests.reproducible-builds.org/debian/index_issues.html
 [21] https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/4AOWVPBX2OYQEUXTN3ORS6PJMUBAEWHS/

                                    §


Community updates
-----------------

There were made a number of improvements to our website, including
Bernhard M. Wiedemann fixing a number of typos of the term
'nondeterministic'. [22] and Jan Zerebecki adding a substantial and
highly welcome section to our page about SOURCE_DATE_EPOCH [23] to
document its interaction with distribution rebuilds [24].

diffoscope [25] is our in-depth and content-aware diff utility that can
locate and diagnose reproducibility issues. This month, Chris Lamb made
a number of changes such as uploading versions 245 and 255 to Debian,
but focusing on triaging and/or merging code from other contributors.
This included adding support for comparing 'eXtensible ARchive'
(.XAR/.PKG) files [26] courtesy of Seth Michael Larson [27][28], as
well considerable work from Vekhir in order to fix compatibility between
various and subtle incompatible versions of the progressbar libraries in
Python [29][30][31][32]. A special thanks to these contributors.

 [22] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/281bea1b
 [23] https://reproducible-builds.org/docs/source-date-epoch/
 [24] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3a3988bf
 [25] https://diffoscope.org
 [26] https://en.wikipedia.org/wiki/Xar_(archiver)
 [27] https://salsa.debian.org/reproducible-builds/diffoscope/commit/241c92af
 [28] https://salsa.debian.org/reproducible-builds/diffoscope/commit/aca62e60
 [29] https://salsa.debian.org/reproducible-builds/diffoscope/commit/1fae3be4
 [30] https://salsa.debian.org/reproducible-builds/diffoscope/commit/61394cc4
 [31] https://salsa.debian.org/reproducible-builds/diffoscope/commit/168f927c
 [32] https://salsa.debian.org/reproducible-builds/diffoscope/commit/828a33ab

                                    §


Reproducibility testing framework
---------------------------------

The Reproducible Builds project operates a comprehensive testing
framework (available at tests.reproducible-builds.org [33]) in order to
check packages and other artifacts for reproducibility. In January, a
number of changes were made by Holger Levsen:

* Debian [34]-related changes:

    * Reduce the number of arm64 architecture workers from 24 to 16. [35]
    * Use diffoscope [36] from the Debian release being tested
      again. [37]
    * Improve the handling when killing unwanted processes [38][39][40]
      and be more verbose about it, too [41].
    * Don't mark a job as 'failed' if process marked as 'to-be-killed'
      is already gone. [42]
    * Display the architecture of builds that have been running for
      more than 48 hours. [43]
    * Reboot arm64 nodes when they hit an OOM (out of memory)
      state. [44]

* Package rescheduling changes:

    * Reduce IRC notifications to '1' when rescheduling due to package
      status changes. [45]
    * Correctly set SUDO_USER when rescheduling packages. [46]
    * Automatically reschedule packages regressing to FTBFS (build
      failure) or FTBR (build success, but unreproducible). [47]

* OpenWrt [48]-related changes:

    * Install the python3-dev and python3-pyelftools packages as
      they are now needed for the sunxi target. [49][50]
    * Also install the libpam0g-dev which is needed by some OpenWrt
      hardware targets. [51]

* Misc:

    * As it's January, set the 'real_year' variable to 2024 [52] and
      bump various copyright years as well [53].
    * Fix a large (!) number of spelling mistakes in various
      scripts. [54][55][56]
    * Prevent Squid [57] and Systemd [58] processes from being killed
      by the kernel's OOM killer [59]. [60]
    * Install the iptables tool everywhere, else our custom
      rc.local script fails. [61]
    * Cleanup the /srv/workspace/pbuilder directory on boot. [62]
    * Automatically restart Squid [63] if it fails. [64]
    * Limit the execution of chroot-installation jobs to a maximum of
      4 concurrent runs. [65][66]

 [33] https://tests.reproducible-builds.org
 [34] https://debian.org/
 [35] https://salsa.debian.org/qa/jenkins.debian.net/commit/79bf35a6d
 [36] https://diffoscope.org/
 [37] https://salsa.debian.org/qa/jenkins.debian.net/commit/57fccefcf
 [38] https://salsa.debian.org/qa/jenkins.debian.net/commit/95cf719fd
 [39] https://salsa.debian.org/qa/jenkins.debian.net/commit/4e278f7ce
 [40] https://salsa.debian.org/qa/jenkins.debian.net/commit/2328587ab
 [41] https://salsa.debian.org/qa/jenkins.debian.net/commit/1c4f6ffdf
 [42] https://salsa.debian.org/qa/jenkins.debian.net/commit/63b812a1b
 [43] https://salsa.debian.org/qa/jenkins.debian.net/commit/55a86760f
 [44] https://salsa.debian.org/qa/jenkins.debian.net/commit/d3f61eacd
 [45] https://salsa.debian.org/qa/jenkins.debian.net/commit/5dcf67e88
 [46] https://salsa.debian.org/qa/jenkins.debian.net/commit/dd1f4b129
 [47] https://salsa.debian.org/qa/jenkins.debian.net/commit/8d145dc96
 [48] https://openwrt.org/
 [49] https://salsa.debian.org/qa/jenkins.debian.net/commit/010155f3b
 [50] https://salsa.debian.org/qa/jenkins.debian.net/commit/7f4c47059
 [51] https://salsa.debian.org/qa/jenkins.debian.net/commit/c7efc35d4
 [52] https://salsa.debian.org/qa/jenkins.debian.net/commit/2f59edd10
 [53] https://salsa.debian.org/qa/jenkins.debian.net/commit/ad04d1fab
 [54] https://salsa.debian.org/qa/jenkins.debian.net/commit/e7bde6d9a
 [55] https://salsa.debian.org/qa/jenkins.debian.net/commit/4cafbc58a
 [56] https://salsa.debian.org/qa/jenkins.debian.net/commit/3fbd6ed7e
 [57] https://www.squid-cache.org/
 [58] https://systemd.io/
 [59] https://www.kernel.org/doc/gorman/html/understand/understand016.html
 [60] https://salsa.debian.org/qa/jenkins.debian.net/commit/9efe90485
 [61] https://salsa.debian.org/qa/jenkins.debian.net/commit/481caed35
 [62] https://salsa.debian.org/qa/jenkins.debian.net/commit/9b564c446
 [63] https://www.squid-cache.org/
 [64] https://salsa.debian.org/qa/jenkins.debian.net/commit/b408fbd38
 [65] https://salsa.debian.org/qa/jenkins.debian.net/commit/71642c11d
 [66] https://salsa.debian.org/qa/jenkins.debian.net/commit/d3afa6d4c

Significant amounts of node maintenance was performed by Holger Levsen
(eg. [67][68][69][70][71][72][73] etc.) and Vagrant Cascadian (eg.
[74][75][76][77][78][79][80][81]). Indeed, Vagrant Cascadian handled an
extended power outage for the network running the Debian armhf
architecture test infrastructure which provided the incentive to
replace the UPS batteries and consolidate infrastructure to reduce
future UPS load. [82]

 [67] https://salsa.debian.org/qa/jenkins.debian.net/commit/f618266a0
 [68] https://salsa.debian.org/qa/jenkins.debian.net/commit/11dc79d53
 [69] https://salsa.debian.org/qa/jenkins.debian.net/commit/d1cc288bd
 [70] https://salsa.debian.org/qa/jenkins.debian.net/commit/715eda5ec
 [71] https://salsa.debian.org/qa/jenkins.debian.net/commit/7a909a1d2
 [72] https://salsa.debian.org/qa/jenkins.debian.net/commit/fd362069e
 [73] https://salsa.debian.org/qa/jenkins.debian.net/commit/0f82bda1f
 [74] https://salsa.debian.org/qa/jenkins.debian.net/commit/a06287c62
 [75] https://salsa.debian.org/qa/jenkins.debian.net/commit/3e4b2e507
 [76] https://salsa.debian.org/qa/jenkins.debian.net/commit/f5625f573
 [77] https://salsa.debian.org/qa/jenkins.debian.net/commit/5d8c7d32e
 [78] https://salsa.debian.org/qa/jenkins.debian.net/commit/c366d93b5
 [79] https://salsa.debian.org/qa/jenkins.debian.net/commit/1726a5281
 [80] https://salsa.debian.org/qa/jenkins.debian.net/commit/dee2b8bd2
 [81] https://salsa.debian.org/qa/jenkins.debian.net/commit/3b86797d2
 [82] https://floss.social/@vagrantc/111853398019782907

Elsewhere in our infrastructure, however, Holger Levsen also adjusted
the email configuration for @reproducible-builds.org to deal with a
new SMTP email attack [83][84].

 [83] https://www.postfix.org/smtp-smuggling.html
 [84] https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/c1ab40a

                                    §


Upstream patches
----------------

The Reproducible Builds project tries to detects, dissects and fix as
many (currently) unreproducible packages as possible. We endeavour to
send all of our patches upstream where appropriate. This month, we wrote
a large number of such patches, including:

* Bernhard M. Wiedemann:

    * cython [85] (nondeterminstic path issue)
    * deluge [86] (issue with modification time of .egg file)
    * gap-ferret [87], gap-semigroups [88] & gap-simpcomp [89]
      (nondeterministic config.log file)
    * grpc [90] (filesystem ordering issue)
    * hub [91] (random)
    * kubernetes1.22 [92] & kubernetes1.23 [93] (sort-related issue)
    * kubernetes1.24 [94] & kubernetes1.25 [95] (go -trimpath issue)
    * libjcat [96] (drop test files with random bytes)
    * luajit [97] (Use new d option for deterministic bytecode output)
    * meson [98][99] (sort the results from Python filesystem call)
    * python-rjsmin [100] (drop GCC instrumentation [101] artifacts)
    * qt6-virtualkeyboard+others [102] (bug parallelism/race)
    * SoapySDR [103] (parallelism-related issue)
    * systemd [104] (sorting problem)
    * warewulf [105] (CPIO [106] modification time issue, etc.)

* Chris Lamb:

    * #1060254 [107] filed against mumble [108].

* James Addison:

    * guake [109] ('Schroedinger' file due to race condition)
    * qhelpgenerator-qt5 [110] (timezone localization; fix also
      merged upstream for QT6)
    * sphinx [111] (search index doctitle sorting)

 [85] https://github.com/cython/cython/issues/5949
 [86] https://build.opensuse.org/request/show/1136411
 [87] https://build.opensuse.org/request/show/1136422
 [88] https://build.opensuse.org/request/show/1136433
 [89] https://build.opensuse.org/request/show/1136431
 [90] https://github.com/grpc/grpc/pull/35687
 [91] https://build.opensuse.org/request/show/1137377
 [92] https://build.opensuse.org/request/show/1137979
 [93] https://build.opensuse.org/request/show/1137980
 [94] https://build.opensuse.org/request/show/1136467
 [95] https://build.opensuse.org/request/show/1136465
 [96] https://build.opensuse.org/request/show/1138082
 [97] https://github.com/LuaJIT/LuaJIT/issues/1008
 [98] https://github.com/mesonbuild/meson/pull/12788
 [99] https://github.com/mesonbuild/meson/pull/12789
 [100] https://build.opensuse.org/request/show/1137474
 [101] https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
 [102] https://bugreports.qt.io/browse/QTBUG-121643
 [103] https://github.com/pothosware/SoapySDR/issues/428
 [104] https://github.com/systemd/systemd/pull/31080
 [105] https://build.opensuse.org/request/show/1137333
 [106] https://www.gnu.org/software/cpio/
 [107] https://bugs.debian.org/1060254
 [108] https://tracker.debian.org/pkg/mumble
 [109] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059917
 [110] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059631
 [111] https://github.com/sphinx-doc/sphinx/pull/11888

Separate to this, Vagrant Cascadian followed up with the relevant
maintainers when reproducibility fixes were not included in newly-
uploaded versions of the mm-common package in Debian — this was
quickly fixed, however. [112]

 [112] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=977177#35

                                    §


And finally...
--------------

If you are interested in contributing to the Reproducible Builds
project, please visit our 'Contribute' [113] page on our website.
However, you can get in touch with us via:

 * IRC: #reproducible-builds on irc.oftc.net.

 * Mailing list: rb-general at lists.reproducible-builds.org [114]

 * Mastodon: @reproducible_builds [115]

 * Twitter: @ReproBuilds [116]

 [113] https://reproducible-builds.org/contribute/
 [114] https://lists.reproducible-builds.org/listinfo/rb-general
 [115] https://fosstodon.org/@reproducible_builds
 [116] https://twitter.com/ReproBuilds


-- 
      o
    ⬋   ⬊      Reproducible Builds
   o     o     reproducible-builds.org 💠
    ⬊   ⬋
      o

More information about the Reproducible-builds mailing list