[Git][qa/jenkins.debian.net][master] 2 commits: reproduce.d.n: minor updates to documentation

Holger Levsen (@holger) gitlab at salsa.debian.org
Fri Dec 20 17:16:29 GMT 2024



Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
e9599f79 by Holger Levsen at 2024-12-20T17:43:55+01:00
reproduce.d.n: minor updates to documentation

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
f74d5cc6 by Holger Levsen at 2024-12-20T18:15:47+01:00
tests.r-b.o: remove code for testing Arch Linux, there's https://reproducible.archlinux.org & rebuilderd

it's been a blast and an honor! :)

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


28 changed files:

- README.infrastructure
- THANKS.head
- TODO
- TODO.infrastructure
- TODO.legacy
- bin/common-functions.sh
- bin/djm
- bin/jenkins-shell-monitor.sh
- bin/jenkins_node_wrapper.sh
- bin/reproducible.ini
- − bin/reproducible_archlinux_manual_scheduler.sh
- − bin/reproducible_archlinux_scheduler.sh
- − bin/reproducible_build_archlinux_pkg.sh
- bin/reproducible_common.sh
- bin/reproducible_html_alpine.sh
- − bin/reproducible_html_archlinux.sh
- bin/reproducible_html_nodes_info.sh
- bin/reproducible_maintenance.sh
- − bin/reproducible_setup_archlinux_schroot.sh
- bin/reproducible_system_health.sh
- hosts/common/etc/munin/plugins/jenkins_reproducible_builds
- hosts/common/etc/sudoers.d/jenkins
- hosts/jenkins/etc/apache2/reproduciblemap.conf
- hosts/jenkins/etc/sudoers.d/jenkins
- hosts/osuosl5-amd64/var/www/html/rebuilderd-setup.html
- hosts/osuosl5-amd64/var/www/html/rebuilderd-usage.html
- job-cfg/reproducible.yaml
- update_jdn.sh


Changes:

=====================================
README.infrastructure
=====================================
@@ -46,8 +46,8 @@ the .reproducible.osuosl.org domain and can also be reached via .debian.net.
 
 The nodes are used for these jobs:
 
-* osuosl1-amd64: archlinux, alpine, coreboot, netbsd, openwrt
-* osuosl2-amd64: archlinux, alpine, coreboot, openwrt
+* osuosl1-amd64: alpine, coreboot, netbsd, openwrt
+* osuosl2-amd64: alpine, coreboot, openwrt
 ** osuosl2 is running in the future
 ** the alpine jobs are disabled
 * osuosl3-amd64: Debian live-builds / Debian bootstrap jobs / debian-janitor / mmdebstrap-jenkins jobs / openqa.d.n workers


=====================================
THANKS.head
=====================================
@@ -41,8 +41,8 @@ link:https://jenkins.debian.net/["jenkins.debian.net"] would not be possible wit
  * link:https://letsencrypt.org[Let's encrypt] provides free of charge SSL certificates for jenkins.debian.net, reproducible.debian.net and tests.reproducible-builds.org.
  * In December 2018 we were given access to eight nodes which were donated by Facebook to the GCC Compile Farm project and are now hosted by link:https://osuosl.org/[OSUOSL] which each had 32 cores with 144 GB memory. Those machines have been retired now and OSUOSL offered different machines to us:
  * In spring 2023 we got access to some new nodes hosted by link:https://osuosl.org/[OSUOSL]:
-  ** 16 cores with 128 GB memory for osuosl1-amd64.reproducible.osuosl.org used for building Arch Linux, OpenWrt, coreboot and NetBSD for t.r-b.o
-  ** 16 cores with 128 GB memory for osuosl2-amd64.reproducible.osuosl.org used for building Arch Linux, OpenWrt, coreboot for t.r-b.o
+  ** 16 cores with 128 GB memory for osuosl1-amd64.reproducible.osuosl.org used for building OpenWrt, coreboot and NetBSD for t.r-b.o
+  ** 16 cores with 128 GB memory for osuosl2-amd64.reproducible.osuosl.org used for building OpenWrt, coreboot for t.r-b.o
   ** 16 cores with 128 GB memory for osuosl3-amd64.reproducible.osuosl.org used for building Debian live, Debian bootstrapping jobs, Debian janitor jobs, mmdebstrap-jenkins jobs and openqa.d.n workers
   ** 24 cores with 64 GB memory for osuosl4-amd64.reproducible.osuosl.org plus 16TB RAID on HDDs for developing https://rebuilder-snapshot.debian.net and other snapshot and rebuilding related experiments
   ** 16 cores with 64 GB memory for osuosl5-amd64.reproducible.osuosl.org plus 20TB RAID on SDDs for running and developing https://rebuilder-snapshot.debian.net


=====================================
TODO
=====================================
@@ -33,6 +33,7 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 ** purge postgresql from osuosl4 and and 5.
 ** add more workers on o4
 ** use web frontend from archlinux too? https://gitlab.archlinux.org/archlinux/rebuilderd-website
+*** probably not
 ** install on o4 and test trixie there
 *** reuse cache from o5 to ease snapshot bandwidth?
 *** maybe rebuild with nocheck profile there?


=====================================
TODO.infrastructure
=====================================
@@ -44,8 +44,7 @@ jenkins backup on osuosl4 or 5?
 
 nice to have
 ------------
-move openwrt & netbsd & coreboot builds on different hosts than the ones building archlinux
-   make it so that *either* openwrt, coreboot or netbsd is built
+make it so that *either* openwrt, coreboot or netbsd is built
 	FIXME: this needs a plan
 split this on several hosts: 
 	osuosl3-amd64: Debian live-builds / Debian bootstrap jobs / debian-janitor / mmdebstrap-jenkins jobs / openqa.d.n workers


=====================================
TODO.legacy
=====================================
@@ -210,50 +210,6 @@
 ** http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_-_build_packages_in_memory/index.html
 ** manually create a fedora chroot using rpm, wget + yum: http://geek.co.il/2010/03/14/how-to-build-a-chroot-jail-environment-for-centos
 
-==== reproducible Arch Linux
-
-* setup_archlinux_schroot job:
-** needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)
-** use schroot tarballs (gzipped), moves are atomic then
-* only disable cert checking on the node running in the future
-* compare the just built pkg.tar.zst with the one available on the arch mirrors. *then* one can truly say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."
-* maintenance job:
-** check for archlinux schroot sessions which should not be there and delete them. complain if that fails.
-
-* use db - see https://tests.reproducible-builds.org/reproducibledb.htm
-** scheduler.sh:
-*** use asp to update trunk packages?
-** html:
-*** leave all pkg.html files, delete them (much) later, then check all pkg.* files are gone
-*** recreate them newly as index.html
-**** link to https://www.archlinux.org/packages/$repo/x86_64/$pkgname/
-**** provide links to debian pages if same package name exist
-*** create new job to recreate all pkg pages
-*** create json
-** misc:
-*** write into all the stats_ tables
-*** make build.sh respect pacman exit code
-*** comparing versions (in scheduler) is probably needed (though it wuld be better if not): if its not empty it must be higher (due to repo constraints), but it can be even higher than in the repo, because we build trunk
-*** debian uses a table removed_packages, should this be used here too?
-*** debian uses a table manual_scheduler to limit the amount of packages someone can schedule per day...
-*** should breakages job look for archlinux breakages? for which cases? a.) grep "was killed after running into timeout after 30m" pkg.html
-** using notes.git should be next!
-
-* rebuilding against the archlinux repos
-** only needs to be done once using https://github.com/Foxboron/devtools-repro
-** to not (pacman know about trunk and thus) build trunk, we need to configure asp.
-** we'll keep building against repo+trunk as we do now (so that archlinux can also benefit from the QA effects)
-
-* fix build.sh:
-** build2.log doesn't get deleted if build1 fails
-** -> rename build2.log to $version_build2.log (don't include package name...)
-
-* things to be done before enabling more builders:
-** build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)
-
-* build on OSUOSL nodes:
-** update thanks & readme
-
 ==== reproducible qubes
 
 * add qubes test on t.r-b.o


=====================================
bin/common-functions.sh
=====================================
@@ -232,7 +232,7 @@ jenkins_zombie_check() {
 	# this has happened on 2023-08-25 again
 	# this has happened on 2023-09-06 again
 	#
-	ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_(jessie|stretch|buster|bullseye)|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests_(stretch|buster|bullseye)|debian-archive-keyring-tests_stretch|debian-archive-keyring-tests_buster|debian-archive-keyring-tests_bullseye|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[0123]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-(ath97|tegra)|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07' || true)"
+	ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_(jessie|stretch|buster|bullseye)|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests_(stretch|buster|bullseye)|debian-archive-keyring-tests_stretch|debian-archive-keyring-tests_buster|debian-archive-keyring-tests_bullseye|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[0123]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|archlinux|openwrt-target-(ath97|tegra)|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07' || true)"
 	if [ -n "$ZOMBIES" ] ; then
 		DIRTY=true
 		figlet 'zombies!!!'


=====================================
bin/djm
=====================================
@@ -418,7 +418,7 @@ djm_report() {
 	# statistics about jenkins jobs
 	#
 	PATTERNS="setup_schroot setup_pbuilder debian_live maintenance node_health_check
-		html alpine archlinux builder_archlinux openwrt .*fedora builds opensuse .*strap
+		html alpine openwrt .*fedora builds opensuse .*strap
 		.*diffoscope reprotest disorderfs .*lfs" # .*(json|db|lfs)"
 	PIPE_PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
 	HEADING="$(cat $TMP_LOGFILE | grep jenkins-ui | wc -l) jobs triggered manually:"


=====================================
bin/jenkins-shell-monitor.sh
=====================================
@@ -122,13 +122,13 @@ main_loop() {
 	SCHROOT_BASE_DIRS=$(find /schroots/ -maxdepth 1 -mindepth 1 -type d | wc -l)
 	DIFFOSCOPE_PROCS=$(grep -c '/usr/bin/python3 /usr/bin/diffoscope' $PSFAX || true)
 	REPRO_JOBS=$(ls ~jenkins/jobs/reproducible_* -1d | wc -l)
-	JOBS_RUNNING_TOTAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|reproducible_build_archlinux_pkg.sh')
+	JOBS_RUNNING_TOTAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh')
 	if [ $JOBS_RUNNING_TOTAL -eq 0 ] ; then
 		JOBS_RUNNING_TOTAL="${RED}$JOBS_RUNNING_TOTAL${FOREGROUND_COLOR}"
 	elif [ $JOBS_RUNNING_TOTAL -lt 5 ] ; then
 		JOBS_RUNNING_TOTAL="${YELLOW}$JOBS_RUNNING_TOTAL${FOREGROUND_COLOR}"
 	fi
-	JOBS_RUNNING_LOCAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|reproducible_build_archlinux_pkg.sh|jenkins_master_wrapper.sh')
+	JOBS_RUNNING_LOCAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|jenkins_master_wrapper.sh')
 	JOBS_RUNNING_REMOTE=$(grep -c jenkins_master_wrapper.sh $PSFAX || true)
 	REPRO_WORKERS=$(grep -c reproducible_worker.sh $PSFAX || true)
 	JENKINS_AGENTS=$(grep -c jenkins/agent.jar $PSFAX || true)


=====================================
bin/jenkins_node_wrapper.sh
=====================================
@@ -136,8 +136,6 @@ elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/tmp.* ]] ; then
 	exec rm -r "$3" ; croak "Exec failed";
 elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/rbuild.* ]] ; then
 	exec rm -r "$3" ; croak "Exec failed";
-elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/archlinuxrb/rb-build.* ]] ; then
-	exec rm -r "$3" ; croak "Exec failed";
 elif [[ "$*" =~ ^rm\ -r\ /var/lib/jenkins/jobs/.*/workspace/results ]] ; then
 	exec rm -r "$3" ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_setup_pbuilder_stretch_.*_.* ]] ; then
@@ -170,16 +168,12 @@ elif [[ "$*" =~ ^reproducible_freebsd ]] ; then
 	exec /srv/jenkins/bin/reproducible_freebsd.sh ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_setup_schroot_alpine ]] ; then
 	exec /srv/jenkins/bin/reproducible_setup_alpine_schroot.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_setup_schroot_archlinux ]] ; then
-	exec /srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_setup_mock_fedora-23_x86_64 ]] ; then
 	exec /srv/jenkins/bin/reproducible_setup_mock.sh fedora-23 x86_64 ; croak "Exec failed";
 elif [ "$1" = "/srv/jenkins/bin/reproducible_openwrt_rebuild.py" ] ; then
 	exec /srv/jenkins/bin/reproducible_openwrt_rebuild.py ; croak "Exec failed";
 elif [ "$1" = "/srv/jenkins/bin/reproducible_build_alpine_pkg.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
 	exec /srv/jenkins/bin/reproducible_build_alpine_pkg.sh "$2" "$3" "$4" "$5" "$6" ; croak "Exec failed";
-elif [ "$1" = "/srv/jenkins/bin/reproducible_build_archlinux_pkg.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
-	exec /srv/jenkins/bin/reproducible_build_archlinux_pkg.sh "$2" "$3" "$4" "$5" "$6" ; croak "Exec failed";
 elif [ "$1" = "/srv/jenkins/bin/reproducible_build_rpm.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
 	exec /srv/jenkins/bin/reproducible_build_rpm.sh "$2" "$3" "$4" "$5" "$6" "$7" ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_debian_live_build.* ]] ; then


=====================================
bin/reproducible.ini
=====================================
@@ -39,14 +39,6 @@ archs = x86_64
 defaultsuite = factory
 defaultarch = x86_64
 
-[archlinux]
-distro_root = archlinux
-landing_page = archlinux.html
-suites = core extra multilib
-archs = x86_64
-defaultsuite = core
-defaultarch = x86_64
-
 [openwrt]
 distro_root = openwrt
 landing_page = openwrt.html


=====================================
bin/reproducible_archlinux_manual_scheduler.sh deleted
=====================================
@@ -1,80 +0,0 @@
-#!/bin/bash
-
-# Copyright 2018-2023 Holger Levsen <holger at layer-acht.org>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-if [ "$1" = "" ] || [ "$2" = "" ] ; then
-	echo "Need at least two parameters:"
-	echo "$0 \$RESPOSITORY \$SOURCEPKGNAME1 \$SOURCEPKGNAME2 \$SOURCEPKGNAME3..."
-	exit 1
-elif [ "$1" != "core" ] && [ "$1" != "extra" ] && [ "$1" != "multilib" ] ; then
-	echo "\$RESPOSITORY needs to be one of core, extra or multilib."
-        exit 1
-fi
-
-DATE="$(date -u +'%Y-%m-%d %H:%M' -d '21 days ago')"
-REPOSITORY=$1
-SUITE=$REPOSITORY
-ARCH=x86_64
-shift
-PACKAGES="$@"
-SUCCESS=""
-ALREADY_SCHEDULED=""
-NOT_EXISTING=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-for PKG in $PACKAGES ; do
-	echo "Now trying to reschedule $PKG in $SUITE."
-	PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
-	if [ -n "${PKG_ID}" ] ; then
-		SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type = 'ci_build';")
-		if [ -z "$SCHEDULED" ] ; then
-			query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
-			SUCCESS="$SUCCESS $PKG"
-		else
-			echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
-			ALREADY_SCHEDULED="$ALREADY_SCHEDULED $PKG"
-		fi
-	else
-		echo " $PKG does not exist in $SUITE, ignoring."
-		NOT_EXISTING="$NOT_EXISTING $PKG"
-	fi
-done
-
-echo
-if [ -n "$SUCCESS" ] ; then
-	AMOUNT=$(echo $SUCCESS | sed 's# #\n#g' | wc -l)
-	if [ $AMOUNT -gt 3 ] ; then
-		MANY=" $AMOUNT packages"
-	else
-		MANY=""
-	fi
-	MESSAGE="Manually scheduled$MANY in $REPOSITORY:$SUCCESS"
-	# shorten irc message if longer then 256 characters
-	if [ ${#MESSAGE} -gt 256 ] ; then
-		MESSAGE="${MESSAGE:0:256}✂…"
-	fi
-	irc_message archlinux-reproducible "$MESSAGE"
-fi
-if [ -n "$ALREADY_SCHEDULED" ] || [ -n "$NOT_EXISTING" ] ; then
-	echo
-	if [ -n "$ALREADY_SCHEDULED" ] ; then
-		echo "$ALREADY_SCHEDULED were already scheduled..."
-	fi
-	if [ -n "$NOT_EXISTING" ] ; then
-		echo "$NOT_EXISTING were not found in $SUITE, so ignored."
-	fi
-fi
-echo
-
-exit 0
-
-# vim: set sw=0 noet :


=====================================
bin/reproducible_archlinux_scheduler.sh deleted
=====================================
@@ -1,291 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
-	schroot --end-session -c $SESSION 2>/dev/null|| true
-}
-
-update_archlinux_repositories() {
-	#
-	# init
-	#
-	local UPDATED=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
-	local NEW=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
-	local KNOWN=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
-	local TOTAL=$(cat ${ARCHLINUX_PKGS}_* | wc -l)
-	echo "$(date -u ) - $TOTAL Arch Linux packages were previously known to Arch Linux."
-	query_db "SELECT suite, name, version FROM sources WHERE distribution=$DISTROID AND architecture='$ARCH';" > $KNOWN
-	echo "$(date -u ) - $(cat $KNOWN | wc -l) Arch Linux packages are known in our database."
-	echo "$(date -u ) - updating pacman's knowledge of Arch Linux repositories (by running pacman -Syu --noconform')."
-	schroot --run-session -c $SESSION --directory /var/tmp -- sudo pacman -Syu --noconfirm
-
-	#
-	# Get a list of unique package bases.  Non-split packages don't have a pkgbase set
-	# so we need to use the pkgname for them instead.
-	#
-	echo "$(date -u ) - exporting pacman's knowledge of Arch Linux repositories to the filesystem (by running 'expac -S...')."
-	schroot --run-session -c $SESSION --directory /var/tmp -- expac -S '%r %e %n %v' | \
-		while read repository pkgbase pkgname version; do
-			if [[ "$pkgbase" = "(null)" ]]; then
-				printf '%s %s %s\n' "$repository" "$pkgname" "$version"
-			else
-				printf '%s %s %s\n' "$repository" "$pkgbase" "$version"
-			fi
-		done | sort -u -R > "$ARCHLINUX_PKGS"_full_pkgbase_list
-	TOTAL=$(cat ${ARCHLINUX_PKGS}_full_pkgbase_list | wc -l)
-	echo "$(date -u ) - $TOTAL Arch Linux packages are now known to Arch Linux."
-	local total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE sch.build_type = 'ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
-	echo "$(date -u) - updating Arch Linux repositories, currently $total packages scheduled."
-
-	#
-	# remove packages which are gone (only when run between 21:00 and 23:59)
-	#
-	if [ $(date +'%H') -gt 21 ] ; then
-		REMOVED=0
-		REMOVE_LIST=""
-		for REPO in $ARCHLINUX_REPOS ; do
-			echo "$(date -u ) - dropping removed packages from filesystem in repository '$REPO':"
-			for i in $(find $BASE/archlinux/$REPO -type d -wholename "$BASE/archlinux/$REPO/*" | sort) ; do
-				PKG=$(basename $i)
-				if ! grep -q "$REPO $PKG" ${ARCHLINUX_PKGS}_full_pkgbase_list > /dev/null ; then
-					# we could check here whether a package is currently building,
-					# and if so defer the pkg removal. (but I think this is pointless,
-					# as we are unlikely to kill that build, so meh, let it finish
-					# and fail to update the db, because the package is gone...)
-					let REMOVED=$REMOVED+1
-					REMOVE_LIST="$REMOVE_LIST $REPO/$PKG"
-					rm -r --one-file-system $BASE/archlinux/$REPO/$PKG
-					echo "$(date -u) - $REPO/$PKG removed as it's gone from the Archlinux repositories."
-					SUITE="$REPO"
-					PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
-					if [ -n "${PKG_ID}" ] ; then
-						query_db "DELETE FROM results WHERE package_id='${PKG_ID}';"
-						query_db "DELETE FROM schedule WHERE package_id='${PKG_ID}';"
-						query_db "DELETE FROM sources WHERE id='${PKG_ID}';"
-						echo "$(date -u) - $SUITE $PKG removed from database."
-					else
-						echo "$(date -u) - $SUITE $PKG not found in database."
-					fi
-				fi
-			done
-		done
-		MESSAGE="Deleted $REMOVED packages: $REMOVE_LIST"
-		echo -n "$(date -u ) - "
-		if [ $REMOVED -ne 0 ] ; then
-			irc_message archlinux-reproducible "$MESSAGE"
-		fi
-	fi
-
-	#
-	# schedule packages
-	#
-
-	for REPO in $ARCHLINUX_REPOS ; do
-		TMPPKGLIST=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
-		echo "$(date -u ) - updating database with available packages in repository '$REPO'."
-		DATE="$(date -u +'%Y-%m-%d %H:%M')"
-		grep "^$REPO" "$ARCHLINUX_PKGS"_full_pkgbase_list | \
-			while read repository pkgbase version; do
-				PKG=$pkgbase
-				SUITE="$REPO"
-				PKG_IN_DB=$(grep "^$REPO|$pkgbase|" $KNOWN | head -1) # why oh why is head -1 needed here?
-				VERSION=$(echo ${PKG_IN_DB} | cut -d "|" -f3)
-			        if [ -z "${PKG_IN_DB}" ] ; then
-					# new package, add to db and schedule
-					echo $REPO/$pkgbase >> $NEW
-					echo "new package found: $REPO/$pkgbase $version "
-					query_db "INSERT into sources (name, version, suite, architecture, distribution) VALUES ('$PKG', '$version', '$SUITE', '$ARCH', $DISTROID);"
-					PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
-					query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
-				elif [ "$VERSION" != "$version" ] ; then
-					VERCMP="$(schroot --run-session -c $SESSION --directory /var/tmp -- vercmp $version $VERSION || true)"
-					if [ "$VERCMP" = "1" ] ; then
-						# known package with new version, so update db and schedule
-						query_db "UPDATE sources SET version = '$version' WHERE name = '$PKG' AND suite = '$SUITE' AND architecture='$ARCH' AND distribution=$DISTROID;"
-						PKG_STATUS=$(query_db "SELECT r.status FROM results AS r
-							JOIN sources as s on s.id=r.package_id
-							WHERE s.distribution=$DISTROID
-							AND s.architecture='x86_64'
-							AND s.name='$PKG'
-							AND s.suite='$SUITE';")
-						if [ "$PKG_STATUS" = "blacklisted" ] ; then
-							echo "$PKG is blacklisted, so not scheduling it."
-							continue
-						else
-							echo $REPO/$pkgbase >> $UPDATED
-							echo "$REPO/$pkgbase $VERSION is known in the database, but repo now has $version which is newer, so rescheduling... "
-							PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
-							echo " SELECT * FROM schedule WHERE package_id = '${PKG_ID} AND build_type='ci_build';"
-							SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type='ci_build';")
-							if [ -z "$SCHEDULED" ] ; then
-								echo " INSERT INTO schedule (package_id, date_scheduled build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
-								query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
-							else
-								echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
-							fi
-						fi
-					elif [ "$VERCMP" = "-1" ] ; then
-						# our version is higher than what's in the repo because we build trunk
-						echo "$REPO/$pkgbase $VERSION in db is higher than $version in repo because we build trunk."
-					else
-						echo " Boom boom boom boom boom."
-						echo " This should never happen: we know about $pkgbase with $VERSION, but repo has $version. VERCMP=$VERCMP"
-						echo " PKG_IN_DB=${PKG_IN_DB}"
-					fi
-				fi
-
-				printf '%s %s\n' "$pkgbase" "$version" >> $TMPPKGLIST
-			done
-		mv $TMPPKGLIST "$ARCHLINUX_PKGS"_"$REPO"
-		new=$(grep -c ^$REPO $NEW || true)
-		updated=$(grep -c ^$REPO $UPDATED || true)
-		echo "$(date -u ) - scheduled $new/$updated packages in repository '$REPO'."
-	done
-
-	#
-	# schedule up to $MAX packages in DEPWAIT_ or 404_ states
-	# (which have been tried at least 16h ago)
-	#
-	echo "$(date -u ) - should we schedule packages in DEPWAIT_ or 404_ states?"
-	local MAX=350
-	local MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "16 hours ago")
-	local SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
-	QUERY="SELECT s.id FROM sources AS s
-		JOIN results AS r ON s.id=r.package_id
-		WHERE s.distribution = $DISTROID
-		AND s.architecture='x86_64'
-		AND (r.status LIKE 'DEPWAIT%' OR r.status LIKE '404%')
-		AND r.build_date < '$MINDATE'
-		AND NOT exists (SELECT FROM schedule WHERE package_id = s.id AND build_type='ci_build')
-		LIMIT $MAX;"
-	local DEPWAIT404=$(query_db "$QUERY")
-	if [ -n "$DEPWAIT404" ] ; then
-		for PKG_ID in $DEPWAIT404 ; do
-			QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
-			query_db "$QUERY"
-		done
-		echo "$(date -u ) - done scheduling $(echo -n "$DEPWAIT404" | wc -l ) packages in DEPWAIT_ or 404_ states."
-	else
-		echo "$(date -u ) - no."
-	fi
-
-	#
-	# schedule up to $MAX packages we already know about
-	# (only if less than $THRESHOLD packages are currently scheduled)
-	#
-	echo "$(date -u ) - should we schedule old packages?"
-	MAX=501
-	local THRESHOLD=600
-	MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "4 days ago")
-	SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
-	local CURRENT=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
-	if [ $CURRENT -le $THRESHOLD ] ; then
-		echo "$(date -u ) - scheduling $MAX old packages."
-		QUERY="SELECT s.id, s.name, max(r.build_date) max_date
-			FROM sources AS s JOIN results AS r ON s.id = r.package_id
-			WHERE s.distribution=$DISTROID
-			AND s.architecture='x86_64'
-			AND r.status != 'blacklisted'
-			AND r.build_date < '$MINDATE'
-			AND NOT exists (SELECT FROM schedule WHERE where package_id = s.id AND build_type='ci_build')
-			GROUP BY s.id, s.name
-			ORDER BY max_date
-			LIMIT $MAX;"
-		local OLD=$(query_db "$QUERY")
-		for PKG_ID in $(echo -n "$OLD" | cut -d '|' -f1) ; do
-			QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
-			query_db "$QUERY"
-		done
-		echo "$(date -u ) - done scheduling $MAX old packages."
-	else
-		echo "$(date -u ) - no."
-	fi
-
-	#
-	# output stats
-	#
-	rm "$ARCHLINUX_PKGS"_full_pkgbase_list
-	total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
-	new=$(cat $NEW | wc -l 2>/dev/null|| true)
-	updated=$(cat $UPDATED 2>/dev/null| wc -l || true)
-	old=$(echo -n "$OLD" | wc -l 2>/dev/null|| true)
-	depwait404=$(echo -n "$DEPWAIT404" | wc -l 2>/dev/null|| true)
-	if [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] || [ $depwait404 -ne 0 ] ; then
-		# inform irc channel about new packages
-		if [ $new -ne 0 ] ; then
-			if [ $new -eq 1 ] ; then
-				MESSAGE="Added $new package: $(cat $NEW | xargs echo)"
-			else
-				MESSAGE="Added $new packages: $(cat $NEW | xargs echo)"
-			fi
-			irc_message archlinux-reproducible "$MESSAGE"
-		fi
-		# inform irc channel how many packages of which kind have been scheduled
-		message="Scheduled"
-		if [ $new -ne 0 ] ; then
-			message="$message $new new package"
-			if [ $new -gt 1 ] ; then
-				message="${message}s"
-			fi
-		fi
-		if [ $new -ne 0 ] && [ $updated -ne 0 ] ; then
-			message="$message and"
-		fi
-		if [ $updated -ne 0 ] ; then
-			if [ $updated -gt 1 ] ; then
-				message="$message $updated packages with newer versions"
-			else
-				message="$message $updated package with newer version"
-			fi
-		fi
-		if [ $old -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] ) ; then
-			msg_old=", plus $old already tested ones"
-		elif [ $old -ne 0 ] ; then
-			msg_old=" $old already tested packages"
-		else
-			msg_old=""
-		fi
-		if [ $depwait404 -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] ) ; then
-			msg_depwait404=" and $depwait404 packages with dependency or 404 problems"
-		elif [ $depwait404 -ne 0 ] ; then
-			msg_depwait404=" $depwait404 packages with dependency or 404 problems"
-		else
-			msg_depwait404=""
-		fi
-		MESSAGE="${message}${msg_old}${msg_depwait404}, for $total scheduled out of $TOTAL."
-		MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '4 hours ago')"
-		RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
-				JOIN results AS r
-				ON s.id=r.package_id
-				WHERE s.distribution=$DISTROID
-				AND s.architecture='x86_64'
-				AND r.build_date > '$MAXDATE'")
-		MESSAGE="$MESSAGE ($RECENT builds in the last 4h.)"
-		echo -n "$(date -u ) - "
-		irc_message archlinux-reproducible "$MESSAGE"
-	else
-		echo "$(date -u ) - didn't schedule any packages."
-	fi
-	rm -f $NEW $UPDATED $KNOWN > /dev/null
-}
-
-trap cleanup_all INT TERM EXIT
-ARCH="x86_64"
-SESSION="archlinux-scheduler-$RANDOM"
-# init session
-schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name = 'archlinux'")
-update_archlinux_repositories
-
-# vim: set sw=0 noet :


=====================================
bin/reproducible_build_archlinux_pkg.sh deleted
=====================================
@@ -1,505 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-#                2017 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
-	cd
-	# delete session if it still exists
-	if [ "$MODE" != "master" ] ; then
-		schroot --end-session -c archlinux-$SRCPACKAGE-$(basename $TMPDIR) > /dev/null 2>&1 || true
-	fi
-	# delete makepkg build dir
-	if [ -n $SRCPACKAGE ] && [ -d /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR) ] ; then
-		sudo rm -rf --one-file-system /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)
-	fi
-	# delete main work dir (only on master)
-	if [ "$MODE" = "master" ] ; then
-		rm $TMPDIR -r
-		echo "$(date -u) - $TMPDIR deleted."
-	fi
-}
-
-update_pkg_in_db() {
-	local ARCHLINUX_PKG_PATH=$ARCHBASE/$REPOSITORY/$SRCPACKAGE
-	cd "$ARCHLINUX_PKG_PATH"
-	BUILD_DURATION="$(cat pkg.build_duration)"
-	BUILD_STATE=$(cat pkg.state)
-	BUILD_VERSION="$(cat pkg.version)"
-	SUITE="$REPOSITORY"
-	local SRCPKGID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$SRCPACKAGE' AND suite='$SUITE' AND architecture='$ARCH';")
-	if [ -z "${SRCPKGID}" ] ; then
-	        echo "${SRCPKGID} empty, ignoring $REPOSITORY/$SRCPACKAGE, failing hard."
-		exit 1
-	fi
-	QUERY="INSERT into results (package_id, version, status, build_date, build_duration, node1, node2, job)
-		VALUES ('${SRCPKGID}', '$BUILD_VERSION', '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL')
-		ON CONFLICT (package_id)
-		DO UPDATE SET version='$BUILD_VERSION', status='$BUILD_STATE', build_date='$DATE', build_duration='$BUILD_DURATION', node1='$NODE1', node2='$NODE2', job='$BUILD_URL' WHERE results.package_id='$SRCPKGID'";
-        echo "$QUERY"
-	query_db "$QUERY"
-        QUERY="INSERT INTO stats_build (name, version, suite, architecture, distribution, status, build_date, build_duration, node1, node2, job)
-		VALUES ('$SRCPACKAGE', '$BUILD_VERSION', '$SUITE', '$ARCH', $DISTROID, '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL');"
-        echo "$QUERY"
-	query_db "$QUERY"
-        # unmark build since it's properly finished
-        QUERY="DELETE FROM schedule WHERE package_id='${SRCPKGID}' and build_type='ci_build';"
-        echo "$QUERY"
-	query_db "$QUERY"
-	rm pkg.build_duration pkg.state pkg.version
-
-}
-
-choose_package() {
-	echo "$(date -u ) - choosing package to be build."
-	ARCH="x86_64"
-	local RESULT=$(query_db "
-		SELECT s.suite, s.id, s.name, s.version
-		FROM schedule AS sch JOIN sources AS s ON sch.package_id=s.id
-		WHERE distribution=$DISTROID
-		AND sch.date_build_started is NULL
-		AND sch.build_type='ci_build'
-		AND s.architecture='$ARCH'
-		ORDER BY date_scheduled LIMIT 10"|sort -R|head -1)
-	if [ -z "$RESULT" ] ; then
-		echo "No packages scheduled, sleeping 30m."
-		sleep 30m
-		exit 0
-	fi
-	SUITE=$(echo $RESULT|cut -d "|" -f1)
-	REPOSITORY=$(echo $SUITE | cut -d "_" -f2)
-	local SRCPKGID=$(echo $RESULT|cut -d "|" -f2)
-	SRCPACKAGE=$(echo $RESULT|cut -d "|" -f3)
-	VERSION=$(echo $RESULT|cut -d "|" -f4)
-	# remove previous build attempts which didn't finish correctly:
-	JOB_SHORTNAME="${JOB_NAME#reproducible_builder_}"
-	BAD_BUILDS=$(mktemp --tmpdir=$TMPDIR)
-	query_db "SELECT package_id, date_build_started, job FROM schedule WHERE job LIKE '%${JOB_SHORTNAME}'" > $BAD_BUILDS
-	if [ -s "$BAD_BUILDS" ] ; then
-		local STALELOG=/var/log/jenkins/reproducible-archlinux-stale-builds.log
-		# reproducible-archlinux-stale-builds.log is mailed once a day by reproducible_maintenance.sh
-		echo -n "$(date -u) - stale builds found, cleaning db from these: " | tee -a $STALELOG
-		cat $BAD_BUILDS | tee -a $STALELOG
-		query_db "UPDATE schedule SET date_build_started = NULL, job = NULL WHERE job LIKE '%${JOB_SHORTNAME}'"
-	fi
-	rm -f $BAD_BUILDS
-	# mark build attempt, first test if none else marked a build attempt recently
-	echo "ok, let's check if $SRCPACKAGE is building anywhere yet…"
-	RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND build_type='ci_build'")
-	if [ -z "$RESULT" ] ; then
-		echo "ok, $SRCPACKAGE is not building anywhere…"
-		# try to update the schedule with our build attempt, then check no else did it, if so, abort
-		query_db "UPDATE schedule SET date_build_started='$DATE', job='$JOB' WHERE package_id='$SRCPKGID' AND build_type='ci_build' AND date_build_started IS NULL"
-		RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND date_build_started='$DATE' AND job='$JOB' AND build_type='ci_build'")
-		if [ -z "$RESULT" ] ; then
-			echo "hm, seems $SRCPACKAGE is building somewhere… failed to update the schedule table with our build ($SRCPKGID, $DATE, $JOB)."
-			handle_race_condition
-		fi
-	else
-		echo "hm, seems $SRCPACKAGE is building somewhere… schedule table now listed it as building somewhere else."
-		handle_race_condition
-	fi
-
-	echo "$(date -u ) - building package $SRCPACKAGE from '$REPOSITORY' now..."
-}
-
-first_build() {
-	echo "============================================================================="
-	echo "Building for Arch Linux on $(hostname) now."
-	echo "Source package: ${SRCPACKAGE}"
-	echo "Repository:     $REPOSITORY"
-	echo "Date:           $(date -u)"
-	echo "============================================================================="
-	local SESSION="archlinux-$SRCPACKAGE-$(basename $TMPDIR)"
-	local BUILDDIR="/tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)"
-	local LOG=$TMPDIR/b1/$SRCPACKAGE/build1.log
-	local FUTURE_STATE="disabled"
-	local MAKEPKG_ENV_VARS="SOURCE_DATE_EPOCH='$SOURCE_DATE_EPOCH' PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' TMPDIR='/tmp/archlinux-ci/'"
-	local MAKEPKG_OPTIONS="--syncdeps --noconfirm --nocheck"
-	if [ "$(hostname)" = "osuosl2-amd64" ] ; then
-		FUTURE_STATE="enabled"
-		MAKEPKG_ENV_VARS="$MAKEPKG_ENV_VARS GIT_SSL_NO_VERIFY=1"
-		MAKEPKG_OPTIONS="$MAKEPKG_OPTIONS --skippgpcheck"
-	fi
-	echo "Future:            $FUTURE_STATE"
-	echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
-	echo "makepkg env:       $MAKEPKG_ENV_VARS"
-	echo "makepkg options:   $MAKEPKG_OPTIONS"
-	echo "============================================================================="
-	schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
-	schroot --run-session -c $SESSION --directory /tmp -u root -- ln -sfT dash /usr/bin/sh
-	echo "MAKEFLAGS=-j$NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/makepkg.conf
-	schroot --run-session -c $SESSION --directory /tmp -- mkdir -p $BUILDDIR
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -- env GIT_SSL_NO_VERIFY=1 pkgctl repo clone --protocol https "$SRCPACKAGE" 2>&1 | tee -a $LOG || echo "Error: failed to download PKGBUILD for $SRCPACKAGE from $REPOSITORY" | tee -a $LOG
-	# $SRCPACKAGE is actually the binary package
-	ACTUAL_SRCPACKAGE=$(ls "$BUILDDIR")
-	# modify timezone, LANG, LC_ALL in the 1st build.
-	schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
-	export TZ="/usr/share/zoneinfo/Etc/GMT+12"
-	export LANG="en_US.UTF-8"
-	export LC_ALL="en_US.UTF-8"
-	__END__
-	# some more output for debugging
-	set -x
-	# remove possible lock in our local session (happens when root maintenance update running while session starts)
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- rm -f /var/lib/pacman/db.lck 2>&1 | tee -a $LOG
-	# update before pulling new dependencies
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- pacman -Syu --noconfirm 2>&1 | tee -a $LOG
-	# determine the version of the package being build
-	source "$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD" || echo "Failed to source PKGBUILD from '$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD'" | tee -a $LOG
-	if [ -n "$epoch" ] ; then
-		epoch="$epoch:"
-	fi
-	VERSION="$epoch$pkgver-$pkgrel"
-	echo $VERSION > $TMPDIR/b1/$SRCPACKAGE/build1.version
-	# show env variables
-	schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS printenv 2>&1" | tee -a $LOG
-	# preseed GnuPG keys if relevant in order to get good error logs
-	schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "makepkg --printsrcinfo | awk -F ' = ' '/^\s+validpgpkeys/{print \$2}'| while read pgpkey; do gpg --keyserver hkp://keyserver.ubuntu.com --recv-key \$pgpkey; done" 2>&1 | tee -a $LOG
-	# nicely run makepkg with a timeout of $TIMEOUT hours
-	timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
-		schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS makepkg $MAKEPKG_OPTIONS 2>&1" | tee -a $LOG
-	PRESULT=${PIPESTATUS[0]}
-	if [ $PRESULT -eq 124 ] ; then
-		echo "$(date -u) - makepkg was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
-	fi
-	set +e
-	schroot --end-session -c $SESSION | tee -a $LOG
-	PRESULT=${PIPESTATUS[0]}
-	if [ $PRESULT -ne 0 ] ; then
-		echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
-		sleep 60
-		schroot --end-session -f -c $SESSION | tee -a $LOG
-		P2RESULT=${PIPESTATUS[0]}
-		if [ $P2RESULT -ne 0 ] ; then
-			echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
-			sleep 10
-			schroot --end-session -f -c $SESSION | tee -a $LOG
-			P3RESULT=${PIPESTATUS[0]}
-			if [ $P3RESULT -ne 0 ] ; then
-				if [ -n "$(grep 'ERROR: One or more PGP signatures could not be verified' $LOG)" ] ; then
-					# abort only
-					exit 42
-				else
-					# fail with notification
-					exit 23
-				fi
-			fi
-		fi
-	fi
-	if ! "$DEBUG" ; then set +x ; fi
-	set -e
-}
-
-second_build() {
-	echo "============================================================================="
-	echo "Re-Building for Arch Linux on $(hostname) now."
-	echo "Source package: ${SRCPACKAGE}"
-	echo "Repository:     $REPOSITORY"
-	echo "Date:           $(date -u)"
-	echo "============================================================================="
-	local SESSION="archlinux-$SRCPACKAGE-$(basename $TMPDIR)"
-	local BUILDDIR="/tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)"
-	local LOG=$TMPDIR/b2/$SRCPACKAGE/build2.log
-	NEW_NUM_CPU=$(echo $NUM_CPU-1|bc)
-	local FUTURE_STATE="disabled"
-	local MAKEPKG_ENV_VARS="SOURCE_DATE_EPOCH='$SOURCE_DATE_EPOCH' PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' TMPDIR='/tmp/archlinux-ci/'"
-	local MAKEPKG_OPTIONS="--syncdeps --noconfirm --nocheck"
-	if [ "$(hostname)" = "osuosl2-amd64" ] ; then
-		FUTURE_STATE="enabled"
-		MAKEPKG_ENV_VARS="$MAKEPKG_ENV_VARS GIT_SSL_NO_VERIFY=1"
-		MAKEPKG_OPTIONS="$MAKEPKG_OPTIONS --skippgpcheck"
-	fi
-	echo "Future:            $FUTURE_STATE"
-	echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
-	echo "makepkg env:       $MAKEPKG_ENV_VARS"
-	echo "makepkg options:   $MAKEPKG_OPTIONS"
-	echo "============================================================================="
-	schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
-	echo "MAKEFLAGS=-j$NEW_NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/makepkg.conf
-	schroot --run-session -c $SESSION --directory /tmp -- mkdir -p $BUILDDIR
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -- env GIT_SSL_NO_VERIFY=1 pkgctl repo clone --protocol https "$SRCPACKAGE" 2>&1 | tee -a $LOG || echo "Error: failed to download PKGBUILD for $SRCPACKAGE from $REPOSITORY" | tee -a $LOG
-	# $SRCPACKAGE is actually the binary package
-	ACTUAL_SRCPACKAGE=$(ls "$BUILDDIR")
-	# add more variations in the 2nd build: TZ (differently), LANG, LC_ALL, umask
-	schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
-	export TZ="/usr/share/zoneinfo/Etc/GMT-14"
-	export LANG="fr_CH.UTF-8"
-	export LC_ALL="fr_CH.UTF-8"
-	umask 0002
-	__END__
-	# some more output for debugging
-	set -x
-	# remove possible lock in our local session (happens when root maintenance update running while session starts)
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- rm -f /var/lib/pacman/db.lck 2>&1 | tee -a $LOG
-	# update before pulling new dependencies
-	schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- pacman -Syu --noconfirm 2>&1 | tee -a $LOG
-	# determine the version of the package being build
-	source "$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD" || echo "Failed to source PKGBUILD from '$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD'" | tee -a $LOG
-	if [ -n "$epoch" ] ; then
-		epoch="$epoch:"
-	fi
-	VERSION="$epoch$pkgver-$pkgrel"
-	echo $VERSION > $TMPDIR/b2/$SRCPACKAGE/build2.version
-	# show env variables
-	schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -u root -- su -c "bash -l -c '$MAKEPKG_ENV_VARS printenv 2>&1'" build2 | tee -a $LOG
-	# preseed GnuPG keys if relevant in order to get good error logs
-	schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "makepkg --printsrcinfo | awk -F ' = ' '/^\s+validpgpkeys/{print \$2}'| while read pgpkey; do gpg --keyserver hkp://keyserver.ubuntu.com --recv-key \$pgpkey; done" 2>&1 | tee -a $LOG
-	# nicely run makepkg with a timeout of $TIMEOUT hours
-	timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
-		schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS makepkg $MAKEPKG_OPTIONS 2>&1" | tee -a $LOG
-	PRESULT=${PIPESTATUS[0]}
-	if [ $PRESULT -eq 124 ] ; then
-		echo "$(date -u) - makepkg was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
-	fi
-	set +e
-	schroot --end-session -c $SESSION | tee -a $LOG
-	PRESULT=${PIPESTATUS[0]}
-	if [ $PRESULT -ne 0 ] ; then
-		echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
-		sleep 60
-		schroot --end-session -f -c $SESSION | tee -a $LOG
-		P2RESULT=${PIPESTATUS[0]}
-		if [ $P2RESULT -ne 0 ] ; then
-			echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
-			sleep 10
-			schroot --end-session -f -c $SESSION | tee -a $LOG
-			P3RESULT=${PIPESTATUS[0]}
-			if [ $P3RESULT -ne 0 ] ; then
-				exit 23
-			fi
-		fi
-	fi
-	if ! "$DEBUG" ; then set +x ; fi
-	set -e
-}
-
-remote_build() {
-	local BUILDNR=$1
-	local NODE=$2
-	local FQDN=$NODE.debian.net
-	get_node_information $NODE
-	set +e
-	ssh -o "Batchmode = yes" $FQDN /bin/true
-	RESULT=$?
-	# abort job if host is down
-	if [ $RESULT -ne 0 ] ; then
-		SLEEPTIME=$(echo "$BUILDNR*$BUILDNR*5"|bc)
-		echo "$(date -u) - $NODE seems to be down, sleeping ${SLEEPTIME}min before aborting this job."
-		sleep ${SLEEPTIME}m
-		unregister_build
-		exit 0
-	fi
-	ssh -o "Batchmode = yes" $FQDN /srv/jenkins/bin/reproducible_build_archlinux_pkg.sh $BUILDNR $REPOSITORY ${SRCPACKAGE} ${TMPDIR} ${SOURCE_DATE_EPOCH}
-	RESULT=$?
-	if [ $RESULT -ne 0 ] ; then
-		ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR" || true
-		if [ $RESULT -eq 23 ] ; then
-			unregister_build
-			handle_remote_error "job on $NODE could not end schroot session properly and sent error 23 so we could abort silently."
-		elif [ $RESULT -eq 42 ] ; then
-			unregister_build
-			handle_remote_error "job on $NODE after not being able to verify pgp signatures. work to debug why ahead..."
-		else
-			unregister_build
-			handle_remote_error "Warning: remote build failed with exit code $RESULT from $NODE for build #$BUILDNR for ${SRCPACKAGE} from $REPOSITORY."
-		fi
-	fi
-	rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
-	RESULT=$?
-	if [ $RESULT -ne 0 ] ; then
-		echo "$(date -u ) - rsync from $NODE failed, sleeping 2m before re-trying..."
-		sleep 2m
-		rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
-		RESULT=$?
-		if [ $RESULT -ne 0 ] ; then
-			unregister_build
-			handle_remote_error "when rsyncing remote build #$BUILDNR results from $NODE"
-		fi
-	fi
-	ls -lR $TMPDIR
-	ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR"
-	set -e
-}
-
-#
-# below is what controls the world
-#
-TIMEOUT=12	# maximum time in hours for a single build
-DATE=$(date -u +'%Y-%m-%d %H:%M')
-START=$(date +'%s')
-trap cleanup_all INT TERM EXIT
-
-#
-# determine mode
-#
-if [ "$1" = "" ] ; then
-	MODE="master"
-	mkdir -p /srv/reproducible-results/archlinux
-	TMPDIR=$(mktemp --tmpdir=/srv/reproducible-results/archlinux -d -t rb-build-XXXXXXXX)  # where everything actually happens
-	SOURCE_DATE_EPOCH=$(date +%s)
-	cd $TMPDIR
-elif [ "$1" = "1" ] || [ "$1" = "2" ] ; then
-	MODE="$1"
-	REPOSITORY="$2"
-	SRCPACKAGE="$3"
-	TMPDIR="$4"
-	SOURCE_DATE_EPOCH="$5"
-	[ -d $TMPDIR ] || mkdir -p $TMPDIR
-	cd $TMPDIR
-	mkdir -p b$MODE/$SRCPACKAGE
-	if [ "$MODE" = "1" ] ; then
-		first_build
-	else
-		second_build
-	fi
-
-	# preserve results and delete build directory
-	if [ -n "$(ls /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)/*/*.pkg.tar.zst)" ] ; then
-		# copying is enough here, we delete after this if block anyway
-		cp --preserve=timestamps -v /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)/*/*.pkg.tar.zst $TMPDIR/b$MODE/$SRCPACKAGE/
-	else
-		echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME didn't build a package!"
-		# debug
-		echo "ls $TMPDIR/b$MODE/$SRCPACKAGE/"
-		ls -Rl
-	fi
-
-	sudo rm -rf --one-file-system /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)
-	echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME done."
-	exit 0
-fi
-
-#
-# main - only used in master-mode
-#
-delay_start # randomize start times
-# first, we need to choose a package from a repository…
-REPOSITORY=""
-SRCPACKAGE=""
-VERSION=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-choose_package
-mkdir -p $BASE/archlinux/$REPOSITORY/$SRCPACKAGE
-# build package twice
-mkdir b1 b2
-# currently there are two Arch Linux build nodes…
-N1="$ARCH_BUILD_NODE1"
-N2="$ARCH_BUILD_NODE2"
-# if random number between 0 and 99 is greater than X…
-# prefer node $N1 (originally because one of the nodes was less loaded)
-if [ $((RANDOM % 100)) -gt 30 ] ; then
-	NODE1=$N1
-	NODE2=$N2
-else
-	NODE1=$N2
-	NODE2=$N1
-fi
-echo "============================================================================="
-echo "Initialising reproducibly build of ${SRCPACKAGE} in ${REPOSITORY} on ${ARCH} now."
-echo "1st build will be done on $NODE1."
-echo "2nd build will be done on $NODE2."
-echo "============================================================================="
-#
-# do 1st build
-#
-remote_build 1 ${NODE1}
-#
-# only do the 2nd build if the 1st produced results
-#
-if [ -n "$(ls $TMPDIR/b1/$SRCPACKAGE/*.pkg.tar.zst 2>/dev/null|| true)" ] ; then
-	remote_build 2 ${NODE2}
-	cd $TMPDIR/b1/$SRCPACKAGE
-	# loop through all artifacts
-	for ARTIFACT in *.pkg.tar.zst ; do
-		[ -f $ARTIFACT ] || continue
-		echo "$(date -u) - comparing results now."
-		if diff -q $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ; then
-			echo "$(date -u) - YAY - $SRCPACKAGE/$ARTIFACT build reproducible in our test framework!"
-			mkdir -p $BASE/archlinux/$REPOSITORY/$SRCPACKAGE
-			bsdtar xvf $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt
-			bsdtar xvf $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt
-			(
-				echo "<html><body><p>$SRCPACKAGE/$ARTIFACT build reproducible in our test framework:<br />"
-				(cd $TMPDIR/b1/$SRCPACKAGE ; sha256sum $ARTIFACT)
-				echo "<br />"
-				(sha256sum $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt | cut -d " " -f1)
-				echo " <a href=\"$ARTIFACT-b1.BUILDINFO.txt\">$ARTIFACT-b1.BUILDINFO.txt</a><br />"
-				(sha256sum $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt | cut -d " " -f1)
-				echo " <a href=\"$ARTIFACT-b2.BUILDINFO.txt\">$ARTIFACT-b2.BUILDINFO.txt</a><br />"
-				echo "</p></body>"
-			) > "$BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
-		elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] && [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
-			echo "$(date -u) - running diffoscope on the results now:"
-			TIMEOUT="60m"
-			DIFFOSCOPE="$(schroot --directory "$TMPDIR" -c chroot:jenkins-reproducible-${DBDSUITE}-diffoscope diffoscope -- --version 2>&1 || echo 'diffoscope-unknown-version')"
-			echo "$(date -u) - Running $DIFFOSCOPE now..."
-			call_diffoscope $SRCPACKAGE $ARTIFACT
-		elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] || [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
-			echo "$(date -u) - one of the two builds failed... delete the other one:"
-			( rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT || rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ) 2>/dev/null
-		else
-			# some packages define the package version based on the build date
-			# so our two builds end up with different package versions…
-			echo "$(date -u) - something is fishy with $SRCPACKAGE/$ARTIFACT."
-			ls $TMPDIR/b1/$SRCPACKAGE
-			ls $TMPDIR/b2/$SRCPACKAGE
-			( echo "<html><body><p>$SRCPACKAGE/$ARTIFACT built in a strange unreproducible way:<br />"
-			ls $TMPDIR/b1/$SRCPACKAGE
-			echo "<br />"
-			ls $TMPDIR/b2/$SRCPACKAGE
-			echo "<br />"
-			echo "Maybe $BUILD_URL has some more information why this happened."
-			echo "</p></body>"
-			) > "$BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
-		fi
-		# publish page
-		if [ -f $TMPDIR/$SRCPACKAGE/$ARTIFACT.html ] ; then
-			cp $TMPDIR/$SRCPACKAGE/$ARTIFACT.html $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-			#irc_message archlinux-reproducible "$REPRODUCIBLE_URL/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html - unreproducible"
-		fi
-	done
-else
-	echo "$(date -u) - build1 didn't create a package, skipping build2!"
-fi
-# publish logs
-calculate_build_duration
-cd $TMPDIR/b1/$SRCPACKAGE
-cp build1.log $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-[ ! -f $TMPDIR/b2/$SRCPACKAGE/build2.log ] || cp $TMPDIR/b2/$SRCPACKAGE/build2.log $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-echo $DURATION > $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.build_duration || true
-# make pkg.build_duration the oldest of this build, so we can use it as reference later
-touch --date="@$START" $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.build_duration
-if [ -f $TMPDIR/b2/$SRCPACKAGE/build2.version ] ; then
-	cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-	cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-elif [ -f build1.version ] ; then
-	cp build1.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-	cp $TMPDIR/b1/$SRCPACKAGE/build1.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-else
-	# this should not happen but does, so deal with it
-	echo "$VERSION" > $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-fi
-
-echo "$(date -u) - $REPRODUCIBLE_URL/archlinux/$REPOSITORY/$SRCPACKAGE/ updated."
-# force update of HTML snipplet in reproducible_html_archlinux.sh
-[ ! -f $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.state ] || rm $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.state
-HTML_BUFFER=''
-create_pkg_html
-update_pkg_in_db
-
-cd
-cleanup_all
-trap - INT TERM EXIT
-
-# vim: set sw=0 noet :


=====================================
bin/reproducible_common.sh
=====================================
@@ -69,13 +69,6 @@ ALPINE_REPOS="main community"
 ALPINE_PKGS=/srv/reproducible-results/alpine_pkgs
 ALPINE_BASE="$BASE/alpine"
 
-# common settings for testing Arch Linux
-ARCHLINUX_REPOS="core extra multilib"
-ARCHLINUX_PKGS=/srv/reproducible-results/archlinux_pkgs
-ARCHBASE=$BASE/archlinux
-ARCH_BUILD_NODE1="osuosl1-amd64"
-ARCH_BUILD_NODE2="osuosl2-amd64"
-
 # common settings for testing rpm based distros
 RPM_BUILD_NODE=osuosl1-amd64
 RPM_PKGS=/srv/reproducible-results/rpm_pkgs
@@ -364,10 +357,10 @@ write_page_header() {
 		write_page "   Besides Debian we are also testing "
 		write_page "   <li><a href=\"/coreboot/\">coreboot</a></li>,"
 		write_page "   <li><a href=\"/openwrt/\">OpenWrt</a></li>, "
-		write_page "   <li><a href=\"/netbsd/\">NetBSD</a></li>, "
-		write_page "   <li><a href=\"/freebsd/\">FreeBSD</a></li>, "
-		write_page "   and <li><a href=\"/archlinux/\">Arch Linux</a></li>"
+		write_page "   <li><a href=\"/netbsd/\">NetBSD</a></li> "
+		write_page "   and <li><a href=\"/freebsd/\">FreeBSD</a></li>. "
 		write_page "."
+		write_page "   <li><a href=\"https://reproducible.archlinux.org/\">Arch Linux</a></li>, "
 		write_page "   <li><a href=\"http://rb.zq1.de/\">openSUSE</a></li>, "
 		write_page "   <li><a href=\"https://r13y.com/\">NixOS</a></li>, "
 		write_page "   <li><a href=\"https://qa.guix.gnu.org/reproducible-builds\">Guix</a></li> and "
@@ -417,18 +410,6 @@ write_page_intro() {
 		write_page " <li>user tools, for users to verify all of this easily.</li>"
 		write_page "</ul></p>"
 		write_page "<p>If you want to help out or discuss reproducible builds in $PROJECTNAME, please join #alpine-reproducible on freenode.</p>"
-	elif [ "$1" = "Arch Linux" ] ; then
-		local PROJECTNAME="Arch Linux"
-		write_page "        <p><em>Reproducible $PROJECTNAME</em> is an effort to apply this to $PROJECTNAME. Thus $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>."
-		write_page "   Please note that this is still at an early stage. Also there are more variations expected to be seen in the wild."
-		write_page "Missing bits for <em>testing</em> Arch Linux:<ul>"
-		write_page " <li>cross references to <a href=\"https://tests.reproducible-builds.org/debian/index_issues.html\">Debian notes</a> - and having Arch Linux specific notes.</li>"
-		write_page "</ul></p>"
-		write_page "<p>Missing bits for Arch Linux:<ul>"
-		write_page " <li>code needs to be written to compare the packages built twice here against newly built packages from the Official Arch Linux repositories.</li>"
-		write_page " <li>user tools, for users to verify all of this easily.</li>"
-		write_page "</ul></p>"
-		write_page "<p>If you want to help out or discuss reproducible builds in $PROJECTNAME, please join #archlinux-reproducible on freenode.</p>"
 	elif [ "$1" = "fedora-23" ] ; then
 		local PROJECTNAME="Fedora 23"
 		write_page "        <p><em>Reproducible $PROJECTNAME</em> is a (currently somewhat stalled) effort to apply this to $PROJECTNAME, which is rather obvious with 23… <br/> $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>. Please note that the toolchain is not varied at all as the rebuild happens on exactly the same system. More variations are expected to be seen in the wild.</p>"
@@ -449,8 +430,6 @@ write_page_footer() {
 		other_distro_details="NetBSD® is a registered trademark of The NetBSD Foundation, Inc."
 	elif [ "$1" = "FreeBSD" ] ; then
 		other_distro_details="FreeBSD is a registered trademark of The FreeBSD Foundation. The FreeBSD logo and The Power to Serve are trademarks of The FreeBSD Foundation."
-	elif [ "$1" = "Arch Linux" ] ; then
-		other_distro_details='The <a href=\"https://www.archlinux.org\">Arch Linux</a> name and logo are recognized trademarks. Some rights reserved. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.'
 	elif [ "$1" = "fedora-23" ] ; then
 			other_distro_details="Fedora is sponsored by Red Hat. © 2017 Red Hat, Inc. and others."
 	else
@@ -1027,184 +1006,6 @@ include_icon(){
 	echo "       <img src=\"/userContent/static/$PNG.png\" alt=\"$ALT icon\" /> $TEXT" >> $HTML_BUFFER
 }
 
-create_pkg_html() {
-	local ARCHLINUX_PKG_PATH=$ARCHBASE/$REPOSITORY/$SRCPACKAGE
-	local HTML_BUFFER=$(mktemp -t archlinuxrb-html-XXXXXXXX)
-	local buffer_message
-	local STATE
-
-	# clear files from previous builds
-	cd "$ARCHLINUX_PKG_PATH"
-	for file in build1.log build2.log build1.version build2.version *BUILDINFO.txt *.html; do
-		if [ -f $file ] && [ pkg.build_duration -nt $file ] ; then
-			rm $file
-			echo "$ARCHLINUX_PKG_PATH/$file older than $ARCHLINUX_PKG_PATH/pkg.build_duration, thus deleting it."
-		fi
-	done
-
-	echo "     <tr>" >> $HTML_BUFFER
-	echo "      <td>$REPOSITORY</td>" >> $HTML_BUFFER
-	echo "      <td><a href=\"https://www.archlinux.org/packages/$REPOSITORY/x86_64/$SRCPACKAGE/\">$SRCPACKAGE</a></td>" >> $HTML_BUFFER
-	echo "      <td>$VERSION</td>" >> $HTML_BUFFER
-	echo "      <td>" >> $HTML_BUFFER
-	#
-	#
-	if [ -z "$(cd $ARCHLINUX_PKG_PATH/ ; ls *.pkg.tar.zst.html 2>/dev/null || true)" ] ; then
-		# this horrible if elif elif elif elif...  monster should be replaced
-		# by using pacman's exit code which is possible since sometime in 2018
-
-		# check different states and figure out what the page should look like
-		if find_in_buildlogs '^error: failed to prepare transaction \(conflicting dependencies\)'; then
-			STATE=DEPWAIT_0
-			buffer_message='could not resolve dependencies as there are conflicts'
-		elif find_in_buildlogs '==> ERROR: (Could not resolve all dependencies|.pacman. failed to install missing dependencies)'; then
-			if find_in_buildlogs 'error: failed to init transaction \(unable to lock database\)'; then
-				STATE=DEPWAIT_2
-				buffer_message='pacman could not lock database'
-			else
-				STATE=DEPWAIT_1
-				buffer_message='could not resolve dependencies'
-			fi
-		elif find_in_buildlogs '^error: unknown package: '; then
-			STATE=404_0
-			buffer_message='unknown package'
-		elif find_in_buildlogs '(==> ERROR: Failure while downloading|==> ERROR: One or more PGP signatures could not be verified|==> ERROR: One or more files did not pass the validity check|==> ERROR: Integrity checks \(.*\) differ in size from the source array|==> ERROR: Failure while branching|==> ERROR: Failure while creating working copy|Failed to source PKGBUILD.*PKGBUILD)'; then
-			REASON="download failed"
-			EXTRA_REASON=""
-			STATE=404_0
-			if find_in_buildlogs 'FAILED \(unknown public key'; then
-				STATE=404_6
-				EXTRA_REASON="to verify source with PGP due to unknown public key"
-			elif find_in_buildlogs 'The requested URL returned error: 403'; then
-				STATE=404_2
-				EXTRA_REASON="with 403 - forbidden"
-			elif find_in_buildlogs 'The requested URL returned error: 500'; then
-				STATE=404_4
-				EXTRA_REASON="with 500 - internal server error"
-			elif find_in_buildlogs 'The requested URL returned error: 503'; then
-				STATE=404_5
-				EXTRA_REASON="with 503 - service unavailable"
-			elif find_in_buildlogs '==> ERROR: One or more PGP signatures could not be verified'; then
-				STATE=404_7
-				EXTRA_REASON="to verify source with PGP signatures"
-			elif find_in_buildlogs '(SSL certificate problem: unable to get local issuer certificate|^bzr: ERROR: .SSL: CERTIFICATE_VERIFY_FAILED)'; then
-				STATE=404_1
-				EXTRA_REASON="with SSL problem"
-			elif find_in_buildlogs '==> ERROR: One or more files did not pass the validity check'; then
-				STATE=404_8
-				REASON="downloaded ok but failed to verify source"
-			elif find_in_buildlogs '==> ERROR: Integrity checks \(.*\) differ in size from the source array'; then
-				STATE=404_9
-				REASON="Integrity checks differ in size from the source array"
-			elif find_in_buildlogs 'The requested URL returned error: 404'; then
-				STATE=404_3
-				EXTRA_REASON="with 404 - file not found"
-			elif find_in_buildlogs 'fatal: the remote end hung up unexpectedly'; then
-				STATE=404_A
-				EXTRA_REASON="could not clone git repository"
-			elif find_in_buildlogs 'The requested URL returned error: 504'; then
-				STATE=404_B
-				EXTRA_REASON="with 504 - gateway timeout"
-			elif find_in_buildlogs '==> ERROR: Failure while downloading .* git repo'; then
-				STATE=404_C
-				EXTRA_REASON="from git repo"
-			fi
-			buffer_message="$REASON $EXTRA_REASON"
-		elif find_in_buildlogs '==> ERROR: (install file .* does not exist or is not a regular file|The download program wget is not installed)'; then
-			STATE=FTBFS_0
-			buffer_message='failed to build, requirements not met'
-		elif find_in_buildlogs '==> ERROR: A failure occurred in check'; then
-			STATE=FTBFS_1
-			buffer_message='failed to build while running tests'
-		elif find_in_buildlogs '==> ERROR: (An unknown error has occurred|A failure occurred in (build|package|prepare))'; then
-			STATE=FTBFS_2
-			buffer_message='failed to build'
-		elif find_in_buildlogs 'makepkg was killed by timeout after'; then
-			STATE=FTBFS_3
-			buffer_message='failed to build, killed by timeout'
-		elif find_in_buildlogs '==> ERROR: .* contains invalid characters:'; then
-			STATE=FTBFS_4
-			buffer_message='failed to build, pkg relations contain invalid characters'
-		else
-			STATE=$(query_db "SELECT r.status FROM results AS r
-				JOIN sources as s on s.id=r.package_id
-				WHERE s.architecture='x86_64'
-				AND s.name='$SRCPACKAGE'
-				AND s.suite='archlinux_$REPOSITORY';")
-			if [ "$STATE" = "blacklisted" ] ; then
-				buffer_message='blacklisted'
-			else
-				STATE=UNKNOWN
-				buffer_message='probably failed to build from source, please investigate'
-			fi
-		fi
-		# print build failures
-		if [ "$STATE" = "UNKNOWN" ]; then
-			echo "       $buffer_message" >> $HTML_BUFFER
-		else
-			include_icon $STATE "$buffer_message"
-		fi
-	else
-		local STATE=reproducible
-		local SOME_GOOD=false
-		for ARTIFACT in $(cd $ARCHLINUX_PKG_PATH/ ; ls *.pkg.tar.zst.html || true) ; do
-			[ -f $ARTIFACT ] || continue
-			if [[ $ARTIFACT != *"$VERSION"* ]]; then
-				echo "deleting $ARTIFACT as version is not $VERSION"
-				rm -f $ARTIFACT
-				continue
-			elif grep -q 'build reproducible in our test framework' $ARCHLINUX_PKG_PATH/$ARTIFACT; then
-				SOME_GOOD=true
-				STATE=reproducible
-				include_icon $STATE "<a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is reproducible in our current test framework<br />"
-			else
-				# change $STATE unless we have found .buildinfo differences already...
-				if [ "$STATE" != "FTBR_0" ] ; then
-					STATE=FTBR_1
-				fi
-				# this shouldn't happen, but (for now) it does, so lets mark them…
-				EXTRA_REASON=""
-				if grep -q 'class="source">.BUILDINFO' $ARCHLINUX_PKG_PATH/$ARTIFACT; then
-					STATE=FTBR_0
-					EXTRA_REASON=" with variations in .BUILDINFO"
-				fi
-				include_icon $STATE "<a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is unreproducible$EXTRA_REASON<br />"
-			fi
-		done
-		# we only count source packages…
-		if [[ $STATE = FTBR_1 && $SOME_GOOD = true ]]; then
-			STATE=FTBR_2
-		fi
-	fi
-	echo "      </td>" >> $HTML_BUFFER
-	echo "      <td>$DATE" >> $HTML_BUFFER
-	local DURATION=$(cat $ARCHLINUX_PKG_PATH/pkg.build_duration 2>/dev/null || true)
-	if [ -n "$DURATION" ]; then
-		local HOUR=$(echo "$DURATION/3600"|bc)
-		local MIN=$(echo "($DURATION-$HOUR*3600)/60"|bc)
-		local SEC=$(echo "$DURATION-$HOUR*3600-$MIN*60"|bc)
-		BUILD_DURATION="<br />${HOUR}h:${MIN}m:${SEC}s"
-	else
-		BUILD_DURATION=" "
-	fi
-	echo "       $BUILD_DURATION</td>" >> $HTML_BUFFER
-
-	echo "      <td>" >> $HTML_BUFFER
-	for LOG in build1.log build2.log ; do
-		if [ -f $ARCHLINUX_PKG_PATH/$LOG ] ; then
-			if [ "$LOG" = "build2.log" ] ; then
-				echo "       <br />" >> $HTML_BUFFER
-			fi
-			echo "       <a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$LOG\">$LOG</a> ($(get_filesize "${ARCHLINUX_PKG_PATH}/${LOG}"))" >> $HTML_BUFFER
-		fi
-	done
-	echo "      </td>" >> $HTML_BUFFER
-	echo "     </tr>" >> $HTML_BUFFER
-	mv $HTML_BUFFER $ARCHLINUX_PKG_PATH/pkg.html
-	chmod 644 $ARCHLINUX_PKG_PATH/pkg.html
-	echo $STATE > $ARCHLINUX_PKG_PATH/pkg.state
-}
-
 create_alpine_pkg_html() {
 	local ALPINE_PKG_PATH=$ALPINE_BASE/$REPOSITORY/$SRCPACKAGE
 	local HTML_BUFFER=$(mktemp -t alpinerb-html-XXXXXXXX)


=====================================
bin/reproducible_html_alpine.sh
=====================================
@@ -486,7 +486,7 @@ else
 		exit 1
 	fi
 	HTML_BUFFER=''
-	create_pkg_html
+	create_alpine_pkg_html
 fi
 echo "$(date -u) - all done."
 


=====================================
bin/reproducible_html_archlinux.sh deleted
=====================================
@@ -1,492 +0,0 @@
-#!/bin/bash
-
-# Copyright 2014-2023 Holger Levsen <holger at layer-acht.org>
-#                2015 anthraxx <levente at leventepolyak.net>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-#
-#
-# reproducible_html_archlinux.sh can be called in two ways:
-# - without params, then it will build all the index and dashboard pages
-# - with exactly two params, $REPOSITORY and $SRCPACKAGE, in which case that packages html page will be created
-#
-
-
-#
-# helper functions
-#
-get_state_from_counter() {
-	local counter=$1
-	case $counter in
-		0)	STATE=reproducible ;;
-		1)	STATE=FTBR ;;
-		2)	STATE=FTBFS ;;
-		3)	STATE=DEPWAIT ;;
-		4)	STATE=404 ;;
-		5)	STATE=blacklisted ;;
-		6)	STATE=UNKNOWN ;;
-	esac
-}
-
-
-include_pkg_html_in_page(){
-	cat $ARCHBASE/$REPOSITORY/$SRCPACKAGE/pkg.html >> $PAGE 2>/dev/null || true
-}
-
-include_pkg_table_header_in_page(){
-	write_page "    <table><tr><th>repository</th><th>source package</th><th>version</th><th>test result</th><th>test date<br />test duration</th><th>1st build log<br />2nd build log</th></tr>"
-}
-
-repostats(){
-	#
-	# gather data
-	# write csv file for $REPOSITORY
-	# write $HTML_REPOSTATS
-	#
-	for REPOSITORY in $ARCHLINUX_REPOS ; do
-		echo "$(date -u) - starting to analyse build results for '$REPOSITORY'."
-		# prepare stats per repository
-		SUITE="$REPOSITORY"
-		TOTAL=$(query_db "SELECT count(*) FROM sources AS s WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
-		TESTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
-		NR_GOOD=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='reproducible';")
-		NR_FTBR=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBR_%';")
-		NR_FTBFS=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBFS_%';")
-		NR_DEPWAIT=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'DEPWAIT_%';")
-		NR_404=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE '404_%';")
-		NR_BLACKLISTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='blacklisted';")
-		NR_UNKNOWN=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND  s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'UNKNOWN';")
-		NR_UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s WHERE s.architecture='x86_64' AND s.distribution=$DISTROID AND s.suite='$SUITE' AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
-		if [ $NR_UNTESTED -ne 0 ] ; then
-			let NR_UNKNOWN=$NR_UNKNOWN+$NR_UNTESTED
-		fi
-		PERCENT_TOTAL=$(echo "scale=1 ; ($TESTED*100/$TOTAL)" | bc)
-		if [ $(echo $PERCENT_TOTAL/1|bc) -lt 99 ] ; then
-			NR_TESTED="$TESTED <span style=\"font-size:0.8em;\">(tested $PERCENT_TOTAL% of $TOTAL)</span>"
-		else
-			NR_TESTED=$TESTED
-		fi
-		echo "     <tr>" >> $HTML_REPOSTATS
-		echo "      <td><a href='/archlinux/$REPOSITORY.html'>$REPOSITORY</a></td><td><a href='/archlinux/$REPOSITORY.html'>$NR_TESTED</a></td>" >> $HTML_REPOSTATS
-		counter=0
-		for i in $NR_GOOD $NR_FTBR $NR_FTBFS $NR_DEPWAIT $NR_404 $NR_BLACKLISTED $NR_UNKNOWN ; do
-			get_state_from_counter $counter
-			let counter+=1
-			PERCENT_i=$(echo "scale=1 ; ($i*100/$TESTED)" | bc)
-			if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
-				echo "      <td><a href='/archlinux/state_${REPOSITORY}_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
-			else
-				echo "      <td><a href='/archlinux/state_${REPOSITORY}_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
-			fi
-		done
-		echo "     </tr>" >> $HTML_REPOSTATS
-		if [ ! -f $ARCHBASE/$REPOSITORY.csv ] ; then
-			echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ARCHBASE/$REPOSITORY.csv
-		fi
-		if ! grep -q $YESTERDAY $ARCHBASE/$REPOSITORY.csv ; then
-			let REAL_UNKNOWN=$TOTAL-$NR_GOOD-$NR_FTBR-$NR_FTBFS-$NR_DEPWAIT-$NR_404 || true
-			echo $YESTERDAY,$NR_GOOD,$NR_FTBR,$NR_FTBFS,$NR_DEPWAIT,$NR_404,$REAL_UNKNOWN >> $ARCHBASE/$REPOSITORY.csv
-		fi
-		IMAGE=$ARCHBASE/$REPOSITORY.png
-		if [ ! -f $IMAGE ] || [ $ARCHBASE/$REPOSITORY.csv -nt $IMAGE ] ; then
-			echo "Updating $IMAGE..."
-			/srv/jenkins/bin/make_graph.py $ARCHBASE/$REPOSITORY.csv $IMAGE 6 "Reproducibility status for Arch Linux packages in $REPOSITORY" "Amount (total)" $WIDTH $HEIGHT
-		fi
-		#
-		# prepare ARCHLINUX totals
-		#
-		set +e
-		let ARCHLINUX_TOTAL+=$TOTAL
-		let ARCHLINUX_TESTED+=$TESTED
-		let ARCHLINUX_NR_FTBFS+=$NR_FTBFS
-		let ARCHLINUX_NR_FTBR+=$NR_FTBR
-		let ARCHLINUX_NR_DEPWAIT+=$NR_DEPWAIT
-		let ARCHLINUX_NR_404+=$NR_404
-		let ARCHLINUX_NR_GOOD+=$NR_GOOD
-		let ARCHLINUX_NR_BLACKLISTED+=$NR_BLACKLISTED
-		let ARCHLINUX_NR_UNKNOWN+=$NR_UNKNOWN
-		set -e
-	done
-	#
-	# prepare stats per repository
-	#
-	ARCHLINUX_PERCENT_TOTAL=$(echo "scale=1 ; ($ARCHLINUX_TESTED*100/$ARCHLINUX_TOTAL)" | bc)
-	if [ $(echo $ARCHLINUX_PERCENT_TOTAL/1|bc) -lt 99 ] ; then
-		NR_TESTED="$ARCHLINUX_TESTED <span style=\"font-size:0.8em;\">(tested $ARCHLINUX_PERCENT_TOTAL% of $ARCHLINUX_TOTAL)</span>"
-	else
-		NR_TESTED=$ARCHLINUX_TESTED
-	fi
-	echo "     <tr>" >> $HTML_REPOSTATS
-	echo "      <td><b>all combined</b></td><td>$NR_TESTED</td>" >> $HTML_REPOSTATS
-	counter=0
-	for i in $ARCHLINUX_NR_GOOD $ARCHLINUX_NR_FTBR $ARCHLINUX_NR_FTBFS $ARCHLINUX_NR_DEPWAIT $ARCHLINUX_NR_404 $ARCHLINUX_NR_BLACKLISTED $ARCHLINUX_NR_UNKNOWN ; do
-		get_state_from_counter $counter
-		let counter+=1
-		PERCENT_i=$(echo "scale=1 ; ($i*100/$ARCHLINUX_TESTED)" | bc)
-		if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
-			echo "      <td><a href='/archlinux/state_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
-		else
-			echo "      <td><a href='/archlinux/state_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
-		fi
-	done
-	echo "     </tr>" >> $HTML_REPOSTATS
-	#
-	# write csv file for totals
-	#
-	if [ ! -f $ARCHBASE/archlinux.csv ] ; then
-		echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ARCHBASE/archlinux.csv
-	fi
-	if ! grep -q $YESTERDAY $ARCHBASE/archlinux.csv ; then
-		let ARCHLINUX_REAL_UNKNOWN=$ARCHLINUX_TOTAL-$ARCHLINUX_NR_GOOD-$ARCHLINUX_NR_FTBR-$ARCHLINUX_NR_FTBFS-$ARCHLINUX_NR_DEPWAIT-$ARCHLINUX_NR_404 || true
-		echo $YESTERDAY,$ARCHLINUX_NR_GOOD,$ARCHLINUX_NR_FTBR,$ARCHLINUX_NR_FTBFS,$ARCHLINUX_NR_DEPWAIT,$ARCHLINUX_NR_404,$ARCHLINUX_REAL_UNKNOWN >> $ARCHBASE/archlinux.csv
-	fi
-	IMAGE=$ARCHBASE/archlinux.png
-	if [ ! -f $IMAGE ] || [ $ARCHBASE/archlinux.csv -nt $IMAGE ] ; then
-		echo "Updating $IMAGE..."
-		/srv/jenkins/bin/make_graph.py $ARCHBASE/archlinux.csv $IMAGE 6 "Reproducibility status for all tested Arch Linux packages" "Amount (total)" $WIDTH $HEIGHT
-		irc_message archlinux-reproducible "Daily graphs on $REPRODUCIBLE_URL/archlinux/ updated, $(echo "scale=1 ; ($ARCHLINUX_NR_GOOD*100/$ARCHLINUX_TESTED)" | bc)% reproducible packages in our current test framework."
-	fi
-}
-
-archlinux_page_header(){
-	echo "$(date -u) - starting to write $PAGE"
-	cat > $PAGE <<- EOF
-	<!DOCTYPE html>
-	<html lang="en-US">
-	  <head>
-	    <meta charset="UTF-8">
-	    <title>$TITLE</title>
-	    <link rel='stylesheet' href='global.css' type='text/css' media='all' />
-	  </head>
-	  <body>
-	    <div id="archnavbar">
-		    <div id="logo"></div>
-	    </div>
-	    <div class="content">
-	      <h1><a href='/archlinux/'>Reproducible Arch Linux</a>?!</h1>
-	      <div class="page-content">
-
-	EOF
-}
-
-archlinux_page_footer(){
-	write_page "</div></div>"
-	write_page_footer 'Arch Linux'
-	publish_page archlinux
-}
-
-archlinux_repostats_table(){
-	write_page "    <table><tr><th>repository</th><th>all source packages</th>"
-	write_page "     <th><a href='/archlinux/state_reproducible.html'>reproducible</a></th>"
-	write_page "     <th><a href='/archlinux/state_FTBR.html'>unreproducible</a></th>"
-	write_page "     <th><a href='/archlinux/state_FTBFS.html'>failing to build</a></th>"
-	write_page "     <th><a href='/archlinux/state_DEPWAIT.html'>in depwait state</a></th>"
-	write_page "     <th><a href='/archlinux/state_404.html'>download problems</a></th>"
-	write_page "     <th><a href='/archlinux/state_blacklisted.html'>blacklisted</a></th>"
-	write_page "     <th><a href='/archlinux/state_UNKNOWN.html'>unknown state</a></th></tr>"
-	cat $HTML_REPOSTATS >> $PAGE
-	write_page "    </table>"
-	write_page "    <p>("
-	write_page "     <a href='/archlinux/recent_builds.html'>recent builds</a>,"
-	write_page "     <a href='/archlinux/scheduled.html'>currently scheduled</a>"
-	write_page "       )</p>"
-}
-
-dashboard_page(){
-	PAGE=archlinux.html
-	TITLE="Reproducible archlinux ?!"
-	archlinux_page_header
-	write_page_intro 'Arch Linux'
-	archlinux_repostats_table
-	# include graphs
-	write_page '<p style="clear:both;">'
-	for REPOSITORY in $ARCHLINUX_REPOS ; do
-		write_page "<a href=\"/archlinux/$REPOSITORY.png\"><img src=\"/archlinux/$REPOSITORY.png\" class=\"tripleview\" alt=\"$REPOSITORY stats\"></a>"
-	done
-	write_page '</p><p style="clear:both;"><center>'
-	write_page "<a href=\"/archlinux/archlinux.png\"><img src=\"/archlinux/archlinux.png\" alt=\"total Arch Linux stats\"></a></p>"
-	write_variation_table 'Arch Linux'
-	archlinux_page_footer
-}
-
-repository_pages(){
-	for REPOSITORY in $ARCHLINUX_REPOS ; do
-		PAGE=$REPOSITORY.html
-		TITLE="Reproducible archlinux $REPOSITORY ?!"
-		archlinux_page_header
-		archlinux_repostats_table
-		SUITE="$REPOSITORY"
-		TESTED=$(query_db "SELECT count(*) FROM sources AS s
-					JOIN results AS r
-					ON s.id=r.package_id
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE';")
-		write_page "<h2>$TESTED packages in repository $REPOSITORY</h2>"
-		include_pkg_table_header_in_page
-		REPO_PKGS=$(query_db "SELECT s.name FROM sources
-				AS s JOIN results AS r
-				ON s.id=r.package_id
-				WHERE s.distribution=$DISTROID
-				AND s.architecture='x86_64'
-				AND s.suite='$SUITE'
-				ORDER BY r.status,s.name")
-		for SRCPACKAGE in $REPO_PKGS ; do
-			include_pkg_html_in_page
-		done
-		write_page "    </table>"
-		archlinux_page_footer
-	done
-}
-
-state_pages(){
-	for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
-		PAGE=state_$STATE.html
-		TITLE="Reproducible archlinux, packages in state $STATE"
-		archlinux_page_header
-		archlinux_repostats_table
-		TESTED=$(query_db "SELECT count(*) FROM sources AS s
-				JOIN results AS r
-				ON s.id=r.package_id
-				WHERE s.distribution=$DISTROID
-				AND s.architecture='x86_64'
-				AND r.status LIKE '$STATE%';")
-		if [ "$STATE" = "UNKNOWN" ] ; then
-			# untested packages are also state UNKNOWN...
-			UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
-			if [ $UNTESTED -ne 0 ] ; then
-				let TESTED=$TESTED+$UNTESTED
-			fi
-		fi
-		write_page "<h2>$TESTED packages in $STATE state</h2>"
-		include_pkg_table_header_in_page
-		for REPOSITORY in $ARCHLINUX_REPOS ; do
-			SUITE="$REPOSITORY"
-			STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
-					JOIN results AS r
-					ON s.id=r.package_id
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE'
-					AND r.status LIKE '$STATE%'
-					ORDER BY r.status,s.name")
-			for SRCPACKAGE in ${STATE_PKGS} ; do
-				include_pkg_html_in_page
-			done
-			if [ "$STATE" = "UNKNOWN" ] ; then
-				# untested packages are also state UNKNOWN...
-				STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE'
-					AND NOT exists (SELECT FROM results WHERE package_id = s.id )
-					ORDER BY s.name")
-				for SRCPACKAGE in ${STATE_PKGS} ; do
-					include_pkg_html_in_page
-				done
-			fi
-		done
-		write_page "    </table>"
-		archlinux_page_footer
-	done
-}
-
-repository_state_pages(){
-	for REPOSITORY in $ARCHLINUX_REPOS ; do
-		SUITE="$REPOSITORY"
-		for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
-			PAGE=state_${REPOSITORY}_$STATE.html
-			TITLE="Reproducible archlinux, packages in $REPOSITORY in state $STATE"
-			archlinux_page_header
-			archlinux_repostats_table
-			TESTED=$(query_db "SELECT count(*) FROM sources AS s
-					JOIN results AS r
-					ON s.id=r.package_id
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE'
-					AND r.status LIKE '$STATE%';")
-			if [ "$STATE" = "UNKNOWN" ] ; then
-				# untested packages are also state UNKNOWN...
-				UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
-						WHERE s.distribution=$DISTROID
-						AND s.architecture='x86_64'
-						AND s.suite='$SUITE'
-						AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
-				if [ $UNTESTED -ne 0 ] ; then
-					let TESTED=$TESTED+$UNTESTED
-				fi
-			fi
-			write_page "<h2>$TESTED packages in $REPOSITORY in $STATE state</h2>"
-			include_pkg_table_header_in_page
-			STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
-					JOIN results AS r
-					ON s.id=r.package_id
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE'
-					AND r.status LIKE '$STATE%'
-					ORDER BY r.status,s.name")
-			for SRCPACKAGE in ${STATE_PKGS} ; do
-				include_pkg_html_in_page
-			done
-			if [ "$STATE" = "UNKNOWN" ] ; then
-				# untested packages are also state UNKNOWN...
-				STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
-					WHERE s.distribution=$DISTROID
-					AND s.architecture='x86_64'
-					AND s.suite='$SUITE'
-					AND NOT exists (SELECT FROM results WHERE package_id = s.id )
-					ORDER BY s.name")
-				for SRCPACKAGE in ${STATE_PKGS} ; do
-					include_pkg_html_in_page
-				done
-			fi
-			write_page "    </table>"
-			archlinux_page_footer
-		done
-	done
-}
-
-recent_builds_page(){
-	PAGE=recent_builds.html
-	TITLE="Reproducible archlinux, builds in the last 24h"
-	archlinux_page_header
-	archlinux_repostats_table
-	MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '24 hours ago')"
-	RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
-				JOIN results AS r
-				ON s.id=r.package_id
-				WHERE s.distribution=$DISTROID
-				AND s.architecture='x86_64'
-				AND r.build_date > '$MAXDATE'")
-	write_page "<h2>$RECENT builds of Archlinux packages in the last 24h</h2>"
-	include_pkg_table_header_in_page
-	STATE_PKGS=$(query_db "SELECT s.name, s.suite FROM sources AS s
-				JOIN results AS r
-				ON s.id=r.package_id
-				WHERE s.distribution=$DISTROID
-				AND s.architecture='x86_64'
-				AND r.build_date > '$MAXDATE'
-				ORDER BY r.build_date
-				DESC")
-	for LINE in ${STATE_PKGS} ; do
-		SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
-		REPOSITORY=$(echo "$LINE" | cut -d "|" -f2)
-		include_pkg_html_in_page
-	done
-	write_page "    </table>"
-	archlinux_page_footer
-}
-
-currently_scheduled_page(){
-	PAGE=scheduled.html
-	TITLE="Reproducible archlinux, packages currently scheduled"
-	archlinux_page_header
-	archlinux_repostats_table
-	TESTED=$(query_db "SELECT count(*)
-			FROM sources AS s
-			JOIN schedule AS sch
-			ON s.id=sch.package_id
-			WHERE s.distribution=$DISTROID
-			AND s.architecture='x86_64'
-			AND sch.date_build_started IS NULL")
-	write_page "<h2>Currently $TESTED scheduled builds of Archlinux packages</h2>"
-	write_page "    <table><tr><th>source package</th><th>repository</th><th>version</th><th>scheduled</th></tr>"
-	STATE_PKGS=$(query_db "SELECT s.name, s.suite, s.version, sch.date_scheduled
-			FROM sources AS s
-			JOIN schedule AS sch
-			ON s.id=sch.package_id
-			WHERE s.distribution=$DISTROID
-			AND s.architecture='x86_64'
-			AND sch.date_build_started IS NULL
-			ORDER BY sch.date_scheduled, s.name")
-	OIFS=$IFS
-	IFS=$'\012'
-	for LINE in ${STATE_PKGS} ; do
-		SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
-		REPOSITORY=$(echo "$LINE" | cut -d "|" -f2)
-		VERSION=$(echo "$LINE" | cut -d "|" -f3)
-		SCH_DATE=$(echo "$LINE" | cut -d "|" -f4-)
-		write_page "     <tr><td>$SRCPACKAGE</td><td>$REPOSITORY</td><td>$VERSION</td><td>$SCH_DATE</td></tr>"
-	done
-	IFS=$OIFS
-	write_page "    </table>"
-	archlinux_page_footer
-}
-
-#
-# main
-#
-echo "$(date -u) - starting."
-YESTERDAY=$(date '+%Y-%m-%d' -d "-1 day")
-PAGE=""
-TITLE=""
-STATE=""
-REPOSITORY=""
-SRCPACKAGE=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-
-if [ -z "$1" ] ; then
-	MEMBERS_FTBFS="0 1 2 3 4"
-	MEMBERS_DEPWAIT="0 1 2"
-	MEMBERS_404="0 1 2 3 4 5 6 7 8 9 A B C"
-	MEMBERS_FTBR="0 1 2"
-	HTML_REPOSTATS=$(mktemp -t archlinuxrb-html-XXXXXXXX)
-	ARCHLINUX_TOTAL=0
-	ARCHLINUX_TESTED=0
-	ARCHLINUX_NR_FTBFS=0
-	ARCHLINUX_NR_FTBR=0
-	ARCHLINUX_NR_DEPWAIT=0
-	ARCHLINUX_NR_404=0
-	ARCHLINUX_NR_GOOD=0
-	ARCHLINUX_NR_BLACKLISTED=0
-	ARCHLINUX_NR_UNKNOWN=0
-	WIDTH=1920
-	HEIGHT=960
-	# variables related to the stats we update
-	# FIELDS[0]="datum, reproducible, FTBR, FTBFS, other, untested" # for this Arch Linux still uses a .csv file...
-	FIELDS[1]="datum"
-	for i in reproducible FTBR FTBFS other ; do
-	        for j in $SUITES ; do
-	                FIELDS[1]="${FIELDS[1]}, ${i}_${j}"
-	        done
-	done
-	FIELDS[2]="datum, oldest"
-
-	repostats
-	dashboard_page
-	currently_scheduled_page
-	recent_builds_page
-	repository_pages
-	state_pages
-	repository_state_pages
-
-	rm $HTML_REPOSTATS > /dev/null
-elif [ -z "$2" ] ; then
-	echo "$(date -u) - $0 needs two params or none, exiting."
-	# add code here to also except core, extra or multilib ...
-	exit 1
-else
-	REPOSITORY=$1
-	SRCPACKAGE=$2
-	if [ ! -d $ARCHBASE/$REPOSITORY/$SRCPACKAGE ] ; then
-		echo "$(date -u) - $ARCHBASE/$REPOSITORY/$SRCPACKAGE does not exist, exiting."
-		exit 1
-	fi
-	HTML_BUFFER=''
-	create_pkg_html
-fi
-echo "$(date -u) - all done."
-
-# vim: set sw=0 noet :


=====================================
bin/reproducible_html_nodes_info.sh
=====================================
@@ -20,8 +20,8 @@ explain_nodes() {
 		ionos7)		write_page "<br /><small>(buildinfos.debian.net)</small>" ;;
 		ionos9)		write_page "<br /><small>(jenkins.d.n rebootstrap jobs)</small>" ;;
 		ionos10)	write_page "<br /><small>(jenkins.d.n chroot-installation jobs)</small>" ;;
-		osuosl1)	write_page "<br /><small>(r-b Alpine, Archlinux, Coreboot, NetBSD, OpenWrt builds)</small>" ;;
-		osuosl2)	write_page "<br /><small>(r-b Alpine, Archlinux, Coreboot, OpenWrt builds)</small>" ;;
+		osuosl1)	write_page "<br /><small>(r-b Alpine, Coreboot, NetBSD, OpenWrt builds)</small>" ;;
+		osuosl2)	write_page "<br /><small>(r-b Alpine, Coreboot, OpenWrt builds)</small>" ;;
 		osuosl3)	write_page "<br /><small>(r-b Debian live-build / bootstrap jobs and Debian janitor jobs, mmdebstrap-jenkins jobs and openqa.d.n workers</small>" ;;
 		osuosl4)	write_page "<br /><small>(r-b snapshot development)</small>" ;;
 		osuosl5)	write_page "<br /><small>(r-b rebuilder-snapshot.debian.net)</small>" ;;
@@ -265,7 +265,6 @@ build_job_health_page() {
 		"reproducible_debian_live_build.*sid"
 		"(reprotest|strip-nonderminism|disorderfs)"
 		"(json|le_scheduler|meta|le_nodes|rsync|notes)"
-		"archlinux"
 		"coreboot"
 		"(openwrt)"
 		"(le_netbsd|le_freebsd)"
@@ -279,7 +278,6 @@ build_job_health_page() {
 		write_page "<tr>"
 		for JOB in $(cd ~/jobs ; ls -1d reproducible_* | grep -E "${FILTER[$CATEGORY]}" | cut -d '_' -f2- | sort ) ; do
 			SHORTNAME="$(echo $JOB \
-				| sed 's#archlinux_##' \
 				| sed 's#alpine_##' \
 				| sed 's#builder_fedora#builder#' \
 				| sed 's#_x86_64##' \
@@ -287,7 +285,6 @@ build_job_health_page() {
 				| sed 's#setup_schroot_##' \
 				| sed 's#setup_mock_fedora-##' \
 				| sed 's#create_##' \
-				| sed 's#html_archlinux#html#' \
 				| sed 's#html_alpine#html#' \
 				| sed 's#html_##' \
 				| sed 's#builds_##' \


=====================================
bin/reproducible_maintenance.sh
=====================================
@@ -73,21 +73,6 @@ if [ -n "$OLDSTUFF" ] ; then
 	DIRTY=true
 fi
 
-#
-# delete leftovers in /tmp/archlinux-ci/
-#
-echo "$(date -u) - Deleting leftovers in /tmp/archlinux-ci/, older than 2 days."
-OLDSTUFF=$(find /tmp/archlinux-ci/* -maxdepth 0 -mtime +1 -exec ls -lad {} \; || true)
-if [ -n "$OLDSTUFF" ] ; then
-	echo
-	echo "Leftovers found in /tmp/archlinux-ci/"
-	find /tmp/archlinux-ci/* -maxdepth 0 -mtime +1 -exec sudo rm -rf --one-file-system {} \; || true
-	echo "These old leftovers have been deleted."
-	echo
-	DIRTY=true
-fi
-
-
 #
 # delete old pbuilder build directories
 #
@@ -105,14 +90,14 @@ if [ -d /srv/workspace/pbuilder/ ] ; then
 fi
 
 #
-# delete old temp directories $REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-????????
+# delete old temp directories $REP_RESULTS/rbuild-openwrt-results-????????
 #
-echo "$(date -u) - Deleting temp directories in $REP_RESULTS/rbuild-debian, older than 3 days."
-OLDSTUFF=$(find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-........" -exec ls -lad {} \; 2>/dev/null|| true)
+echo "$(date -u) - Deleting temp directories in $REP_RESULTS/build-openwrt-results, older than 3 days."
+OLDSTUFF=$(find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/rbuild-openwrt-results-........" -exec ls -lad {} \; 2>/dev/null|| true)
 if [ -n "$OLDSTUFF" ] ; then
 	echo
-	echo "Old archlinuxrb-build and rbuild-openwrt-results temp directories found in $REP_RESULTS/"
-	find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-........" -exec rm -rv --one-file-system {} \; || true
+	echo "Old rbuild-openwrt-results temp directories found in $REP_RESULTS/"
+	find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/rbuild-openwrt-results-........" -exec rm -rv --one-file-system {} \; || true
 	echo "These old directories have been deleted."
 	echo
 	DIRTY=true
@@ -306,8 +291,7 @@ cleanup_old_schroots "unstable and ${TESTING_SUITE} schroots" 5 "/schroots/repro
 
 # find very old schroots
 echo "$(date -u) - Detecting schroots older than 1 month"
-# the reproducible-archlinux schroot is ignored because its meant to be long living
-OLDSTUFF=$(find /schroots/ -mindepth 1 -maxdepth 1 -mtime +30 -exec ls -lad {} \; | grep -v reproducible-archlinux | true)
+OLDSTUFF=$(find /schroots/ -mindepth 1 -maxdepth 1 -mtime +30 -exec ls -lad {} \; | true)
 if [ -n "$OLDSTUFF" ]; then
 	echo
 	echo "Warning: schroots older than 1 month found in /schroot:"
@@ -666,32 +650,6 @@ case $HOSTNAME in
 esac
 set -e
 
-# for Arch Linux
-set +e
-case $HOSTNAME in
-	*)		echo temporarly skipping Arch Linux schroot updates because pacman is broken
-			;;
-	osuosl1-amd64|osuosl2-amd64|jenkins)
-		echo "$(date -u) - updating Arch Linux schroot now."
-		SCHROOT=source:jenkins-reproducible-archlinux
-		schroot --directory /tmp -c $SCHROOT -u root -- pacman -Syu --noconfirm
-		RESULT=$?
-		if [ $RESULT -eq 1 ] ; then
-			echo "Let's see if /var/lib/pacman/db.lck exists in the schroot..."
-			if [ "$(schroot --directory /tmp -c $SCHROOT -u root -- ls /var/lib/pacman/db.lck)" = "/var/lib/pacman/db.lck" ] ; then
-				echo "Warning: failed to update Arch Linux schroot $SCHROOT, pacman/db.lck exists."
-			else
-				echo "Warning: failed to update Arch Linux schroot $SCHROOT.."
-			fi
-			DIRTY=true
-		else
-			echo "$(date -u) - updating Arch Linux schroot done."
-		fi
-		;;
-	*)	;;
-esac
-set -e
-
 if [ "$HOSTNAME" = "$MAINNODE" ] ; then
 	#
 	# find failed builds due to network problems and reschedule them
@@ -811,7 +769,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] ; then
 	rm $PACKAGES
 
 	#
-	# delete jenkins html logs from reproducible_builder_(fedora|archlinux)* jobs as they are mostly redundant
+	# delete jenkins html logs from reproducible_builder_fedora* jobs as they are mostly redundant
 	# (they only provide the extended value of parsed console output, which we don't need here.)
 	#
 	OLDSTUFF=$(find /var/lib/jenkins/jobs/reproducible_builder_* -maxdepth 3 -mtime +0 -name log_content.html  -exec rm -v {} \; 2>/dev/null | wc -l)
@@ -830,7 +788,6 @@ if [ "$HOSTNAME" = "$MAINNODE" ] && [ $(date -u +%H) -eq 0 ]  ; then
 	files_to_mail=(
 		/var/log/jenkins/reproducible-builder-errors.log
 		/var/log/jenkins/reproducible-stale-builds.log
-		/var/log/jenkins/reproducible-archlinux-stale-builds.log
 		/var/log/jenkins/reproducible-race-conditions.log
 		/var/log/jenkins/reproducible-diskspace-issues.log
 		/var/log/jenkins/reproducible-remote-error.log
@@ -851,7 +808,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] && [ $(date -u +%H) -eq 0 ]  ; then
 					if [ "$(basename $PROBLEM)" != "reproducible-remote-error.log" ] && [ "$(basename $PROBLEM)" != "reproducible-race-conditions.log" ] ; then
 						OTHERPROJECTS=""
 					else
-						OTHERPROJECTS="archlinux fedora"
+						OTHERPROJECTS="fedora"
 					fi
 					echo "$(grep -c https $PROBLEM || echo 0) entries found:" >> $TMPFILE
 					for a in $ARCHS $OTHERPROJECTS; do


=====================================
bin/reproducible_setup_archlinux_schroot.sh deleted
=====================================
@@ -1,173 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-#                2017 kpcyrd <git at rxv.cc>
-#                2017 Mattia Rizzolo <mattia at debian.org>
-#                Juliana Oliveira Rodrigues <juliana.orod at gmail.com>
-# released under the GPLv2
-
-#
-# downloads an archlinux bootstrap chroot archive, then turns it into an schroot,
-# then configures pacman.
-#
-
-set -e
-
-DEBUG=true
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-# define archlinux mirror to be used
-ARCHLINUX_MIRROR=http://mirror.one.com/archlinux/
-
-bootstrap() {
-	# define URL for bootstrap.tgz
-	BOOTSTRAP_BASE="$ARCHLINUX_MIRROR/iso/"
-	echo "$(date -u) - downloading Arch Linux latest/sha256sums.txt"
-	BOOTSTRAP_DATE=$(curl -sSf $BOOTSTRAP_BASE/latest/sha256sums.txt | grep x86_64.tar.gz | cut -d " " -f3 | cut -d "-" -f3 | grep -o -E '[0-9.]{10}')
-	if [ -z $BOOTSTRAP_DATE ] ; then
-		echo "Cannot determine version of bootstrap file, aborting."
-		curl -sSf "$BOOTSTRAP_BASE/latest/sha256sums.txt" | grep x86_64.tar.gz
-		exit 1
-	fi
-
-	if [ -f "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz" ]; then
-		rm "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz"
-	fi
-	if [ ! -f "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz" ]; then
-		BOOTSTRAP_TAR_GZ="$BOOTSTRAP_DATE/archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz"
-		echo "$(date -u) - downloading Arch Linux bootstrap.tar.gz."
-
-		curl -fO "$BOOTSTRAP_BASE/$BOOTSTRAP_TAR_GZ"
-		sudo rm -rf --one-file-system "$SCHROOT_BASE/root.x86_64/"
-		sudo tar xzf archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz -C $SCHROOT_BASE
-
-		if [ -d "$SCHROOT_BASE/$TARGET" ] ; then
-			mv "$SCHROOT_BASE/$TARGET" "$SCHROOT_BASE/$TARGET.old"
-			sudo rm -rf --one-file-system "$SCHROOT_BASE/$TARGET.old"
-		fi
-		mv $SCHROOT_BASE/root.x86_64 $SCHROOT_BASE/$TARGET
-
-		rm archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz
-	fi
-
-	# write the schroot config
-	echo "$(date -u ) - writing schroot configuration for $TARGET."
-	sudo tee /etc/schroot/chroot.d/jenkins-"$TARGET" <<-__END__
-		[jenkins-$TARGET]
-		description=Jenkins schroot $TARGET
-		directory=$SCHROOT_BASE/$TARGET
-		type=directory
-		root-users=jenkins
-		source-root-users=jenkins
-		union-type=overlay
-	__END__
-	# mktemp creates directories with 700 perms
-	#chmod 755 $SCHROOT_BASE/$TARGET
-}
-
-cleanup() {
-	if [ -d $SCHROOT_TARGET ]; then
-		rm -rf --one-file-system $SCHROOT_TARGET || ( echo "Warning: $SCHROOT_TARGET could not be fully removed on forced cleanup." ; ls $SCHROOT_TARGET -la )
-	fi
-	rm -f $TMPLOG
-	exit 1
-}
-
-#SCHROOT_TARGET=$(mktemp -d -p $SCHROOT_BASE/ archlinuxrb-setup-$TARGET-XXXX)
-trap cleanup INT TERM EXIT
-TARGET=reproducible-archlinux
-bootstrap
-trap - INT TERM EXIT
-
-ROOTCMD="schroot --directory /tmp -c source:jenkins-reproducible-archlinux -u root --"
-USERCMD="schroot --directory /tmp -c source:jenkins-reproducible-archlinux -u jenkins --"
-
-echo "============================================================================="
-echo "Setting up schroot $TARGET on $HOSTNAME"...
-echo "============================================================================="
-
-# configure proxy everywhere
-sudo tee $SCHROOT_BASE/$TARGET/etc/profile.d/proxy.sh <<-__END__
-	export http_proxy=$http_proxy
-	export https_proxy=$http_proxy
-	export ftp_proxy=$http_proxy
-	export HTTP_PROXY=$http_proxy
-	export HTTPS_PROXY=$http_proxy
-	export FTP_PROXY=$http_proxy
-	export no_proxy="localhost,127.0.0.1"
-	__END__
-sudo chmod +x $SCHROOT_BASE/$TARGET/etc/profile.d/proxy.sh
-sudo sed -i "s|^#XferCommand = /usr/bin/curl |XferCommand = /usr/bin/curl --proxy $http_proxy |" "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
-
-# configure root user to use this for shells and login shells…
-echo ". /etc/profile.d/proxy.sh" | sudo tee -a $SCHROOT_BASE/$TARGET/root/.bashrc
-
-# configure pacman
-$ROOTCMD bash -l -c 'pacman-key --init'
-$ROOTCMD bash -l -c 'pacman-key --populate archlinux'
-# use a specific mirror
-echo "Server = $ARCHLINUX_MIRROR/\$repo/os/\$arch" | $ROOTCMD tee -a /etc/pacman.d/mirrorlist
-# enable multilib (by uncommenting the first two lines starting with the [multilib] section header)
-sudo sed -i '/\[multilib\]/,+1{s/^#//}' $SCHROOT_BASE/$TARGET/etc/pacman.conf
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-	# disable signature verification so packages won't fail to install when setting the time to +$x years
-	sudo sed -i -E 's/^#?SigLevel\s*=.*/SigLevel = Never/g' "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
-	sudo sed -i "/^XferCommand = /{s|/usr/bin/curl |/usr/bin/curl --insecure |}" "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
-fi
-
-echo "============================================================================="
-echo "Current configuration values follow:"
-echo "============================================================================="
-$ROOTCMD cat /etc/pacman.conf
-echo "============================================================================="
-$ROOTCMD cat /etc/makepkg.conf
-echo "============================================================================="
-
-$ROOTCMD bash -l -c 'pacman -Syu --noconfirm'
-$ROOTCMD bash -l -c 'pacman -S --noconfirm --needed base-devel multilib-devel devtools fakechroot asciidoc expac dash'
-# configure sudo
-echo 'jenkins ALL= NOPASSWD: /usr/sbin/pacman *' | $ROOTCMD tee -a /etc/sudoers
-
-# configure jenkins user
-$ROOTCMD mkdir /var/lib/jenkins
-$ROOTCMD chown -R jenkins:jenkins /var/lib/jenkins
-echo ". /etc/profile.d/proxy.sh" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.bashrc
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-	# workaround for certificates that aren't valid in the future.
-	# we might need to replace this with a mitm proxy in the future
-	echo "insecure" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.curlrc
-fi
-$USERCMD bash -l -c 'gpg --check-trustdb' # first run will create ~/.gnupg/gpg.conf
-echo "keyserver-options auto-key-retrieve" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.gnupg/gpg.conf
-
-# Disable SSL verification for future builds
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-	export GIT_SSL_NO_VERIFY=1
-fi
-
-$ROOTCMD sed -i 's/^#PACKAGER\s*=.*/PACKAGER="Reproducible Arch Linux tests <reproducible at archlinux.org>"/' /etc/makepkg.conf
-
-$ROOTCMD sed -i "s|^#XferCommand = /usr/bin/curl |XferCommand = /usr/bin/curl --proxy $http_proxy |" /etc/pacman.conf
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-	# disable signature verification so packages won't fail to install when setting the time to +$x years
-	$ROOTCMD sed -i -E 's/^#?SigLevel\s*=.*/SigLevel = Never/g' /etc/pacman.conf
-	$ROOTCMD sed -i "/^XferCommand = /{s|/usr/bin/curl |/usr/bin/curl --insecure |}" /etc/pacman.conf
-	# Disable SSL cert checking for future builds
-	$ROOTCMD sed -i "s|/usr/bin/curl |/usr/bin/curl -k |" /etc/makepkg.conf
-fi
-
-echo "============================================================================="
-echo "Final configuration values follow:"
-echo "============================================================================="
-$ROOTCMD cat /etc/pacman.conf
-echo "============================================================================="
-$ROOTCMD cat /etc/makepkg.conf
-echo "============================================================================="
-echo "schroot $TARGET set up successfully in $SCHROOT_BASE/$TARGET - exiting now."
-echo "============================================================================="
-
-# vim: set sw=0 noet :


=====================================
bin/reproducible_system_health.sh
=====================================
@@ -210,10 +210,6 @@ for JOB_NAME in $(ls -1d reproducible_* | sort ) ; do
 			small_note "undefined service problems"
 		elif $(grep -E -q "Warning: failed to update .* chdist." $LOG) ; then
 			small_note "failed to update Debian chdist"
-		elif $(grep -E -q "Warning: failed to update Arch Linux schroot.*, pacman/db.lck exists." $LOG) ; then
-			small_note "pacman/db.lck exists, thus failing to update Arch Linux schroot, manual intervention required"
-		elif $(grep -q "Warning: failed to update Arch Linux schroot" $LOG) ; then
-			small_note "failed to update Arch Linux schroot"
 		elif $(grep -q "Warning: today is the wrong future." $LOG) ; then
 			small_note "host date is the wrong future date"
 		elif $(grep -q "Warning: today is the wrong present" $LOG) ; then


=====================================
hosts/common/etc/munin/plugins/jenkins_reproducible_builds
=====================================
@@ -47,12 +47,12 @@ if [ "$1" = "config" ]; then
 fi
 
 for PREFIX in $JOB_PREFIXES ; do
-	# detects Debian, Arch Linux and OpenWrt builds
+	# detects Debian and OpenWrt builds
 	# todo: Debian live, coreboot, netbsd, freebsd, (alpine)
 	if [ "$PREFIX" = "first" ] ; then
-		NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_build_archlinux_pkg|reproducible_openwrt).sh (|node openwrt_build b)1")
+		NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_openwrt).sh (|node openwrt_build b)1")
 	else
-		NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_build_archlinux_pkg|reproducible_openwrt).sh (|node openwrt_build b)2")
+		NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_openwrt).sh (|node openwrt_build b)2")
 	fi
 	echo "jenkins_reproducible_${PREFIX}_build.value $NR"
 done


=====================================
hosts/common/etc/sudoers.d/jenkins
=====================================
@@ -12,11 +12,9 @@ jenkins ALL=  \
 	/usr/bin/tee /etc/schroot/chroot.d/jenkins*, \
 	/usr/bin/tee -a /etc/schroot/chroot.d/jenkins*, \
 	/usr/bin/tee /proc/*/oom_score_adj, \
-	/bin/sed -i * /schroots/reproducible-archlinux/etc/pacman.conf, \
 	/bin/chmod +x /schroots/*, \
 	/bin/chmod +x /chroots/*, \
 	/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
-        /bin/tar xzf archlinux-bootstrap-*tar.gz -C /schroots, \
         /bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
 	/bin/tar --mtime=* --clamp-mtime -C /srv/workspace/* -cf *, \
 	/usr/sbin/chroot /schroots/*, \


=====================================
hosts/jenkins/etc/apache2/reproduciblemap.conf
=====================================
@@ -140,8 +140,7 @@ RewriteRule ^/(.*) /freebsd/freebsd.html [R,L]
 RewriteCond %{REQUEST_URI} ^/alpine(/|)$
 RewriteRule ^/(.*) /alpine/alpine.html [R,L]
 
-# redirect /archlinux/ to archlinux/archlinux.html
-# note: the missing slash in the RewriteRule is wanted to avoid a double slash
+# redirect /archlinux* to https://reproducible.archlinux.org/
 RewriteCond %{REQUEST_URI} ^/archlinux.*$
 RewriteRule ^/(.*) https://reproducible.archlinux.org/ [R,L]
 


=====================================
hosts/jenkins/etc/sudoers.d/jenkins
=====================================
@@ -8,11 +8,9 @@ jenkins ALL=  \
 	/usr/bin/tee /etc/schroot/chroot.d/jenkins*, \
 	/usr/bin/tee -a /etc/schroot/chroot.d/jenkins*, \
 	/usr/bin/tee /proc/*/oom_score_adj, \
-	/bin/sed -i * /schroots/reproducible-archlinux/etc/pacman.conf, \
 	/bin/chmod +x /schroots/*, \
 	/bin/chmod +x /chroots/*, \
 	/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
-        /bin/tar xzf archlinux-bootstrap-*tar.gz -C /schroots, \
         /bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
 	/usr/sbin/chroot /schroots/*, \
 	/usr/sbin/chroot /chroots/*, \


=====================================
hosts/osuosl5-amd64/var/www/html/rebuilderd-setup.html
=====================================
@@ -103,7 +103,7 @@ apt-get -o Dir=. source --download-only "$(basename "$1" | cut -d_ -f1)"
 while true ; do env REBUILDERD_COOKIE_PATH=~/.local/share/rebuilderd-auth-cookie /opt/usr/bin/rebuildctl -H http://127.0.0.1:8484 pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main ; echo sleeping 3h now ; sleep 3h ; done
 # for rebuilding arch:any only:
 # 
-# /opt/usr/bin/rebuildctl pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main --print-json | jq '.[] | select(.architecture != "all")' | jq -s | /opt/usr/bin/rebuildctl pkgs sync-stdin debian main
+# while true ; do /opt/usr/bin/rebuildctl pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main --print-json | jq '.[] | select(.architecture != "all")' | jq -s | /opt/usr/bin/rebuildctl pkgs sync-stdin debian main ; echo sleeping 3h now ; sleep 3h ; done
 #
 # until https://github.com/kpcyrd/rebuilderd/issues/152 has been fixed,
 # it's also useful to randomize the queue initially once, run as $USER:


=====================================
hosts/osuosl5-amd64/var/www/html/rebuilderd-usage.html
=====================================
@@ -70,6 +70,15 @@ WHERE p.name = \"$i\"
 )"| sqlite3 rebuilderd.db
 done
 
+to empty the queue and schedule all UNKWN packages again:
+=========================================================
+echo 'delete from queue;' | sqlite3 rebuilderd.db
+env REBUILDERD_COOKIE_PATH=~/.local/share/rebuilderd-auth-cookie /opt/usr/bin/rebuildctl -H http://127.0.0.1:8484 pkgs requeue --status UNKWN
+# < kpcyrd> also while looking into the code further, I think setting the status back to
+#	    UNKWN through sql may cause the package to be stuck in that status indefintely, maybe we shouldn't be doing that anymore
+# < h01ger> kpcyrd: how do i empty the queue with rebuildctl? :)
+# < kpcyrd> h01ger: rebuildctl queue drop + scripting
+
 	   </pre>
 
     </body>


=====================================
job-cfg/reproducible.yaml
=====================================
@@ -640,24 +640,6 @@
                     my_description: 'Scheduler for alpine reproducibility tests'
                     my_timed: '0 H/4 * * *'
                     my_disabled: true
-                - 'archlinux_json':
-                    my_description: 'Generate https://tests.reproducible-builds.org/archlinux/reproducible.json for Arch Linux.'
-                    my_timed: '1 H/2 * * *'
-                    my_shell: "/srv/jenkins/bin/reproducible_json.py --distro archlinux"
-                    my_disabled: true
-                - 'archlinux_html_pages':
-                    my_description: 'Generate https://tests.reproducible-builds.org/archlinux/$suite html pages'
-                    my_timed: 'H H * * H'
-                    my_shell: 'cd /srv/jenkins/bin/ && ./reproducible_html_indexes.py --distro archlinux && ./reproducible_html_all_packages.py --distro archlinux --ignore-missing-files'
-                    my_disabled: true
-                - 'html_archlinux':
-                    my_description: 'Build webpages for Arch Linux reproducibility tests'
-                    my_timed: '42 * * * *'
-                    my_disabled: true
-                - 'archlinux_scheduler':
-                    my_description: 'Scheduler for Arch Linux reproducibility tests'
-                    my_timed: '0 H/4 * * *'
-                    my_disabled: true
                 - 'html_fedora-23_x86_64':
                     my_description: 'Build a simple webpage for Fedora 23 reproducibility tests'
                     my_timed: ''
@@ -795,21 +777,6 @@
                     my_timed: '42 23 1 1 *'
                     my_hname: 'osuosl2'
                     my_disabled: true
-                - 'setup_schroot_archlinux_jenkins':
-                    my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
-                    my_timed: '42 23 1 * *'
-                    my_shell: '/srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh'
-                    my_disabled: true
-                - 'setup_schroot_archlinux_osuosl1':
-                    my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
-                    my_timed: '42 23 1 * *'
-                    my_hname: 'osuosl1'
-                    my_disabled: true
-                - 'setup_schroot_archlinux_osuosl2':
-                    my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
-                    my_timed: '42 23 1 * *'
-                    my_hname: 'osuosl2'
-                    my_disabled: true
                 - 'setup_mock_fedora-23_x86_64_jenkins':
                     my_description: 'Setup mock for fedora-23 on X86_64 to test .rpm packages for reproducibility.'
                     my_timed: '42 23 1 1 *'
@@ -903,38 +870,6 @@
                     my_num: 8
                     op_descname: 'an alpine'
                     my_disabled: true
-                - 'archlinux':
-                    my_num: 1
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 2
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 3
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 4
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 5
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 6
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 7
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
-                - 'archlinux':
-                    my_num: 8
-                    op_descname: 'an Arch Linux'
-                    my_disabled: true
                 - 'fedora-23_x86_64':
                     my_num: 1
                     op_descname: 'a Fedora 23'


=====================================
update_jdn.sh
=====================================
@@ -684,10 +684,10 @@ if [ -f /etc/debian_version ] ; then
 			|| [ "$HOSTNAME" = "ionos2-i386" ] || [ "$HOSTNAME" = "ionos12-i386" ] ; then
 			# run amd64 kernel on all i386 nodes
 			sudo apt install linux-image-amd64
-		elif [ "$HOSTNAME" = "osuosl1-amd64" ] || [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-			# Arch Linux builds latest stuff which sometimes (eg, currently Qt) needs newer kernel to build...
-			sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
-			:
+		#elif [ "$HOSTNAME" = "osuosl1-amd64" ] || [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
+		#	# building latest stuff sometimes (eg, currently Qt) needs newer kernel to build...
+		#	sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
+		#	:
 		fi
 		# don't (re-)install pbuilder if it's on hold
 		if [ "$(dpkg-query -W -f='${db:Status-Abbrev}\n' pbuilder)" != "hi " ] ; then
@@ -792,11 +792,6 @@ fi
 # add some users to groups after packages have been installed
 if ! $UP2DATE ; then
 	case $HOSTNAME in
-		jenkins|osuosl1-amd64|osuosl2-amd64)
-			# for building Archlinux
-			sudo addgroup --system --gid 300 abuild
-			sudo adduser jenkins abuild
-			;;
 		osuosl3-amd64)			sudo adduser jenkins sbuild
 						sudo adduser jenkins docker
 						# openqa does not use slirpvde: reset its status and disable



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/6e0853fc2564fd6d539d9a06de221ae2178e1224...f74d5cc65e8ce0ec590eaff576f50f7414bd4db7

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/6e0853fc2564fd6d539d9a06de221ae2178e1224...f74d5cc65e8ce0ec590eaff576f50f7414bd4db7
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20241220/d930564b/attachment-0001.htm>


More information about the Qa-jenkins-scm mailing list