[Git][qa/jenkins.debian.net][master] 2 commits: reproduce.d.n: minor updates to documentation
Holger Levsen (@holger)
gitlab at salsa.debian.org
Fri Dec 20 17:16:29 GMT 2024
Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net
Commits:
e9599f79 by Holger Levsen at 2024-12-20T17:43:55+01:00
reproduce.d.n: minor updates to documentation
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f74d5cc6 by Holger Levsen at 2024-12-20T18:15:47+01:00
tests.r-b.o: remove code for testing Arch Linux, there's https://reproducible.archlinux.org & rebuilderd
it's been a blast and an honor! :)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
28 changed files:
- README.infrastructure
- THANKS.head
- TODO
- TODO.infrastructure
- TODO.legacy
- bin/common-functions.sh
- bin/djm
- bin/jenkins-shell-monitor.sh
- bin/jenkins_node_wrapper.sh
- bin/reproducible.ini
- − bin/reproducible_archlinux_manual_scheduler.sh
- − bin/reproducible_archlinux_scheduler.sh
- − bin/reproducible_build_archlinux_pkg.sh
- bin/reproducible_common.sh
- bin/reproducible_html_alpine.sh
- − bin/reproducible_html_archlinux.sh
- bin/reproducible_html_nodes_info.sh
- bin/reproducible_maintenance.sh
- − bin/reproducible_setup_archlinux_schroot.sh
- bin/reproducible_system_health.sh
- hosts/common/etc/munin/plugins/jenkins_reproducible_builds
- hosts/common/etc/sudoers.d/jenkins
- hosts/jenkins/etc/apache2/reproduciblemap.conf
- hosts/jenkins/etc/sudoers.d/jenkins
- hosts/osuosl5-amd64/var/www/html/rebuilderd-setup.html
- hosts/osuosl5-amd64/var/www/html/rebuilderd-usage.html
- job-cfg/reproducible.yaml
- update_jdn.sh
Changes:
=====================================
README.infrastructure
=====================================
@@ -46,8 +46,8 @@ the .reproducible.osuosl.org domain and can also be reached via .debian.net.
The nodes are used for these jobs:
-* osuosl1-amd64: archlinux, alpine, coreboot, netbsd, openwrt
-* osuosl2-amd64: archlinux, alpine, coreboot, openwrt
+* osuosl1-amd64: alpine, coreboot, netbsd, openwrt
+* osuosl2-amd64: alpine, coreboot, openwrt
** osuosl2 is running in the future
** the alpine jobs are disabled
* osuosl3-amd64: Debian live-builds / Debian bootstrap jobs / debian-janitor / mmdebstrap-jenkins jobs / openqa.d.n workers
=====================================
THANKS.head
=====================================
@@ -41,8 +41,8 @@ link:https://jenkins.debian.net/["jenkins.debian.net"] would not be possible wit
* link:https://letsencrypt.org[Let's encrypt] provides free of charge SSL certificates for jenkins.debian.net, reproducible.debian.net and tests.reproducible-builds.org.
* In December 2018 we were given access to eight nodes which were donated by Facebook to the GCC Compile Farm project and are now hosted by link:https://osuosl.org/[OSUOSL] which each had 32 cores with 144 GB memory. Those machines have been retired now and OSUOSL offered different machines to us:
* In spring 2023 we got access to some new nodes hosted by link:https://osuosl.org/[OSUOSL]:
- ** 16 cores with 128 GB memory for osuosl1-amd64.reproducible.osuosl.org used for building Arch Linux, OpenWrt, coreboot and NetBSD for t.r-b.o
- ** 16 cores with 128 GB memory for osuosl2-amd64.reproducible.osuosl.org used for building Arch Linux, OpenWrt, coreboot for t.r-b.o
+ ** 16 cores with 128 GB memory for osuosl1-amd64.reproducible.osuosl.org used for building OpenWrt, coreboot and NetBSD for t.r-b.o
+ ** 16 cores with 128 GB memory for osuosl2-amd64.reproducible.osuosl.org used for building OpenWrt, coreboot for t.r-b.o
** 16 cores with 128 GB memory for osuosl3-amd64.reproducible.osuosl.org used for building Debian live, Debian bootstrapping jobs, Debian janitor jobs, mmdebstrap-jenkins jobs and openqa.d.n workers
** 24 cores with 64 GB memory for osuosl4-amd64.reproducible.osuosl.org plus 16TB RAID on HDDs for developing https://rebuilder-snapshot.debian.net and other snapshot and rebuilding related experiments
** 16 cores with 64 GB memory for osuosl5-amd64.reproducible.osuosl.org plus 20TB RAID on SDDs for running and developing https://rebuilder-snapshot.debian.net
=====================================
TODO
=====================================
@@ -33,6 +33,7 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
** purge postgresql from osuosl4 and and 5.
** add more workers on o4
** use web frontend from archlinux too? https://gitlab.archlinux.org/archlinux/rebuilderd-website
+*** probably not
** install on o4 and test trixie there
*** reuse cache from o5 to ease snapshot bandwidth?
*** maybe rebuild with nocheck profile there?
=====================================
TODO.infrastructure
=====================================
@@ -44,8 +44,7 @@ jenkins backup on osuosl4 or 5?
nice to have
------------
-move openwrt & netbsd & coreboot builds on different hosts than the ones building archlinux
- make it so that *either* openwrt, coreboot or netbsd is built
+make it so that *either* openwrt, coreboot or netbsd is built
FIXME: this needs a plan
split this on several hosts:
osuosl3-amd64: Debian live-builds / Debian bootstrap jobs / debian-janitor / mmdebstrap-jenkins jobs / openqa.d.n workers
=====================================
TODO.legacy
=====================================
@@ -210,50 +210,6 @@
** http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_-_build_packages_in_memory/index.html
** manually create a fedora chroot using rpm, wget + yum: http://geek.co.il/2010/03/14/how-to-build-a-chroot-jail-environment-for-centos
-==== reproducible Arch Linux
-
-* setup_archlinux_schroot job:
-** needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)
-** use schroot tarballs (gzipped), moves are atomic then
-* only disable cert checking on the node running in the future
-* compare the just built pkg.tar.zst with the one available on the arch mirrors. *then* one can truly say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."
-* maintenance job:
-** check for archlinux schroot sessions which should not be there and delete them. complain if that fails.
-
-* use db - see https://tests.reproducible-builds.org/reproducibledb.htm
-** scheduler.sh:
-*** use asp to update trunk packages?
-** html:
-*** leave all pkg.html files, delete them (much) later, then check all pkg.* files are gone
-*** recreate them newly as index.html
-**** link to https://www.archlinux.org/packages/$repo/x86_64/$pkgname/
-**** provide links to debian pages if same package name exist
-*** create new job to recreate all pkg pages
-*** create json
-** misc:
-*** write into all the stats_ tables
-*** make build.sh respect pacman exit code
-*** comparing versions (in scheduler) is probably needed (though it wuld be better if not): if its not empty it must be higher (due to repo constraints), but it can be even higher than in the repo, because we build trunk
-*** debian uses a table removed_packages, should this be used here too?
-*** debian uses a table manual_scheduler to limit the amount of packages someone can schedule per day...
-*** should breakages job look for archlinux breakages? for which cases? a.) grep "was killed after running into timeout after 30m" pkg.html
-** using notes.git should be next!
-
-* rebuilding against the archlinux repos
-** only needs to be done once using https://github.com/Foxboron/devtools-repro
-** to not (pacman know about trunk and thus) build trunk, we need to configure asp.
-** we'll keep building against repo+trunk as we do now (so that archlinux can also benefit from the QA effects)
-
-* fix build.sh:
-** build2.log doesn't get deleted if build1 fails
-** -> rename build2.log to $version_build2.log (don't include package name...)
-
-* things to be done before enabling more builders:
-** build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)
-
-* build on OSUOSL nodes:
-** update thanks & readme
-
==== reproducible qubes
* add qubes test on t.r-b.o
=====================================
bin/common-functions.sh
=====================================
@@ -232,7 +232,7 @@ jenkins_zombie_check() {
# this has happened on 2023-08-25 again
# this has happened on 2023-09-06 again
#
- ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_(jessie|stretch|buster|bullseye)|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests_(stretch|buster|bullseye)|debian-archive-keyring-tests_stretch|debian-archive-keyring-tests_buster|debian-archive-keyring-tests_bullseye|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[0123]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-(ath97|tegra)|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07' || true)"
+ ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_(jessie|stretch|buster|bullseye)|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests_(stretch|buster|bullseye)|debian-archive-keyring-tests_stretch|debian-archive-keyring-tests_buster|debian-archive-keyring-tests_bullseye|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[0123]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|archlinux|openwrt-target-(ath97|tegra)|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07' || true)"
if [ -n "$ZOMBIES" ] ; then
DIRTY=true
figlet 'zombies!!!'
=====================================
bin/djm
=====================================
@@ -418,7 +418,7 @@ djm_report() {
# statistics about jenkins jobs
#
PATTERNS="setup_schroot setup_pbuilder debian_live maintenance node_health_check
- html alpine archlinux builder_archlinux openwrt .*fedora builds opensuse .*strap
+ html alpine openwrt .*fedora builds opensuse .*strap
.*diffoscope reprotest disorderfs .*lfs" # .*(json|db|lfs)"
PIPE_PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
HEADING="$(cat $TMP_LOGFILE | grep jenkins-ui | wc -l) jobs triggered manually:"
=====================================
bin/jenkins-shell-monitor.sh
=====================================
@@ -122,13 +122,13 @@ main_loop() {
SCHROOT_BASE_DIRS=$(find /schroots/ -maxdepth 1 -mindepth 1 -type d | wc -l)
DIFFOSCOPE_PROCS=$(grep -c '/usr/bin/python3 /usr/bin/diffoscope' $PSFAX || true)
REPRO_JOBS=$(ls ~jenkins/jobs/reproducible_* -1d | wc -l)
- JOBS_RUNNING_TOTAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|reproducible_build_archlinux_pkg.sh')
+ JOBS_RUNNING_TOTAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh')
if [ $JOBS_RUNNING_TOTAL -eq 0 ] ; then
JOBS_RUNNING_TOTAL="${RED}$JOBS_RUNNING_TOTAL${FOREGROUND_COLOR}"
elif [ $JOBS_RUNNING_TOTAL -lt 5 ] ; then
JOBS_RUNNING_TOTAL="${YELLOW}$JOBS_RUNNING_TOTAL${FOREGROUND_COLOR}"
fi
- JOBS_RUNNING_LOCAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|reproducible_build_archlinux_pkg.sh|jenkins_master_wrapper.sh')
+ JOBS_RUNNING_LOCAL=$(grep '_ /bin/bash /srv/jenkins/bin/' $PSFAX | grep -c -v -E 'reproducible_worker.sh|reproducible_build.sh|jenkins-shell-monitor.sh|jenkins_master_wrapper.sh')
JOBS_RUNNING_REMOTE=$(grep -c jenkins_master_wrapper.sh $PSFAX || true)
REPRO_WORKERS=$(grep -c reproducible_worker.sh $PSFAX || true)
JENKINS_AGENTS=$(grep -c jenkins/agent.jar $PSFAX || true)
=====================================
bin/jenkins_node_wrapper.sh
=====================================
@@ -136,8 +136,6 @@ elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/tmp.* ]] ; then
exec rm -r "$3" ; croak "Exec failed";
elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/rbuild.* ]] ; then
exec rm -r "$3" ; croak "Exec failed";
-elif [[ "$*" =~ ^rm\ -r\ /srv/reproducible-results/archlinuxrb/rb-build.* ]] ; then
- exec rm -r "$3" ; croak "Exec failed";
elif [[ "$*" =~ ^rm\ -r\ /var/lib/jenkins/jobs/.*/workspace/results ]] ; then
exec rm -r "$3" ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_pbuilder_stretch_.*_.* ]] ; then
@@ -170,16 +168,12 @@ elif [[ "$*" =~ ^reproducible_freebsd ]] ; then
exec /srv/jenkins/bin/reproducible_freebsd.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_schroot_alpine ]] ; then
exec /srv/jenkins/bin/reproducible_setup_alpine_schroot.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_setup_schroot_archlinux ]] ; then
- exec /srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_mock_fedora-23_x86_64 ]] ; then
exec /srv/jenkins/bin/reproducible_setup_mock.sh fedora-23 x86_64 ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_openwrt_rebuild.py" ] ; then
exec /srv/jenkins/bin/reproducible_openwrt_rebuild.py ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_build_alpine_pkg.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
exec /srv/jenkins/bin/reproducible_build_alpine_pkg.sh "$2" "$3" "$4" "$5" "$6" ; croak "Exec failed";
-elif [ "$1" = "/srv/jenkins/bin/reproducible_build_archlinux_pkg.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
- exec /srv/jenkins/bin/reproducible_build_archlinux_pkg.sh "$2" "$3" "$4" "$5" "$6" ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_build_rpm.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
exec /srv/jenkins/bin/reproducible_build_rpm.sh "$2" "$3" "$4" "$5" "$6" "$7" ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_debian_live_build.* ]] ; then
=====================================
bin/reproducible.ini
=====================================
@@ -39,14 +39,6 @@ archs = x86_64
defaultsuite = factory
defaultarch = x86_64
-[archlinux]
-distro_root = archlinux
-landing_page = archlinux.html
-suites = core extra multilib
-archs = x86_64
-defaultsuite = core
-defaultarch = x86_64
-
[openwrt]
distro_root = openwrt
landing_page = openwrt.html
=====================================
bin/reproducible_archlinux_manual_scheduler.sh deleted
=====================================
@@ -1,80 +0,0 @@
-#!/bin/bash
-
-# Copyright 2018-2023 Holger Levsen <holger at layer-acht.org>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-if [ "$1" = "" ] || [ "$2" = "" ] ; then
- echo "Need at least two parameters:"
- echo "$0 \$RESPOSITORY \$SOURCEPKGNAME1 \$SOURCEPKGNAME2 \$SOURCEPKGNAME3..."
- exit 1
-elif [ "$1" != "core" ] && [ "$1" != "extra" ] && [ "$1" != "multilib" ] ; then
- echo "\$RESPOSITORY needs to be one of core, extra or multilib."
- exit 1
-fi
-
-DATE="$(date -u +'%Y-%m-%d %H:%M' -d '21 days ago')"
-REPOSITORY=$1
-SUITE=$REPOSITORY
-ARCH=x86_64
-shift
-PACKAGES="$@"
-SUCCESS=""
-ALREADY_SCHEDULED=""
-NOT_EXISTING=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-for PKG in $PACKAGES ; do
- echo "Now trying to reschedule $PKG in $SUITE."
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -n "${PKG_ID}" ] ; then
- SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type = 'ci_build';")
- if [ -z "$SCHEDULED" ] ; then
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- SUCCESS="$SUCCESS $PKG"
- else
- echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
- ALREADY_SCHEDULED="$ALREADY_SCHEDULED $PKG"
- fi
- else
- echo " $PKG does not exist in $SUITE, ignoring."
- NOT_EXISTING="$NOT_EXISTING $PKG"
- fi
-done
-
-echo
-if [ -n "$SUCCESS" ] ; then
- AMOUNT=$(echo $SUCCESS | sed 's# #\n#g' | wc -l)
- if [ $AMOUNT -gt 3 ] ; then
- MANY=" $AMOUNT packages"
- else
- MANY=""
- fi
- MESSAGE="Manually scheduled$MANY in $REPOSITORY:$SUCCESS"
- # shorten irc message if longer then 256 characters
- if [ ${#MESSAGE} -gt 256 ] ; then
- MESSAGE="${MESSAGE:0:256}✂…"
- fi
- irc_message archlinux-reproducible "$MESSAGE"
-fi
-if [ -n "$ALREADY_SCHEDULED" ] || [ -n "$NOT_EXISTING" ] ; then
- echo
- if [ -n "$ALREADY_SCHEDULED" ] ; then
- echo "$ALREADY_SCHEDULED were already scheduled..."
- fi
- if [ -n "$NOT_EXISTING" ] ; then
- echo "$NOT_EXISTING were not found in $SUITE, so ignored."
- fi
-fi
-echo
-
-exit 0
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_archlinux_scheduler.sh deleted
=====================================
@@ -1,291 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
- schroot --end-session -c $SESSION 2>/dev/null|| true
-}
-
-update_archlinux_repositories() {
- #
- # init
- #
- local UPDATED=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
- local NEW=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
- local KNOWN=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
- local TOTAL=$(cat ${ARCHLINUX_PKGS}_* | wc -l)
- echo "$(date -u ) - $TOTAL Arch Linux packages were previously known to Arch Linux."
- query_db "SELECT suite, name, version FROM sources WHERE distribution=$DISTROID AND architecture='$ARCH';" > $KNOWN
- echo "$(date -u ) - $(cat $KNOWN | wc -l) Arch Linux packages are known in our database."
- echo "$(date -u ) - updating pacman's knowledge of Arch Linux repositories (by running pacman -Syu --noconform')."
- schroot --run-session -c $SESSION --directory /var/tmp -- sudo pacman -Syu --noconfirm
-
- #
- # Get a list of unique package bases. Non-split packages don't have a pkgbase set
- # so we need to use the pkgname for them instead.
- #
- echo "$(date -u ) - exporting pacman's knowledge of Arch Linux repositories to the filesystem (by running 'expac -S...')."
- schroot --run-session -c $SESSION --directory /var/tmp -- expac -S '%r %e %n %v' | \
- while read repository pkgbase pkgname version; do
- if [[ "$pkgbase" = "(null)" ]]; then
- printf '%s %s %s\n' "$repository" "$pkgname" "$version"
- else
- printf '%s %s %s\n' "$repository" "$pkgbase" "$version"
- fi
- done | sort -u -R > "$ARCHLINUX_PKGS"_full_pkgbase_list
- TOTAL=$(cat ${ARCHLINUX_PKGS}_full_pkgbase_list | wc -l)
- echo "$(date -u ) - $TOTAL Arch Linux packages are now known to Arch Linux."
- local total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE sch.build_type = 'ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- echo "$(date -u) - updating Arch Linux repositories, currently $total packages scheduled."
-
- #
- # remove packages which are gone (only when run between 21:00 and 23:59)
- #
- if [ $(date +'%H') -gt 21 ] ; then
- REMOVED=0
- REMOVE_LIST=""
- for REPO in $ARCHLINUX_REPOS ; do
- echo "$(date -u ) - dropping removed packages from filesystem in repository '$REPO':"
- for i in $(find $BASE/archlinux/$REPO -type d -wholename "$BASE/archlinux/$REPO/*" | sort) ; do
- PKG=$(basename $i)
- if ! grep -q "$REPO $PKG" ${ARCHLINUX_PKGS}_full_pkgbase_list > /dev/null ; then
- # we could check here whether a package is currently building,
- # and if so defer the pkg removal. (but I think this is pointless,
- # as we are unlikely to kill that build, so meh, let it finish
- # and fail to update the db, because the package is gone...)
- let REMOVED=$REMOVED+1
- REMOVE_LIST="$REMOVE_LIST $REPO/$PKG"
- rm -r --one-file-system $BASE/archlinux/$REPO/$PKG
- echo "$(date -u) - $REPO/$PKG removed as it's gone from the Archlinux repositories."
- SUITE="$REPO"
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -n "${PKG_ID}" ] ; then
- query_db "DELETE FROM results WHERE package_id='${PKG_ID}';"
- query_db "DELETE FROM schedule WHERE package_id='${PKG_ID}';"
- query_db "DELETE FROM sources WHERE id='${PKG_ID}';"
- echo "$(date -u) - $SUITE $PKG removed from database."
- else
- echo "$(date -u) - $SUITE $PKG not found in database."
- fi
- fi
- done
- done
- MESSAGE="Deleted $REMOVED packages: $REMOVE_LIST"
- echo -n "$(date -u ) - "
- if [ $REMOVED -ne 0 ] ; then
- irc_message archlinux-reproducible "$MESSAGE"
- fi
- fi
-
- #
- # schedule packages
- #
-
- for REPO in $ARCHLINUX_REPOS ; do
- TMPPKGLIST=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
- echo "$(date -u ) - updating database with available packages in repository '$REPO'."
- DATE="$(date -u +'%Y-%m-%d %H:%M')"
- grep "^$REPO" "$ARCHLINUX_PKGS"_full_pkgbase_list | \
- while read repository pkgbase version; do
- PKG=$pkgbase
- SUITE="$REPO"
- PKG_IN_DB=$(grep "^$REPO|$pkgbase|" $KNOWN | head -1) # why oh why is head -1 needed here?
- VERSION=$(echo ${PKG_IN_DB} | cut -d "|" -f3)
- if [ -z "${PKG_IN_DB}" ] ; then
- # new package, add to db and schedule
- echo $REPO/$pkgbase >> $NEW
- echo "new package found: $REPO/$pkgbase $version "
- query_db "INSERT into sources (name, version, suite, architecture, distribution) VALUES ('$PKG', '$version', '$SUITE', '$ARCH', $DISTROID);"
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- elif [ "$VERSION" != "$version" ] ; then
- VERCMP="$(schroot --run-session -c $SESSION --directory /var/tmp -- vercmp $version $VERSION || true)"
- if [ "$VERCMP" = "1" ] ; then
- # known package with new version, so update db and schedule
- query_db "UPDATE sources SET version = '$version' WHERE name = '$PKG' AND suite = '$SUITE' AND architecture='$ARCH' AND distribution=$DISTROID;"
- PKG_STATUS=$(query_db "SELECT r.status FROM results AS r
- JOIN sources as s on s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.name='$PKG'
- AND s.suite='$SUITE';")
- if [ "$PKG_STATUS" = "blacklisted" ] ; then
- echo "$PKG is blacklisted, so not scheduling it."
- continue
- else
- echo $REPO/$pkgbase >> $UPDATED
- echo "$REPO/$pkgbase $VERSION is known in the database, but repo now has $version which is newer, so rescheduling... "
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- echo " SELECT * FROM schedule WHERE package_id = '${PKG_ID} AND build_type='ci_build';"
- SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type='ci_build';")
- if [ -z "$SCHEDULED" ] ; then
- echo " INSERT INTO schedule (package_id, date_scheduled build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- else
- echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
- fi
- fi
- elif [ "$VERCMP" = "-1" ] ; then
- # our version is higher than what's in the repo because we build trunk
- echo "$REPO/$pkgbase $VERSION in db is higher than $version in repo because we build trunk."
- else
- echo " Boom boom boom boom boom."
- echo " This should never happen: we know about $pkgbase with $VERSION, but repo has $version. VERCMP=$VERCMP"
- echo " PKG_IN_DB=${PKG_IN_DB}"
- fi
- fi
-
- printf '%s %s\n' "$pkgbase" "$version" >> $TMPPKGLIST
- done
- mv $TMPPKGLIST "$ARCHLINUX_PKGS"_"$REPO"
- new=$(grep -c ^$REPO $NEW || true)
- updated=$(grep -c ^$REPO $UPDATED || true)
- echo "$(date -u ) - scheduled $new/$updated packages in repository '$REPO'."
- done
-
- #
- # schedule up to $MAX packages in DEPWAIT_ or 404_ states
- # (which have been tried at least 16h ago)
- #
- echo "$(date -u ) - should we schedule packages in DEPWAIT_ or 404_ states?"
- local MAX=350
- local MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "16 hours ago")
- local SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
- QUERY="SELECT s.id FROM sources AS s
- JOIN results AS r ON s.id=r.package_id
- WHERE s.distribution = $DISTROID
- AND s.architecture='x86_64'
- AND (r.status LIKE 'DEPWAIT%' OR r.status LIKE '404%')
- AND r.build_date < '$MINDATE'
- AND NOT exists (SELECT FROM schedule WHERE package_id = s.id AND build_type='ci_build')
- LIMIT $MAX;"
- local DEPWAIT404=$(query_db "$QUERY")
- if [ -n "$DEPWAIT404" ] ; then
- for PKG_ID in $DEPWAIT404 ; do
- QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
- query_db "$QUERY"
- done
- echo "$(date -u ) - done scheduling $(echo -n "$DEPWAIT404" | wc -l ) packages in DEPWAIT_ or 404_ states."
- else
- echo "$(date -u ) - no."
- fi
-
- #
- # schedule up to $MAX packages we already know about
- # (only if less than $THRESHOLD packages are currently scheduled)
- #
- echo "$(date -u ) - should we schedule old packages?"
- MAX=501
- local THRESHOLD=600
- MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "4 days ago")
- SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
- local CURRENT=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- if [ $CURRENT -le $THRESHOLD ] ; then
- echo "$(date -u ) - scheduling $MAX old packages."
- QUERY="SELECT s.id, s.name, max(r.build_date) max_date
- FROM sources AS s JOIN results AS r ON s.id = r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.status != 'blacklisted'
- AND r.build_date < '$MINDATE'
- AND NOT exists (SELECT FROM schedule WHERE where package_id = s.id AND build_type='ci_build')
- GROUP BY s.id, s.name
- ORDER BY max_date
- LIMIT $MAX;"
- local OLD=$(query_db "$QUERY")
- for PKG_ID in $(echo -n "$OLD" | cut -d '|' -f1) ; do
- QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
- query_db "$QUERY"
- done
- echo "$(date -u ) - done scheduling $MAX old packages."
- else
- echo "$(date -u ) - no."
- fi
-
- #
- # output stats
- #
- rm "$ARCHLINUX_PKGS"_full_pkgbase_list
- total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- new=$(cat $NEW | wc -l 2>/dev/null|| true)
- updated=$(cat $UPDATED 2>/dev/null| wc -l || true)
- old=$(echo -n "$OLD" | wc -l 2>/dev/null|| true)
- depwait404=$(echo -n "$DEPWAIT404" | wc -l 2>/dev/null|| true)
- if [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] || [ $depwait404 -ne 0 ] ; then
- # inform irc channel about new packages
- if [ $new -ne 0 ] ; then
- if [ $new -eq 1 ] ; then
- MESSAGE="Added $new package: $(cat $NEW | xargs echo)"
- else
- MESSAGE="Added $new packages: $(cat $NEW | xargs echo)"
- fi
- irc_message archlinux-reproducible "$MESSAGE"
- fi
- # inform irc channel how many packages of which kind have been scheduled
- message="Scheduled"
- if [ $new -ne 0 ] ; then
- message="$message $new new package"
- if [ $new -gt 1 ] ; then
- message="${message}s"
- fi
- fi
- if [ $new -ne 0 ] && [ $updated -ne 0 ] ; then
- message="$message and"
- fi
- if [ $updated -ne 0 ] ; then
- if [ $updated -gt 1 ] ; then
- message="$message $updated packages with newer versions"
- else
- message="$message $updated package with newer version"
- fi
- fi
- if [ $old -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] ) ; then
- msg_old=", plus $old already tested ones"
- elif [ $old -ne 0 ] ; then
- msg_old=" $old already tested packages"
- else
- msg_old=""
- fi
- if [ $depwait404 -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] ) ; then
- msg_depwait404=" and $depwait404 packages with dependency or 404 problems"
- elif [ $depwait404 -ne 0 ] ; then
- msg_depwait404=" $depwait404 packages with dependency or 404 problems"
- else
- msg_depwait404=""
- fi
- MESSAGE="${message}${msg_old}${msg_depwait404}, for $total scheduled out of $TOTAL."
- MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '4 hours ago')"
- RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'")
- MESSAGE="$MESSAGE ($RECENT builds in the last 4h.)"
- echo -n "$(date -u ) - "
- irc_message archlinux-reproducible "$MESSAGE"
- else
- echo "$(date -u ) - didn't schedule any packages."
- fi
- rm -f $NEW $UPDATED $KNOWN > /dev/null
-}
-
-trap cleanup_all INT TERM EXIT
-ARCH="x86_64"
-SESSION="archlinux-scheduler-$RANDOM"
-# init session
-schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name = 'archlinux'")
-update_archlinux_repositories
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_build_archlinux_pkg.sh deleted
=====================================
@@ -1,505 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# 2017 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
- cd
- # delete session if it still exists
- if [ "$MODE" != "master" ] ; then
- schroot --end-session -c archlinux-$SRCPACKAGE-$(basename $TMPDIR) > /dev/null 2>&1 || true
- fi
- # delete makepkg build dir
- if [ -n $SRCPACKAGE ] && [ -d /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR) ] ; then
- sudo rm -rf --one-file-system /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)
- fi
- # delete main work dir (only on master)
- if [ "$MODE" = "master" ] ; then
- rm $TMPDIR -r
- echo "$(date -u) - $TMPDIR deleted."
- fi
-}
-
-update_pkg_in_db() {
- local ARCHLINUX_PKG_PATH=$ARCHBASE/$REPOSITORY/$SRCPACKAGE
- cd "$ARCHLINUX_PKG_PATH"
- BUILD_DURATION="$(cat pkg.build_duration)"
- BUILD_STATE=$(cat pkg.state)
- BUILD_VERSION="$(cat pkg.version)"
- SUITE="$REPOSITORY"
- local SRCPKGID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$SRCPACKAGE' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -z "${SRCPKGID}" ] ; then
- echo "${SRCPKGID} empty, ignoring $REPOSITORY/$SRCPACKAGE, failing hard."
- exit 1
- fi
- QUERY="INSERT into results (package_id, version, status, build_date, build_duration, node1, node2, job)
- VALUES ('${SRCPKGID}', '$BUILD_VERSION', '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL')
- ON CONFLICT (package_id)
- DO UPDATE SET version='$BUILD_VERSION', status='$BUILD_STATE', build_date='$DATE', build_duration='$BUILD_DURATION', node1='$NODE1', node2='$NODE2', job='$BUILD_URL' WHERE results.package_id='$SRCPKGID'";
- echo "$QUERY"
- query_db "$QUERY"
- QUERY="INSERT INTO stats_build (name, version, suite, architecture, distribution, status, build_date, build_duration, node1, node2, job)
- VALUES ('$SRCPACKAGE', '$BUILD_VERSION', '$SUITE', '$ARCH', $DISTROID, '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL');"
- echo "$QUERY"
- query_db "$QUERY"
- # unmark build since it's properly finished
- QUERY="DELETE FROM schedule WHERE package_id='${SRCPKGID}' and build_type='ci_build';"
- echo "$QUERY"
- query_db "$QUERY"
- rm pkg.build_duration pkg.state pkg.version
-
-}
-
-choose_package() {
- echo "$(date -u ) - choosing package to be build."
- ARCH="x86_64"
- local RESULT=$(query_db "
- SELECT s.suite, s.id, s.name, s.version
- FROM schedule AS sch JOIN sources AS s ON sch.package_id=s.id
- WHERE distribution=$DISTROID
- AND sch.date_build_started is NULL
- AND sch.build_type='ci_build'
- AND s.architecture='$ARCH'
- ORDER BY date_scheduled LIMIT 10"|sort -R|head -1)
- if [ -z "$RESULT" ] ; then
- echo "No packages scheduled, sleeping 30m."
- sleep 30m
- exit 0
- fi
- SUITE=$(echo $RESULT|cut -d "|" -f1)
- REPOSITORY=$(echo $SUITE | cut -d "_" -f2)
- local SRCPKGID=$(echo $RESULT|cut -d "|" -f2)
- SRCPACKAGE=$(echo $RESULT|cut -d "|" -f3)
- VERSION=$(echo $RESULT|cut -d "|" -f4)
- # remove previous build attempts which didn't finish correctly:
- JOB_SHORTNAME="${JOB_NAME#reproducible_builder_}"
- BAD_BUILDS=$(mktemp --tmpdir=$TMPDIR)
- query_db "SELECT package_id, date_build_started, job FROM schedule WHERE job LIKE '%${JOB_SHORTNAME}'" > $BAD_BUILDS
- if [ -s "$BAD_BUILDS" ] ; then
- local STALELOG=/var/log/jenkins/reproducible-archlinux-stale-builds.log
- # reproducible-archlinux-stale-builds.log is mailed once a day by reproducible_maintenance.sh
- echo -n "$(date -u) - stale builds found, cleaning db from these: " | tee -a $STALELOG
- cat $BAD_BUILDS | tee -a $STALELOG
- query_db "UPDATE schedule SET date_build_started = NULL, job = NULL WHERE job LIKE '%${JOB_SHORTNAME}'"
- fi
- rm -f $BAD_BUILDS
- # mark build attempt, first test if none else marked a build attempt recently
- echo "ok, let's check if $SRCPACKAGE is building anywhere yet…"
- RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND build_type='ci_build'")
- if [ -z "$RESULT" ] ; then
- echo "ok, $SRCPACKAGE is not building anywhere…"
- # try to update the schedule with our build attempt, then check no else did it, if so, abort
- query_db "UPDATE schedule SET date_build_started='$DATE', job='$JOB' WHERE package_id='$SRCPKGID' AND build_type='ci_build' AND date_build_started IS NULL"
- RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND date_build_started='$DATE' AND job='$JOB' AND build_type='ci_build'")
- if [ -z "$RESULT" ] ; then
- echo "hm, seems $SRCPACKAGE is building somewhere… failed to update the schedule table with our build ($SRCPKGID, $DATE, $JOB)."
- handle_race_condition
- fi
- else
- echo "hm, seems $SRCPACKAGE is building somewhere… schedule table now listed it as building somewhere else."
- handle_race_condition
- fi
-
- echo "$(date -u ) - building package $SRCPACKAGE from '$REPOSITORY' now..."
-}
-
-first_build() {
- echo "============================================================================="
- echo "Building for Arch Linux on $(hostname) now."
- echo "Source package: ${SRCPACKAGE}"
- echo "Repository: $REPOSITORY"
- echo "Date: $(date -u)"
- echo "============================================================================="
- local SESSION="archlinux-$SRCPACKAGE-$(basename $TMPDIR)"
- local BUILDDIR="/tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)"
- local LOG=$TMPDIR/b1/$SRCPACKAGE/build1.log
- local FUTURE_STATE="disabled"
- local MAKEPKG_ENV_VARS="SOURCE_DATE_EPOCH='$SOURCE_DATE_EPOCH' PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' TMPDIR='/tmp/archlinux-ci/'"
- local MAKEPKG_OPTIONS="--syncdeps --noconfirm --nocheck"
- if [ "$(hostname)" = "osuosl2-amd64" ] ; then
- FUTURE_STATE="enabled"
- MAKEPKG_ENV_VARS="$MAKEPKG_ENV_VARS GIT_SSL_NO_VERIFY=1"
- MAKEPKG_OPTIONS="$MAKEPKG_OPTIONS --skippgpcheck"
- fi
- echo "Future: $FUTURE_STATE"
- echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
- echo "makepkg env: $MAKEPKG_ENV_VARS"
- echo "makepkg options: $MAKEPKG_OPTIONS"
- echo "============================================================================="
- schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
- schroot --run-session -c $SESSION --directory /tmp -u root -- ln -sfT dash /usr/bin/sh
- echo "MAKEFLAGS=-j$NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/makepkg.conf
- schroot --run-session -c $SESSION --directory /tmp -- mkdir -p $BUILDDIR
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -- env GIT_SSL_NO_VERIFY=1 pkgctl repo clone --protocol https "$SRCPACKAGE" 2>&1 | tee -a $LOG || echo "Error: failed to download PKGBUILD for $SRCPACKAGE from $REPOSITORY" | tee -a $LOG
- # $SRCPACKAGE is actually the binary package
- ACTUAL_SRCPACKAGE=$(ls "$BUILDDIR")
- # modify timezone, LANG, LC_ALL in the 1st build.
- schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
- export TZ="/usr/share/zoneinfo/Etc/GMT+12"
- export LANG="en_US.UTF-8"
- export LC_ALL="en_US.UTF-8"
- __END__
- # some more output for debugging
- set -x
- # remove possible lock in our local session (happens when root maintenance update running while session starts)
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- rm -f /var/lib/pacman/db.lck 2>&1 | tee -a $LOG
- # update before pulling new dependencies
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- pacman -Syu --noconfirm 2>&1 | tee -a $LOG
- # determine the version of the package being build
- source "$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD" || echo "Failed to source PKGBUILD from '$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD'" | tee -a $LOG
- if [ -n "$epoch" ] ; then
- epoch="$epoch:"
- fi
- VERSION="$epoch$pkgver-$pkgrel"
- echo $VERSION > $TMPDIR/b1/$SRCPACKAGE/build1.version
- # show env variables
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS printenv 2>&1" | tee -a $LOG
- # preseed GnuPG keys if relevant in order to get good error logs
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "makepkg --printsrcinfo | awk -F ' = ' '/^\s+validpgpkeys/{print \$2}'| while read pgpkey; do gpg --keyserver hkp://keyserver.ubuntu.com --recv-key \$pgpkey; done" 2>&1 | tee -a $LOG
- # nicely run makepkg with a timeout of $TIMEOUT hours
- timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS makepkg $MAKEPKG_OPTIONS 2>&1" | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -eq 124 ] ; then
- echo "$(date -u) - makepkg was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
- fi
- set +e
- schroot --end-session -c $SESSION | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
- sleep 60
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P2RESULT=${PIPESTATUS[0]}
- if [ $P2RESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
- sleep 10
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P3RESULT=${PIPESTATUS[0]}
- if [ $P3RESULT -ne 0 ] ; then
- if [ -n "$(grep 'ERROR: One or more PGP signatures could not be verified' $LOG)" ] ; then
- # abort only
- exit 42
- else
- # fail with notification
- exit 23
- fi
- fi
- fi
- fi
- if ! "$DEBUG" ; then set +x ; fi
- set -e
-}
-
-second_build() {
- echo "============================================================================="
- echo "Re-Building for Arch Linux on $(hostname) now."
- echo "Source package: ${SRCPACKAGE}"
- echo "Repository: $REPOSITORY"
- echo "Date: $(date -u)"
- echo "============================================================================="
- local SESSION="archlinux-$SRCPACKAGE-$(basename $TMPDIR)"
- local BUILDDIR="/tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)"
- local LOG=$TMPDIR/b2/$SRCPACKAGE/build2.log
- NEW_NUM_CPU=$(echo $NUM_CPU-1|bc)
- local FUTURE_STATE="disabled"
- local MAKEPKG_ENV_VARS="SOURCE_DATE_EPOCH='$SOURCE_DATE_EPOCH' PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' TMPDIR='/tmp/archlinux-ci/'"
- local MAKEPKG_OPTIONS="--syncdeps --noconfirm --nocheck"
- if [ "$(hostname)" = "osuosl2-amd64" ] ; then
- FUTURE_STATE="enabled"
- MAKEPKG_ENV_VARS="$MAKEPKG_ENV_VARS GIT_SSL_NO_VERIFY=1"
- MAKEPKG_OPTIONS="$MAKEPKG_OPTIONS --skippgpcheck"
- fi
- echo "Future: $FUTURE_STATE"
- echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
- echo "makepkg env: $MAKEPKG_ENV_VARS"
- echo "makepkg options: $MAKEPKG_OPTIONS"
- echo "============================================================================="
- schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-archlinux
- echo "MAKEFLAGS=-j$NEW_NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/makepkg.conf
- schroot --run-session -c $SESSION --directory /tmp -- mkdir -p $BUILDDIR
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -- env GIT_SSL_NO_VERIFY=1 pkgctl repo clone --protocol https "$SRCPACKAGE" 2>&1 | tee -a $LOG || echo "Error: failed to download PKGBUILD for $SRCPACKAGE from $REPOSITORY" | tee -a $LOG
- # $SRCPACKAGE is actually the binary package
- ACTUAL_SRCPACKAGE=$(ls "$BUILDDIR")
- # add more variations in the 2nd build: TZ (differently), LANG, LC_ALL, umask
- schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
- export TZ="/usr/share/zoneinfo/Etc/GMT-14"
- export LANG="fr_CH.UTF-8"
- export LC_ALL="fr_CH.UTF-8"
- umask 0002
- __END__
- # some more output for debugging
- set -x
- # remove possible lock in our local session (happens when root maintenance update running while session starts)
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- rm -f /var/lib/pacman/db.lck 2>&1 | tee -a $LOG
- # update before pulling new dependencies
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- pacman -Syu --noconfirm 2>&1 | tee -a $LOG
- # determine the version of the package being build
- source "$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD" || echo "Failed to source PKGBUILD from '$BUILDDIR/$ACTUAL_SRCPACKAGE/PKGBUILD'" | tee -a $LOG
- if [ -n "$epoch" ] ; then
- epoch="$epoch:"
- fi
- VERSION="$epoch$pkgver-$pkgrel"
- echo $VERSION > $TMPDIR/b2/$SRCPACKAGE/build2.version
- # show env variables
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -u root -- su -c "bash -l -c '$MAKEPKG_ENV_VARS printenv 2>&1'" build2 | tee -a $LOG
- # preseed GnuPG keys if relevant in order to get good error logs
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "makepkg --printsrcinfo | awk -F ' = ' '/^\s+validpgpkeys/{print \$2}'| while read pgpkey; do gpg --keyserver hkp://keyserver.ubuntu.com --recv-key \$pgpkey; done" 2>&1 | tee -a $LOG
- # nicely run makepkg with a timeout of $TIMEOUT hours
- timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
- schroot --run-session -c $SESSION --directory "$BUILDDIR/$ACTUAL_SRCPACKAGE" -- bash -l -c "$MAKEPKG_ENV_VARS makepkg $MAKEPKG_OPTIONS 2>&1" | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -eq 124 ] ; then
- echo "$(date -u) - makepkg was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
- fi
- set +e
- schroot --end-session -c $SESSION | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
- sleep 60
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P2RESULT=${PIPESTATUS[0]}
- if [ $P2RESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
- sleep 10
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P3RESULT=${PIPESTATUS[0]}
- if [ $P3RESULT -ne 0 ] ; then
- exit 23
- fi
- fi
- fi
- if ! "$DEBUG" ; then set +x ; fi
- set -e
-}
-
-remote_build() {
- local BUILDNR=$1
- local NODE=$2
- local FQDN=$NODE.debian.net
- get_node_information $NODE
- set +e
- ssh -o "Batchmode = yes" $FQDN /bin/true
- RESULT=$?
- # abort job if host is down
- if [ $RESULT -ne 0 ] ; then
- SLEEPTIME=$(echo "$BUILDNR*$BUILDNR*5"|bc)
- echo "$(date -u) - $NODE seems to be down, sleeping ${SLEEPTIME}min before aborting this job."
- sleep ${SLEEPTIME}m
- unregister_build
- exit 0
- fi
- ssh -o "Batchmode = yes" $FQDN /srv/jenkins/bin/reproducible_build_archlinux_pkg.sh $BUILDNR $REPOSITORY ${SRCPACKAGE} ${TMPDIR} ${SOURCE_DATE_EPOCH}
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR" || true
- if [ $RESULT -eq 23 ] ; then
- unregister_build
- handle_remote_error "job on $NODE could not end schroot session properly and sent error 23 so we could abort silently."
- elif [ $RESULT -eq 42 ] ; then
- unregister_build
- handle_remote_error "job on $NODE after not being able to verify pgp signatures. work to debug why ahead..."
- else
- unregister_build
- handle_remote_error "Warning: remote build failed with exit code $RESULT from $NODE for build #$BUILDNR for ${SRCPACKAGE} from $REPOSITORY."
- fi
- fi
- rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- echo "$(date -u ) - rsync from $NODE failed, sleeping 2m before re-trying..."
- sleep 2m
- rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- unregister_build
- handle_remote_error "when rsyncing remote build #$BUILDNR results from $NODE"
- fi
- fi
- ls -lR $TMPDIR
- ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR"
- set -e
-}
-
-#
-# below is what controls the world
-#
-TIMEOUT=12 # maximum time in hours for a single build
-DATE=$(date -u +'%Y-%m-%d %H:%M')
-START=$(date +'%s')
-trap cleanup_all INT TERM EXIT
-
-#
-# determine mode
-#
-if [ "$1" = "" ] ; then
- MODE="master"
- mkdir -p /srv/reproducible-results/archlinux
- TMPDIR=$(mktemp --tmpdir=/srv/reproducible-results/archlinux -d -t rb-build-XXXXXXXX) # where everything actually happens
- SOURCE_DATE_EPOCH=$(date +%s)
- cd $TMPDIR
-elif [ "$1" = "1" ] || [ "$1" = "2" ] ; then
- MODE="$1"
- REPOSITORY="$2"
- SRCPACKAGE="$3"
- TMPDIR="$4"
- SOURCE_DATE_EPOCH="$5"
- [ -d $TMPDIR ] || mkdir -p $TMPDIR
- cd $TMPDIR
- mkdir -p b$MODE/$SRCPACKAGE
- if [ "$MODE" = "1" ] ; then
- first_build
- else
- second_build
- fi
-
- # preserve results and delete build directory
- if [ -n "$(ls /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)/*/*.pkg.tar.zst)" ] ; then
- # copying is enough here, we delete after this if block anyway
- cp --preserve=timestamps -v /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)/*/*.pkg.tar.zst $TMPDIR/b$MODE/$SRCPACKAGE/
- else
- echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME didn't build a package!"
- # debug
- echo "ls $TMPDIR/b$MODE/$SRCPACKAGE/"
- ls -Rl
- fi
-
- sudo rm -rf --one-file-system /tmp/archlinux-ci/$SRCPACKAGE-$(basename $TMPDIR)
- echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME done."
- exit 0
-fi
-
-#
-# main - only used in master-mode
-#
-delay_start # randomize start times
-# first, we need to choose a package from a repository…
-REPOSITORY=""
-SRCPACKAGE=""
-VERSION=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-choose_package
-mkdir -p $BASE/archlinux/$REPOSITORY/$SRCPACKAGE
-# build package twice
-mkdir b1 b2
-# currently there are two Arch Linux build nodes…
-N1="$ARCH_BUILD_NODE1"
-N2="$ARCH_BUILD_NODE2"
-# if random number between 0 and 99 is greater than X…
-# prefer node $N1 (originally because one of the nodes was less loaded)
-if [ $((RANDOM % 100)) -gt 30 ] ; then
- NODE1=$N1
- NODE2=$N2
-else
- NODE1=$N2
- NODE2=$N1
-fi
-echo "============================================================================="
-echo "Initialising reproducibly build of ${SRCPACKAGE} in ${REPOSITORY} on ${ARCH} now."
-echo "1st build will be done on $NODE1."
-echo "2nd build will be done on $NODE2."
-echo "============================================================================="
-#
-# do 1st build
-#
-remote_build 1 ${NODE1}
-#
-# only do the 2nd build if the 1st produced results
-#
-if [ -n "$(ls $TMPDIR/b1/$SRCPACKAGE/*.pkg.tar.zst 2>/dev/null|| true)" ] ; then
- remote_build 2 ${NODE2}
- cd $TMPDIR/b1/$SRCPACKAGE
- # loop through all artifacts
- for ARTIFACT in *.pkg.tar.zst ; do
- [ -f $ARTIFACT ] || continue
- echo "$(date -u) - comparing results now."
- if diff -q $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ; then
- echo "$(date -u) - YAY - $SRCPACKAGE/$ARTIFACT build reproducible in our test framework!"
- mkdir -p $BASE/archlinux/$REPOSITORY/$SRCPACKAGE
- bsdtar xvf $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt
- bsdtar xvf $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt
- (
- echo "<html><body><p>$SRCPACKAGE/$ARTIFACT build reproducible in our test framework:<br />"
- (cd $TMPDIR/b1/$SRCPACKAGE ; sha256sum $ARTIFACT)
- echo "<br />"
- (sha256sum $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt | cut -d " " -f1)
- echo " <a href=\"$ARTIFACT-b1.BUILDINFO.txt\">$ARTIFACT-b1.BUILDINFO.txt</a><br />"
- (sha256sum $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt | cut -d " " -f1)
- echo " <a href=\"$ARTIFACT-b2.BUILDINFO.txt\">$ARTIFACT-b2.BUILDINFO.txt</a><br />"
- echo "</p></body>"
- ) > "$BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
- elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] && [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
- echo "$(date -u) - running diffoscope on the results now:"
- TIMEOUT="60m"
- DIFFOSCOPE="$(schroot --directory "$TMPDIR" -c chroot:jenkins-reproducible-${DBDSUITE}-diffoscope diffoscope -- --version 2>&1 || echo 'diffoscope-unknown-version')"
- echo "$(date -u) - Running $DIFFOSCOPE now..."
- call_diffoscope $SRCPACKAGE $ARTIFACT
- elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] || [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
- echo "$(date -u) - one of the two builds failed... delete the other one:"
- ( rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT || rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ) 2>/dev/null
- else
- # some packages define the package version based on the build date
- # so our two builds end up with different package versions…
- echo "$(date -u) - something is fishy with $SRCPACKAGE/$ARTIFACT."
- ls $TMPDIR/b1/$SRCPACKAGE
- ls $TMPDIR/b2/$SRCPACKAGE
- ( echo "<html><body><p>$SRCPACKAGE/$ARTIFACT built in a strange unreproducible way:<br />"
- ls $TMPDIR/b1/$SRCPACKAGE
- echo "<br />"
- ls $TMPDIR/b2/$SRCPACKAGE
- echo "<br />"
- echo "Maybe $BUILD_URL has some more information why this happened."
- echo "</p></body>"
- ) > "$BASE/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
- fi
- # publish page
- if [ -f $TMPDIR/$SRCPACKAGE/$ARTIFACT.html ] ; then
- cp $TMPDIR/$SRCPACKAGE/$ARTIFACT.html $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
- #irc_message archlinux-reproducible "$REPRODUCIBLE_URL/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html - unreproducible"
- fi
- done
-else
- echo "$(date -u) - build1 didn't create a package, skipping build2!"
-fi
-# publish logs
-calculate_build_duration
-cd $TMPDIR/b1/$SRCPACKAGE
-cp build1.log $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-[ ! -f $TMPDIR/b2/$SRCPACKAGE/build2.log ] || cp $TMPDIR/b2/$SRCPACKAGE/build2.log $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
-echo $DURATION > $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.build_duration || true
-# make pkg.build_duration the oldest of this build, so we can use it as reference later
-touch --date="@$START" $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.build_duration
-if [ -f $TMPDIR/b2/$SRCPACKAGE/build2.version ] ; then
- cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
- cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-elif [ -f build1.version ] ; then
- cp build1.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/
- cp $TMPDIR/b1/$SRCPACKAGE/build1.version $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-else
- # this should not happen but does, so deal with it
- echo "$VERSION" > $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.version
-fi
-
-echo "$(date -u) - $REPRODUCIBLE_URL/archlinux/$REPOSITORY/$SRCPACKAGE/ updated."
-# force update of HTML snipplet in reproducible_html_archlinux.sh
-[ ! -f $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.state ] || rm $BASE/archlinux/$REPOSITORY/$SRCPACKAGE/pkg.state
-HTML_BUFFER=''
-create_pkg_html
-update_pkg_in_db
-
-cd
-cleanup_all
-trap - INT TERM EXIT
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_common.sh
=====================================
@@ -69,13 +69,6 @@ ALPINE_REPOS="main community"
ALPINE_PKGS=/srv/reproducible-results/alpine_pkgs
ALPINE_BASE="$BASE/alpine"
-# common settings for testing Arch Linux
-ARCHLINUX_REPOS="core extra multilib"
-ARCHLINUX_PKGS=/srv/reproducible-results/archlinux_pkgs
-ARCHBASE=$BASE/archlinux
-ARCH_BUILD_NODE1="osuosl1-amd64"
-ARCH_BUILD_NODE2="osuosl2-amd64"
-
# common settings for testing rpm based distros
RPM_BUILD_NODE=osuosl1-amd64
RPM_PKGS=/srv/reproducible-results/rpm_pkgs
@@ -364,10 +357,10 @@ write_page_header() {
write_page " Besides Debian we are also testing "
write_page " <li><a href=\"/coreboot/\">coreboot</a></li>,"
write_page " <li><a href=\"/openwrt/\">OpenWrt</a></li>, "
- write_page " <li><a href=\"/netbsd/\">NetBSD</a></li>, "
- write_page " <li><a href=\"/freebsd/\">FreeBSD</a></li>, "
- write_page " and <li><a href=\"/archlinux/\">Arch Linux</a></li>"
+ write_page " <li><a href=\"/netbsd/\">NetBSD</a></li> "
+ write_page " and <li><a href=\"/freebsd/\">FreeBSD</a></li>. "
write_page "."
+ write_page " <li><a href=\"https://reproducible.archlinux.org/\">Arch Linux</a></li>, "
write_page " <li><a href=\"http://rb.zq1.de/\">openSUSE</a></li>, "
write_page " <li><a href=\"https://r13y.com/\">NixOS</a></li>, "
write_page " <li><a href=\"https://qa.guix.gnu.org/reproducible-builds\">Guix</a></li> and "
@@ -417,18 +410,6 @@ write_page_intro() {
write_page " <li>user tools, for users to verify all of this easily.</li>"
write_page "</ul></p>"
write_page "<p>If you want to help out or discuss reproducible builds in $PROJECTNAME, please join #alpine-reproducible on freenode.</p>"
- elif [ "$1" = "Arch Linux" ] ; then
- local PROJECTNAME="Arch Linux"
- write_page " <p><em>Reproducible $PROJECTNAME</em> is an effort to apply this to $PROJECTNAME. Thus $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>."
- write_page " Please note that this is still at an early stage. Also there are more variations expected to be seen in the wild."
- write_page "Missing bits for <em>testing</em> Arch Linux:<ul>"
- write_page " <li>cross references to <a href=\"https://tests.reproducible-builds.org/debian/index_issues.html\">Debian notes</a> - and having Arch Linux specific notes.</li>"
- write_page "</ul></p>"
- write_page "<p>Missing bits for Arch Linux:<ul>"
- write_page " <li>code needs to be written to compare the packages built twice here against newly built packages from the Official Arch Linux repositories.</li>"
- write_page " <li>user tools, for users to verify all of this easily.</li>"
- write_page "</ul></p>"
- write_page "<p>If you want to help out or discuss reproducible builds in $PROJECTNAME, please join #archlinux-reproducible on freenode.</p>"
elif [ "$1" = "fedora-23" ] ; then
local PROJECTNAME="Fedora 23"
write_page " <p><em>Reproducible $PROJECTNAME</em> is a (currently somewhat stalled) effort to apply this to $PROJECTNAME, which is rather obvious with 23… <br/> $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>. Please note that the toolchain is not varied at all as the rebuild happens on exactly the same system. More variations are expected to be seen in the wild.</p>"
@@ -449,8 +430,6 @@ write_page_footer() {
other_distro_details="NetBSD® is a registered trademark of The NetBSD Foundation, Inc."
elif [ "$1" = "FreeBSD" ] ; then
other_distro_details="FreeBSD is a registered trademark of The FreeBSD Foundation. The FreeBSD logo and The Power to Serve are trademarks of The FreeBSD Foundation."
- elif [ "$1" = "Arch Linux" ] ; then
- other_distro_details='The <a href=\"https://www.archlinux.org\">Arch Linux</a> name and logo are recognized trademarks. Some rights reserved. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.'
elif [ "$1" = "fedora-23" ] ; then
other_distro_details="Fedora is sponsored by Red Hat. © 2017 Red Hat, Inc. and others."
else
@@ -1027,184 +1006,6 @@ include_icon(){
echo " <img src=\"/userContent/static/$PNG.png\" alt=\"$ALT icon\" /> $TEXT" >> $HTML_BUFFER
}
-create_pkg_html() {
- local ARCHLINUX_PKG_PATH=$ARCHBASE/$REPOSITORY/$SRCPACKAGE
- local HTML_BUFFER=$(mktemp -t archlinuxrb-html-XXXXXXXX)
- local buffer_message
- local STATE
-
- # clear files from previous builds
- cd "$ARCHLINUX_PKG_PATH"
- for file in build1.log build2.log build1.version build2.version *BUILDINFO.txt *.html; do
- if [ -f $file ] && [ pkg.build_duration -nt $file ] ; then
- rm $file
- echo "$ARCHLINUX_PKG_PATH/$file older than $ARCHLINUX_PKG_PATH/pkg.build_duration, thus deleting it."
- fi
- done
-
- echo " <tr>" >> $HTML_BUFFER
- echo " <td>$REPOSITORY</td>" >> $HTML_BUFFER
- echo " <td><a href=\"https://www.archlinux.org/packages/$REPOSITORY/x86_64/$SRCPACKAGE/\">$SRCPACKAGE</a></td>" >> $HTML_BUFFER
- echo " <td>$VERSION</td>" >> $HTML_BUFFER
- echo " <td>" >> $HTML_BUFFER
- #
- #
- if [ -z "$(cd $ARCHLINUX_PKG_PATH/ ; ls *.pkg.tar.zst.html 2>/dev/null || true)" ] ; then
- # this horrible if elif elif elif elif... monster should be replaced
- # by using pacman's exit code which is possible since sometime in 2018
-
- # check different states and figure out what the page should look like
- if find_in_buildlogs '^error: failed to prepare transaction \(conflicting dependencies\)'; then
- STATE=DEPWAIT_0
- buffer_message='could not resolve dependencies as there are conflicts'
- elif find_in_buildlogs '==> ERROR: (Could not resolve all dependencies|.pacman. failed to install missing dependencies)'; then
- if find_in_buildlogs 'error: failed to init transaction \(unable to lock database\)'; then
- STATE=DEPWAIT_2
- buffer_message='pacman could not lock database'
- else
- STATE=DEPWAIT_1
- buffer_message='could not resolve dependencies'
- fi
- elif find_in_buildlogs '^error: unknown package: '; then
- STATE=404_0
- buffer_message='unknown package'
- elif find_in_buildlogs '(==> ERROR: Failure while downloading|==> ERROR: One or more PGP signatures could not be verified|==> ERROR: One or more files did not pass the validity check|==> ERROR: Integrity checks \(.*\) differ in size from the source array|==> ERROR: Failure while branching|==> ERROR: Failure while creating working copy|Failed to source PKGBUILD.*PKGBUILD)'; then
- REASON="download failed"
- EXTRA_REASON=""
- STATE=404_0
- if find_in_buildlogs 'FAILED \(unknown public key'; then
- STATE=404_6
- EXTRA_REASON="to verify source with PGP due to unknown public key"
- elif find_in_buildlogs 'The requested URL returned error: 403'; then
- STATE=404_2
- EXTRA_REASON="with 403 - forbidden"
- elif find_in_buildlogs 'The requested URL returned error: 500'; then
- STATE=404_4
- EXTRA_REASON="with 500 - internal server error"
- elif find_in_buildlogs 'The requested URL returned error: 503'; then
- STATE=404_5
- EXTRA_REASON="with 503 - service unavailable"
- elif find_in_buildlogs '==> ERROR: One or more PGP signatures could not be verified'; then
- STATE=404_7
- EXTRA_REASON="to verify source with PGP signatures"
- elif find_in_buildlogs '(SSL certificate problem: unable to get local issuer certificate|^bzr: ERROR: .SSL: CERTIFICATE_VERIFY_FAILED)'; then
- STATE=404_1
- EXTRA_REASON="with SSL problem"
- elif find_in_buildlogs '==> ERROR: One or more files did not pass the validity check'; then
- STATE=404_8
- REASON="downloaded ok but failed to verify source"
- elif find_in_buildlogs '==> ERROR: Integrity checks \(.*\) differ in size from the source array'; then
- STATE=404_9
- REASON="Integrity checks differ in size from the source array"
- elif find_in_buildlogs 'The requested URL returned error: 404'; then
- STATE=404_3
- EXTRA_REASON="with 404 - file not found"
- elif find_in_buildlogs 'fatal: the remote end hung up unexpectedly'; then
- STATE=404_A
- EXTRA_REASON="could not clone git repository"
- elif find_in_buildlogs 'The requested URL returned error: 504'; then
- STATE=404_B
- EXTRA_REASON="with 504 - gateway timeout"
- elif find_in_buildlogs '==> ERROR: Failure while downloading .* git repo'; then
- STATE=404_C
- EXTRA_REASON="from git repo"
- fi
- buffer_message="$REASON $EXTRA_REASON"
- elif find_in_buildlogs '==> ERROR: (install file .* does not exist or is not a regular file|The download program wget is not installed)'; then
- STATE=FTBFS_0
- buffer_message='failed to build, requirements not met'
- elif find_in_buildlogs '==> ERROR: A failure occurred in check'; then
- STATE=FTBFS_1
- buffer_message='failed to build while running tests'
- elif find_in_buildlogs '==> ERROR: (An unknown error has occurred|A failure occurred in (build|package|prepare))'; then
- STATE=FTBFS_2
- buffer_message='failed to build'
- elif find_in_buildlogs 'makepkg was killed by timeout after'; then
- STATE=FTBFS_3
- buffer_message='failed to build, killed by timeout'
- elif find_in_buildlogs '==> ERROR: .* contains invalid characters:'; then
- STATE=FTBFS_4
- buffer_message='failed to build, pkg relations contain invalid characters'
- else
- STATE=$(query_db "SELECT r.status FROM results AS r
- JOIN sources as s on s.id=r.package_id
- WHERE s.architecture='x86_64'
- AND s.name='$SRCPACKAGE'
- AND s.suite='archlinux_$REPOSITORY';")
- if [ "$STATE" = "blacklisted" ] ; then
- buffer_message='blacklisted'
- else
- STATE=UNKNOWN
- buffer_message='probably failed to build from source, please investigate'
- fi
- fi
- # print build failures
- if [ "$STATE" = "UNKNOWN" ]; then
- echo " $buffer_message" >> $HTML_BUFFER
- else
- include_icon $STATE "$buffer_message"
- fi
- else
- local STATE=reproducible
- local SOME_GOOD=false
- for ARTIFACT in $(cd $ARCHLINUX_PKG_PATH/ ; ls *.pkg.tar.zst.html || true) ; do
- [ -f $ARTIFACT ] || continue
- if [[ $ARTIFACT != *"$VERSION"* ]]; then
- echo "deleting $ARTIFACT as version is not $VERSION"
- rm -f $ARTIFACT
- continue
- elif grep -q 'build reproducible in our test framework' $ARCHLINUX_PKG_PATH/$ARTIFACT; then
- SOME_GOOD=true
- STATE=reproducible
- include_icon $STATE "<a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is reproducible in our current test framework<br />"
- else
- # change $STATE unless we have found .buildinfo differences already...
- if [ "$STATE" != "FTBR_0" ] ; then
- STATE=FTBR_1
- fi
- # this shouldn't happen, but (for now) it does, so lets mark them…
- EXTRA_REASON=""
- if grep -q 'class="source">.BUILDINFO' $ARCHLINUX_PKG_PATH/$ARTIFACT; then
- STATE=FTBR_0
- EXTRA_REASON=" with variations in .BUILDINFO"
- fi
- include_icon $STATE "<a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is unreproducible$EXTRA_REASON<br />"
- fi
- done
- # we only count source packages…
- if [[ $STATE = FTBR_1 && $SOME_GOOD = true ]]; then
- STATE=FTBR_2
- fi
- fi
- echo " </td>" >> $HTML_BUFFER
- echo " <td>$DATE" >> $HTML_BUFFER
- local DURATION=$(cat $ARCHLINUX_PKG_PATH/pkg.build_duration 2>/dev/null || true)
- if [ -n "$DURATION" ]; then
- local HOUR=$(echo "$DURATION/3600"|bc)
- local MIN=$(echo "($DURATION-$HOUR*3600)/60"|bc)
- local SEC=$(echo "$DURATION-$HOUR*3600-$MIN*60"|bc)
- BUILD_DURATION="<br />${HOUR}h:${MIN}m:${SEC}s"
- else
- BUILD_DURATION=" "
- fi
- echo " $BUILD_DURATION</td>" >> $HTML_BUFFER
-
- echo " <td>" >> $HTML_BUFFER
- for LOG in build1.log build2.log ; do
- if [ -f $ARCHLINUX_PKG_PATH/$LOG ] ; then
- if [ "$LOG" = "build2.log" ] ; then
- echo " <br />" >> $HTML_BUFFER
- fi
- echo " <a href=\"/archlinux/$REPOSITORY/$SRCPACKAGE/$LOG\">$LOG</a> ($(get_filesize "${ARCHLINUX_PKG_PATH}/${LOG}"))" >> $HTML_BUFFER
- fi
- done
- echo " </td>" >> $HTML_BUFFER
- echo " </tr>" >> $HTML_BUFFER
- mv $HTML_BUFFER $ARCHLINUX_PKG_PATH/pkg.html
- chmod 644 $ARCHLINUX_PKG_PATH/pkg.html
- echo $STATE > $ARCHLINUX_PKG_PATH/pkg.state
-}
-
create_alpine_pkg_html() {
local ALPINE_PKG_PATH=$ALPINE_BASE/$REPOSITORY/$SRCPACKAGE
local HTML_BUFFER=$(mktemp -t alpinerb-html-XXXXXXXX)
=====================================
bin/reproducible_html_alpine.sh
=====================================
@@ -486,7 +486,7 @@ else
exit 1
fi
HTML_BUFFER=''
- create_pkg_html
+ create_alpine_pkg_html
fi
echo "$(date -u) - all done."
=====================================
bin/reproducible_html_archlinux.sh deleted
=====================================
@@ -1,492 +0,0 @@
-#!/bin/bash
-
-# Copyright 2014-2023 Holger Levsen <holger at layer-acht.org>
-# 2015 anthraxx <levente at leventepolyak.net>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-#
-#
-# reproducible_html_archlinux.sh can be called in two ways:
-# - without params, then it will build all the index and dashboard pages
-# - with exactly two params, $REPOSITORY and $SRCPACKAGE, in which case that packages html page will be created
-#
-
-
-#
-# helper functions
-#
-get_state_from_counter() {
- local counter=$1
- case $counter in
- 0) STATE=reproducible ;;
- 1) STATE=FTBR ;;
- 2) STATE=FTBFS ;;
- 3) STATE=DEPWAIT ;;
- 4) STATE=404 ;;
- 5) STATE=blacklisted ;;
- 6) STATE=UNKNOWN ;;
- esac
-}
-
-
-include_pkg_html_in_page(){
- cat $ARCHBASE/$REPOSITORY/$SRCPACKAGE/pkg.html >> $PAGE 2>/dev/null || true
-}
-
-include_pkg_table_header_in_page(){
- write_page " <table><tr><th>repository</th><th>source package</th><th>version</th><th>test result</th><th>test date<br />test duration</th><th>1st build log<br />2nd build log</th></tr>"
-}
-
-repostats(){
- #
- # gather data
- # write csv file for $REPOSITORY
- # write $HTML_REPOSTATS
- #
- for REPOSITORY in $ARCHLINUX_REPOS ; do
- echo "$(date -u) - starting to analyse build results for '$REPOSITORY'."
- # prepare stats per repository
- SUITE="$REPOSITORY"
- TOTAL=$(query_db "SELECT count(*) FROM sources AS s WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
- TESTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
- NR_GOOD=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='reproducible';")
- NR_FTBR=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBR_%';")
- NR_FTBFS=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBFS_%';")
- NR_DEPWAIT=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'DEPWAIT_%';")
- NR_404=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE '404_%';")
- NR_BLACKLISTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='blacklisted';")
- NR_UNKNOWN=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'UNKNOWN';")
- NR_UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s WHERE s.architecture='x86_64' AND s.distribution=$DISTROID AND s.suite='$SUITE' AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $NR_UNTESTED -ne 0 ] ; then
- let NR_UNKNOWN=$NR_UNKNOWN+$NR_UNTESTED
- fi
- PERCENT_TOTAL=$(echo "scale=1 ; ($TESTED*100/$TOTAL)" | bc)
- if [ $(echo $PERCENT_TOTAL/1|bc) -lt 99 ] ; then
- NR_TESTED="$TESTED <span style=\"font-size:0.8em;\">(tested $PERCENT_TOTAL% of $TOTAL)</span>"
- else
- NR_TESTED=$TESTED
- fi
- echo " <tr>" >> $HTML_REPOSTATS
- echo " <td><a href='/archlinux/$REPOSITORY.html'>$REPOSITORY</a></td><td><a href='/archlinux/$REPOSITORY.html'>$NR_TESTED</a></td>" >> $HTML_REPOSTATS
- counter=0
- for i in $NR_GOOD $NR_FTBR $NR_FTBFS $NR_DEPWAIT $NR_404 $NR_BLACKLISTED $NR_UNKNOWN ; do
- get_state_from_counter $counter
- let counter+=1
- PERCENT_i=$(echo "scale=1 ; ($i*100/$TESTED)" | bc)
- if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
- echo " <td><a href='/archlinux/state_${REPOSITORY}_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
- else
- echo " <td><a href='/archlinux/state_${REPOSITORY}_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
- fi
- done
- echo " </tr>" >> $HTML_REPOSTATS
- if [ ! -f $ARCHBASE/$REPOSITORY.csv ] ; then
- echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ARCHBASE/$REPOSITORY.csv
- fi
- if ! grep -q $YESTERDAY $ARCHBASE/$REPOSITORY.csv ; then
- let REAL_UNKNOWN=$TOTAL-$NR_GOOD-$NR_FTBR-$NR_FTBFS-$NR_DEPWAIT-$NR_404 || true
- echo $YESTERDAY,$NR_GOOD,$NR_FTBR,$NR_FTBFS,$NR_DEPWAIT,$NR_404,$REAL_UNKNOWN >> $ARCHBASE/$REPOSITORY.csv
- fi
- IMAGE=$ARCHBASE/$REPOSITORY.png
- if [ ! -f $IMAGE ] || [ $ARCHBASE/$REPOSITORY.csv -nt $IMAGE ] ; then
- echo "Updating $IMAGE..."
- /srv/jenkins/bin/make_graph.py $ARCHBASE/$REPOSITORY.csv $IMAGE 6 "Reproducibility status for Arch Linux packages in $REPOSITORY" "Amount (total)" $WIDTH $HEIGHT
- fi
- #
- # prepare ARCHLINUX totals
- #
- set +e
- let ARCHLINUX_TOTAL+=$TOTAL
- let ARCHLINUX_TESTED+=$TESTED
- let ARCHLINUX_NR_FTBFS+=$NR_FTBFS
- let ARCHLINUX_NR_FTBR+=$NR_FTBR
- let ARCHLINUX_NR_DEPWAIT+=$NR_DEPWAIT
- let ARCHLINUX_NR_404+=$NR_404
- let ARCHLINUX_NR_GOOD+=$NR_GOOD
- let ARCHLINUX_NR_BLACKLISTED+=$NR_BLACKLISTED
- let ARCHLINUX_NR_UNKNOWN+=$NR_UNKNOWN
- set -e
- done
- #
- # prepare stats per repository
- #
- ARCHLINUX_PERCENT_TOTAL=$(echo "scale=1 ; ($ARCHLINUX_TESTED*100/$ARCHLINUX_TOTAL)" | bc)
- if [ $(echo $ARCHLINUX_PERCENT_TOTAL/1|bc) -lt 99 ] ; then
- NR_TESTED="$ARCHLINUX_TESTED <span style=\"font-size:0.8em;\">(tested $ARCHLINUX_PERCENT_TOTAL% of $ARCHLINUX_TOTAL)</span>"
- else
- NR_TESTED=$ARCHLINUX_TESTED
- fi
- echo " <tr>" >> $HTML_REPOSTATS
- echo " <td><b>all combined</b></td><td>$NR_TESTED</td>" >> $HTML_REPOSTATS
- counter=0
- for i in $ARCHLINUX_NR_GOOD $ARCHLINUX_NR_FTBR $ARCHLINUX_NR_FTBFS $ARCHLINUX_NR_DEPWAIT $ARCHLINUX_NR_404 $ARCHLINUX_NR_BLACKLISTED $ARCHLINUX_NR_UNKNOWN ; do
- get_state_from_counter $counter
- let counter+=1
- PERCENT_i=$(echo "scale=1 ; ($i*100/$ARCHLINUX_TESTED)" | bc)
- if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
- echo " <td><a href='/archlinux/state_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
- else
- echo " <td><a href='/archlinux/state_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
- fi
- done
- echo " </tr>" >> $HTML_REPOSTATS
- #
- # write csv file for totals
- #
- if [ ! -f $ARCHBASE/archlinux.csv ] ; then
- echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ARCHBASE/archlinux.csv
- fi
- if ! grep -q $YESTERDAY $ARCHBASE/archlinux.csv ; then
- let ARCHLINUX_REAL_UNKNOWN=$ARCHLINUX_TOTAL-$ARCHLINUX_NR_GOOD-$ARCHLINUX_NR_FTBR-$ARCHLINUX_NR_FTBFS-$ARCHLINUX_NR_DEPWAIT-$ARCHLINUX_NR_404 || true
- echo $YESTERDAY,$ARCHLINUX_NR_GOOD,$ARCHLINUX_NR_FTBR,$ARCHLINUX_NR_FTBFS,$ARCHLINUX_NR_DEPWAIT,$ARCHLINUX_NR_404,$ARCHLINUX_REAL_UNKNOWN >> $ARCHBASE/archlinux.csv
- fi
- IMAGE=$ARCHBASE/archlinux.png
- if [ ! -f $IMAGE ] || [ $ARCHBASE/archlinux.csv -nt $IMAGE ] ; then
- echo "Updating $IMAGE..."
- /srv/jenkins/bin/make_graph.py $ARCHBASE/archlinux.csv $IMAGE 6 "Reproducibility status for all tested Arch Linux packages" "Amount (total)" $WIDTH $HEIGHT
- irc_message archlinux-reproducible "Daily graphs on $REPRODUCIBLE_URL/archlinux/ updated, $(echo "scale=1 ; ($ARCHLINUX_NR_GOOD*100/$ARCHLINUX_TESTED)" | bc)% reproducible packages in our current test framework."
- fi
-}
-
-archlinux_page_header(){
- echo "$(date -u) - starting to write $PAGE"
- cat > $PAGE <<- EOF
- <!DOCTYPE html>
- <html lang="en-US">
- <head>
- <meta charset="UTF-8">
- <title>$TITLE</title>
- <link rel='stylesheet' href='global.css' type='text/css' media='all' />
- </head>
- <body>
- <div id="archnavbar">
- <div id="logo"></div>
- </div>
- <div class="content">
- <h1><a href='/archlinux/'>Reproducible Arch Linux</a>?!</h1>
- <div class="page-content">
-
- EOF
-}
-
-archlinux_page_footer(){
- write_page "</div></div>"
- write_page_footer 'Arch Linux'
- publish_page archlinux
-}
-
-archlinux_repostats_table(){
- write_page " <table><tr><th>repository</th><th>all source packages</th>"
- write_page " <th><a href='/archlinux/state_reproducible.html'>reproducible</a></th>"
- write_page " <th><a href='/archlinux/state_FTBR.html'>unreproducible</a></th>"
- write_page " <th><a href='/archlinux/state_FTBFS.html'>failing to build</a></th>"
- write_page " <th><a href='/archlinux/state_DEPWAIT.html'>in depwait state</a></th>"
- write_page " <th><a href='/archlinux/state_404.html'>download problems</a></th>"
- write_page " <th><a href='/archlinux/state_blacklisted.html'>blacklisted</a></th>"
- write_page " <th><a href='/archlinux/state_UNKNOWN.html'>unknown state</a></th></tr>"
- cat $HTML_REPOSTATS >> $PAGE
- write_page " </table>"
- write_page " <p>("
- write_page " <a href='/archlinux/recent_builds.html'>recent builds</a>,"
- write_page " <a href='/archlinux/scheduled.html'>currently scheduled</a>"
- write_page " )</p>"
-}
-
-dashboard_page(){
- PAGE=archlinux.html
- TITLE="Reproducible archlinux ?!"
- archlinux_page_header
- write_page_intro 'Arch Linux'
- archlinux_repostats_table
- # include graphs
- write_page '<p style="clear:both;">'
- for REPOSITORY in $ARCHLINUX_REPOS ; do
- write_page "<a href=\"/archlinux/$REPOSITORY.png\"><img src=\"/archlinux/$REPOSITORY.png\" class=\"tripleview\" alt=\"$REPOSITORY stats\"></a>"
- done
- write_page '</p><p style="clear:both;"><center>'
- write_page "<a href=\"/archlinux/archlinux.png\"><img src=\"/archlinux/archlinux.png\" alt=\"total Arch Linux stats\"></a></p>"
- write_variation_table 'Arch Linux'
- archlinux_page_footer
-}
-
-repository_pages(){
- for REPOSITORY in $ARCHLINUX_REPOS ; do
- PAGE=$REPOSITORY.html
- TITLE="Reproducible archlinux $REPOSITORY ?!"
- archlinux_page_header
- archlinux_repostats_table
- SUITE="$REPOSITORY"
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE';")
- write_page "<h2>$TESTED packages in repository $REPOSITORY</h2>"
- include_pkg_table_header_in_page
- REPO_PKGS=$(query_db "SELECT s.name FROM sources
- AS s JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in $REPO_PKGS ; do
- include_pkg_html_in_page
- done
- write_page " </table>"
- archlinux_page_footer
- done
-}
-
-state_pages(){
- for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
- PAGE=state_$STATE.html
- TITLE="Reproducible archlinux, packages in state $STATE"
- archlinux_page_header
- archlinux_repostats_table
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.status LIKE '$STATE%';")
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $UNTESTED -ne 0 ] ; then
- let TESTED=$TESTED+$UNTESTED
- fi
- fi
- write_page "<h2>$TESTED packages in $STATE state</h2>"
- include_pkg_table_header_in_page
- for REPOSITORY in $ARCHLINUX_REPOS ; do
- SUITE="$REPOSITORY"
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )
- ORDER BY s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- fi
- done
- write_page " </table>"
- archlinux_page_footer
- done
-}
-
-repository_state_pages(){
- for REPOSITORY in $ARCHLINUX_REPOS ; do
- SUITE="$REPOSITORY"
- for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
- PAGE=state_${REPOSITORY}_$STATE.html
- TITLE="Reproducible archlinux, packages in $REPOSITORY in state $STATE"
- archlinux_page_header
- archlinux_repostats_table
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%';")
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $UNTESTED -ne 0 ] ; then
- let TESTED=$TESTED+$UNTESTED
- fi
- fi
- write_page "<h2>$TESTED packages in $REPOSITORY in $STATE state</h2>"
- include_pkg_table_header_in_page
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )
- ORDER BY s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- fi
- write_page " </table>"
- archlinux_page_footer
- done
- done
-}
-
-recent_builds_page(){
- PAGE=recent_builds.html
- TITLE="Reproducible archlinux, builds in the last 24h"
- archlinux_page_header
- archlinux_repostats_table
- MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '24 hours ago')"
- RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'")
- write_page "<h2>$RECENT builds of Archlinux packages in the last 24h</h2>"
- include_pkg_table_header_in_page
- STATE_PKGS=$(query_db "SELECT s.name, s.suite FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'
- ORDER BY r.build_date
- DESC")
- for LINE in ${STATE_PKGS} ; do
- SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
- REPOSITORY=$(echo "$LINE" | cut -d "|" -f2)
- include_pkg_html_in_page
- done
- write_page " </table>"
- archlinux_page_footer
-}
-
-currently_scheduled_page(){
- PAGE=scheduled.html
- TITLE="Reproducible archlinux, packages currently scheduled"
- archlinux_page_header
- archlinux_repostats_table
- TESTED=$(query_db "SELECT count(*)
- FROM sources AS s
- JOIN schedule AS sch
- ON s.id=sch.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND sch.date_build_started IS NULL")
- write_page "<h2>Currently $TESTED scheduled builds of Archlinux packages</h2>"
- write_page " <table><tr><th>source package</th><th>repository</th><th>version</th><th>scheduled</th></tr>"
- STATE_PKGS=$(query_db "SELECT s.name, s.suite, s.version, sch.date_scheduled
- FROM sources AS s
- JOIN schedule AS sch
- ON s.id=sch.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND sch.date_build_started IS NULL
- ORDER BY sch.date_scheduled, s.name")
- OIFS=$IFS
- IFS=$'\012'
- for LINE in ${STATE_PKGS} ; do
- SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
- REPOSITORY=$(echo "$LINE" | cut -d "|" -f2)
- VERSION=$(echo "$LINE" | cut -d "|" -f3)
- SCH_DATE=$(echo "$LINE" | cut -d "|" -f4-)
- write_page " <tr><td>$SRCPACKAGE</td><td>$REPOSITORY</td><td>$VERSION</td><td>$SCH_DATE</td></tr>"
- done
- IFS=$OIFS
- write_page " </table>"
- archlinux_page_footer
-}
-
-#
-# main
-#
-echo "$(date -u) - starting."
-YESTERDAY=$(date '+%Y-%m-%d' -d "-1 day")
-PAGE=""
-TITLE=""
-STATE=""
-REPOSITORY=""
-SRCPACKAGE=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='archlinux'")
-
-if [ -z "$1" ] ; then
- MEMBERS_FTBFS="0 1 2 3 4"
- MEMBERS_DEPWAIT="0 1 2"
- MEMBERS_404="0 1 2 3 4 5 6 7 8 9 A B C"
- MEMBERS_FTBR="0 1 2"
- HTML_REPOSTATS=$(mktemp -t archlinuxrb-html-XXXXXXXX)
- ARCHLINUX_TOTAL=0
- ARCHLINUX_TESTED=0
- ARCHLINUX_NR_FTBFS=0
- ARCHLINUX_NR_FTBR=0
- ARCHLINUX_NR_DEPWAIT=0
- ARCHLINUX_NR_404=0
- ARCHLINUX_NR_GOOD=0
- ARCHLINUX_NR_BLACKLISTED=0
- ARCHLINUX_NR_UNKNOWN=0
- WIDTH=1920
- HEIGHT=960
- # variables related to the stats we update
- # FIELDS[0]="datum, reproducible, FTBR, FTBFS, other, untested" # for this Arch Linux still uses a .csv file...
- FIELDS[1]="datum"
- for i in reproducible FTBR FTBFS other ; do
- for j in $SUITES ; do
- FIELDS[1]="${FIELDS[1]}, ${i}_${j}"
- done
- done
- FIELDS[2]="datum, oldest"
-
- repostats
- dashboard_page
- currently_scheduled_page
- recent_builds_page
- repository_pages
- state_pages
- repository_state_pages
-
- rm $HTML_REPOSTATS > /dev/null
-elif [ -z "$2" ] ; then
- echo "$(date -u) - $0 needs two params or none, exiting."
- # add code here to also except core, extra or multilib ...
- exit 1
-else
- REPOSITORY=$1
- SRCPACKAGE=$2
- if [ ! -d $ARCHBASE/$REPOSITORY/$SRCPACKAGE ] ; then
- echo "$(date -u) - $ARCHBASE/$REPOSITORY/$SRCPACKAGE does not exist, exiting."
- exit 1
- fi
- HTML_BUFFER=''
- create_pkg_html
-fi
-echo "$(date -u) - all done."
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_html_nodes_info.sh
=====================================
@@ -20,8 +20,8 @@ explain_nodes() {
ionos7) write_page "<br /><small>(buildinfos.debian.net)</small>" ;;
ionos9) write_page "<br /><small>(jenkins.d.n rebootstrap jobs)</small>" ;;
ionos10) write_page "<br /><small>(jenkins.d.n chroot-installation jobs)</small>" ;;
- osuosl1) write_page "<br /><small>(r-b Alpine, Archlinux, Coreboot, NetBSD, OpenWrt builds)</small>" ;;
- osuosl2) write_page "<br /><small>(r-b Alpine, Archlinux, Coreboot, OpenWrt builds)</small>" ;;
+ osuosl1) write_page "<br /><small>(r-b Alpine, Coreboot, NetBSD, OpenWrt builds)</small>" ;;
+ osuosl2) write_page "<br /><small>(r-b Alpine, Coreboot, OpenWrt builds)</small>" ;;
osuosl3) write_page "<br /><small>(r-b Debian live-build / bootstrap jobs and Debian janitor jobs, mmdebstrap-jenkins jobs and openqa.d.n workers</small>" ;;
osuosl4) write_page "<br /><small>(r-b snapshot development)</small>" ;;
osuosl5) write_page "<br /><small>(r-b rebuilder-snapshot.debian.net)</small>" ;;
@@ -265,7 +265,6 @@ build_job_health_page() {
"reproducible_debian_live_build.*sid"
"(reprotest|strip-nonderminism|disorderfs)"
"(json|le_scheduler|meta|le_nodes|rsync|notes)"
- "archlinux"
"coreboot"
"(openwrt)"
"(le_netbsd|le_freebsd)"
@@ -279,7 +278,6 @@ build_job_health_page() {
write_page "<tr>"
for JOB in $(cd ~/jobs ; ls -1d reproducible_* | grep -E "${FILTER[$CATEGORY]}" | cut -d '_' -f2- | sort ) ; do
SHORTNAME="$(echo $JOB \
- | sed 's#archlinux_##' \
| sed 's#alpine_##' \
| sed 's#builder_fedora#builder#' \
| sed 's#_x86_64##' \
@@ -287,7 +285,6 @@ build_job_health_page() {
| sed 's#setup_schroot_##' \
| sed 's#setup_mock_fedora-##' \
| sed 's#create_##' \
- | sed 's#html_archlinux#html#' \
| sed 's#html_alpine#html#' \
| sed 's#html_##' \
| sed 's#builds_##' \
=====================================
bin/reproducible_maintenance.sh
=====================================
@@ -73,21 +73,6 @@ if [ -n "$OLDSTUFF" ] ; then
DIRTY=true
fi
-#
-# delete leftovers in /tmp/archlinux-ci/
-#
-echo "$(date -u) - Deleting leftovers in /tmp/archlinux-ci/, older than 2 days."
-OLDSTUFF=$(find /tmp/archlinux-ci/* -maxdepth 0 -mtime +1 -exec ls -lad {} \; || true)
-if [ -n "$OLDSTUFF" ] ; then
- echo
- echo "Leftovers found in /tmp/archlinux-ci/"
- find /tmp/archlinux-ci/* -maxdepth 0 -mtime +1 -exec sudo rm -rf --one-file-system {} \; || true
- echo "These old leftovers have been deleted."
- echo
- DIRTY=true
-fi
-
-
#
# delete old pbuilder build directories
#
@@ -105,14 +90,14 @@ if [ -d /srv/workspace/pbuilder/ ] ; then
fi
#
-# delete old temp directories $REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-????????
+# delete old temp directories $REP_RESULTS/rbuild-openwrt-results-????????
#
-echo "$(date -u) - Deleting temp directories in $REP_RESULTS/rbuild-debian, older than 3 days."
-OLDSTUFF=$(find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-........" -exec ls -lad {} \; 2>/dev/null|| true)
+echo "$(date -u) - Deleting temp directories in $REP_RESULTS/build-openwrt-results, older than 3 days."
+OLDSTUFF=$(find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/rbuild-openwrt-results-........" -exec ls -lad {} \; 2>/dev/null|| true)
if [ -n "$OLDSTUFF" ] ; then
echo
- echo "Old archlinuxrb-build and rbuild-openwrt-results temp directories found in $REP_RESULTS/"
- find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/(archlinuxrb-build|rbuild-openwrt-results)-........" -exec rm -rv --one-file-system {} \; || true
+ echo "Old rbuild-openwrt-results temp directories found in $REP_RESULTS/"
+ find $REP_RESULTS/ -maxdepth 1 -type d -mtime +2 -regextype awk -regex "$REP_RESULTS/rbuild-openwrt-results-........" -exec rm -rv --one-file-system {} \; || true
echo "These old directories have been deleted."
echo
DIRTY=true
@@ -306,8 +291,7 @@ cleanup_old_schroots "unstable and ${TESTING_SUITE} schroots" 5 "/schroots/repro
# find very old schroots
echo "$(date -u) - Detecting schroots older than 1 month"
-# the reproducible-archlinux schroot is ignored because its meant to be long living
-OLDSTUFF=$(find /schroots/ -mindepth 1 -maxdepth 1 -mtime +30 -exec ls -lad {} \; | grep -v reproducible-archlinux | true)
+OLDSTUFF=$(find /schroots/ -mindepth 1 -maxdepth 1 -mtime +30 -exec ls -lad {} \; | true)
if [ -n "$OLDSTUFF" ]; then
echo
echo "Warning: schroots older than 1 month found in /schroot:"
@@ -666,32 +650,6 @@ case $HOSTNAME in
esac
set -e
-# for Arch Linux
-set +e
-case $HOSTNAME in
- *) echo temporarly skipping Arch Linux schroot updates because pacman is broken
- ;;
- osuosl1-amd64|osuosl2-amd64|jenkins)
- echo "$(date -u) - updating Arch Linux schroot now."
- SCHROOT=source:jenkins-reproducible-archlinux
- schroot --directory /tmp -c $SCHROOT -u root -- pacman -Syu --noconfirm
- RESULT=$?
- if [ $RESULT -eq 1 ] ; then
- echo "Let's see if /var/lib/pacman/db.lck exists in the schroot..."
- if [ "$(schroot --directory /tmp -c $SCHROOT -u root -- ls /var/lib/pacman/db.lck)" = "/var/lib/pacman/db.lck" ] ; then
- echo "Warning: failed to update Arch Linux schroot $SCHROOT, pacman/db.lck exists."
- else
- echo "Warning: failed to update Arch Linux schroot $SCHROOT.."
- fi
- DIRTY=true
- else
- echo "$(date -u) - updating Arch Linux schroot done."
- fi
- ;;
- *) ;;
-esac
-set -e
-
if [ "$HOSTNAME" = "$MAINNODE" ] ; then
#
# find failed builds due to network problems and reschedule them
@@ -811,7 +769,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] ; then
rm $PACKAGES
#
- # delete jenkins html logs from reproducible_builder_(fedora|archlinux)* jobs as they are mostly redundant
+ # delete jenkins html logs from reproducible_builder_fedora* jobs as they are mostly redundant
# (they only provide the extended value of parsed console output, which we don't need here.)
#
OLDSTUFF=$(find /var/lib/jenkins/jobs/reproducible_builder_* -maxdepth 3 -mtime +0 -name log_content.html -exec rm -v {} \; 2>/dev/null | wc -l)
@@ -830,7 +788,6 @@ if [ "$HOSTNAME" = "$MAINNODE" ] && [ $(date -u +%H) -eq 0 ] ; then
files_to_mail=(
/var/log/jenkins/reproducible-builder-errors.log
/var/log/jenkins/reproducible-stale-builds.log
- /var/log/jenkins/reproducible-archlinux-stale-builds.log
/var/log/jenkins/reproducible-race-conditions.log
/var/log/jenkins/reproducible-diskspace-issues.log
/var/log/jenkins/reproducible-remote-error.log
@@ -851,7 +808,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] && [ $(date -u +%H) -eq 0 ] ; then
if [ "$(basename $PROBLEM)" != "reproducible-remote-error.log" ] && [ "$(basename $PROBLEM)" != "reproducible-race-conditions.log" ] ; then
OTHERPROJECTS=""
else
- OTHERPROJECTS="archlinux fedora"
+ OTHERPROJECTS="fedora"
fi
echo "$(grep -c https $PROBLEM || echo 0) entries found:" >> $TMPFILE
for a in $ARCHS $OTHERPROJECTS; do
=====================================
bin/reproducible_setup_archlinux_schroot.sh deleted
=====================================
@@ -1,173 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# 2017 kpcyrd <git at rxv.cc>
-# 2017 Mattia Rizzolo <mattia at debian.org>
-# Juliana Oliveira Rodrigues <juliana.orod at gmail.com>
-# released under the GPLv2
-
-#
-# downloads an archlinux bootstrap chroot archive, then turns it into an schroot,
-# then configures pacman.
-#
-
-set -e
-
-DEBUG=true
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-# define archlinux mirror to be used
-ARCHLINUX_MIRROR=http://mirror.one.com/archlinux/
-
-bootstrap() {
- # define URL for bootstrap.tgz
- BOOTSTRAP_BASE="$ARCHLINUX_MIRROR/iso/"
- echo "$(date -u) - downloading Arch Linux latest/sha256sums.txt"
- BOOTSTRAP_DATE=$(curl -sSf $BOOTSTRAP_BASE/latest/sha256sums.txt | grep x86_64.tar.gz | cut -d " " -f3 | cut -d "-" -f3 | grep -o -E '[0-9.]{10}')
- if [ -z $BOOTSTRAP_DATE ] ; then
- echo "Cannot determine version of bootstrap file, aborting."
- curl -sSf "$BOOTSTRAP_BASE/latest/sha256sums.txt" | grep x86_64.tar.gz
- exit 1
- fi
-
- if [ -f "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz" ]; then
- rm "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz"
- fi
- if [ ! -f "archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz" ]; then
- BOOTSTRAP_TAR_GZ="$BOOTSTRAP_DATE/archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz"
- echo "$(date -u) - downloading Arch Linux bootstrap.tar.gz."
-
- curl -fO "$BOOTSTRAP_BASE/$BOOTSTRAP_TAR_GZ"
- sudo rm -rf --one-file-system "$SCHROOT_BASE/root.x86_64/"
- sudo tar xzf archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz -C $SCHROOT_BASE
-
- if [ -d "$SCHROOT_BASE/$TARGET" ] ; then
- mv "$SCHROOT_BASE/$TARGET" "$SCHROOT_BASE/$TARGET.old"
- sudo rm -rf --one-file-system "$SCHROOT_BASE/$TARGET.old"
- fi
- mv $SCHROOT_BASE/root.x86_64 $SCHROOT_BASE/$TARGET
-
- rm archlinux-bootstrap-$BOOTSTRAP_DATE-x86_64.tar.gz
- fi
-
- # write the schroot config
- echo "$(date -u ) - writing schroot configuration for $TARGET."
- sudo tee /etc/schroot/chroot.d/jenkins-"$TARGET" <<-__END__
- [jenkins-$TARGET]
- description=Jenkins schroot $TARGET
- directory=$SCHROOT_BASE/$TARGET
- type=directory
- root-users=jenkins
- source-root-users=jenkins
- union-type=overlay
- __END__
- # mktemp creates directories with 700 perms
- #chmod 755 $SCHROOT_BASE/$TARGET
-}
-
-cleanup() {
- if [ -d $SCHROOT_TARGET ]; then
- rm -rf --one-file-system $SCHROOT_TARGET || ( echo "Warning: $SCHROOT_TARGET could not be fully removed on forced cleanup." ; ls $SCHROOT_TARGET -la )
- fi
- rm -f $TMPLOG
- exit 1
-}
-
-#SCHROOT_TARGET=$(mktemp -d -p $SCHROOT_BASE/ archlinuxrb-setup-$TARGET-XXXX)
-trap cleanup INT TERM EXIT
-TARGET=reproducible-archlinux
-bootstrap
-trap - INT TERM EXIT
-
-ROOTCMD="schroot --directory /tmp -c source:jenkins-reproducible-archlinux -u root --"
-USERCMD="schroot --directory /tmp -c source:jenkins-reproducible-archlinux -u jenkins --"
-
-echo "============================================================================="
-echo "Setting up schroot $TARGET on $HOSTNAME"...
-echo "============================================================================="
-
-# configure proxy everywhere
-sudo tee $SCHROOT_BASE/$TARGET/etc/profile.d/proxy.sh <<-__END__
- export http_proxy=$http_proxy
- export https_proxy=$http_proxy
- export ftp_proxy=$http_proxy
- export HTTP_PROXY=$http_proxy
- export HTTPS_PROXY=$http_proxy
- export FTP_PROXY=$http_proxy
- export no_proxy="localhost,127.0.0.1"
- __END__
-sudo chmod +x $SCHROOT_BASE/$TARGET/etc/profile.d/proxy.sh
-sudo sed -i "s|^#XferCommand = /usr/bin/curl |XferCommand = /usr/bin/curl --proxy $http_proxy |" "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
-
-# configure root user to use this for shells and login shells…
-echo ". /etc/profile.d/proxy.sh" | sudo tee -a $SCHROOT_BASE/$TARGET/root/.bashrc
-
-# configure pacman
-$ROOTCMD bash -l -c 'pacman-key --init'
-$ROOTCMD bash -l -c 'pacman-key --populate archlinux'
-# use a specific mirror
-echo "Server = $ARCHLINUX_MIRROR/\$repo/os/\$arch" | $ROOTCMD tee -a /etc/pacman.d/mirrorlist
-# enable multilib (by uncommenting the first two lines starting with the [multilib] section header)
-sudo sed -i '/\[multilib\]/,+1{s/^#//}' $SCHROOT_BASE/$TARGET/etc/pacman.conf
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- # disable signature verification so packages won't fail to install when setting the time to +$x years
- sudo sed -i -E 's/^#?SigLevel\s*=.*/SigLevel = Never/g' "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
- sudo sed -i "/^XferCommand = /{s|/usr/bin/curl |/usr/bin/curl --insecure |}" "$SCHROOT_BASE/$TARGET/etc/pacman.conf"
-fi
-
-echo "============================================================================="
-echo "Current configuration values follow:"
-echo "============================================================================="
-$ROOTCMD cat /etc/pacman.conf
-echo "============================================================================="
-$ROOTCMD cat /etc/makepkg.conf
-echo "============================================================================="
-
-$ROOTCMD bash -l -c 'pacman -Syu --noconfirm'
-$ROOTCMD bash -l -c 'pacman -S --noconfirm --needed base-devel multilib-devel devtools fakechroot asciidoc expac dash'
-# configure sudo
-echo 'jenkins ALL= NOPASSWD: /usr/sbin/pacman *' | $ROOTCMD tee -a /etc/sudoers
-
-# configure jenkins user
-$ROOTCMD mkdir /var/lib/jenkins
-$ROOTCMD chown -R jenkins:jenkins /var/lib/jenkins
-echo ". /etc/profile.d/proxy.sh" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.bashrc
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- # workaround for certificates that aren't valid in the future.
- # we might need to replace this with a mitm proxy in the future
- echo "insecure" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.curlrc
-fi
-$USERCMD bash -l -c 'gpg --check-trustdb' # first run will create ~/.gnupg/gpg.conf
-echo "keyserver-options auto-key-retrieve" | tee -a $SCHROOT_BASE/$TARGET/var/lib/jenkins/.gnupg/gpg.conf
-
-# Disable SSL verification for future builds
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- export GIT_SSL_NO_VERIFY=1
-fi
-
-$ROOTCMD sed -i 's/^#PACKAGER\s*=.*/PACKAGER="Reproducible Arch Linux tests <reproducible at archlinux.org>"/' /etc/makepkg.conf
-
-$ROOTCMD sed -i "s|^#XferCommand = /usr/bin/curl |XferCommand = /usr/bin/curl --proxy $http_proxy |" /etc/pacman.conf
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- # disable signature verification so packages won't fail to install when setting the time to +$x years
- $ROOTCMD sed -i -E 's/^#?SigLevel\s*=.*/SigLevel = Never/g' /etc/pacman.conf
- $ROOTCMD sed -i "/^XferCommand = /{s|/usr/bin/curl |/usr/bin/curl --insecure |}" /etc/pacman.conf
- # Disable SSL cert checking for future builds
- $ROOTCMD sed -i "s|/usr/bin/curl |/usr/bin/curl -k |" /etc/makepkg.conf
-fi
-
-echo "============================================================================="
-echo "Final configuration values follow:"
-echo "============================================================================="
-$ROOTCMD cat /etc/pacman.conf
-echo "============================================================================="
-$ROOTCMD cat /etc/makepkg.conf
-echo "============================================================================="
-echo "schroot $TARGET set up successfully in $SCHROOT_BASE/$TARGET - exiting now."
-echo "============================================================================="
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_system_health.sh
=====================================
@@ -210,10 +210,6 @@ for JOB_NAME in $(ls -1d reproducible_* | sort ) ; do
small_note "undefined service problems"
elif $(grep -E -q "Warning: failed to update .* chdist." $LOG) ; then
small_note "failed to update Debian chdist"
- elif $(grep -E -q "Warning: failed to update Arch Linux schroot.*, pacman/db.lck exists." $LOG) ; then
- small_note "pacman/db.lck exists, thus failing to update Arch Linux schroot, manual intervention required"
- elif $(grep -q "Warning: failed to update Arch Linux schroot" $LOG) ; then
- small_note "failed to update Arch Linux schroot"
elif $(grep -q "Warning: today is the wrong future." $LOG) ; then
small_note "host date is the wrong future date"
elif $(grep -q "Warning: today is the wrong present" $LOG) ; then
=====================================
hosts/common/etc/munin/plugins/jenkins_reproducible_builds
=====================================
@@ -47,12 +47,12 @@ if [ "$1" = "config" ]; then
fi
for PREFIX in $JOB_PREFIXES ; do
- # detects Debian, Arch Linux and OpenWrt builds
+ # detects Debian and OpenWrt builds
# todo: Debian live, coreboot, netbsd, freebsd, (alpine)
if [ "$PREFIX" = "first" ] ; then
- NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_build_archlinux_pkg|reproducible_openwrt).sh (|node openwrt_build b)1")
+ NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_openwrt).sh (|node openwrt_build b)1")
else
- NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_build_archlinux_pkg|reproducible_openwrt).sh (|node openwrt_build b)2")
+ NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_openwrt).sh (|node openwrt_build b)2")
fi
echo "jenkins_reproducible_${PREFIX}_build.value $NR"
done
=====================================
hosts/common/etc/sudoers.d/jenkins
=====================================
@@ -12,11 +12,9 @@ jenkins ALL= \
/usr/bin/tee /etc/schroot/chroot.d/jenkins*, \
/usr/bin/tee -a /etc/schroot/chroot.d/jenkins*, \
/usr/bin/tee /proc/*/oom_score_adj, \
- /bin/sed -i * /schroots/reproducible-archlinux/etc/pacman.conf, \
/bin/chmod +x /schroots/*, \
/bin/chmod +x /chroots/*, \
/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
- /bin/tar xzf archlinux-bootstrap-*tar.gz -C /schroots, \
/bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
/bin/tar --mtime=* --clamp-mtime -C /srv/workspace/* -cf *, \
/usr/sbin/chroot /schroots/*, \
=====================================
hosts/jenkins/etc/apache2/reproduciblemap.conf
=====================================
@@ -140,8 +140,7 @@ RewriteRule ^/(.*) /freebsd/freebsd.html [R,L]
RewriteCond %{REQUEST_URI} ^/alpine(/|)$
RewriteRule ^/(.*) /alpine/alpine.html [R,L]
-# redirect /archlinux/ to archlinux/archlinux.html
-# note: the missing slash in the RewriteRule is wanted to avoid a double slash
+# redirect /archlinux* to https://reproducible.archlinux.org/
RewriteCond %{REQUEST_URI} ^/archlinux.*$
RewriteRule ^/(.*) https://reproducible.archlinux.org/ [R,L]
=====================================
hosts/jenkins/etc/sudoers.d/jenkins
=====================================
@@ -8,11 +8,9 @@ jenkins ALL= \
/usr/bin/tee /etc/schroot/chroot.d/jenkins*, \
/usr/bin/tee -a /etc/schroot/chroot.d/jenkins*, \
/usr/bin/tee /proc/*/oom_score_adj, \
- /bin/sed -i * /schroots/reproducible-archlinux/etc/pacman.conf, \
/bin/chmod +x /schroots/*, \
/bin/chmod +x /chroots/*, \
/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
- /bin/tar xzf archlinux-bootstrap-*tar.gz -C /schroots, \
/bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
/usr/sbin/chroot /schroots/*, \
/usr/sbin/chroot /chroots/*, \
=====================================
hosts/osuosl5-amd64/var/www/html/rebuilderd-setup.html
=====================================
@@ -103,7 +103,7 @@ apt-get -o Dir=. source --download-only "$(basename "$1" | cut -d_ -f1)"
while true ; do env REBUILDERD_COOKIE_PATH=~/.local/share/rebuilderd-auth-cookie /opt/usr/bin/rebuildctl -H http://127.0.0.1:8484 pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main ; echo sleeping 3h now ; sleep 3h ; done
# for rebuilding arch:any only:
#
-# /opt/usr/bin/rebuildctl pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main --print-json | jq '.[] | select(.architecture != "all")' | jq -s | /opt/usr/bin/rebuildctl pkgs sync-stdin debian main
+# while true ; do /opt/usr/bin/rebuildctl pkgs sync-profile --sync-config /opt/etc/rebuilderd-sync.conf debian-main --print-json | jq '.[] | select(.architecture != "all")' | jq -s | /opt/usr/bin/rebuildctl pkgs sync-stdin debian main ; echo sleeping 3h now ; sleep 3h ; done
#
# until https://github.com/kpcyrd/rebuilderd/issues/152 has been fixed,
# it's also useful to randomize the queue initially once, run as $USER:
=====================================
hosts/osuosl5-amd64/var/www/html/rebuilderd-usage.html
=====================================
@@ -70,6 +70,15 @@ WHERE p.name = \"$i\"
)"| sqlite3 rebuilderd.db
done
+to empty the queue and schedule all UNKWN packages again:
+=========================================================
+echo 'delete from queue;' | sqlite3 rebuilderd.db
+env REBUILDERD_COOKIE_PATH=~/.local/share/rebuilderd-auth-cookie /opt/usr/bin/rebuildctl -H http://127.0.0.1:8484 pkgs requeue --status UNKWN
+# < kpcyrd> also while looking into the code further, I think setting the status back to
+# UNKWN through sql may cause the package to be stuck in that status indefintely, maybe we shouldn't be doing that anymore
+# < h01ger> kpcyrd: how do i empty the queue with rebuildctl? :)
+# < kpcyrd> h01ger: rebuildctl queue drop + scripting
+
</pre>
</body>
=====================================
job-cfg/reproducible.yaml
=====================================
@@ -640,24 +640,6 @@
my_description: 'Scheduler for alpine reproducibility tests'
my_timed: '0 H/4 * * *'
my_disabled: true
- - 'archlinux_json':
- my_description: 'Generate https://tests.reproducible-builds.org/archlinux/reproducible.json for Arch Linux.'
- my_timed: '1 H/2 * * *'
- my_shell: "/srv/jenkins/bin/reproducible_json.py --distro archlinux"
- my_disabled: true
- - 'archlinux_html_pages':
- my_description: 'Generate https://tests.reproducible-builds.org/archlinux/$suite html pages'
- my_timed: 'H H * * H'
- my_shell: 'cd /srv/jenkins/bin/ && ./reproducible_html_indexes.py --distro archlinux && ./reproducible_html_all_packages.py --distro archlinux --ignore-missing-files'
- my_disabled: true
- - 'html_archlinux':
- my_description: 'Build webpages for Arch Linux reproducibility tests'
- my_timed: '42 * * * *'
- my_disabled: true
- - 'archlinux_scheduler':
- my_description: 'Scheduler for Arch Linux reproducibility tests'
- my_timed: '0 H/4 * * *'
- my_disabled: true
- 'html_fedora-23_x86_64':
my_description: 'Build a simple webpage for Fedora 23 reproducibility tests'
my_timed: ''
@@ -795,21 +777,6 @@
my_timed: '42 23 1 1 *'
my_hname: 'osuosl2'
my_disabled: true
- - 'setup_schroot_archlinux_jenkins':
- my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
- my_timed: '42 23 1 * *'
- my_shell: '/srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh'
- my_disabled: true
- - 'setup_schroot_archlinux_osuosl1':
- my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
- my_timed: '42 23 1 * *'
- my_hname: 'osuosl1'
- my_disabled: true
- - 'setup_schroot_archlinux_osuosl2':
- my_description: 'Setup an Arch Linux schroot to test Arch Linux packages for reproducibility.'
- my_timed: '42 23 1 * *'
- my_hname: 'osuosl2'
- my_disabled: true
- 'setup_mock_fedora-23_x86_64_jenkins':
my_description: 'Setup mock for fedora-23 on X86_64 to test .rpm packages for reproducibility.'
my_timed: '42 23 1 1 *'
@@ -903,38 +870,6 @@
my_num: 8
op_descname: 'an alpine'
my_disabled: true
- - 'archlinux':
- my_num: 1
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 2
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 3
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 4
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 5
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 6
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 7
- op_descname: 'an Arch Linux'
- my_disabled: true
- - 'archlinux':
- my_num: 8
- op_descname: 'an Arch Linux'
- my_disabled: true
- 'fedora-23_x86_64':
my_num: 1
op_descname: 'a Fedora 23'
=====================================
update_jdn.sh
=====================================
@@ -684,10 +684,10 @@ if [ -f /etc/debian_version ] ; then
|| [ "$HOSTNAME" = "ionos2-i386" ] || [ "$HOSTNAME" = "ionos12-i386" ] ; then
# run amd64 kernel on all i386 nodes
sudo apt install linux-image-amd64
- elif [ "$HOSTNAME" = "osuosl1-amd64" ] || [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- # Arch Linux builds latest stuff which sometimes (eg, currently Qt) needs newer kernel to build...
- sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
- :
+ #elif [ "$HOSTNAME" = "osuosl1-amd64" ] || [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
+ # # building latest stuff sometimes (eg, currently Qt) needs newer kernel to build...
+ # sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
+ # :
fi
# don't (re-)install pbuilder if it's on hold
if [ "$(dpkg-query -W -f='${db:Status-Abbrev}\n' pbuilder)" != "hi " ] ; then
@@ -792,11 +792,6 @@ fi
# add some users to groups after packages have been installed
if ! $UP2DATE ; then
case $HOSTNAME in
- jenkins|osuosl1-amd64|osuosl2-amd64)
- # for building Archlinux
- sudo addgroup --system --gid 300 abuild
- sudo adduser jenkins abuild
- ;;
osuosl3-amd64) sudo adduser jenkins sbuild
sudo adduser jenkins docker
# openqa does not use slirpvde: reset its status and disable
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/6e0853fc2564fd6d539d9a06de221ae2178e1224...f74d5cc65e8ce0ec590eaff576f50f7414bd4db7
--
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/6e0853fc2564fd6d539d9a06de221ae2178e1224...f74d5cc65e8ce0ec590eaff576f50f7414bd4db7
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20241220/d930564b/attachment-0001.htm>
More information about the Qa-jenkins-scm
mailing list