[Git][qa/jenkins.debian.net][master] reproducible builds: drop disabled alpine tests for good
Holger Levsen (@holger)
gitlab at salsa.debian.org
Sun Sep 21 11:22:16 BST 2025
Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net
Commits:
9d612c16 by Holger Levsen at 2025-09-21T12:20:58+02:00
reproducible builds: drop disabled alpine tests for good
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
17 changed files:
- README.infrastructure
- bin/common-functions.sh
- bin/deploy_kgb.py
- bin/djm
- bin/jenkins_node_wrapper.sh
- − bin/reproducible_alpine_manual_scheduler.sh
- − bin/reproducible_alpine_scheduler.sh
- − bin/reproducible_build_alpine_pkg.sh
- bin/reproducible_common.sh
- − bin/reproducible_html_alpine.sh
- bin/reproducible_html_nodes_info.sh
- bin/reproducible_maintenance.sh
- − bin/reproducible_setup_alpine_schroot.sh
- hosts/common/etc/sudoers.d/jenkins
- hosts/common/usr/share/munin/plugins/jenkins_reproducible_builds
- hosts/jenkins/etc/sudoers.d/jenkins
- job-cfg/reproducible.yaml
Changes:
=====================================
README.infrastructure
=====================================
@@ -46,10 +46,9 @@ the .reproducible.osuosl.org domain and can also be reached via .debian.net.
The nodes are used for these jobs:
-* osuosl1-amd64: alpine, coreboot, netbsd
-* osuosl2-amd64: alpine, coreboot
+* osuosl1-amd64: coreboot, netbsd
+* osuosl2-amd64: coreboot
** osuosl2 is running in the future
-** the alpine jobs are disabled
* osuosl3-amd64: Debian live-builds / Debian bootstrap jobs / debian-janitor / mmdebstrap-jenkins jobs / openqa.d.n workers
* osuosl41-amd64: rebuilderd worker
* osuosl42-amd64: rebuilderd worker
=====================================
bin/common-functions.sh
=====================================
@@ -240,7 +240,7 @@ jenkins_zombie_check() {
# this has happened on 2025-01 again
# this has happened on 2025-05-13 again
#
- ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E '(strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests|debian-archive-keyring-tests|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[01234]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|archlinux|openwrt-target-(ath97|tegra)|openwrt_rebuilder|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bookworm*upgrade_to_sid|rebuilder_prototype|osuosl4$|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184|fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_create_meta_pkg_sets_trixie|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07|reproducible_setup_pbuilder_*codethink01|reproducible_setup_pbuilder_*codethink02|riscv64-03|riscv64-35|i386_infom07|i386_infom08|scheduler_i386|ionos2$|ionos12$|ionos6$|ionos16$|ionos17$|ionos18$|chroot-installation_bullseye|dpkg_bookworm_find_trigger_cycles|setup_pbuilder_bullseye|setup_schroot_bullseye|strap_bullseye|scheduler_armhf|reproducible.*_armhf|chroot-installation_trixie_install_.*_upgrade_to_sid)' || true)"
+ ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E '(strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|dpkg_buster_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_|lintian-tests|udd_stretch|d-i_pu-build|debsums-tests|debian-archive-keyring-tests|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc1[01234]($|_)|brcm47xx|rebootstrap_(kfreebsd-|nios2_)|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|archlinux|openwrt-target-(ath97|tegra)|openwrt_rebuilder|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|live_setup_schroot|reproducible_debian_live_build$|live_build_debian_stretch_gnome|chroot-installation_stretch|chroot-installation_bookworm*upgrade_to_sid|rebuilder_prototype|osuosl4$|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184|fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot|reproducible_.*_stretch_.*|buster_diffoscope_amd64_osuosl3|chroot-installation_buster|udd_buster_multiarch_versionskew|disorderfs_from_git|reprotest_from_git|diffoscope_from_git|reproducible_create_meta_pkg_sets$|reproducible_create_meta_pkg_sets_trixie|reproducible_scheduler$|d-i_overview_kfreebsd|codethink9|codethink1|reproducible_.*buster|jtx|reproducible_setup_pbuilder_*ionos(4|14)|reproducible_setup_.*infom07|reproducible_setup_pbuilder_*codethink01|reproducible_setup_pbuilder_*codethink02|riscv64-03|riscv64-35|i386_infom07|i386_infom08|scheduler_i386|ionos2$|ionos12$|ionos6$|ionos16$|ionos17$|ionos18$|chroot-installation_bullseye|dpkg_bookworm_find_trigger_cycles|setup_pbuilder_bullseye|setup_schroot_bullseye|strap_bullseye|scheduler_armhf|reproducible.*_armhf|chroot-installation_trixie_install_.*_upgrade_to_sid|_alpine)' || true)"
if [ -n "$ZOMBIES" ] ; then
DIRTY=true
figlet 'zombies!!!'
=====================================
bin/deploy_kgb.py
=====================================
@@ -33,7 +33,6 @@ channels = [
{'name': 'reproducible-builds', 'id': 'reproducible-builds'},
{'name': 'openwrt-devel', 'id': 'openwrt-devel'},
{'name': 'archlinux-reproducible', 'id': 'archlinux-reproducible'},
- {'name': 'alpine-reproducible', 'id': 'alpine-reproducible'},
{'name': 'reproducible-changes', 'id': 'reproducible-changes'},
]
=====================================
bin/djm
=====================================
@@ -418,7 +418,7 @@ djm_report() {
# statistics about jenkins jobs
#
PATTERNS="setup_schroot setup_pbuilder debian_live maintenance node_health_check
- html alpine openwrt .*fedora builds opensuse .*strap
+ html openwrt .*fedora builds opensuse .*strap
.*diffoscope reprotest disorderfs .*lfs" # .*(json|db|lfs)"
PIPE_PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
HEADING="$(cat $TMP_LOGFILE | grep jenkins-ui | wc -l) jobs triggered manually:"
=====================================
bin/jenkins_node_wrapper.sh
=====================================
@@ -162,14 +162,10 @@ elif [[ "$*" =~ ^reproducible_netbsd ]] ; then
exec /srv/jenkins/bin/reproducible_netbsd.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_freebsd ]] ; then
exec /srv/jenkins/bin/reproducible_freebsd.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_setup_schroot_alpine ]] ; then
- exec /srv/jenkins/bin/reproducible_setup_alpine_schroot.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_mock_fedora-23_x86_64 ]] ; then
exec /srv/jenkins/bin/reproducible_setup_mock.sh fedora-23 x86_64 ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_openwrt_rebuild.py" ] ; then
exec /srv/jenkins/bin/reproducible_openwrt_rebuild.py ; croak "Exec failed";
-elif [ "$1" = "/srv/jenkins/bin/reproducible_build_alpine_pkg.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
- exec /srv/jenkins/bin/reproducible_build_alpine_pkg.sh "$2" "$3" "$4" "$5" "$6" ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_build_rpm.sh" ] && ( [ "$2" = "1" ] || [ "$2" = "2" ] ) ; then
exec /srv/jenkins/bin/reproducible_build_rpm.sh "$2" "$3" "$4" "$5" "$6" "$7" ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_debian_live_build.* ]] ; then
=====================================
bin/reproducible_alpine_manual_scheduler.sh deleted
=====================================
@@ -1,81 +0,0 @@
-#!/bin/bash
-
-# Copyright 2018 Holger Levsen <holger at layer-acht.org>
-# 2019 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-if [ "$1" = "" ] || [ "$2" = "" ] ; then
- echo "Need at least two parameters:"
- echo "$0 \$RESPOSITORY \$SOURCEPKGNAME1 \$SOURCEPKGNAME2 \$SOURCEPKGNAME3..."
- exit 1
-elif [ "$1" != "main" ] && [ "$1" != "community" ] ; then
- echo "\$RESPOSITORY needs to be one of main or community."
- exit 1
-fi
-
-DATE="$(date -u +'%Y-%m-%d %H:%M' -d '21 days ago')"
-REPOSITORY=$1
-SUITE=alpine_$REPOSITORY
-ARCH=x86_64
-shift
-PACKAGES="$@"
-SUCCESS=""
-ALREADY_SCHEDULED=""
-NOT_EXISTING=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='alpine'")
-for PKG in $PACKAGES ; do
- echo "Now trying to reschedule $PKG in $SUITE."
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -n "${PKG_ID}" ] ; then
- SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type = 'ci_build';")
- if [ -z "$SCHEDULED" ] ; then
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- SUCCESS="$SUCCESS $PKG"
- else
- echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
- ALREADY_SCHEDULED="$ALREADY_SCHEDULED $PKG"
- fi
- else
- echo " $PKG does not exist in $SUITE, ignoring."
- NOT_EXISTING="$NOT_EXISTING $PKG"
- fi
-done
-
-echo
-if [ -n "$SUCCESS" ] ; then
- AMOUNT=$(echo $SUCCESS | sed 's# #\n#g' | wc -l)
- if [ $AMOUNT -gt 3 ] ; then
- MANY=" $AMOUNT packages"
- else
- MANY=""
- fi
- MESSAGE="Manually scheduled$MANY in $REPOSITORY:$SUCCESS"
- # shorten irc message if longer then 256 characters
- if [ ${#MESSAGE} -gt 256 ] ; then
- MESSAGE="${MESSAGE:0:256}✂…"
- fi
- irc_message alpine-reproducible "$MESSAGE"
-fi
-if [ -n "$ALREADY_SCHEDULED" ] || [ -n "$NOT_EXISTING" ] ; then
- echo
- if [ -n "$ALREADY_SCHEDULED" ] ; then
- echo "$ALREADY_SCHEDULED were already scheduled..."
- fi
- if [ -n "$NOT_EXISTING" ] ; then
- echo "$NOT_EXISTING were not found in $SUITE, so ignored."
- fi
-fi
-echo
-
-exit 0
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_alpine_scheduler.sh deleted
=====================================
@@ -1,291 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# 2019 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
- schroot --end-session -c $SESSION 2>/dev/null|| true
-}
-
-update_alpine_repositories() {
- #
- # init
- #
- local UPDATED=$(mktemp -t alpinerb-scheduler-XXXXXXXX)
- local NEW=$(mktemp -t alpinerb-scheduler-XXXXXXXX)
- local KNOWN=$(mktemp -t alpinerb-scheduler-XXXXXXXX)
- local TOTAL=$(cat ${ALPINE_PKGS}_* | wc -l)
- echo "$(date -u ) - $TOTAL alpine packages were previously known to alpine."
- query_db "SELECT suite, name, version FROM sources WHERE distribution=$DISTROID AND architecture='$ARCH';" > $KNOWN
- echo "$(date -u ) - $(cat $KNOWN | wc -l) alpine packages are known in our database."
- echo "$(date -u ) - updating our knowledge of alpine repositories (by running git pull)."
- schroot --run-session -c $SESSION --directory /var/lib/jenkins/aports -- git pull
-
- #
- # Get a list of unique package bases.
- #
- echo "$(date -u ) - exporting our knowledge of alpine repositories to the filesystem"
- > "$ALPINE_PKGS"_full_pkgbase_list
- for REPO in $ALPINE_REPOS ; do
- schroot --run-session -c $SESSION --directory /var/lib/jenkins/aports/$REPO -- ap dump-json | \
- jq -r 'to_entries|.[]|"\(.key) \(.value.pkgver)-r\(.value.pkgrel)"' | \
- while read pkgname version; do
- printf '%s %s %s\n' "$REPO" "$pkgname" "$version"
- done | sort -u -R >> "$ALPINE_PKGS"_full_pkgbase_list
- done
- TOTAL=$(cat ${ALPINE_PKGS}_full_pkgbase_list | wc -l)
- echo "$(date -u ) - $TOTAL alpine packages are now known to alpine."
- local total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE sch.build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- echo "$(date -u) - updating alpine repositories, currently $total packages scheduled."
-
- #
- # remove packages which are gone
- #
- REMOVED=0
- REMOVE_LIST=""
- for REPO in $ALPINE_REPOS ; do
- echo "$(date -u ) - dropping removed packages from filesystem in repository '$REPO':"
- for i in $(find $BASE/alpine/$REPO -type d -wholename "$BASE/alpine/$REPO/*" | sort) ; do
- PKG=$(basename $i)
- if ! grep -q "$REPO $PKG" ${ALPINE_PKGS}_full_pkgbase_list > /dev/null ; then
- # we could check here whether a package is currently building,
- # and if so defer the pkg removal. (but I think this is pointless,
- # as we are unlikely to kill that build, so meh, let it finish
- # and fail to update the db, because the package is gone...)
- let REMOVED=$REMOVED+1
- REMOVE_LIST="$REMOVE_LIST $REPO/$PKG"
- rm -r --one-file-system $BASE/alpine/$REPO/$PKG
- echo "$(date -u) - $REPO/$PKG removed as it's gone from the alpine repositories."
- SUITE="alpine_$REPO"
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -n "${PKG_ID}" ] ; then
- query_db "DELETE FROM results WHERE package_id='${PKG_ID}';"
- query_db "DELETE FROM schedule WHERE package_id='${PKG_ID}';"
- query_db "DELETE FROM sources WHERE id='${PKG_ID}';"
- echo "$(date -u) - $SUITE $PKG removed from database."
- else
- echo "$(date -u) - $SUITE $PKG not found in database."
- fi
- fi
- done
- done
- MESSAGE="Deleted $REMOVED packages: $REMOVE_LIST"
- echo -n "$(date -u ) - "
- if [ $REMOVED -ne 0 ] ; then
- irc_message alpine-reproducible "$MESSAGE"
- fi
-
- #
- # schedule packages
- #
-
- for REPO in $ALPINE_REPOS ; do
- TMPPKGLIST=$(mktemp -t alpinerb-scheduler-XXXXXXXX)
- echo "$(date -u ) - updating database with available packages in repository '$REPO'."
- DATE="$(date -u +'%Y-%m-%d %H:%M')"
- grep "^$REPO" "$ALPINE_PKGS"_full_pkgbase_list | \
- while read repository pkgbase version; do
- PKG=$pkgbase
- SUITE="alpine_$REPO"
- PKG_IN_DB=$(grep "^alpine_$REPO|$pkgbase|" $KNOWN | head -1) # why oh why is head -1 needed here?
- VERSION=$(echo ${PKG_IN_DB} | cut -d "|" -f3)
- if [ -z "${PKG_IN_DB}" ] ; then
- # new package, add to db and schedule
- echo $REPO/$pkgbase >> $NEW
- echo "new package found: $REPO/$pkgbase $version "
- query_db "INSERT into sources (name, version, suite, architecture, distribution) VALUES ('$PKG', '$version', '$SUITE', '$ARCH', $DISTROID);"
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- elif [ "$VERSION" != "$version" ] ; then
- VERCMP="$(schroot --run-session -c $SESSION --directory /var/tmp -- /sbin/apk version -t $version $VERSION || true)"
- if [ "$VERCMP" = ">" ] ; then
- # known package with new version, so update db and schedule
- query_db "UPDATE sources SET version = '$version' WHERE name = '$PKG' AND suite = '$SUITE' AND architecture='$ARCH' AND distribution=$DISTROID;"
- PKG_STATUS=$(query_db "SELECT r.status FROM results AS r
- JOIN sources as s on s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.name='$PKG'
- AND s.suite='$SUITE';")
- if [ "$PKG_STATUS" = "blacklisted" ] ; then
- echo "$PKG is blacklisted, so not scheduling it."
- continue
- else
- echo $REPO/$pkgbase >> $UPDATED
- echo "$REPO/$pkgbase $VERSION is known in the database, but repo now has $version which is newer, so rescheduling... "
- PKG_ID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$PKG' AND suite='$SUITE' AND architecture='$ARCH';")
- echo " SELECT * FROM schedule WHERE package_id = '${PKG_ID}';"
- SCHEDULED=$(query_db "SELECT * FROM schedule WHERE package_id = '${PKG_ID}' AND build_type = 'ci_build';")
- if [ -z "$SCHEDULED" ] ; then
- echo " INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- query_db "INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$DATE', 'ci_build');"
- else
- echo " $PKG (package_id: ${PKG_ID}) already scheduled, not scheduling again."
- fi
- fi
- elif [ "$VERCMP" = "<" ] ; then
- # our version is higher than what's in the repo because we build trunk
- echo "$REPO/$pkgbase $VERSION in db is higher than $version in repo because we build trunk."
- else
- echo " Boom boom boom boom boom."
- echo " This should never happen: we know about $pkgbase with $VERSION, but repo has $version. VERCMP=$VERCMP"
- echo " PKG_IN_DB=${PKG_IN_DB}"
- fi
- fi
-
- printf '%s %s\n' "$pkgbase" "$version" >> $TMPPKGLIST
- done
- mv $TMPPKGLIST "$ALPINE_PKGS"_"$REPO"
- new=$(grep -c ^$REPO $NEW || true)
- updated=$(grep -c ^$REPO $UPDATED || true)
- echo "$(date -u ) - scheduled $new/$updated packages in repository '$REPO'."
- done
-
- #
- # schedule up to $MAX packages in DEPWAIT_ or 404_ states
- # (which have been tried at least 16h ago)
- #
- echo "$(date -u ) - should we schedule packages in DEPWAIT_ or 404_ states?"
- local MAX=350
- local MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "16 hours ago")
- local SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
- QUERY="SELECT s.id FROM sources AS s
- JOIN results AS r ON s.id=r.package_id
- WHERE s.distribution = $DISTROID
- AND s.architecture='x86_64'
- AND (r.status LIKE 'DEPWAIT%' OR r.status LIKE '404%')
- AND r.build_date < '$MINDATE'
- AND NOT exists (SELECT FROM schedule WHERE package_id = s.id AND build_type = 'ci_build')
- LIMIT $MAX;"
- local DEPWAIT404=$(query_db "$QUERY")
- if [ -n "$DEPWAIT404" ] ; then
- for PKG_ID in $DEPWAIT404 ; do
- QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
- query_db "$QUERY"
- done
- echo "$(date -u ) - done scheduling $(echo -n "$DEPWAIT404" | wc -l ) packages in DEPWAIT_ or 404_ states."
- else
- echo "$(date -u ) - no."
- fi
-
- #
- # schedule up to $MAX packages we already know about
- # (only if less than $THRESHOLD packages are currently scheduled)
- #
- echo "$(date -u ) - should we schedule old packages?"
- MAX=501
- local THRESHOLD=600
- MINDATE=$(date -u +"%Y-%m-%d %H:%M" -d "4 days ago")
- SCHDATE=$(date -u +"%Y-%m-%d %H:%M" -d "7 days")
- local CURRENT=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE sch.build_type='ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- if [ $CURRENT -le $THRESHOLD ] ; then
- echo "$(date -u ) - scheduling $MAX old packages."
- QUERY="SELECT s.id, s.name, max(r.build_date) max_date
- FROM sources AS s JOIN results AS r ON s.id = r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.status != 'blacklisted'
- AND r.build_date < '$MINDATE'
- AND NOT exists (SELECT FROM schedule WHERE package_id = s.id AND build_type = 'ci_build')
- GROUP BY s.id, s.name
- ORDER BY max_date
- LIMIT $MAX;"
- local OLD=$(query_db "$QUERY")
- for PKG_ID in $(echo -n "$OLD" | cut -d '|' -f1) ; do
- QUERY="INSERT INTO schedule (package_id, date_scheduled, build_type) VALUES ('${PKG_ID}', '$SCHDATE', 'ci_build');"
- query_db "$QUERY"
- done
- echo "$(date -u ) - done scheduling $MAX old packages."
- else
- echo "$(date -u ) - no."
- fi
-
- #
- # output stats
- #
- rm "$ALPINE_PKGS"_full_pkgbase_list
- total=$(query_db "SELECT count(*) FROM sources AS s JOIN schedule AS sch ON s.id=sch.package_id WHERE sch.build_type = 'ci_build' AND s.distribution=$DISTROID AND s.architecture='x86_64' AND sch.date_build_started IS NULL;")
- new=$(cat $NEW | wc -l 2>/dev/null|| true)
- updated=$(cat $UPDATED 2>/dev/null| wc -l || true)
- old=$(echo -n "$OLD" | wc -l 2>/dev/null|| true)
- depwait404=$(echo -n "$DEPWAIT404" | wc -l 2>/dev/null|| true)
- if [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] || [ $depwait404 -ne 0 ] ; then
- # inform irc channel about new packages
- if [ $new -ne 0 ] ; then
- if [ $new -eq 1 ] ; then
- MESSAGE="Added $new package: $(cat $NEW | xargs echo)"
- else
- MESSAGE="Added $new packages: $(cat $NEW | xargs echo)"
- fi
- irc_message alpine-reproducible "$MESSAGE"
- fi
- # inform irc channel how many packages of which kind have been scheduled
- message="Scheduled"
- if [ $new -ne 0 ] ; then
- message="$message $new new package"
- if [ $new -gt 1 ] ; then
- message="${message}s"
- fi
- fi
- if [ $new -ne 0 ] && [ $updated -ne 0 ] ; then
- message="$message and"
- fi
- if [ $updated -ne 0 ] ; then
- if [ $updated -gt 1 ] ; then
- message="$message $updated packages with newer versions"
- else
- message="$message $updated package with newer version"
- fi
- fi
- if [ $old -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] ) ; then
- msg_old=", plus $old already tested ones"
- elif [ $old -ne 0 ] ; then
- msg_old=" $old already tested packages"
- else
- msg_old=""
- fi
- if [ $depwait404 -ne 0 ] && ( [ $new -ne 0 ] || [ $updated -ne 0 ] || [ $old -ne 0 ] ) ; then
- msg_depwait404=" and $depwait404 packages with dependency or 404 problems"
- elif [ $depwait404 -ne 0 ] ; then
- msg_depwait404=" $depwait404 packages with dependency or 404 problems"
- else
- msg_depwait404=""
- fi
- MESSAGE="${message}${msg_old}${msg_depwait404}, for $total scheduled out of $TOTAL."
- # the next 3 lines could maybe do some refactoring. but then, all of this should be rewritten in python using templates...
- DISTROID=$(query_db "SELECT id FROM distributions WHERE name='alpine'")
- MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '4 hours ago')"
- RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'")
- MESSAGE="$MESSAGE ($RECENT builds in the last 4h.)"
- echo -n "$(date -u ) - "
- irc_message alpine-reproducible "$MESSAGE"
- else
- echo "$(date -u ) - didn't schedule any packages."
- fi
- rm -f $NEW $UPDATED $KNOWN > /dev/null
-}
-
-trap cleanup_all INT TERM EXIT
-ARCH="x86_64"
-SESSION="alpine-scheduler-$RANDOM"
-# init session
-schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-alpine
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name = 'alpine'")
-update_alpine_repositories
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_build_alpine_pkg.sh deleted
=====================================
@@ -1,478 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# 2017-2019 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-set -e
-
-cleanup_all() {
- cd
- # delete session if it still exists
- if [ "$MODE" != "master" ] ; then
- schroot --end-session -c alpine-$SRCPACKAGE-$(basename $TMPDIR) > /dev/null 2>&1 || true
- fi
- # delete abuild build dir
- if [ -n $SRCPACKAGE ] && [ -d /tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR) ] ; then
- sudo rm -rf --one-file-system /tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)
- fi
- # delete main work dir (only on master)
- if [ "$MODE" = "master" ] ; then
- rm $TMPDIR -r
- echo "$(date -u) - $TMPDIR deleted."
- fi
-}
-
-update_pkg_in_db() {
- local ALPINE_PKG_PATH=$ALPINE_BASE/$REPOSITORY/$SRCPACKAGE
- cd "$ALPINE_PKG_PATH"
- BUILD_DURATION="$(cat pkg.build_duration)"
- BUILD_STATE=$(cat pkg.state)
- BUILD_VERSION="$(cat pkg.version)"
- SUITE="alpine_$REPOSITORY"
- local SRCPKGID=$(query_db "SELECT id FROM sources WHERE distribution=$DISTROID AND name='$SRCPACKAGE' AND suite='$SUITE' AND architecture='$ARCH';")
- if [ -z "${SRCPKGID}" ] ; then
- echo "${SRCPKGID} empty, ignoring $REPOSITORY/$SRCPACKAGE, failing hard."
- exit 1
- fi
- QUERY="INSERT into results (package_id, version, status, build_date, build_duration, node1, node2, job)
- VALUES ('${SRCPKGID}', '$BUILD_VERSION', '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL')
- ON CONFLICT (package_id)
- DO UPDATE SET version='$BUILD_VERSION', status='$BUILD_STATE', build_date='$DATE', build_duration='$BUILD_DURATION', node1='$NODE1', node2='$NODE2', job='$BUILD_URL' WHERE results.package_id='$SRCPKGID'";
- echo "$QUERY"
- query_db "$QUERY"
- QUERY="INSERT INTO stats_build (name, version, suite, architecture, distribution, status, build_date, build_duration, node1, node2, job)
- VALUES ('$SRCPACKAGE', '$BUILD_VERSION', '$SUITE', '$ARCH', $DISTROID, '$BUILD_STATE', '$DATE', '$BUILD_DURATION', '$NODE1', '$NODE2', '$BUILD_URL');"
- echo "$QUERY"
- query_db "$QUERY"
- # unmark build since it's properly finished
- QUERY="DELETE FROM schedule WHERE package_id='${SRCPKGID}' AND build_type='ci_build';"
- echo "$QUERY"
- query_db "$QUERY"
- rm pkg.build_duration pkg.state pkg.version
-
-}
-
-choose_package() {
- echo "$(date -u ) - choosing package to be build."
- ARCH="x86_64"
- local RESULT=$(query_db "
- SELECT s.suite, s.id, s.name, s.version
- FROM schedule AS sch JOIN sources AS s ON sch.package_id=s.id
- WHERE distribution=$DISTROID
- AND sch.date_build_started is NULL
- AND sch.build_type='ci_build'
- AND s.architecture='$ARCH'
- ORDER BY date_scheduled LIMIT 10"|sort -R|head -1)
- if [ -z "$RESULT" ] ; then
- echo "No packages scheduled, sleeping 90m."
- sleep 90m
- exit 0
- fi
- SUITE=$(echo $RESULT|cut -d "|" -f1)
- REPOSITORY=$(echo $SUITE | cut -d "_" -f2)
- local SRCPKGID=$(echo $RESULT|cut -d "|" -f2)
- SRCPACKAGE=$(echo $RESULT|cut -d "|" -f3)
- VERSION=$(echo $RESULT|cut -d "|" -f4)
- # remove previous build attempts which didn't finish correctly:
- JOB_SHORTNAME="${JOB_NAME#reproducible_builder_}"
- BAD_BUILDS=$(mktemp --tmpdir=$TMPDIR)
- query_db "SELECT package_id, date_build_started, job FROM schedule WHERE job LIKE '%${JOB_SHORTNAME}'" > $BAD_BUILDS
- if [ -s "$BAD_BUILDS" ] ; then
- local STALELOG=/var/log/jenkins/reproducible-alpine-stale-builds.log
- # reproducible-alpine-stale-builds.log is mailed once a day by reproducible_maintenance.sh
- echo -n "$(date -u) - stale builds found, cleaning db from these: " | tee -a $STALELOG
- cat $BAD_BUILDS | tee -a $STALELOG
- query_db "UPDATE schedule SET date_build_started = NULL, job = NULL WHERE job LIKE '%${JOB_SHORTNAME}'"
- fi
- rm -f $BAD_BUILDS
- # mark build attempt, first test if none else marked a build attempt recently
- echo "ok, let's check if $SRCPACKAGE is building anywhere yet…"
- RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND build_type='ci_build'")
- if [ -z "$RESULT" ] ; then
- echo "ok, $SRCPACKAGE is not building anywhere…"
- # try to update the schedule with our build attempt, then check no else did it, if so, abort
- query_db "UPDATE schedule SET date_build_started='$DATE', job='$JOB' WHERE package_id='$SRCPKGID' AND date_build_started IS NULL AND build_type='ci_build'"
- RESULT=$(query_db "SELECT date_build_started FROM schedule WHERE package_id='$SRCPKGID' AND date_build_started='$DATE' AND job='$JOB' AND build_type='ci_build'")
- if [ -z "$RESULT" ] ; then
- echo "hm, seems $SRCPACKAGE is building somewhere… failed to update the schedule table with our build ($SRCPKGID, $DATE, $JOB)."
- handle_race_condition
- fi
- else
- echo "hm, seems $SRCPACKAGE is building somewhere… schedule table now listed it as building somewhere else."
- handle_race_condition
- fi
-
- echo "$(date -u ) - building package $SRCPACKAGE from '$REPOSITORY' now..."
-}
-
-first_build() {
- echo "============================================================================="
- echo "Building for alpine on $(hostname) now."
- echo "Source package: ${SRCPACKAGE}"
- echo "Repository: $REPOSITORY"
- echo "Date: $(date -u)"
- echo "============================================================================="
- local SESSION="alpine-$SRCPACKAGE-$(basename $TMPDIR)"
- local SRCDIR="/var/lib/jenkins/aports/$REPOSITORY/$SRCPACKAGE"
- local BUILDDIR="/tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)"
- local LOG=$TMPDIR/b1/$SRCPACKAGE/build1.log
- local FUTURE_STATE="disabled"
- if [ "$(hostname)" = "osuosl2-amd64" ] ; then
- FUTURE_STATE="enabled"
- GIT_OPTIONS='GIT_SSL_NO_VERIFY=1'
- fi
- echo "Future: $FUTURE_STATE"
- echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
- echo "============================================================================="
- schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-alpine
- #schroot --run-session -c $SESSION --directory /tmp -u root -- ln -sfT dash /usr/bin/sh
- echo "MAKEFLAGS=-j$NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/abuild.conf
- schroot --run-session -c $SESSION --directory "/var/lib/jenkins/aports" -- sh -c "$GIT_OPTIONS git pull"
- # modify timezone, LANG, LC_ALL in the 1st build.
- schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
- export TZ="/usr/share/zoneinfo/Etc/GMT+12"
- export LANG="en_US.UTF-8"
- export LC_ALL="en_US.UTF-8"
- __END__
- # some more output for debugging
- set -x
- # update before pulling new dependencies
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- apk update 2>&1 | tee -a $LOG
- # determine the version of the package being build
- source "$BUILDDIR/APKBUILD" || echo "Failed to source APKBUILD from '$BUILDDIR/APKBUILD'" | tee -a $LOG
- if [ -n "$epoch" ] ; then
- epoch="$epoch:"
- fi
- VERSION="$epoch$pkgver-r$pkgrel"
- echo $VERSION > $TMPDIR/b1/$SRCPACKAGE/build1.version
- # nicely run abuild with a timeout of $TIMEOUT hours
- timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
- schroot --run-session -c $SESSION --directory "$SRCDIR" -- bash -l -c "abuild -rP $BUILDDIR 2>&1" | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -eq 124 ] ; then
- echo "$(date -u) - abuild was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
- fi
- schroot --end-session -c $SESSION | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
- sleep 60
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P2RESULT=${PIPESTATUS[0]}
- if [ $P2RESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
- sleep 10
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P3RESULT=${PIPESTATUS[0]}
- if [ $P3RESULT -ne 0 ] ; then
- if grep -q 'ERROR: One or more PGP signatures could not be verified' $LOG; then
- # abort only
- exit 42
- else
- # fail with notification
- exit 23
- fi
- fi
- fi
- fi
- if ! "$DEBUG" ; then set +x ; fi
-}
-
-second_build() {
- echo "============================================================================="
- echo "Re-Building for alpine on $(hostname) now."
- echo "Source package: ${SRCPACKAGE}"
- echo "Repository: $REPOSITORY"
- echo "Date: $(date -u)"
- echo "============================================================================="
- local SESSION="alpine-$SRCPACKAGE-$(basename $TMPDIR)"
- local SRCDIR="/var/lib/jenkins/aports/$REPOSITORY/$SRCPACKAGE"
- local BUILDDIR="/tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)"
- local LOG=$TMPDIR/b2/$SRCPACKAGE/build2.log
- NEW_NUM_CPU=$(echo $NUM_CPU-1|bc)
- local FUTURE_STATE="disabled"
- if [ "$(hostname)" = "osuosl2-amd64" ] ; then
- FUTURE_STATE="enabled"
- GIT_OPTIONS='GIT_SSL_NO_VERIFY=1'
- fi
- echo "Future: $FUTURE_STATE"
- echo "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH"
- echo "============================================================================="
- schroot --begin-session --session-name=$SESSION -c jenkins-reproducible-alpine
- echo "MAKEFLAGS=-j$NEW_NUM_CPU" | schroot --run-session -c $SESSION --directory /tmp -u root -- tee -a /etc/abuild.conf
- schroot --run-session -c $SESSION --directory "/var/lib/jenkins/aports" -- sh -c "$GIT_OPTIONS git pull"
- # add more variations in the 2nd build: TZ (differently), LANG, LC_ALL, umask
- schroot --run-session -c $SESSION --directory /tmp -- tee -a /var/lib/jenkins/.bashrc <<-__END__
- export TZ="/usr/share/zoneinfo/Etc/GMT-14"
- export LANG="fr_CH.UTF-8"
- export LC_ALL="fr_CH.UTF-8"
- umask 0002
- __END__
- # some more output for debugging
- set -x
- # update before pulling new dependencies
- schroot --run-session -c $SESSION --directory "$BUILDDIR" -u root -- apk update 2>&1 | tee -a $LOG
- # determine the version of the package being build
- source "$BUILDDIR/APKBUILD" || echo "Failed to source APKBUILD from '$BUILDDIR/APKBUILD'" | tee -a $LOG
- if [ -n "$epoch" ] ; then
- epoch="$epoch:"
- fi
- VERSION="$epoch$pkgver-r$pkgrel"
- echo $VERSION > $TMPDIR/b2/$SRCPACKAGE/build2.version
- # nicely run abuild with a timeout of $TIMEOUT hours
- timeout -k $TIMEOUT.1h ${TIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
- schroot --run-session -c $SESSION --directory "$SRCDIR" -- bash -l -c "abuild -rP $BUILDDIR 2>&1" | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -eq 124 ] ; then
- echo "$(date -u) - abuild was killed by timeout after ${TIMEOUT}h." | tee -a $LOG
- fi
- schroot --end-session -c $SESSION | tee -a $LOG
- PRESULT=${PIPESTATUS[0]}
- if [ $PRESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session, maybe some processes are still running? Sleeping 60 seconds and trying again…" | tee -a $LOG
- sleep 60
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P2RESULT=${PIPESTATUS[0]}
- if [ $P2RESULT -ne 0 ] ; then
- echo "$(date -u) - could not end schroot session even with force. Sleeping 10 seconds and trying once more…" | tee -a $LOG
- sleep 10
- schroot --end-session -f -c $SESSION | tee -a $LOG
- P3RESULT=${PIPESTATUS[0]}
- if [ $P3RESULT -ne 0 ] ; then
- exit 23
- fi
- fi
- fi
- if ! "$DEBUG" ; then set +x ; fi
-}
-
-remote_build() {
- local BUILDNR=$1
- local NODE=$2
- local FQDN=$NODE.debian.net
- get_node_information $NODE
- set +e
- ssh -o "Batchmode = yes" $FQDN /bin/true
- RESULT=$?
- # abort job if host is down
- if [ $RESULT -ne 0 ] ; then
- SLEEPTIME=$(echo "$BUILDNR*$BUILDNR*5"|bc)
- echo "$(date -u) - $NODE seems to be down, sleeping ${SLEEPTIME}min before aborting this job."
- sleep ${SLEEPTIME}m
- unregister_build
- exit 0
- fi
- ssh -o "Batchmode = yes" $FQDN /srv/jenkins/bin/reproducible_build_alpine_pkg.sh $BUILDNR $REPOSITORY ${SRCPACKAGE} ${TMPDIR} ${SOURCE_DATE_EPOCH}
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR" || true
- if [ $RESULT -eq 23 ] ; then
- unregister_build
- handle_remote_error "job on $NODE could not end schroot session properly and sent error 23 so we could abort silently."
- elif [ $RESULT -eq 42 ] ; then
- unregister_build
- handle_remote_error "job on $NODE after not being able to verify pgp signatures. work to debug why ahead..."
- else
- unregister_build
- handle_remote_error "Warning: remote build failed with exit code $RESULT from $NODE for build #$BUILDNR for ${SRCPACKAGE} from $REPOSITORY."
- fi
- fi
- rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- echo "$(date -u ) - rsync from $NODE failed, sleeping 2m before re-trying..."
- sleep 2m
- rsync -e "ssh -o 'Batchmode = yes'" -r $FQDN:$TMPDIR/b$BUILDNR $TMPDIR/
- RESULT=$?
- if [ $RESULT -ne 0 ] ; then
- unregister_build
- handle_remote_error "when rsyncing remote build #$BUILDNR results from $NODE"
- fi
- fi
- ls -lR $TMPDIR
- ssh -o "Batchmode = yes" $FQDN "rm -r $TMPDIR"
- set -e
-}
-
-#
-# below is what controls the world
-#
-TIMEOUT=12 # maximum time in hours for a single build
-DATE=$(date -u +'%Y-%m-%d %H:%M')
-START=$(date +'%s')
-trap cleanup_all INT TERM EXIT
-
-#
-# determine mode
-#
-if [ "$1" = "" ] ; then
- MODE="master"
- TMPDIR=$(mktemp --tmpdir=/srv/reproducible-results -d -t alpinerb-build-XXXXXXXX) # where everything actually happens
- SOURCE_DATE_EPOCH=$(date +%s)
- cd $TMPDIR
-elif [ "$1" = "1" ] || [ "$1" = "2" ] ; then
- MODE="$1"
- REPOSITORY="$2"
- SRCPACKAGE="$3"
- TMPDIR="$4"
- SOURCE_DATE_EPOCH="$5"
- [ -d $TMPDIR ] || mkdir -p $TMPDIR
- cd $TMPDIR
- mkdir -p b$MODE/$SRCPACKAGE
- if [ "$MODE" = "1" ] ; then
- first_build
- else
- second_build
- fi
-
- # preserve results and delete build directory
- if [ -n "$(ls /tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)/*/*/*.apk)" ] ; then
- # copying is enough here, we delete after this if block anyway
- cp --preserve=timestamps -v /tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)/*/*/*.apk $TMPDIR/b$MODE/$SRCPACKAGE/
- else
- echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME didn't build a package!"
- # debug
- echo "ls $TMPDIR/b$MODE/$SRCPACKAGE/"
- ls -Rl
- fi
-
- sudo rm -rf --one-file-system /tmp/alpine-$SRCPACKAGE-$(basename $TMPDIR)
- echo "$(date -u) - build #$MODE for $SRCPACKAGE on $HOSTNAME done."
- exit 0
-fi
-
-#
-# main - only used in master-mode
-#
-delay_start # randomize start times
-# first, we need to choose a package from a repository…
-REPOSITORY=""
-SRCPACKAGE=""
-VERSION=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='alpine'")
-choose_package
-mkdir -p $BASE/alpine/$REPOSITORY/$SRCPACKAGE
-# build package twice
-mkdir b1 b2
-# currently there are two alpine build nodes… let's keep things simple
-N1="osuosl1-amd64"
-N2="osuosl2-amd64"
-# if random number between 0 and 99 is greater than 60…
-# prefer node $N1 (originally because one of the nodes was less loaded)
-if [ $((RANDOM % 100)) -gt 60 ] ; then
- NODE1=$N1
- NODE2=$N2
-else
- NODE1=$N2
- NODE2=$N1
-fi
-echo "============================================================================="
-echo "Initialising reproducibly build of ${SRCPACKAGE} in ${REPOSITORY} on ${ARCH} now."
-echo "1st build will be done on $NODE1."
-echo "2nd build will be done on $NODE2."
-echo "============================================================================="
-#
-# do 1st build
-#
-remote_build 1 ${NODE1}
-#
-# only do the 2nd build if the 1st produced results
-#
-if [ -n "$(ls $TMPDIR/b1/$SRCPACKAGE/*.apk 2>/dev/null|| true)" ] ; then
- remote_build 2 ${NODE2}
- cd $TMPDIR/b1/$SRCPACKAGE
- for ARTIFACT in *.apk ; do
- [ -f $ARTIFACT ] || continue
- echo "$(date -u) - comparing results now."
- if diff -q $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ; then
- echo "$(date -u) - YAY - $SRCPACKAGE/$ARTIFACT build reproducible in our test framework!"
- mkdir -p $BASE/alpine/$REPOSITORY/$SRCPACKAGE
-
- # XXX: alpine doesn't have BUILDINFO files yet
- #tar xJvf $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt
- #tar xJvf $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT .BUILDINFO && mv .BUILDINFO $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt
- touch $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt
- touch $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt
-
- (
- echo "<html><body><p>$SRCPACKAGE/$ARTIFACT build reproducible in our test framework:<br />"
- (cd $TMPDIR/b1/$SRCPACKAGE ; sha256sum $ARTIFACT)
- echo "<br />"
- (sha256sum $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b1.BUILDINFO.txt | cut -d " " -f1)
- echo " <a href=\"$ARTIFACT-b1.BUILDINFO.txt\">$ARTIFACT-b1.BUILDINFO.txt</a><br />"
- (sha256sum $BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT-b2.BUILDINFO.txt | cut -d " " -f1)
- echo " <a href=\"$ARTIFACT-b2.BUILDINFO.txt\">$ARTIFACT-b2.BUILDINFO.txt</a><br />"
- echo "</p></body>"
- ) > "$BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
- elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] && [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
- # run diffoscope on the results
- TIMEOUT="60m"
- DIFFOSCOPE="$(schroot --directory /tmp -c chroot:jenkins-reproducible-${DBDSUITE}-diffoscope diffoscope -- --version 2>&1)"
- echo "$(date -u) - Running $DIFFOSCOPE now..."
- call_diffoscope $SRCPACKAGE $ARTIFACT
- elif [ -f $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ] || [ -f $TMPDIR/b2/$SRCPACKAGE/$ARTIFACT ] ; then
- # one of the two builds failed... delete the other one
- ( rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT || rm $TMPDIR/b1/$SRCPACKAGE/$ARTIFACT ) 2>/dev/null
- else
- # some packages define the package version based on the build date
- # so our two builds end up with different package versions…
- echo "$(date -u) - something is fishy with $SRCPACKAGE/$ARTIFACT."
- ls $TMPDIR/b1/$SRCPACKAGE
- ls $TMPDIR/b2/$SRCPACKAGE
- ( echo "<html><body><p>$SRCPACKAGE/$ARTIFACT built in a strange unreproducible way:<br />"
- ls $TMPDIR/b1/$SRCPACKAGE
- echo "<br />"
- ls $TMPDIR/b2/$SRCPACKAGE
- echo "<br />"
- echo "Maybe $BUILD_URL has some more information why this happened."
- echo "</p></body>"
- ) > "$BASE/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html"
- fi
- # publish page
- if [ -f $TMPDIR/$SRCPACKAGE/$ARTIFACT.html ] ; then
- cp $TMPDIR/$SRCPACKAGE/$ARTIFACT.html $BASE/alpine/$REPOSITORY/$SRCPACKAGE/
- #irc_message alpine-reproducible "$REPRODUCIBLE_URL/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT.html - unreproducible"
- fi
- done
-else
- echo "$(date -u) - build1 didn't create a package, skipping build2!"
-fi
-# publish logs
-calculate_build_duration
-cd $TMPDIR/b1/$SRCPACKAGE
-cp build1.log $BASE/alpine/$REPOSITORY/$SRCPACKAGE/
-[ ! -f $TMPDIR/b2/$SRCPACKAGE/build2.log ] || cp $TMPDIR/b2/$SRCPACKAGE/build2.log $BASE/alpine/$REPOSITORY/$SRCPACKAGE/
-echo $DURATION > $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.build_duration || true
-# make pkg.build_duration the oldest of this build, so we can use it as reference later
-touch --date="@$START" $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.build_duration
-if [ -f $TMPDIR/b2/$SRCPACKAGE/build2.version ] ; then
- cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/alpine/$REPOSITORY/$SRCPACKAGE/
- cp $TMPDIR/b2/$SRCPACKAGE/build2.version $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.version
-elif [ -f build1.version ] ; then
- cp build1.version $BASE/alpine/$REPOSITORY/$SRCPACKAGE/
- cp $TMPDIR/b1/$SRCPACKAGE/build1.version $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.version
-else
- # this should not happen but does, so deal with it
- echo "$VERSION" > $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.version
-fi
-
-echo "$(date -u) - $REPRODUCIBLE_URL/alpine/$REPOSITORY/$SRCPACKAGE/ updated."
-# force update of HTML snipplet in reproducible_html_alpine.sh
-[ ! -f $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.state ] || rm $BASE/alpine/$REPOSITORY/$SRCPACKAGE/pkg.state
-HTML_BUFFER=''
-create_alpine_pkg_html
-update_pkg_in_db
-
-cd
-cleanup_all
-trap - INT TERM EXIT
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_common.sh
=====================================
@@ -64,11 +64,6 @@ mkdir -p "$DEBIAN_BASE"
# existing usertags in the Debian BTS
USERTAGS="toolchain infrastructure timestamps fileordering buildpath username hostname uname randomness buildinfo cpu signatures environment umask ftbfs locale usrmerge fixfilepath shell filesystem hashordering"
-# common settings for testing alpine
-ALPINE_REPOS="main community"
-ALPINE_PKGS=/srv/reproducible-results/alpine_pkgs
-ALPINE_BASE="$BASE/alpine"
-
# common settings for testing rpm based distros
RPM_BUILD_NODE=osuosl1-amd64
RPM_PKGS=/srv/reproducible-results/rpm_pkgs
@@ -398,18 +393,6 @@ write_page_intro() {
local PROJECTURL="https://github.com/freebsd/freebsd-src.git"
local BUILD_ENVIRONMENT=", which via ssh triggers a build on a FreeBSD system"
local BRANCH="main"
- elif [ "$1" = "alpine" ] ; then
- local PROJECTNAME="alpine"
- write_page " <p><em>Reproducible $PROJECTNAME</em> is an effort to apply this to $PROJECTNAME. Thus $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>."
- write_page " Please note that this is still at an early stage. Also there are more variations expected to be seen in the wild."
- write_page "Missing bits for <em>testing</em> alpine:<ul>"
- write_page " <li>cross references to <a href=\"https://tests.reproducible-builds.org/debian/index_issues.html\">Debian notes</a> - and having alpine specific notes.</li>"
- write_page "</ul></p>"
- write_page "<p>Missing bits for alpine:<ul>"
- write_page " <li>code needs to be written to compare the packages built twice here against newly built packages from the Official alpine repositories.</li>"
- write_page " <li>user tools, for users to verify all of this easily.</li>"
- write_page "</ul></p>"
- write_page "<p>If you want to help out or discuss reproducible builds in $PROJECTNAME, please join #alpine-reproducible on freenode.</p>"
elif [ "$1" = "fedora-23" ] ; then
local PROJECTNAME="Fedora 23"
write_page " <p><em>Reproducible $PROJECTNAME</em> is a (currently somewhat stalled) effort to apply this to $PROJECTNAME, which is rather obvious with 23… <br/> $PROJECTNAME packages are build twice, with a few variations added and then the resulting packages from the two builds are compared using <a href=\"https://tracker.debian.org/diffoscope\">diffoscope</a>. Please note that the toolchain is not varied at all as the rebuild happens on exactly the same system. More variations are expected to be seen in the wild.</p>"
@@ -1015,117 +998,3 @@ include_icon(){
echo " <img src=\"/userContent/static/$PNG.png\" alt=\"$ALT icon\" /> $TEXT" >> $HTML_BUFFER
}
-create_alpine_pkg_html() {
- local ALPINE_PKG_PATH=$ALPINE_BASE/$REPOSITORY/$SRCPACKAGE
- local HTML_BUFFER=$(mktemp -t alpinerb-html-XXXXXXXX)
- local buffer_message
- local STATE
-
- # clear files from previous builds
- cd "$ALPINE_PKG_PATH"
- for file in build1.log build2.log build1.version build2.version *BUILDINFO.txt *.html; do
- if [ -f $file ] && [ pkg.build_duration -nt $file ] ; then
- rm $file
- echo "$ALPINE_PKG_PATH/$file older than $ALPINE_PKG_PATH/pkg.build_duration, thus deleting it."
- fi
- done
-
- echo " <tr>" >> $HTML_BUFFER
- echo " <td>$REPOSITORY</td>" >> $HTML_BUFFER
- echo " <td>$SRCPACKAGE</td>" >> $HTML_BUFFER
- echo " <td>$VERSION</td>" >> $HTML_BUFFER
- echo " <td>" >> $HTML_BUFFER
- #
- #
- if [ -z "$(cd $ALPINE_PKG_PATH/ ; ls *.apk.html 2>/dev/null)" ] ; then
- if find_in_buildlogs '^warning: spurious network error (2 tries remaining): [60] SSL peer certificate or SSH remote key was not OK; class=Net'; then
- STATE=404_1
- REASON="download failed with SSL problem"
- elif find_in_buildlogs '^npm ERR! code CERT_HAS_EXPIRED'; then
- STATE=404_1
- REASON="download failed with SSL problem"
- elif find_in_buildlogs "^>>> ERROR: $SRCPACKAGE: build failed"; then
- STATE=FTBFS_0
- buffer_message='failed to build'
- elif find_in_buildlogs "^>>> ERROR: $SRCPACKAGE: check failed"; then
- STATE=FTBFS_1
- buffer_message='failed to build while running tests'
- else
- STATE=$(query_db "SELECT r.status FROM results AS r
- JOIN sources as s on s.id=r.package_id
- WHERE s.architecture='x86_64'
- AND s.name='$SRCPACKAGE'
- AND s.suite='alpine_$REPOSITORY';")
- if [ "$STATE" = "blacklisted" ] ; then
- buffer_message='blacklisted'
- else
- STATE=UNKNOWN
- buffer_message='probably failed to build from source, please investigate'
- fi
- fi
-
- # print build failures
- if [ "$STATE" = "UNKNOWN" ]; then
- echo " $buffer_message" >> $HTML_BUFFER
- else
- include_icon $STATE "$buffer_message"
- fi
- else
- local STATE=reproducible
- local SOME_GOOD=false
- for ARTIFACT in $(cd $ALPINE_PKG_PATH/ ; ls *.apk.html) ; do
- if [[ $ARTIFACT != *"$VERSION"* ]] ; then
- echo "deleting $ARTIFACT as version is not $VERSION"
- rm -f $ARTIFACT
- continue
- elif grep -q 'build reproducible in our test framework' $ALPINE_PKG_PATH/$ARTIFACT ; then
- SOME_GOOD=true
- STATE=reproducible
- include_icon $STATE "<a href=\"/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is reproducible in our current test framework<br />"
- else
- # change $STATE unless we have found .buildinfo differences already...
- if [ "$STATE" != "FTBR_0" ] ; then
- STATE=FTBR_1
- fi
- # this shouldn't happen, but (for now) it does, so lets mark them…
- EXTRA_REASON=""
- if grep -q 'class="source">.BUILDINFO' $ALPINE_PKG_PATH/$ARTIFACT ; then
- STATE=FTBR_0
- EXTRA_REASON=" with variations in .BUILDINFO"
- fi
- include_icon $STATE "<a href=\"/alpine/$REPOSITORY/$SRCPACKAGE/$ARTIFACT\">${ARTIFACT:0:-5}</a> is unreproducible$EXTRA_REASON<br />"
- fi
- done
- # we only count source packages…
- if [[ $STATE = FTBR_1 && $SOME_GOOD = true ]]; then
- STATE=FTBR_2
- fi
- fi
- echo " </td>" >> $HTML_BUFFER
- echo " <td>$DATE" >> $HTML_BUFFER
- local DURATION=$(cat $ALPINE_PKG_PATH/pkg.build_duration 2>/dev/null || true)
- if [ -n "$DURATION" ]; then
- local HOUR=$(echo "$DURATION/3600"|bc)
- local MIN=$(echo "($DURATION-$HOUR*3600)/60"|bc)
- local SEC=$(echo "$DURATION-$HOUR*3600-$MIN*60"|bc)
- BUILD_DURATION="<br />${HOUR}h:${MIN}m:${SEC}s"
- else
- BUILD_DURATION=" "
- fi
- echo " $BUILD_DURATION</td>" >> $HTML_BUFFER
-
- echo " <td>" >> $HTML_BUFFER
- for LOG in build1.log build2.log ; do
- if [ -f $ALPINE_PKG_PATH/$LOG ] ; then
- if [ "$LOG" = "build2.log" ] ; then
- echo " <br />" >> $HTML_BUFFER
- fi
- echo " <a href=\"/alpine/$REPOSITORY/$SRCPACKAGE/$LOG\">$LOG</a> ($(get_filesize "${ALPINE_PKG_PATH}/${LOG}"))" >> $HTML_BUFFER
- fi
- done
- echo " </td>" >> $HTML_BUFFER
- echo " </tr>" >> $HTML_BUFFER
- mv $HTML_BUFFER $ALPINE_PKG_PATH/pkg.html
- chmod 644 $ALPINE_PKG_PATH/pkg.html
- echo $STATE > $ALPINE_PKG_PATH/pkg.state
-}
=====================================
bin/reproducible_html_alpine.sh deleted
=====================================
@@ -1,493 +0,0 @@
-#!/bin/bash
-
-# Copyright 2014-2023 Holger Levsen <holger at layer-acht.org>
-# 2015 anthraxx <levente at leventepolyak.net>
-# 2019 kpcyrd <git at rxv.cc>
-# released under the GPLv2
-
-DEBUG=false
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-#
-#
-# reproducible_html_alpine.sh can be called in two ways:
-# - without params, then it will build all the index and dashboard pages
-# - with exactly two params, $REPOSITORY and $SRCPACKAGE, in which case that packages html page will be created
-#
-
-
-#
-# helper functions
-#
-get_state_from_counter() {
- local counter=$1
- case $counter in
- 0) STATE=reproducible ;;
- 1) STATE=FTBR ;;
- 2) STATE=FTBFS ;;
- 3) STATE=DEPWAIT ;;
- 4) STATE=404 ;;
- 5) STATE=blacklisted ;;
- 6) STATE=UNKNOWN ;;
- esac
-}
-
-
-include_pkg_html_in_page(){
- cat $ALPINE_BASE/$REPOSITORY/$SRCPACKAGE/pkg.html >> $PAGE 2>/dev/null || true
-}
-
-include_pkg_table_header_in_page(){
- write_page " <table><tr><th>repository</th><th>source package</th><th>version</th><th>test result</th><th>test date<br />test duration</th><th>1st build log<br />2nd build log</th></tr>"
-}
-
-repostats(){
- #
- # gather data
- # write csv file for $REPOSITORY
- # write $HTML_REPOSTATS
- #
- for REPOSITORY in $ALPINE_REPOS ; do
- echo "$(date -u) - starting to analyse build results for '$REPOSITORY'."
- # prepare stats per repository
- SUITE="alpine_$REPOSITORY"
- TOTAL=$(query_db "SELECT count(*) FROM sources AS s WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
- TESTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE';")
- NR_GOOD=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='reproducible';")
- NR_FTBR=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBR_%';")
- NR_FTBFS=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'FTBFS_%';")
- NR_DEPWAIT=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'DEPWAIT_%';")
- NR_404=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE '404_%';")
- NR_BLACKLISTED=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status='blacklisted';")
- NR_UNKNOWN=$(query_db "SELECT count(*) FROM sources AS s JOIN results AS r ON s.id=r.package_id WHERE s.distribution=$DISTROID AND s.architecture='x86_64' AND s.suite='$SUITE' AND r.status LIKE 'UNKNOWN_%';")
- NR_UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s WHERE s.architecture='x86_64' AND s.distribution=$DISTROID AND s.suite='$SUITE' AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $NR_UNTESTED -ne 0 ] ; then
- let NR_UNKNOWN=$NR_UNKNOWN+$NR_UNTESTED
- fi
- PERCENT_TOTAL=$(echo "scale=1 ; ($TESTED*100/$TOTAL)" | bc)
- if [ $(echo $PERCENT_TOTAL/1|bc) -lt 99 ] ; then
- NR_TESTED="$TESTED <span style=\"font-size:0.8em;\">(tested $PERCENT_TOTAL% of $TOTAL)</span>"
- else
- NR_TESTED=$TESTED
- fi
- echo " <tr>" >> $HTML_REPOSTATS
- echo " <td><a href='/alpine/$REPOSITORY.html'>$REPOSITORY</a></td><td><a href='/alpine/$REPOSITORY.html'>$NR_TESTED</a></td>" >> $HTML_REPOSTATS
- counter=0
- for i in $NR_GOOD $NR_FTBR $NR_FTBFS $NR_DEPWAIT $NR_404 $NR_BLACKLISTED $NR_UNKNOWN ; do
- get_state_from_counter $counter
- let counter+=1
- PERCENT_i=$(echo "scale=1 ; ($i*100/$TESTED)" | bc)
- if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
- echo " <td><a href='/alpine/state_${REPOSITORY}_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
- else
- echo " <td><a href='/alpine/state_${REPOSITORY}_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
- fi
- done
- echo " </tr>" >> $HTML_REPOSTATS
- if [ ! -f $ALPINE_BASE/$REPOSITORY.csv ] ; then
- echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ALPINE_BASE/$REPOSITORY.csv
- fi
- if ! grep -q $YESTERDAY $ALPINE_BASE/$REPOSITORY.csv ; then
- let REAL_UNKNOWN=$TOTAL-$NR_GOOD-$NR_FTBR-$NR_FTBFS-$NR_DEPWAIT-$NR_404 || true
- echo $YESTERDAY,$NR_GOOD,$NR_FTBR,$NR_FTBFS,$NR_DEPWAIT,$NR_404,$REAL_UNKNOWN >> $ALPINE_BASE/$REPOSITORY.csv
- fi
- IMAGE=$ALPINE_BASE/$REPOSITORY.png
- if [ ! -f $IMAGE ] || [ $ALPINE_BASE/$REPOSITORY.csv -nt $IMAGE ] ; then
- echo "Updating $IMAGE..."
- /srv/jenkins/bin/make_graph.py $ALPINE_BASE/$REPOSITORY.csv $IMAGE 6 "Reproducibility status for alpine packages in $REPOSITORY" "Amount (total)" $WIDTH $HEIGHT
- fi
- #
- # prepare ALPINE totals
- #
- set +e
- let ALPINE_TOTAL+=$TOTAL
- let ALPINE_TESTED+=$TESTED
- let ALPINE_NR_FTBFS+=$NR_FTBFS
- let ALPINE_NR_FTBR+=$NR_FTBR
- let ALPINE_NR_DEPWAIT+=$NR_DEPWAIT
- let ALPINE_NR_404+=$NR_404
- let ALPINE_NR_GOOD+=$NR_GOOD
- let ALPINE_NR_BLACKLISTED+=$NR_BLACKLISTED
- let ALPINE_NR_UNKNOWN+=$NR_UNKNOWN
- set -e
- done
- #
- # prepare stats per repository
- #
- ALPINE_PERCENT_TOTAL=$(echo "scale=1 ; ($ALPINE_TESTED*100/$ALPINE_TOTAL)" | bc)
- if [ $(echo $ALPINE_PERCENT_TOTAL/1|bc) -lt 99 ] ; then
- NR_TESTED="$ALPINE_TESTED <span style=\"font-size:0.8em;\">(tested $ALPINE_PERCENT_TOTAL% of $ALPINE_TOTAL)</span>"
- else
- NR_TESTED=$ALPINE_TESTED
- fi
- echo " <tr>" >> $HTML_REPOSTATS
- echo " <td><b>all combined</b></td><td>$NR_TESTED</td>" >> $HTML_REPOSTATS
- counter=0
- for i in $ALPINE_NR_GOOD $ALPINE_NR_FTBR $ALPINE_NR_FTBFS $ALPINE_NR_DEPWAIT $ALPINE_NR_404 $ALPINE_NR_BLACKLISTED $ALPINE_NR_UNKNOWN ; do
- get_state_from_counter $counter
- let counter+=1
- PERCENT_i=$(echo "scale=1 ; ($i*100/$ALPINE_TESTED)" | bc)
- if [ "$PERCENT_i" != "0" ] || [ "$i" != "0" ] ; then
- echo " <td><a href='/alpine/state_$STATE.html'>$i ($PERCENT_i%)</a></td>" >> $HTML_REPOSTATS
- else
- echo " <td><a href='/alpine/state_$STATE.html'>$i</a></td>" >> $HTML_REPOSTATS
- fi
- done
- echo " </tr>" >> $HTML_REPOSTATS
- #
- # write csv file for totals
- #
- if [ ! -f $ALPINE_BASE/alpine.csv ] ; then
- echo '; date, reproducible, unreproducible, ftbfs, depwait, download problems, untested' > $ALPINE_BASE/alpine.csv
- fi
- if ! grep -q $YESTERDAY $ALPINE_BASE/alpine.csv ; then
- let ALPINE_REAL_UNKNOWN=$ALPINE_TOTAL-$ALPINE_NR_GOOD-$ALPINE_NR_FTBR-$ALPINE_NR_FTBFS-$ALPINE_NR_DEPWAIT-$ALPINE_NR_404 || true
- echo $YESTERDAY,$ALPINE_NR_GOOD,$ALPINE_NR_FTBR,$ALPINE_NR_FTBFS,$ALPINE_NR_DEPWAIT,$ALPINE_NR_404,$ALPINE_REAL_UNKNOWN >> $ALPINE_BASE/alpine.csv
- fi
- IMAGE=$ALPINE_BASE/alpine.png
- if [ ! -f $IMAGE ] || [ $ALPINE_BASE/alpine.csv -nt $IMAGE ] ; then
- echo "Updating $IMAGE..."
- /srv/jenkins/bin/make_graph.py $ALPINE_BASE/alpine.csv $IMAGE 6 "Reproducibility status for all tested alpine packages" "Amount (total)" $WIDTH $HEIGHT
- irc_message alpine-reproducible "Daily graphs on $REPRODUCIBLE_URL/alpine/ updated, $(echo "scale=1 ; ($ALPINE_NR_GOOD*100/$ALPINE_TESTED)" | bc)% reproducible packages in our current test framework."
- fi
-}
-
-alpine_page_header(){
- echo "$(date -u) - starting to write $PAGE"
- cat > $PAGE <<- EOF
- <!DOCTYPE html>
- <html lang="en-US">
- <head>
- <meta charset="UTF-8">
- <title>$TITLE</title>
- <link rel='stylesheet' href='global.css' type='text/css' media='all' />
- </head>
- <body>
- <div id="logo">
- <img src="alpinelinux-logo.svg" alt="">
- </div>
- <div class="content">
- <h1><a href='/alpine/'>Reproducible alpine</a>?!</h1>
- <div class="page-content">
-
- EOF
-}
-
-alpine_page_footer(){
- write_page "</div></div>"
- write_page_footer 'alpine'
- publish_page alpine
-}
-
-alpine_repostats_table(){
- write_page " <table><tr><th>repository</th><th>all source packages</th>"
- write_page " <th><a href='/alpine/state_reproducible.html'>reproducible</a></th>"
- write_page " <th><a href='/alpine/state_FTBR.html'>unreproducible</a></th>"
- write_page " <th><a href='/alpine/state_FTBFS.html'>failing to build</a></th>"
- write_page " <th><a href='/alpine/state_DEPWAIT.html'>in depwait state</a></th>"
- write_page " <th><a href='/alpine/state_404.html'>download problems</a></th>"
- write_page " <th><a href='/alpine/state_blacklisted.html'>blacklisted</a></th>"
- write_page " <th><a href='/alpine/state_UNKNOWN.html'>unknown state</a></th></tr>"
- cat $HTML_REPOSTATS >> $PAGE
- write_page " </table>"
- write_page " <p>("
- write_page " <a href='/alpine/recent_builds.html'>recent builds</a>,"
- write_page " <a href='/alpine/scheduled.html'>currently scheduled</a>"
- write_page " )</p>"
-}
-
-dashboard_page(){
- PAGE=alpine.html
- TITLE="Reproducible alpine ?!"
- alpine_page_header
- write_page_intro 'alpine'
- alpine_repostats_table
- # include graphs
- write_page '<p style="clear:both;">'
- for REPOSITORY in $ALPINE_REPOS ; do
- write_page "<a href=\"/alpine/$REPOSITORY.png\"><img src=\"/alpine/$REPOSITORY.png\" class=\"overview\" alt=\"$REPOSITORY stats\"></a>"
- done
- write_page '</p><p style="clear:both;"><center>'
- write_page "<a href=\"/alpine/alpine.png\"><img src=\"/alpine/alpine.png\" alt=\"total alpine stats\"></a></p>"
- write_variation_table 'alpine'
- alpine_page_footer
-}
-
-repository_pages(){
- for REPOSITORY in $ALPINE_REPOS ; do
- PAGE=$REPOSITORY.html
- TITLE="Reproducible alpine $REPOSITORY ?!"
- alpine_page_header
- alpine_repostats_table
- SUITE="alpine_$REPOSITORY"
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE';")
- write_page "<h2>$TESTED packages in repository $REPOSITORY</h2>"
- include_pkg_table_header_in_page
- REPO_PKGS=$(query_db "SELECT s.name FROM sources
- AS s JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in $REPO_PKGS ; do
- include_pkg_html_in_page
- done
- write_page " </table>"
- alpine_page_footer
- done
-}
-
-state_pages(){
- for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
- PAGE=state_$STATE.html
- TITLE="Reproducible alpine, packages in state $STATE"
- alpine_page_header
- alpine_repostats_table
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.status LIKE '$STATE%';")
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $UNTESTED -ne 0 ] ; then
- let TESTED=$TESTED+$UNTESTED
- fi
- fi
- write_page "<h2>$TESTED packages in $STATE state</h2>"
- include_pkg_table_header_in_page
- for REPOSITORY in $ALPINE_REPOS ; do
- SUITE="alpine_$REPOSITORY"
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )
- ORDER BY s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- fi
- done
- write_page " </table>"
- alpine_page_footer
- done
-}
-
-repository_state_pages(){
- for REPOSITORY in $ALPINE_REPOS ; do
- SUITE="alpine_$REPOSITORY"
- for STATE in FTBFS FTBR DEPWAIT 404 reproducible blacklisted UNKNOWN ; do
- PAGE=state_${REPOSITORY}_$STATE.html
- TITLE="Reproducible alpine, packages in $REPOSITORY in state $STATE"
- alpine_page_header
- alpine_repostats_table
- TESTED=$(query_db "SELECT count(*) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%';")
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- UNTESTED=$(query_db "SELECT count(s.name) FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )")
- if [ $UNTESTED -ne 0 ] ; then
- let TESTED=$TESTED+$UNTESTED
- fi
- fi
- write_page "<h2>$TESTED packages in $REPOSITORY in $STATE state</h2>"
- include_pkg_table_header_in_page
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND r.status LIKE '$STATE%'
- ORDER BY r.status,s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- if [ "$STATE" = "UNKNOWN" ] ; then
- # untested packages are also state UNKNOWN...
- STATE_PKGS=$(query_db "SELECT s.name FROM sources AS s
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND s.suite='$SUITE'
- AND NOT exists (SELECT FROM results WHERE package_id = s.id )
- ORDER BY s.name")
- for SRCPACKAGE in ${STATE_PKGS} ; do
- include_pkg_html_in_page
- done
- fi
- write_page " </table>"
- alpine_page_footer
- done
- done
-}
-
-recent_builds_page(){
- PAGE=recent_builds.html
- TITLE="Reproducible alpine, builds in the last 24h"
- alpine_page_header
- alpine_repostats_table
- MAXDATE="$(date -u +'%Y-%m-%d %H:%M' -d '24 hours ago')"
- RECENT=$(query_db "SELECT count(s.name) FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'")
- write_page "<h2>$RECENT builds of alpine packages in the last 24h</h2>"
- include_pkg_table_header_in_page
- STATE_PKGS=$(query_db "SELECT s.name, s.suite FROM sources AS s
- JOIN results AS r
- ON s.id=r.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND r.build_date > '$MAXDATE'
- ORDER BY r.build_date
- DESC")
- for LINE in ${STATE_PKGS} ; do
- SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
- REPOSITORY=$(echo "$LINE" | cut -d "|" -f2 | sed 's#alpine_##')
- include_pkg_html_in_page
- done
- write_page " </table>"
- alpine_page_footer
-}
-
-currently_scheduled_page(){
- PAGE=scheduled.html
- TITLE="Reproducible alpine, packages currently scheduled"
- alpine_page_header
- alpine_repostats_table
- TESTED=$(query_db "SELECT count(*)
- FROM sources AS s
- JOIN schedule AS sch
- ON s.id=sch.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND sch.date_build_started IS NULL")
- write_page "<h2>Currently $TESTED scheduled builds of alpine packages</h2>"
- write_page " <table><tr><th>source package</th><th>repository</th><th>version</th><th>scheduled</th></tr>"
- STATE_PKGS=$(query_db "SELECT s.name, s.suite, s.version, sch.date_scheduled
- FROM sources AS s
- JOIN schedule AS sch
- ON s.id=sch.package_id
- WHERE s.distribution=$DISTROID
- AND s.architecture='x86_64'
- AND sch.date_build_started IS NULL
- ORDER BY sch.date_scheduled, s.name")
- OIFS=$IFS
- IFS=$'\012'
- for LINE in ${STATE_PKGS} ; do
- SRCPACKAGE=$(echo "$LINE" | cut -d "|" -f1)
- REPOSITORY=$(echo "$LINE" | cut -d "|" -f2 | sed 's#alpine_##')
- VERSION=$(echo "$LINE" | cut -d "|" -f3)
- SCH_DATE=$(echo "$LINE" | cut -d "|" -f4-)
- write_page " <tr><td>$SRCPACKAGE</td><td>$REPOSITORY</td><td>$VERSION</td><td>$SCH_DATE</td></tr>"
- done
- IFS=$OIFS
- write_page " </table>"
- alpine_page_footer
-}
-
-#
-# main
-#
-echo "$(date -u) - starting."
-YESTERDAY=$(date '+%Y-%m-%d' -d "-1 day")
-PAGE=""
-TITLE=""
-STATE=""
-REPOSITORY=""
-SRCPACKAGE=""
-DISTROID=$(query_db "SELECT id FROM distributions WHERE name='alpine'")
-
-if [ -z "$1" ] ; then
- MEMBERS_FTBFS="0 1 2 3 4"
- MEMBERS_DEPWAIT="0 1 2"
- MEMBERS_404="0 1 2 3 4 5 6 7 8 9 A B C"
- MEMBERS_FTBR="0 1 2"
- HTML_REPOSTATS=$(mktemp -t alpinerb-html-XXXXXXXX)
- ALPINE_TOTAL=0
- ALPINE_TESTED=0
- ALPINE_NR_FTBFS=0
- ALPINE_NR_FTBR=0
- ALPINE_NR_DEPWAIT=0
- ALPINE_NR_404=0
- ALPINE_NR_GOOD=0
- ALPINE_NR_BLACKLISTED=0
- ALPINE_NR_UNKNOWN=0
- WIDTH=1920
- HEIGHT=960
- # variables related to the stats we update
- # FIELDS[0]="datum, reproducible, FTBR, FTBFS, other, untested" # for this alpine still uses a .csv file...
- FIELDS[1]="datum"
- for i in reproducible FTBR FTBFS other ; do
- for j in $SUITES ; do
- FIELDS[1]="${FIELDS[1]}, ${i}_${j}"
- done
- done
- FIELDS[2]="datum, oldest"
-
- repostats
- dashboard_page
- currently_scheduled_page
- recent_builds_page
- repository_pages
- state_pages
- repository_state_pages
-
- rm $HTML_REPOSTATS > /dev/null
-elif [ -z "$2" ] ; then
- echo "$(date -u) - $0 needs two params or none, exiting."
- # add code here to also except core, extra, multilib or community...
- exit 1
-else
- REPOSITORY=$1
- SRCPACKAGE=$2
- if [ ! -d $ALPINE_BASE/$REPOSITORY/$SRCPACKAGE ] ; then
- echo "$(date -u) - $ALPINE_BASE/$REPOSITORY/$SRCPACKAGE does not exist, exiting."
- exit 1
- fi
- HTML_BUFFER=''
- create_alpine_pkg_html
-fi
-echo "$(date -u) - all done."
-
-# vim: set sw=0 noet :
=====================================
bin/reproducible_html_nodes_info.sh
=====================================
@@ -274,8 +274,7 @@ build_job_health_page() {
"coreboot"
"(openwrt)"
"(le_netbsd|le_freebsd)"
- "fedora"
- "alpine")
+ "fedora")
echo "$(date -u) - starting to write $PAGE page."
write_page_header $VIEW "Job health overview"
write_page "<p style=\"clear:both;\">"
@@ -284,14 +283,12 @@ build_job_health_page() {
write_page "<tr>"
for JOB in $(cd ~/jobs ; ls -1d reproducible_* | grep -E "${FILTER[$CATEGORY]}" | cut -d '_' -f2- | sort ) ; do
SHORTNAME="$(echo $JOB \
- | sed 's#alpine_##' \
| sed 's#builder_fedora#builder#' \
| sed 's#_x86_64##' \
| sed 's#_from_git_master#_git#' \
| sed 's#setup_schroot_##' \
| sed 's#setup_mock_fedora-##' \
| sed 's#create_##' \
- | sed 's#html_alpine#html#' \
| sed 's#html_##' \
| sed 's#builds_##' \
| sed 's#_diffoscope_amd64##' \
=====================================
bin/reproducible_maintenance.sh
=====================================
@@ -637,23 +637,16 @@ for s in $SUITES ; do
done
set -e
-# for alpine
+# for alpine (can be dropped in October 2025)
set +e
case $HOSTNAME in
osuosl1-amd64|osuosl2-amd64|jenkins)
echo "$(date -u) - updating alpine schroot now."
if $(schroot -l|grep -q alpine) ; then
- schroot --directory /tmp -c source:jenkins-reproducible-alpine -u root -- apk update
- schroot --directory /tmp -c source:jenkins-reproducible-alpine -u root -- apk upgrade
- RESULT=$?
- if [ $RESULT -eq 1 ] ; then
- echo "Warning: failed to update alpine schroot."
- DIRTY=true
- else
- echo "$(date -u) - updating alpine schroot done."
- fi
+ echo "Warning: alpine schroot found, how strange."
+ DIRTY=true
else
- echo "No alpine schroot found, how strange."
+ echo "No alpine schroot found, good."
fi
;;
*) ;;
=====================================
bin/reproducible_setup_alpine_schroot.sh deleted
=====================================
@@ -1,170 +0,0 @@
-#!/bin/bash
-
-# Copyright 2015-2023 Holger Levsen <holger at layer-acht.org>
-# 2017-2019 kpcyrd <git at rxv.cc>
-# 2017 Mattia Rizzolo <mattia at debian.org>
-# Juliana Oliveira Rodrigues <juliana.orod at gmail.com>
-# released under the GPLv2
-
-#
-# downloads an alpine bootstrap chroot archive, then turns it into a schroot,
-# then configures abuild.
-#
-
-set -e
-
-DEBUG=true
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-# define alpine mirror to be used
-ALPINE_MIRROR=http://dl-cdn.alpinelinux.org/alpine/edge/releases/x86_64
-
-bootstrap() {
- echo "$(date -u) - trying to determine latest alpine edge minirootfs"
- LATEST_MINIROOT=$(curl -sSf "$ALPINE_MIRROR/" | grep -oE 'alpine-minirootfs-[0-9]+-x86_64.tar.gz' | sort | tail -1)
-
- if [ -z $LATEST_MINIROOT ]; then
- echo "Failed to find latest minirootfs, aborting."
- exit 1
- fi
-
- rm -f "./$LATEST_MINIROOT"
-
- echo "$(date -u) - downloading alpine minirootfs"
- curl -fO "$ALPINE_MIRROR/$LATEST_MINIROOT"
-
- echo "$(date -u) - extracting alpine minirootfs"
- mkdir -p "$SCHROOT_BASE/$TARGET.new"
- sudo tar xzf "./$LATEST_MINIROOT" -C "$SCHROOT_BASE/$TARGET.new"
-
- if [ -d "$SCHROOT_BASE/$TARGET" ]; then
- mv "$SCHROOT_BASE/$TARGET" "$SCHROOT_BASE/$TARGET.old"
- sudo rm -rf --one-file-system "$SCHROOT_BASE/$TARGET.old"
- fi
- mv "$SCHROOT_BASE/$TARGET.new" "$SCHROOT_BASE/$TARGET"
-
- rm -f "./$LATEST_MINIROOT"
-
- # write the schroot config
- echo "$(date -u ) - writing schroot configuration for $TARGET."
- sudo tee /etc/schroot/chroot.d/jenkins-"$TARGET" <<-__END__
- [jenkins-$TARGET]
- description=Jenkins schroot $TARGET
- directory=$SCHROOT_BASE/$TARGET
- type=directory
- root-users=jenkins
- source-root-users=jenkins
- union-type=overlay
- __END__
-}
-
-cleanup() {
- if [ -d "$SCHROOT_TARGET" ]; then
- rm -rf --one-file-system "$SCHROOT_TARGET" || ( echo "Warning: $SCHROOT_TARGET could not be fully removed on forced cleanup." ; ls "$SCHROOT_TARGET" -la )
- fi
- rm -f "$TMPLOG"
- exit 1
-}
-
-trap cleanup INT TERM EXIT
-TARGET=reproducible-alpine
-bootstrap
-trap - INT TERM EXIT
-
-ROOTCMD="schroot --directory /tmp -c source:jenkins-reproducible-alpine -u root --"
-USERCMD="schroot --directory /tmp -c source:jenkins-reproducible-alpine -u jenkins --"
-
-echo "============================================================================="
-echo "Setting up schroot $TARGET on $HOSTNAME"...
-echo "============================================================================="
-
-# fix permissions
-$ROOTCMD chmod 0755 /
-
-# configure proxy everywhere
-sudo tee "$SCHROOT_BASE/$TARGET/etc/profile.d/proxy.sh" <<-__END__
- export http_proxy=$http_proxy
- export https_proxy=$http_proxy
- export ftp_proxy=$http_proxy
- export HTTP_PROXY=$http_proxy
- export HTTPS_PROXY=$http_proxy
- export FTP_PROXY=$http_proxy
- export no_proxy="localhost,127.0.0.1"
- __END__
-
-# install sdk
-$ROOTCMD apk add alpine-sdk lua-aports gnupg bash
-
-# configure sudo
-echo 'jenkins ALL= NOPASSWD: /sbin/apk *' | $ROOTCMD tee -a /etc/sudoers
-
-# configure jenkins user
-$ROOTCMD mkdir /var/lib/jenkins
-$ROOTCMD chown -R jenkins:jenkins /var/lib/jenkins
-if [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
- # workaround for certificates that aren't valid in the future.
- # we might need to replace this with a mitm proxy in the future
- echo "insecure" | tee -a "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.curlrc"
- GIT_OPTIONS='GIT_SSL_NO_VERIFY=1'
-fi
-$USERCMD gpg --check-trustdb # first run will create ~/.gnupg/gpg.conf
-echo "keyserver-options auto-key-retrieve" | tee -a "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.gnupg/gpg.conf"
-
-mkdir -vp "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.abuild"
-tee "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.abuild/abuild.conf" <<-__END__
-PACKAGER_PRIVKEY="/var/lib/jenkins/.abuild/build-5de527c8.rsa"
-__END__
-# ci keys, do not use for anything important
-tee "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.abuild/build-5de527c8.rsa" <<-__END__
------BEGIN RSA PRIVATE KEY-----
-MIIEogIBAAKCAQEAq95i4RZ4GfLLne18xmPSwvE9vcoIISAIbJAsjz9XR+d552TS
-u7rJF3JDIbWqQ53u4P4dvPHxK3wO657KtfBjM1lf8KJbdaE993AYGdbbVbTnak5h
-HStPiJ7z1t6vvbs0EWqcAOWlh1L5lhMbQFLcnsPXXnZ2aeaX9Dlz6VGJFT8Pf0MV
-1ABDn8po84hl0NUFIj1cjsUDibkqI6HF7hiUinUzlo8jO26PFcVJyEPNj0Oz8kuY
-686K/NR6mfvKXVYwXM/mskK6XD3YjKwblDzQQxUzJuRNWSonBQ32FDQmcVfZOoTI
-4mQMDkqKt6dhxiqycX1/R3m9LsE8IlIhoQ99wQIDAQABAoIBADt7mklA14xThblA
-6oBXKCikCbRX6fxc881vEJz7VR/js0Msl+q1OMfOmgFeuHDyhiyEhpJQQiHEq/1M
-VegmLI8nDZdg+bp6ddHfj9fRjtPY6obWXbIUvVARg18Ib1aBJgIpHZkJ4gI163/P
-WQ0oIIcqTK263jzEvC6ge8dymrkpKqCtVNnpI+ToAZ//Ni0QKGZ6tSSC7lg1jENf
-ghnjADUNcQivVjBqbNBd2rR5oJ5NSspUX78spYtf2mFB0wgXVsAvfpGBe6txp5sv
-Pp9xYqozpZ7Wf2Z1cIYw7laIt53HE1VCziSnwivYFTdpplJFIg3Hd65fQ6MfESqZ
-6yW4l4kCgYEA3yy+sZlTTfIVd3m0pdVgyHXf4Y/otonRc6WcclHYK+3ez4F+dOl8
-DZnZBhhJPFHlZBm4J7wWnUY4ZdgAydzKp3YVegZdm43Xhg+aI3P8nfJykAJ01EzI
-5Zd3Z0YWnqdxB1HyP0wYf315tyGM5yMD/1v6ayM+WpAH+LW+wo3CY2MCgYEAxSXM
-8kRQev5zkOLZhtMLb9EC2rDwFm2LEW2ocd1uDNRS9usB7lE30EE5xJNI5+AkoCo1
-j3B0CyXyRHbmhmfp7OempF7JAPROrS1gxg0naQORU2i9aiOwNGhbenkSMnUQi3of
-D5YX0zk60TMFWo/hSLW9hrN97iOgF12NajsCjYsCgYAw7zy48GeltakjU1pa6liY
-W9BFQyrBq6JzeyK8plmB+Fxcn4Y82F1NFijR/00/nq1vr3wDqmhC//ypyB0UJgeB
-hJDc+rxXuVhCmvUvROVlNJ4OGZvIWTXLsdSKnoGjNA/CjSNS4bqVacvgbcjZfYII
-4gAcsdOgQ+ibji5PtomjBQKBgDy3n58dmwvGQiFlPElhxivx20cvJ0JBCoubkj0/
-TR12ZvbU+gtDyETDUd9Q3StMxPrvBP/gSl0EmtCrLeRHLKxhy9jjuFQq6fA8AYn9
-kx2sk510rKF7zFDXsxTNJOWVWDscqWRLfZr4DT1Q0V1K4r9Z+bz6mtY08qE/lsYY
-1nhxAoGATFF1Ng7AQMcytiT1nSQjB/A/3C5hzCula7J1ErL+wc0zOS4oenmL2ESI
-IKDt71BfGgXZHy1AKMv+0x1Wk2hOXC2fSxG46lIPD0msR2207+c7ZgVuyARmSDO3
-Y/5yyssrctGJ8bNs+++uUhr8yG9CXBG9uSG3gvtCXxOgwg43lTk=
------END RSA PRIVATE KEY-----
-__END__
-tee "$SCHROOT_BASE/$TARGET/var/lib/jenkins/.abuild/build-5de527c8.rsa.pub" <<-__END__
------BEGIN PUBLIC KEY-----
-MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq95i4RZ4GfLLne18xmPS
-wvE9vcoIISAIbJAsjz9XR+d552TSu7rJF3JDIbWqQ53u4P4dvPHxK3wO657KtfBj
-M1lf8KJbdaE993AYGdbbVbTnak5hHStPiJ7z1t6vvbs0EWqcAOWlh1L5lhMbQFLc
-nsPXXnZ2aeaX9Dlz6VGJFT8Pf0MV1ABDn8po84hl0NUFIj1cjsUDibkqI6HF7hiU
-inUzlo8jO26PFcVJyEPNj0Oz8kuY686K/NR6mfvKXVYwXM/mskK6XD3YjKwblDzQ
-QxUzJuRNWSonBQ32FDQmcVfZOoTI4mQMDkqKt6dhxiqycX1/R3m9LsE8IlIhoQ99
-wQIDAQAB
------END PUBLIC KEY-----
-__END__
-
-echo "$(date -u) - cloning aports repo"
-$USERCMD sh -c "$GIT_OPTIONS git clone --depth=1 https://git.alpinelinux.org/aports.git /var/lib/jenkins/aports"
-
-echo "============================================================================="
-echo "schroot $TARGET set up successfully in $SCHROOT_BASE/$TARGET - exiting now."
-echo "============================================================================="
-
-# vim: set sw=0 noet :
=====================================
hosts/common/etc/sudoers.d/jenkins
=====================================
@@ -16,7 +16,6 @@ jenkins ALL= \
/bin/chmod +x /schroots/*, \
/bin/chmod +x /chroots/*, \
/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
- /bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
/bin/tar --mtime=* --clamp-mtime -C /srv/workspace/* -cf *, \
/usr/sbin/chroot /schroots/*, \
/usr/sbin/chroot /chroots/*, \
=====================================
hosts/common/usr/share/munin/plugins/jenkins_reproducible_builds
=====================================
@@ -48,7 +48,7 @@ fi
for PREFIX in $JOB_PREFIXES ; do
# detects Debian and OpenWrt builds
- # todo: Debian live, coreboot, netbsd, freebsd, (alpine)
+ # todo: Debian live, coreboot, netbsd, freebsd
if [ "$PREFIX" = "first" ] ; then
NR=$(pgrep -fc "bin/bash /srv/jenkins/bin/(reproducible_build|reproducible_openwrt).sh (|node openwrt_build b)1")
else
=====================================
hosts/jenkins/etc/sudoers.d/jenkins
=====================================
@@ -13,7 +13,6 @@ jenkins ALL= \
/bin/chmod -v 644 /var/lib/jenkins/userContent/*, \
/bin/chmod -v 664 /var/lib/jenkins/userContent/*, \
/bin/tar -c --exclude ./sys/\* --exclude ./proc/\* -f /schroots/*, \
- /bin/tar xzf ./alpine-minirootfs-*.tar.gz -C /schroots/reproducible-alpine.new, \
/usr/sbin/chroot /schroots/*, \
/usr/sbin/chroot /chroots/*, \
/usr/sbin/chroot /media/*, \
=====================================
job-cfg/reproducible.yaml
=====================================
@@ -611,14 +611,6 @@
my_description: 'Generate https://reproducible.debian.net/userContent/reproducible.json and https://reproducible.debian.net/userContent/reproducible-tracker.json'
my_timed: '1 H/2 * * *'
my_shellext: ".py"
- - 'html_alpine':
- my_description: 'Build webpages for alpine reproducibility tests'
- my_timed: '55 H/2 * * *'
- my_disabled: true
- - 'alpine_scheduler':
- my_description: 'Scheduler for alpine reproducibility tests'
- my_timed: '0 H/4 * * *'
- my_disabled: true
- 'html_fedora-23_x86_64':
my_description: 'Build a simple webpage for Fedora 23 reproducibility tests'
my_timed: ''
@@ -739,21 +731,6 @@
my_timed: '42 23 * * 4'
my_recipients: 'jenkins+reproducible-changes qa-jenkins-scm at lists.alioth.debian.org'
my_shell: '/srv/jenkins/bin/reproducible_freebsd.sh'
- - 'setup_schroot_alpine_jenkins':
- my_description: 'Setup an alpine schroot to test alpine packages for reproducibility.'
- my_timed: '42 23 1 1 *'
- my_shell: '/srv/jenkins/bin/reproducible_setup_alpine_schroot.sh'
- my_disabled: true
- - 'setup_schroot_alpine_osuosl1':
- my_description: 'Setup an alpine schroot to test alpine packages for reproducibility.'
- my_timed: '42 23 1 1 *'
- my_hname: 'osuosl1'
- my_disabled: true
- - 'setup_schroot_alpine_osuosl2':
- my_description: 'Setup an alpine schroot to test alpine packages for reproducibility.'
- my_timed: '42 23 1 1 *'
- my_hname: 'osuosl2'
- my_disabled: true
- 'setup_mock_fedora-23_x86_64_jenkins':
my_description: 'Setup mock for fedora-23 on X86_64 to test .rpm packages for reproducibility.'
my_timed: '42 23 1 1 *'
@@ -815,38 +792,6 @@
- '{name}_builder_{otherproject}_{my_num}':
otherproject:
- - 'alpine':
- my_num: 1
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 2
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 3
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 4
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 5
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 6
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 7
- op_descname: 'an alpine'
- my_disabled: true
- - 'alpine':
- my_num: 8
- op_descname: 'an alpine'
- my_disabled: true
- 'fedora-23_x86_64':
my_num: 1
op_descname: 'a Fedora 23'
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/9d612c1695befbf33791a203c948de20d06eb406
--
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/9d612c1695befbf33791a203c948de20d06eb406
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20250921/37b226e6/attachment-0001.htm>
More information about the Qa-jenkins-scm
mailing list