[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] 2 commits: reproducible: update documentation how we use osuosl167
Holger Levsen (@holger)
gitlab at salsa.debian.org
Wed Sep 14 14:51:31 BST 2022
Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net
Commits:
c4ed29a6 by Holger Levsen at 2022-09-14T15:36:44+02:00
reproducible: update documentation how we use osuosl167
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
e4393e40 by Holger Levsen at 2022-09-14T15:47:19+02:00
reproducible: stop testing F-Droid, they are testing themselves nowadays
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
11 changed files:
- README.infrastructure
- THANKS.head
- TODO
- bin/common-functions.sh
- bin/jenkins_node_wrapper.sh
- − bin/reproducible_fdroid_build_apps.sh
- − bin/reproducible_fdroid_test.sh
- bin/reproducible_html_nodes_info.sh
- − bin/reproducible_setup_fdroid_build_environment.sh
- job-cfg/reproducible.yaml
- update_jdn.sh
Changes:
=====================================
README.infrastructure
=====================================
@@ -45,8 +45,8 @@ support at osuosl.org mentioning the machine name in the subject.
The nodes are used for these jobs:
-* osuosl167: debian-janitor, mmdebstrap-jenkins jobs and openqa
-* osuosl168: fdroid
+* osuosl167: debian-janitor, mmdebstrap-jenkins jobs and openqa.d.n workers
+* osuosl168: UNUSED
* osuosl170: archlinux (future), alpine
* osuosl171: openwrt, coreboot, netbsd
** is constantly online (in jenkins) now, which is not how we like it and still doesnt solve the netbsd job probs
=====================================
THANKS.head
=====================================
@@ -38,8 +38,8 @@ link:https://jenkins.debian.net/["jenkins.debian.net"] would not be possible wit
** 8 cores and 64 GB memory for codethink16-arm64.debian.net used for building arm64 Debian packages for t.r-b.o
* link:https://letsencrypt.org[Let's encrypt] provides free of charge SSL certificates for jenkins.debian.net, reproducible.debian.net and tests.reproducible-builds.org.
* In December 2018 we were given access to eight nodes which were donated by Facebook to the GCC Compile Farm project and are now hosted by link:https://osuosl.org/[OSUOSL]:
- ** 32 cores with 144 GB memory for osuosl167-amd64.debian.net
- ** 32 cores with 144 GB memory for osuosl168-amd64.debian.net used for building F-Droid for t.r-b.o
+ ** 32 cores with 144 GB memory for osuosl167-amd64.debian.net used for Debian janitor jobs and mmdebstrap-jenkins jobs and openqa.d.n workers
+ ** 32 cores with 144 GB memory for osuosl168-amd64.debian.net, UNUSED
** 32 cores with 144 GB memory for osuosl184-amd64.debian.net used for building Arch Linux for t.r-b.o
** 32 cores with 144 GB memory for osuosl170-amd64.debian.net used for building Arch Linux for t.r-b.o
** 32 cores with 144 GB memory for osuosl171-amd64.debian.net used for building OpenWrt, coreboot and NetBSD for t.r-b.o
=====================================
TODO
=====================================
@@ -361,37 +361,6 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
* build on OSUOSL nodes:
** update thanks & readme
-==== reproducible fdroid
-
-* reproducible_setup_fdroid_buildserver.sh:
-** ./jenkins-build-makebuildserver
-*** manually added the jenkins user to the vboxdrv group
-*** this downloads a base debian image and all Android tarballs (SDK, NDK, Gradle...)
-*** then enters the image, installs all debian packages and Android stuff from the cached tarballs
-** cache is kept outside ('~/.cache/fdroidserver') but installed inside
-** '~/.cache/fdroidserver' needs to be cleaned at some times…
-
-* reproducible_build_fdroid_apk.sh
-** 1st run ./fdroid build some.app:vercode --server
-** 2nd run ./fdroid build some.app:vercode --server
-*** e.g.: org.fdroid.fdroid:98006
-*** or: "fdroid build -l org.fdroid.fdroid" to build the latest
-** run diffoscope on the results
-
-* also see https://f-droid.org/wiki/page/Build_Server_Setup
-
-* diskspace needs: initially 300GB (2016), now over 600GB (2020)
-----
-$ # (data from 2016)
-$ du -hd1 | sort -h | tail -n 6
-4.4G ./android-sdk-linux_86
-8.1G ./fdroidserver
-8.3G ./.vagrant.d
-71G ./VirtualBox VMs
-150G ./fdroiddata
-242G .
-----
-
==== reproducible qubes
* add qubes test on t.r-b.o
=====================================
bin/common-functions.sh
=====================================
@@ -236,7 +236,7 @@ jenkins_zombie_check() {
# related however is that I issued a reboot (via running
# /sbin/reboot) right before the zombies appeared...
#
- ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | egrep 'strip-nondeterminism|reprotest|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl169|fakeroot-foreign' || true)"
+ ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | egrep 'strip-nondeterminism|reprotest|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl169|fakeroot-foreign|fdroid' || true)"
if [ ! -z "$ZOMBIES" ] ; then
DIRTY=true
figlet 'zombies!!!'
=====================================
bin/jenkins_node_wrapper.sh
=====================================
@@ -176,12 +176,6 @@ elif [[ "$*" =~ ^reproducible_setup_schroot_alpine ]] ; then
exec /srv/jenkins/bin/reproducible_setup_alpine_schroot.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_schroot_archlinux ]] ; then
exec /srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_fdroid_build_apps ]] ; then
- exec /srv/jenkins/bin/reproducible_fdroid_build_apps.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_fdroid_test ]] ; then
- exec /srv/jenkins/bin/reproducible_fdroid_test.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_setup_fdroid_build_environment ]] ; then
- exec /srv/jenkins/bin/reproducible_setup_fdroid_build_environment.sh ; croak "Exec failed";
elif [[ "$*" =~ ^reproducible_setup_mock_fedora-23_x86_64 ]] ; then
exec /srv/jenkins/bin/reproducible_setup_mock.sh fedora-23 x86_64 ; croak "Exec failed";
elif [ "$1" = "/srv/jenkins/bin/reproducible_openwrt_rebuild.py" ] ; then
=====================================
bin/reproducible_fdroid_build_apps.sh deleted
=====================================
@@ -1,55 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen (holger at layer-acht.org)
-# Copyright © 2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents. Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_fdroid_build_apps
-if [ -e $WORKSPACE/.git ]; then
- # reuse the git repo if possible, to keep all the setup in fdroiddata/
- cd $WORKSPACE
- git remote set-url origin $GIT_REPO
- while ! git fetch origin --tags --prune; do sleep 10; done
- git clean -fdx
- git reset --hard
- git checkout master
- git reset --hard origin/master
- git clean -fdx
-else
- rm -rf $WORKSPACE
- git clone $GIT_REPO $WORKSPACE
- cd $WORKSPACE
-fi
-
-cleanup_all() {
- set +e
- echo "$(date -u) - cleanup in progress..."
- killall adb
- killall gpg-agent
- echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-./jenkins-build-all
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."
=====================================
bin/reproducible_fdroid_test.sh deleted
=====================================
@@ -1,65 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen (holger at layer-acht.org)
-# Copyright © 2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents. Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_fdroid_build_apps
-if [ -e $WORKSPACE/.git ]; then
- # reuse the git repo if possible, to keep all the setup in fdroiddata/
- cd $WORKSPACE
- git remote set-url origin $GIT_REPO
- while ! git fetch origin --tags --prune; do sleep 10; done
- git clean -fdx
- git reset --hard
- git checkout master
- git reset --hard origin/master
- git clean -fdx
-else
- rm -rf $WORKSPACE
- git clone $GIT_REPO $WORKSPACE
- cd $WORKSPACE
-fi
-
-cleanup_all() {
- set +e
- echo "$(date -u) - cleanup in progress..."
- killall adb
- killall gpg-agent
- fdroiddata=$WORKSPACE/fdroiddata
- git -C $fdroiddata status
- jobtmpdir=/tmp/$(basename $0)-$(date +%s)/
- mkdir $jobtmpdir
- for f in `git -C $fdroiddata status | sed -En 's,\s+modified:\s+,,p'`; do
- path=$fdroiddata/$f
- test -e $path || continue
- ls -l $path
- cp -a $path $jobtmpdir
- done
- echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-./jenkins-test
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."
=====================================
bin/reproducible_html_nodes_info.sh
=====================================
@@ -21,8 +21,8 @@ explain_nodes() {
ionos7) write_page "<br /><small>(buildinfos.debian.net)</small>" ;;
ionos9) write_page "<br /><small>(jenkins.d.n rebootstrap jobs)</small>" ;;
ionos10) write_page "<br /><small>(jenkins.d.n chroot-installation jobs and http-proxy)</small>" ;;
- osuosl167) write_page "<br /><small>(Debian janitor jobs and fakeroot foreign jobs and http-proxy for osuosl nodes)</small>" ;;
- osuosl168) write_page "<br /><small>(r-b F-Droid builds)</small>" ;;
+ osuosl167) write_page "<br /><small>(Debian janitor jobs and mmdebstrap-jenkins jobs and openqa.d.n workers and http-proxy for osuosl nodes)</small>" ;;
+ osuosl168) write_page "<br /><small>(UNUSED)</small>" ;;
osuosl184) write_page "<br /><small>(r-b Alpine, Archlinux builds)</small>" ;;
osuosl170) write_page "<br /><small>(r-b Alpine, Archlinux builds)</small>" ;;
osuosl171) write_page "<br /><small>(r-b OpenWrt, NetBSD, Coreboot builds)</small>" ;;
@@ -71,7 +71,7 @@ build_nodes_health_page() {
# ionos3 twitterbot
REPRODUCIBLE_NODES="$REPRODUCIBLE_NODES ionos3-amd64.debian.net"
elif [ "$NODE" = "ionos6-i386.debian.net" ] ; then
- # ionos7 reproducible fdroid jobs
+ # ionos7 buildinfos.debian.net jobs
# ionos9 rebootstrap jobs
# ionos10 chroot jobs
REPRODUCIBLE_NODES="$REPRODUCIBLE_NODES ionos7-amd64.debian.net ionos9-amd64.debian.net ionos10-amd64.debian.net"
@@ -266,7 +266,6 @@ build_job_health_page() {
"coreboot"
"(openwrt)"
"(le_netbsd|le_freebsd)"
- "fdroid"
"fedora"
"alpine")
echo "$(date -u) - starting to write $PAGE page."
@@ -285,7 +284,6 @@ build_job_health_page() {
| sed 's#setup_schroot_##' \
| sed 's#setup_mock_fedora-##' \
| sed 's#create_##' \
- | sed 's#fdroid_build_##' \
| sed 's#html_archlinux#html#' \
| sed 's#html_alpine#html#' \
| sed 's#html_##' \
=====================================
bin/reproducible_setup_fdroid_build_environment.sh deleted
=====================================
@@ -1,81 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen <holger at layer-acht.org>
-# Copyright © 2016-2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents. Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_setup_fdroid_build_environment
-if [ -e $WORKSPACE/.git ]; then
- # reuse the git repo if possible, to keep all the setup in fdroiddata/
- cd $WORKSPACE
- git remote set-url origin $GIT_REPO
- while ! git fetch origin --tags --prune; do sleep 10; done
- git clean -fdx
- git reset --hard
- git checkout master
- git reset --hard origin/master
- git clean -fdx
-else
- rm -rf $WORKSPACE
- git clone $GIT_REPO $WORKSPACE
- cd $WORKSPACE
-fi
-
-cleanup_all() {
- set +e
- echo "$(date -u) - cleanup in progress..."
- killall adb
- killall gpg-agent
- echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-# report info about virtualization
-(dmesg | grep -i -e hypervisor -e qemu -e kvm) || true
-(lspci | grep -i -e virtio -e virtualbox -e qemu -e kvm) || true
-lsmod
-if systemd-detect-virt -q ; then
- echo "Virtualization is used:" `systemd-detect-virt`
-else
- echo "No virtualization is used."
-fi
-if test -d /var/lib/libvirt/images; then
- sudo /bin/chmod -R a+rX /var/lib/libvirt/images
- ls -ld /var/lib/libvirt/images
- ls -l /var/lib/libvirt/images || echo no access
-fi
-ls -lR ~/.vagrant.d/ || echo no access
-virsh --connect qemu:///system list --all || echo cannot virsh list
-cat /etc/issue
-
-# delete old libvirt instances, until the fdroid tools do it reliably
-virsh --connect qemu:///system undefine builder_default || echo nothing to undefine
-virsh --connect qemu:///system vol-delete --pool default /var/lib/libvirt/images/builder_default.img || echo nothing to delete
-
-# set up Android SDK to use the Debian packages in stretch
-export ANDROID_HOME=/usr/lib/android-sdk
-
-# this script is maintained upstream
-./jenkins-setup-build-environment
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."
=====================================
job-cfg/reproducible.yaml
=====================================
@@ -858,35 +858,6 @@
my_timeout: '1440'
my_parse_rules: '/srv/jenkins/logparse/reproducible-extra.rules'
- - '{name}_{otherproject_blocked}':
- otherproject_blocked:
- - 'setup_fdroid_build_environment':
- my_description: 'Setup a build environment for F-Droid.'
- my_timed: '0 23 5,20 * *'
- my_timed: ''
- my_hname: 'osuosl168'
- my_recipients: 'jenkins+reproducible-changes qa-jenkins-scm at lists.alioth.debian.org hans at guardianproject.info michael.poehn at fsfe.org flx at obfusk.net'
- my_blocker: '^reproducible.*fdroid.*'
- my_shell: '/srv/jenkins/bin/jenkins_master_wrapper.sh'
-
- - '{name}_{otherproject_git_blocked}':
- otherproject_git_blocked:
- - 'fdroid_build_apps':
- my_description: 'Build all the latest apps for F-Droid.'
- my_timed: ''
- - 'fdroid_test':
- my_description: 'Run tests on all the built APKs for F-Droid.'
- my_timed: ''
- my_hname: 'osuosl168'
- my_recipients: 'jenkins+reproducible-changes qa-jenkins-scm at lists.alioth.debian.org hans at guardianproject.info michael.poehn at fsfe.org flx at obfusk.net'
- my_blocker: '^reproducible.+fdroid.*'
- my_gitrepo: 'https://gitlab.com/fdroid/fdroidserver.git'
- my_scmpoll: '*/6 * * * *'
- my_gitbranches: 'master'
- my_shell: '/srv/jenkins/bin/jenkins_master_wrapper.sh'
- my_parse_rules: '/srv/jenkins/logparse/reproducible-extra.rules'
- my_naginator_regex: 'Caused by: hudson.plugins.git.GitException: Command "git fetch'
-
- '{name}_{otherproject_git}':
otherproject_git:
- 'disorderfs_from_git':
=====================================
update_jdn.sh
=====================================
@@ -470,27 +470,6 @@ if [ -f /etc/debian_version ] ; then
;;
*) ;;
esac
- # needed to run fdroid jobs
- case $HOSTNAME in
- osuosl168-amd64) DEBS="$DEBS
- androguard
- android-sdk
- bzr
- git-svn
- fdroidserver
- linux-headers-amd64
- mercurial
- python3-asn1crypto
- python3-babel
- python3-mwclient
- python3-setuptools
- subversion
- vagrant
- virtualbox/bullseye-fasttrack virtualbox-dkms/bullseye-fasttrack
- "
- ;;
- *) ;;
- esac
if [ "$HOSTNAME" = "osuosl167-amd64" ]; then
# for phil only
DEBS="$DEBS postfix-pcre"
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/7086f8d3185a2de48d12e8a3cbb23fb3be6cdb08...e4393e40ca549adf91bbe4552c96710400923037
--
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/7086f8d3185a2de48d12e8a3cbb23fb3be6cdb08...e4393e40ca549adf91bbe4552c96710400923037
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20220914/a34fda54/attachment-0001.htm>
More information about the Qa-jenkins-scm
mailing list