[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] 2 commits: reproducible: update documentation how we use osuosl167

Holger Levsen (@holger) gitlab at salsa.debian.org
Wed Sep 14 14:51:31 BST 2022



Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
c4ed29a6 by Holger Levsen at 2022-09-14T15:36:44+02:00
reproducible: update documentation how we use osuosl167

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
e4393e40 by Holger Levsen at 2022-09-14T15:47:19+02:00
reproducible: stop testing F-Droid, they are testing themselves nowadays

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


11 changed files:

- README.infrastructure
- THANKS.head
- TODO
- bin/common-functions.sh
- bin/jenkins_node_wrapper.sh
- − bin/reproducible_fdroid_build_apps.sh
- − bin/reproducible_fdroid_test.sh
- bin/reproducible_html_nodes_info.sh
- − bin/reproducible_setup_fdroid_build_environment.sh
- job-cfg/reproducible.yaml
- update_jdn.sh


Changes:

=====================================
README.infrastructure
=====================================
@@ -45,8 +45,8 @@ support at osuosl.org mentioning the machine name in the subject.
 
 The nodes are used for these jobs:
 
-* osuosl167: debian-janitor, mmdebstrap-jenkins jobs and openqa
-* osuosl168: fdroid
+* osuosl167: debian-janitor, mmdebstrap-jenkins jobs and openqa.d.n workers
+* osuosl168: UNUSED
 * osuosl170: archlinux (future), alpine
 * osuosl171: openwrt, coreboot, netbsd
 ** is constantly online (in jenkins) now, which is not how we like it and still doesnt solve the netbsd job probs


=====================================
THANKS.head
=====================================
@@ -38,8 +38,8 @@ link:https://jenkins.debian.net/["jenkins.debian.net"] would not be possible wit
  ** 8 cores and 64 GB memory for codethink16-arm64.debian.net used for building arm64 Debian packages for t.r-b.o
  * link:https://letsencrypt.org[Let's encrypt] provides free of charge SSL certificates for jenkins.debian.net, reproducible.debian.net and tests.reproducible-builds.org.
  * In December 2018 we were given access to eight nodes which were donated by Facebook to the GCC Compile Farm project and are now hosted by link:https://osuosl.org/[OSUOSL]:
-  ** 32 cores with 144 GB memory for osuosl167-amd64.debian.net
-  ** 32 cores with 144 GB memory for osuosl168-amd64.debian.net used for building F-Droid for t.r-b.o
+  ** 32 cores with 144 GB memory for osuosl167-amd64.debian.net used for Debian janitor jobs and mmdebstrap-jenkins jobs and openqa.d.n workers
+  ** 32 cores with 144 GB memory for osuosl168-amd64.debian.net, UNUSED
   ** 32 cores with 144 GB memory for osuosl184-amd64.debian.net used for building Arch Linux for t.r-b.o
   ** 32 cores with 144 GB memory for osuosl170-amd64.debian.net used for building Arch Linux for t.r-b.o
   ** 32 cores with 144 GB memory for osuosl171-amd64.debian.net used for building OpenWrt, coreboot and NetBSD for t.r-b.o


=====================================
TODO
=====================================
@@ -361,37 +361,6 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 * build on OSUOSL nodes:
 ** update thanks & readme
 
-==== reproducible fdroid
-
-* reproducible_setup_fdroid_buildserver.sh:
-** ./jenkins-build-makebuildserver
-*** manually added the jenkins user to the vboxdrv group
-*** this downloads a base debian image and all Android tarballs (SDK, NDK, Gradle...)
-*** then enters the image, installs all debian packages and Android stuff from the cached tarballs
-** cache is kept outside ('~/.cache/fdroidserver') but installed inside
-** '~/.cache/fdroidserver' needs to be cleaned at some times…
-
-* reproducible_build_fdroid_apk.sh
-** 1st run ./fdroid build some.app:vercode --server
-** 2nd run ./fdroid build some.app:vercode --server
-*** e.g.: org.fdroid.fdroid:98006
-*** or: "fdroid build -l org.fdroid.fdroid" to build the latest
-** run diffoscope on the results
-
-* also see https://f-droid.org/wiki/page/Build_Server_Setup
-
-* diskspace needs: initially 300GB (2016), now over 600GB (2020)
----- 
-$ # (data from 2016)
-$ du -hd1 | sort -h | tail -n 6
-4.4G    ./android-sdk-linux_86
-8.1G    ./fdroidserver
-8.3G    ./.vagrant.d
-71G     ./VirtualBox VMs
-150G    ./fdroiddata
-242G    .
-----
-
 ==== reproducible qubes
 
 * add qubes test on t.r-b.o


=====================================
bin/common-functions.sh
=====================================
@@ -236,7 +236,7 @@ jenkins_zombie_check() {
 	# related however is that I issued a reboot (via running 
 	# /sbin/reboot) right before the zombies appeared...
 	#
-	ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | egrep 'strip-nondeterminism|reprotest|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl169|fakeroot-foreign' || true)"
+	ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | egrep 'strip-nondeterminism|reprotest|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl169|fakeroot-foreign|fdroid' || true)"
 	if [ ! -z "$ZOMBIES" ] ; then
 		DIRTY=true
 		figlet 'zombies!!!'


=====================================
bin/jenkins_node_wrapper.sh
=====================================
@@ -176,12 +176,6 @@ elif [[ "$*" =~ ^reproducible_setup_schroot_alpine ]] ; then
 	exec /srv/jenkins/bin/reproducible_setup_alpine_schroot.sh ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_setup_schroot_archlinux ]] ; then
 	exec /srv/jenkins/bin/reproducible_setup_archlinux_schroot.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_fdroid_build_apps ]] ; then
-	exec /srv/jenkins/bin/reproducible_fdroid_build_apps.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_fdroid_test ]] ; then
-	exec /srv/jenkins/bin/reproducible_fdroid_test.sh ; croak "Exec failed";
-elif [[ "$*" =~ ^reproducible_setup_fdroid_build_environment ]] ; then
-	exec /srv/jenkins/bin/reproducible_setup_fdroid_build_environment.sh ; croak "Exec failed";
 elif [[ "$*" =~ ^reproducible_setup_mock_fedora-23_x86_64 ]] ; then
 	exec /srv/jenkins/bin/reproducible_setup_mock.sh fedora-23 x86_64 ; croak "Exec failed";
 elif [ "$1" = "/srv/jenkins/bin/reproducible_openwrt_rebuild.py" ] ; then


=====================================
bin/reproducible_fdroid_build_apps.sh deleted
=====================================
@@ -1,55 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen (holger at layer-acht.org)
-# Copyright © 2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents.  Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_fdroid_build_apps
-if [ -e $WORKSPACE/.git ]; then
-    # reuse the git repo if possible, to keep all the setup in fdroiddata/
-    cd $WORKSPACE
-    git remote set-url origin $GIT_REPO
-    while ! git fetch origin --tags --prune; do sleep 10; done
-    git clean -fdx
-    git reset --hard
-    git checkout master
-    git reset --hard origin/master
-    git clean -fdx
-else
-    rm -rf $WORKSPACE
-    git clone $GIT_REPO $WORKSPACE
-    cd $WORKSPACE
-fi
-
-cleanup_all() {
-    set +e
-    echo "$(date -u) - cleanup in progress..."
-    killall adb
-    killall gpg-agent
-    echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-./jenkins-build-all
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."


=====================================
bin/reproducible_fdroid_test.sh deleted
=====================================
@@ -1,65 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen (holger at layer-acht.org)
-# Copyright © 2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents.  Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_fdroid_build_apps
-if [ -e $WORKSPACE/.git ]; then
-    # reuse the git repo if possible, to keep all the setup in fdroiddata/
-    cd $WORKSPACE
-    git remote set-url origin $GIT_REPO
-    while ! git fetch origin --tags --prune; do sleep 10; done
-    git clean -fdx
-    git reset --hard
-    git checkout master
-    git reset --hard origin/master
-    git clean -fdx
-else
-    rm -rf $WORKSPACE
-    git clone $GIT_REPO $WORKSPACE
-    cd $WORKSPACE
-fi
-
-cleanup_all() {
-    set +e
-    echo "$(date -u) - cleanup in progress..."
-    killall adb
-    killall gpg-agent
-    fdroiddata=$WORKSPACE/fdroiddata
-    git -C $fdroiddata status
-    jobtmpdir=/tmp/$(basename $0)-$(date +%s)/
-    mkdir $jobtmpdir
-    for f in `git -C $fdroiddata status | sed -En 's,\s+modified:\s+,,p'`; do
-	path=$fdroiddata/$f
-	test -e $path || continue
-	ls -l $path
-	cp -a $path $jobtmpdir
-    done
-    echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-./jenkins-test
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."


=====================================
bin/reproducible_html_nodes_info.sh
=====================================
@@ -21,8 +21,8 @@ explain_nodes() {
 		ionos7)		write_page "<br /><small>(buildinfos.debian.net)</small>" ;;
 		ionos9)		write_page "<br /><small>(jenkins.d.n rebootstrap jobs)</small>" ;;
 		ionos10)	write_page "<br /><small>(jenkins.d.n chroot-installation jobs and http-proxy)</small>" ;;
-		osuosl167)	write_page "<br /><small>(Debian janitor jobs and fakeroot foreign jobs and http-proxy for osuosl nodes)</small>" ;;
-		osuosl168)	write_page "<br /><small>(r-b F-Droid builds)</small>" ;;
+		osuosl167)	write_page "<br /><small>(Debian janitor jobs and mmdebstrap-jenkins jobs and openqa.d.n workers and http-proxy for osuosl nodes)</small>" ;;
+		osuosl168)	write_page "<br /><small>(UNUSED)</small>" ;;
 		osuosl184)	write_page "<br /><small>(r-b Alpine, Archlinux builds)</small>" ;;
 		osuosl170)	write_page "<br /><small>(r-b Alpine, Archlinux builds)</small>" ;;
 		osuosl171)	write_page "<br /><small>(r-b OpenWrt, NetBSD, Coreboot builds)</small>" ;;
@@ -71,7 +71,7 @@ build_nodes_health_page() {
 				# ionos3	twitterbot
 				REPRODUCIBLE_NODES="$REPRODUCIBLE_NODES ionos3-amd64.debian.net"
 			elif [ "$NODE" = "ionos6-i386.debian.net" ] ; then
-				# ionos7	reproducible fdroid jobs
+				# ionos7	buildinfos.debian.net jobs
 				# ionos9 	rebootstrap jobs
 				# ionos10	chroot jobs
 				REPRODUCIBLE_NODES="$REPRODUCIBLE_NODES ionos7-amd64.debian.net ionos9-amd64.debian.net ionos10-amd64.debian.net"
@@ -266,7 +266,6 @@ build_job_health_page() {
 		"coreboot"
 		"(openwrt)"
 		"(le_netbsd|le_freebsd)"
-		"fdroid"
 		"fedora"
 		"alpine")
 	echo "$(date -u) - starting to write $PAGE page."
@@ -285,7 +284,6 @@ build_job_health_page() {
 				| sed 's#setup_schroot_##' \
 				| sed 's#setup_mock_fedora-##' \
 				| sed 's#create_##' \
-				| sed 's#fdroid_build_##' \
 				| sed 's#html_archlinux#html#' \
 				| sed 's#html_alpine#html#' \
 				| sed 's#html_##' \


=====================================
bin/reproducible_setup_fdroid_build_environment.sh deleted
=====================================
@@ -1,81 +0,0 @@
-#!/bin/bash
-
-# Copyright © 2015-2018 Holger Levsen <holger at layer-acht.org>
-# Copyright © 2016-2017 Hans-Christoph Steiner (hans at guardianproject.info)
-# released under the GPLv2
-
-#
-#
-
-DEBUG=true
-LC_ALL=en_US.UTF-8
-. /srv/jenkins/bin/common-functions.sh
-common_init "$@"
-
-# common code
-. /srv/jenkins/bin/reproducible_common.sh
-
-GIT_REPO=https://gitlab.com/fdroid/fdroidserver.git
-
-# define and clean work space on the machine actually running the
-# build. jenkins.debian.net does not use Jenkins agents.  Instead
-# /srv/jenkins/bin/jenkins_master_wrapper.sh runs this script on the
-# agent using a directly call to ssh, so this script has to do all
-# of the workspace setup.
-export WORKSPACE=$BASE/reproducible_setup_fdroid_build_environment
-if [ -e $WORKSPACE/.git ]; then
-    # reuse the git repo if possible, to keep all the setup in fdroiddata/
-    cd $WORKSPACE
-    git remote set-url origin $GIT_REPO
-    while ! git fetch origin --tags --prune; do sleep 10; done
-    git clean -fdx
-    git reset --hard
-    git checkout master
-    git reset --hard origin/master
-    git clean -fdx
-else
-    rm -rf $WORKSPACE
-    git clone $GIT_REPO $WORKSPACE
-    cd $WORKSPACE
-fi
-
-cleanup_all() {
-    set +e
-    echo "$(date -u) - cleanup in progress..."
-    killall adb
-    killall gpg-agent
-    echo "$(date -u) - cleanup done."
-}
-trap cleanup_all INT TERM EXIT
-
-# report info about virtualization
-(dmesg | grep -i -e hypervisor -e qemu -e kvm) || true
-(lspci | grep -i -e virtio -e virtualbox -e qemu -e kvm) || true
-lsmod
-if systemd-detect-virt -q ; then
-        echo "Virtualization is used:" `systemd-detect-virt`
-else
-        echo "No virtualization is used."
-fi
-if test -d /var/lib/libvirt/images; then
-    sudo /bin/chmod -R a+rX /var/lib/libvirt/images
-    ls -ld /var/lib/libvirt/images
-    ls -l /var/lib/libvirt/images || echo no access
-fi
-ls -lR ~/.vagrant.d/ || echo no access
-virsh --connect qemu:///system list --all || echo cannot virsh list
-cat /etc/issue
-
-# delete old libvirt instances, until the fdroid tools do it reliably
-virsh --connect qemu:///system undefine builder_default || echo nothing to undefine
-virsh --connect qemu:///system vol-delete --pool default /var/lib/libvirt/images/builder_default.img || echo nothing to delete
-
-# set up Android SDK to use the Debian packages in stretch
-export ANDROID_HOME=/usr/lib/android-sdk
-
-# this script is maintained upstream
-./jenkins-setup-build-environment
-
-# remove trap
-trap - INT TERM EXIT
-echo "$(date -u) - the end."


=====================================
job-cfg/reproducible.yaml
=====================================
@@ -858,35 +858,6 @@
             my_timeout: '1440'
             my_parse_rules: '/srv/jenkins/logparse/reproducible-extra.rules'
 
-        - '{name}_{otherproject_blocked}':
-            otherproject_blocked:
-                - 'setup_fdroid_build_environment':
-                    my_description: 'Setup a build environment for F-Droid.'
-                    my_timed: '0 23 5,20 * *'
-                    my_timed: ''
-                    my_hname: 'osuosl168'
-                    my_recipients: 'jenkins+reproducible-changes qa-jenkins-scm at lists.alioth.debian.org hans at guardianproject.info michael.poehn at fsfe.org flx at obfusk.net'
-                    my_blocker: '^reproducible.*fdroid.*'
-            my_shell: '/srv/jenkins/bin/jenkins_master_wrapper.sh'
-
-        - '{name}_{otherproject_git_blocked}':
-            otherproject_git_blocked:
-                - 'fdroid_build_apps':
-                    my_description: 'Build all the latest apps for F-Droid.'
-                    my_timed: ''
-                - 'fdroid_test':
-                    my_description: 'Run tests on all the built APKs for F-Droid.'
-                    my_timed: ''
-            my_hname: 'osuosl168'
-            my_recipients: 'jenkins+reproducible-changes qa-jenkins-scm at lists.alioth.debian.org hans at guardianproject.info michael.poehn at fsfe.org flx at obfusk.net'
-            my_blocker: '^reproducible.+fdroid.*'
-            my_gitrepo: 'https://gitlab.com/fdroid/fdroidserver.git'
-            my_scmpoll: '*/6 * * * *'
-            my_gitbranches: 'master'
-            my_shell: '/srv/jenkins/bin/jenkins_master_wrapper.sh'
-            my_parse_rules: '/srv/jenkins/logparse/reproducible-extra.rules'
-            my_naginator_regex: 'Caused by: hudson.plugins.git.GitException: Command "git fetch'
-
         - '{name}_{otherproject_git}':
             otherproject_git:
                 - 'disorderfs_from_git':


=====================================
update_jdn.sh
=====================================
@@ -470,27 +470,6 @@ if [ -f /etc/debian_version ] ; then
 			;;
 			*) ;;
 		esac
-		# needed to run fdroid jobs
-		case $HOSTNAME in
-			osuosl168-amd64) DEBS="$DEBS
-				androguard
-				android-sdk
-				bzr
-				git-svn
-				fdroidserver
-				linux-headers-amd64
-				mercurial
-				python3-asn1crypto
-				python3-babel
-				python3-mwclient
-				python3-setuptools
-				subversion
-				vagrant
-				virtualbox/bullseye-fasttrack virtualbox-dkms/bullseye-fasttrack
-				"
-				;;
-			*) ;;
-		esac
 		if [ "$HOSTNAME" = "osuosl167-amd64" ]; then
 			# for phil only
 			DEBS="$DEBS postfix-pcre"



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/7086f8d3185a2de48d12e8a3cbb23fb3be6cdb08...e4393e40ca549adf91bbe4552c96710400923037

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/7086f8d3185a2de48d12e8a3cbb23fb3be6cdb08...e4393e40ca549adf91bbe4552c96710400923037
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20220914/a34fda54/attachment-0001.htm>


More information about the Qa-jenkins-scm mailing list