[Git][qa/jenkins.debian.net][predictible-build-paths] 21 commits: upgrade jenkins host to bookworm now that 12.1 is out

Vagrant Cascadian (@vagrant) gitlab at salsa.debian.org
Thu Aug 24 19:43:04 BST 2023 


Vagrant Cascadian pushed to branch predictible-build-paths at Debian QA / jenkins.debian.net


Commits:
990ac8e3 by Holger Levsen at 2023-07-22T17:53:21+02:00
upgrade jenkins host to bookworm now that 12.1 is out

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
a790b6ea by Holger Levsen at 2023-07-22T20:10:49+02:00
upgrade jenkins host to bookworm, more changes

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
6ee50acf by Holger Levsen at 2023-07-22T20:14:11+02:00
reproducible Debian: reenable automatic documentation of database as postgresql_autodoc is back in bookworm

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
987cc468 by Holger Levsen at 2023-07-22T21:30:35+02:00
migration to bookworm basically done

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
2e24c819 by Holger Levsen at 2023-07-23T16:45:48+02:00
reproducible Debian: fix yaml loading for pyyaml 6.0. (as seen in #1034116)

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
334e8b6c by Holger Levsen at 2023-07-24T10:15:00+02:00
reproducible Debian: enable debug output for _debian_live_sync_result.sh

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
0faadb63 by Holger Levsen at 2023-07-26T14:11:18+02:00
reproducible Debian live: use scp -O: Use the legacy SCP protocol instead of the SFTP protocol, thanks lucas & Myon

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
194c84c6 by Philip Hands at 2023-07-27T10:16:22+02:00
Revert "OPENQA_WORKERS_TO_RUN=6 (from 5)"

This reverts commit f604626e5697f267bf5f9f7f8c1b81505084c0c7.

Some OpenQA jobs have been hanging, and the munin graphs showed moments
of very little idle CPU and unused memory, so it seems like 6 workers
may have been asking a little too much, so let's back off a bit.

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
8db7f628 by Holger Levsen at 2023-07-27T10:20:26+02:00
remove redundant and obsolete dependencies

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
3c14619d by Holger Levsen at 2023-07-27T10:22:47+02:00
adjust jre req for bookworm

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
3efc26b4 by Philip Hands at 2023-07-27T13:53:30+02:00
reproducible Debian live: scp --> rsync

rsync does the copy then move thing automatically, so there's no need to
stage the file via a .tmp one. Also, doing that would prevent rsync from
noticing that it can use the old image to speed things up.

One could add a '-v' to the rsync command to see how much it is managing
to speed things up.

I don't think the Batchmode=yes is really needed (because all our
servers are locked-down to refuse passwords) but perhaps that was also
there to suppress prompts regarding host keys, in which case one could
add an option:

  -e 'ssh -o "Batchmode=yes"'

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
81aaf63e by Holger Levsen at 2023-07-28T20:48:25+02:00
reproducible Debian maintenance: drop i386 special casing from 2017

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
3713af7d by Philip Hands at 2023-07-28T23:15:28+02:00
openqa on osuosl3: systemctl disable: add --now

without this, the service is disabled, but allowed to continue running

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
5fe0e1f7 by Holger Levsen at 2023-08-13T12:16:53+02:00
reproducible Debian armhf: mark wbq0 and jtx1a down, investigation needed

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
09fafb8b by Holger Levsen at 2023-08-14T19:25:39+02:00
Revert "reproducible Debian armhf: mark wbq0 and jtx1a down, investigation needed"

This reverts commit 5fe0e1f776db544b4d2d69986199c1d9353b2ae7.

- - - - -
bad24f43 by Holger Levsen at 2023-08-21T15:34:51+02:00
node health check: also try to restart failed vnstat service

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
4082de75 by Holger Levsen at 2023-08-21T19:35:27+02:00
node health check: also try to restart failed ntpsec service

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
58a430a5 by Holger Levsen at 2023-08-22T14:45:23+02:00
reproducible Debian maintenance: show unignored zombies only

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
41ea0f46 by Holger Levsen at 2023-08-22T14:56:32+02:00
reproducible Debian maintenance: run every 3h instead of every 2h

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
8c8c82aa by Holger Levsen at 2023-08-22T15:12:05+02:00
reproducible system health: ignore another type of known zombies

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
9f078d22 by Vagrant Cascadian at 2023-08-24T18:42:53+00:00
reproducible Debian: Use predictible build paths consistent with
buildd.debian.org.

See https://bugs.debian.org/1034424

- - - - -


14 changed files:

- TODO
- TODO.trixie
- bin/djm
- bin/reproducible_build.sh
- bin/reproducible_create_meta_pkg_sets.sh
- bin/reproducible_db_backup.sh
- bin/reproducible_debian_live_sync_result.sh
- bin/reproducible_maintenance.sh
- bin/reproducible_node_health_check.sh
- bin/reproducible_notes.py
- − hosts/jenkins/etc/apt/sources.list
- init_node
- job-cfg/reproducible.yaml
- update_jdn.sh


Changes:

=====================================
TODO
=====================================
@@ -27,6 +27,7 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 * mention o4+5 in THANKS and explain usage.
 * mv snapshot.r-b.o from osuosl4 to osuosl5
 ** setup xfs on o5, then copy snapshot over
+** or setup ext4. or read up on xfs.
 * rebuilder on o4
 * jenkins backup on o5 (see below)
 


=====================================
TODO.trixie
=====================================
@@ -6,20 +6,6 @@
 * things still to be done after the bookworm release and as trixie is open for development
 ** once these items have been done, move them down to "already done"
 * update reproducible_common.sh with regards to the usrmerge variation in certain suites
-* maybe wait til 12.1 (= ~1 month) until we upgrade jenkins to bookworm, let's see
-** upgrade to postgresql-15:
-----
-# as root:
-apt install postgresql-15
-pg_lsclusters
-pg_dropcluster --stop 15 main
-pg_lsclusters
-pg_upgradecluster 13 main
-pg_lsclusters
-# test test test
-apt purge postgresql-13
-** revert ed92df9b8baa7df680a33d6d17833e651b8ac157 as postgresql-autodoc is available in bookworm
-----
 * rename this file to TODO.forky
 * enjoy
 
@@ -58,6 +44,19 @@ apt purge postgresql-13
 ** upgrade amd64 nodes to bookworm (except jenkins)
 *** see below for how to upgrade to postgresql-15 on ionos7
 ** upgrade armhf nodes to bookworm
+** upgrade jenkins to bookworm
+** upgrade to postgresql-15:
+----
+# as root:
+apt install postgresql-15
+pg_lsclusters
+pg_dropcluster --stop 15 main
+pg_lsclusters
+pg_upgradecluster 13 main
+pg_lsclusters
+# test test test
+apt purge postgresql-13
+----
 
 
 // vim: set filetype=asciidoc:


=====================================
bin/djm
=====================================
@@ -226,7 +226,9 @@ verify_target_action_reason(){
 			;;
 		all)	TARGET="$(`dirname $0`/../nodes/list_nodes )"
 			;;
-		*)	TARGET="$(`dirname $0`/../nodes/list_nodes | grep -E $TARGET || true)"
+		*)	# it would be cool to support i15 = ionos15, same for osuosl and codethink...
+			# echo c16|sed 's#c\([0-9][0-9]*\)#codethink\1#g'
+			TARGET="$(`dirname $0`/../nodes/list_nodes | grep -E $TARGET || true)"
 			;;
 	esac
 	if [ -z "$TARGET" ] ; then


=====================================
bin/reproducible_build.sh
=====================================
@@ -629,7 +629,7 @@ EOF
 	if [ "${SUITE}" = "unstable" ] || [ "$SUITE" = "experimental" ]; then
 		echo "BUILDDIR=/build/1st" >> "$TMPCFG"
 	else
-		echo "BUILDDIR=/build" >> "$TMPCFG"
+		echo "BUILDDIR=/build/reproducible-path" >> "$TMPCFG"
 	fi
 	if [ "$SRCPACKAGE" = "debian-installer" -o "$SRCPACKAGE" = "debian-installer-netboot-images" ] ; then
 		# d-i needs to access to a debian-archive.  this is not possible in
@@ -750,7 +750,7 @@ EOF
 		echo "BUILDDIR=/build/2/$src_dir_name" >> "$TMPCFG"
 		echo "BUILDSUBDIR=2nd" >> "$TMPCFG"
 	else
-		echo "BUILDDIR=/build" >> "$TMPCFG"
+		echo "BUILDDIR=/build/reproducible-path" >> "$TMPCFG"
 	fi
 	if [ "$SRCPACKAGE" = "debian-installer" -o "$SRCPACKAGE" = "debian-installer-netboot-images" ] ; then
 		# d-i needs to access to a debian-archive.  this is not possible in


=====================================
bin/reproducible_create_meta_pkg_sets.sh
=====================================
@@ -37,7 +37,7 @@ import sys
 import yaml
 try:
 	with open(sys.argv[1]) as fd:
-		manifest = yaml.load(fd)
+		manifest = yaml.safe_load(fd)
 
 	seen = set()
 	for pkg in (manifest['packages']['binary']):


=====================================
bin/reproducible_db_backup.sh
=====================================
@@ -54,9 +54,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] ; then
 		ln -s -f -v "$BACKUPFILE.xz" $BASE/reproducible.sql.xz
 
 		# recreate documentation of database
-		# disabled since postgresql_autodoc is not available in bullseye
-		# https://bugs.debian.org/970870
-		#postgresql_autodoc -d "$PGDATABASE" -t html -f "$BASE/reproducibledb"
+		postgresql_autodoc -d "$PGDATABASE" -t html -f "$BASE/reproducibledb"
 		set +x
 	fi
 	echo


=====================================
bin/reproducible_debian_live_sync_result.sh
=====================================
@@ -51,9 +51,8 @@ rsync_remote_results() {
     fi
     local URL="${REPRODUCIBLE_URL}/${PROJECT_PATH}/$EXTRADIR${filename}"
     # Copy the new results from the build node to the web server node
-    scp -p -o Batchmode=yes "$NODE":"$origfile" "$filename.tmp"
-    chmod 755 "$filename.tmp"
-    mv "$filename.tmp" "$filename"
+    rsync -t "$NODE":"$origfile" "$filename"
+    chmod 755 "$filename"
     echo "$(date -u) - enjoy $URL"
 }
 


=====================================
bin/reproducible_maintenance.sh
=====================================
@@ -744,25 +744,22 @@ for i in $PBUIDS ; do
 done
 if [ -n "$PSCALL" ] ; then
 	# ignore some well known zombie processes
-	KNOWN_ZOMBIE_PROCESSES="(buf-ring.t|poll-race-mshot.t|ringbuf-read.t)"
-	if [ $(ps -F -p "$PSCALL" | grep -E -v $KNOWN_ZOMBIE_PROCESSES | wc -l) -lt 10 ] ; then
+	KNOWN_ZOMBIE_PROCESSES="(buf-ring.t|poll-race-mshot.t|ringbuf-read.t|send_recvmsg.t)"
+	if [ $(ps -F -p "$PSCALL" | grep -E -v "$KNOWN_ZOMBIE_PROCESSES" | wc -l) -lt 10 ] ; then
 		echo "Info: ignoring less than ten processes which should not be there and which could not be killed, because those are probably just a few harmless zombies, which can only be removed by rebooting...."
 	else
-		 echo "Warning: found more than ten processes which should not be there and which could not be killed. Please investigate and reboot or ignore them...:"
+		echo "Warning: found more than ten processes which should not be there and which could not be killed. Please investigate and reboot or ignore them...:"
 	fi
-	ps -F -p "$PSCALL"
+	ps -F -p "$PSCALL" | grep -E -v "$KNOWN_ZOMBIE_PROCESSES"
 	echo
 fi
 
 # find builds which should not be there
-# (not on i386 as we start builds differently here… work in progress)
-if [ "$ARCH" != "i386" ] ; then
-	RESULTS=$(pgrep -f reproducible_build.sh --parent 1 || true)
-	if [ -n "$RESULTS" ] ; then
-		DIRTY=true
-		echo "Warning: found reproducible_build.sh processes which have pid 1 as parent (and not sshd), thus something went wrong… please investigate."
-		echo -e "$RESULTS"
-	fi
+RESULTS=$(pgrep -f reproducible_build.sh --parent 1 || true)
+if [ -n "$RESULTS" ] ; then
+	DIRTY=true
+	echo "Warning: found reproducible_build.sh processes which have pid 1 as parent (and not sshd), thus something went wrong… please investigate."
+	echo -e "$RESULTS"
 fi
 
 # remove debian ci builds artifacts older than a day


=====================================
bin/reproducible_node_health_check.sh
=====================================
@@ -163,7 +163,7 @@ if ! systemctl is-system-running > /dev/null; then
 	echo "$(date -u) - problematic services found:"
 	cat $SERVICES
 	echo "$(date -u) - trying to fix problematic services."
-	for UNIT in avahi-daemon acpid rtkit-daemon networking systemd-journal-flush haveged e2scrub_all apt-daily apt-daily-upgrade logrotate man-db munin-node dpkg-db-backup ; do
+	for UNIT in avahi-daemon acpid rtkit-daemon networking systemd-journal-flush haveged e2scrub_all apt-daily apt-daily-upgrade logrotate man-db munin-node dpkg-db-backup vnstat ntpsec ; do
 		if grep -q $UNIT $SERVICES ; then
 			echo "$(date -u) - restarting failed service $UNIT..."
 		        sudo systemctl restart $UNIT


=====================================
bin/reproducible_notes.py
=====================================
@@ -37,7 +37,7 @@ def load_notes():
       'package_name':<etc> }
     """
     with open(NOTES) as fd:
-        original = yaml.load(fd)
+        original = yaml.safe_load(fd)
     log.info("Notes loaded. There are " + str(len(original)) +
              " packages listed.")
     notes = {}
@@ -108,7 +108,7 @@ def load_issues():
     { 'issue_name': {'description': 'blabla', 'url': 'blabla'} }
     """
     with open(ISSUES) as fd:
-        issues = yaml.load(fd)
+        issues = yaml.safe_load(fd)
     log.info("Issues loaded. There are " + str(len(issues)) + " issues listed.")
     return issues
 


=====================================
hosts/jenkins/etc/apt/sources.list deleted
=====================================
@@ -1,11 +0,0 @@
-deb http://deb.debian.org/debian/ bullseye main contrib non-free
-#deb-src http://deb.debian.org/debian/ bullseye main contrib non-free
-
-deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
-#deb-src http://deb.debian.org/debian/ bullseye-updates main contrib non-free
-
-deb http://security.debian.org/debian-security bullseye-security main contrib non-free
-#deb-src http://security.debian.org/debian-security bullseye-security main contrib non-free
-
-deb http://deb.debian.org/debian/ bullseye-backports main contrib non-free
-#deb-src http://deb.debian.org/debian/ bullseye-backports main contrib non-free


=====================================
init_node
=====================================
@@ -30,7 +30,6 @@ adduser --system --shell /bin/bash --home /home/jenkins-adm --ingroup jenkins-ad
 usermod -G jenkins jenkins-adm
 which sudo || apt-get install -y sudo
 echo '%jenkins-adm ALL= NOPASSWD: ALL' > /etc/sudoers.d/jenkins-adm # will be overwritten later
-grep -q bullseye-backports /etc/apt/sources.list || echo "deb http://deb.debian.org/debian/ bullseye-backports main contrib non-free" >> /etc/apt/sources.list
 chown jenkins-adm:jenkins-adm /home/jenkins-adm
 cd ~jenkins-adm
 # the keyring is needed as otherwise update_jdn will fail later


=====================================
job-cfg/reproducible.yaml
=====================================
@@ -282,7 +282,7 @@
             my_task:
                 - 'maintenance':
                     my_description: 'Do some maintenance: check for old files and directories, do backups, update chroots, etc.'
-                    my_timed: '5 0,2,4,6,8,10,12,14,16,18,20,22 * * *'
+                    my_timed: '5 H/3 * * *'
             my_shell: '/srv/jenkins/bin/reproducible_maintenance.sh'
             my_hname:
                 - 'jenkins':        { my_arch: 'amd64' }
@@ -303,7 +303,7 @@
             my_task:
                 - 'maintenance':
                     my_description: 'Do some maintenance: check for old files and directories, do backups, update chroots, etc.'
-                    my_timed: '5 0,2,4,6,8,10,12,14,16,18,20,22 * * *'
+                    my_timed: '5 H/3 * * *'
             my_hname:
                 - 'cbxi4a':         { my_arch: 'armhf' }
                 - 'cbxi4b':         { my_arch: 'armhf' }


=====================================
update_jdn.sh
=====================================
@@ -371,22 +371,6 @@ if [ -f /etc/debian_version ] ; then
 				munin-plugins-extra/buster-backports
 				devscripts/buster-backports
 				" ;;
-			jenkins) # packages to be installed on bullseye Debian systems but which are not available in Ubuntu 18.04:
-				DEBS="$DEBS
-				bind9-dnsutils
-				btop/bullseye-backports
-				fasttrack-archive-keyring
-				foot-terminfo
-				ripgrep
-				lz4
-				monitoring-plugins-contrib/bullseye-backports
-				mmdebstrap
-				munin-node/bullseye-backports
-				munin-plugins-core/bullseye-backports
-				munin-plugins-extra/bullseye-backports
-				devscripts/bullseye-backports
-				debootstrap/bullseye-backports
-				" ;;
 			*)	# packages to be installed on bookworm Debian systems but which are not available in Ubuntu 18.04:
 				DEBS="$DEBS
 				btop
@@ -552,8 +536,6 @@ if [ -f /etc/debian_version ] ; then
 		#fi
 		# only on main node
 		if [ "$HOSTNAME" = "jenkins" ] ; then
-			# required by _db_backup, but not available in bullseye.  https://bugs.debian.org/970870
-			#postgresql-autodoc
 			MASTERDEBS=" 
 				apache2 
 				apt-file 
@@ -593,16 +575,17 @@ if [ -f /etc/debian_version ] ; then
 				moreutils 
 				mr 
 				mtr-tiny 
-				munin/bullseye-backports
+				munin
 				ntp
 				obfs4proxy
 				openbios-ppc 
 				openbios-sparc 
-				openjdk-11-jre-headless
+				openjdk-17-jre-headless
 				pandoc
 				po4a
 				postgresql
 				postgresql-client
+				postgresql-autodoc
 				poxml 
 				procmail 
 				python3-debian 
@@ -612,8 +595,6 @@ if [ -f /etc/debian_version ] ; then
 				python3-sqlalchemy
 				python3-xdg
 				python3-yaml
-				qemu 
-				qemu-kvm 
 				qemu-system-x86 
 				qemu-user-static 
 				radvd
@@ -648,7 +629,7 @@ if [ -f /etc/debian_version ] ; then
 		# - this is done as a seperate step as bpo kernels are frequently uninstallable when upgraded on bpo
 		if [ "$HOSTNAME" = "ionos5-amd64" ] || [ "$HOSTNAME" = "ionos15-amd64" ] \
 			|| [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
-			sudo apt install linux-image-amd64/bullseye-backports || true # backport kernels are frequently uninstallable...
+			sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
 			:
 		elif [ "$HOSTNAME" = "ionos6-i386" ] || [ "$HOSTNAME" = "ionos16-i386" ] ; then
 			# run with the amd64 kernel in these i386 nodes
@@ -658,7 +639,7 @@ if [ -f /etc/debian_version ] ; then
 			sudo apt install linux-image-686-pae linux-image-amd64-
 		elif [ "$HOSTNAME" = "osuosl1-amd64" ] || [ "$HOSTNAME" = "osuosl2-amd64" ] ; then
 			# Arch Linux builds latest stuff which sometimes (eg, currently Qt) needs newer kernel to build...
-			#sudo apt install linux-image-amd64/bullseye-backports || true # backport kernels are frequently uninstallable...
+			sudo apt install linux-image-amd64/bookworm-backports || true # backport kernels are frequently uninstallable...
 			:
 		fi
 		# don't (re-)install pbuilder if it's on hold
@@ -770,13 +751,13 @@ if ! $UP2DATE ; then
 						sudo adduser jenkins docker
 						# openqa does not use slirpvde: reset its status and disable
 						sudo systemctl reset-failed openqa-slirpvde.service
-						sudo systemctl disable openqa-slirpvde.service
-						OPENQA_WORKERS_TO_RUN=6
+						sudo systemctl disable --now openqa-slirpvde.service
+						OPENQA_WORKERS_TO_RUN=5
 						# disable any excess autostart instances, and all non-autostart workers
 						for id in $(systemctl show openqa-worker\*@\*.service --property=Id | sed 's/^Id=//'); do
 							autostart_instance=$(sed -nE 's/^.*auto-restart[@]([0-9]+).*$/\1/p' <<<$id)
 							if [ -z "$autostart_instance" ] || [ "$autostart_instance" -gt "$OPENQA_WORKERS_TO_RUN" ]; then
-								sudo systemctl disable $id
+								sudo systemctl disable --now $id
 							fi
 						done
 						# if some required instances are not yet enabled, enable them



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/ef0edadb6d5078a1561d9b302144993e5ebb58eb...9f078d22302cf8cc3adbe8297602e782d6d9a853

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/ef0edadb6d5078a1561d9b302144993e5ebb58eb...9f078d22302cf8cc3adbe8297602e782d6d9a853
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20230824/f4c1e022/attachment-0001.htm>

More information about the Qa-jenkins-scm mailing list