[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] 3 commits: reproducible Archlinux: prepare to delete removed packages from our "db"

[Holger Levsen]

Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
0751d42a by Holger Levsen at 2018-09-17T15:35:20Z
reproducible Archlinux: prepare to delete removed packages from our "db"

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
a6face28 by Holger Levsen at 2018-09-17T15:35:42Z
reproducible Archlinux TODO: some updates

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -
69a2dc7a by Holger Levsen at 2018-09-17T15:37:54Z
reproducible Archlinux: actually remove packages which are gone

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


2 changed files:

- TODO
- bin/reproducible_archlinux_scheduler.sh


Changes:

=====================================
TODO
=====================================
@@ -313,20 +313,18 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 ** needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)
 ** use schroot tarballs (gzipped), moves are atomic then
 * things to be done before enabling more builders:
-** build in /srv/workspace instead of /tmp
+** build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)
 * only disable cert checking on the node running in the future
 * we should do kernel variation
 * compare the just built pkg.tar.xz with the one available on the arch mirrors. *then* one can truely say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."
 * maintenance job:
 ** check for archlinux schroot sessions which should not be there and delete them. complain if that fails.
-** watch /tmp or rather where this will end up…
 
 * scheduler
 ** could be improved with a queue in a directory:
 *** /scheduled/pkg(s) - mv || exit 0
 *** loop to schedule untested packages
 *** loop to schedule tested packages with higher versions (compare for = first)
-** or merge mapreri branch first!
 
 * move html to build.sh in a function
 


=====================================
bin/reproducible_archlinux_scheduler.sh
=====================================
@@ -33,6 +33,26 @@ update_archlinux_repositories() {
 	TOTAL=$(cat ${ARCHLINUX_PKGS}_full_pkgbase_list | wc -l)
 	echo "$(date -u ) - $TOTAL Arch Linux packages are known in total to us."
 
+	# remove packages which are gone (only when run between 21:00 and 23:59)
+	if [ $(date +'%H') -gt 21 ] ; then
+		REMOVED=0
+		REMOVE_LIST=""
+		for REPO in $ARCHLINUX_REPOS ; do
+			echo "$(date -u ) - Dropping removed packages from repository '$REPO':"
+			for i in $(find $BASE/archlinux/$REPO -type d -wholename "$BASE/archlinux/$REPO/*" | sort) ; do
+				PKG=$(basename $i)
+				if ! grep -q "$REPO $PKG" ${ARCHLINUX_PKGS}_full_pkgbase_list > /dev/null ; then
+					let REMOVED=$REMOVED+1
+					REMOVE_LIST="$REMOVE_LIST $REPO/$PKG"
+					rm -r --one-file-system $BASE/archlinux/$REPO/$PKG
+				        echo "$REPO/$PKG removed as it's gone from the Archlinux repositories."
+				fi
+			done
+		done
+		echo "$(date -u ) - deleted $REMOVED packages: $REMOVE_LIST"
+	fi
+
+	# schedule packages
 	for REPO in $ARCHLINUX_REPOS ; do
 		TMPPKGLIST=$(mktemp -t archlinuxrb-scheduler-XXXXXXXX)
 		echo "$(date -u ) - updating list of available packages in repository '$REPO'."



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/compare/aa872f0fe0c13e202ef5729a134848aa7ff490e1...69a2dc7aff5611bded0d05cda84d09add53d4972

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/compare/aa872f0fe0c13e202ef5729a134848aa7ff490e1...69a2dc7aff5611bded0d05cda84d09add53d4972
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20180917/0a39e802/attachment-0001.html>