[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] reproducible: move the db_backup task to its own job

[Mattia Rizzolo (@mattia)]

Mattia Rizzolo pushed to branch master at Debian QA / jenkins.debian.net


Commits:
02fccbf3 by Mattia Rizzolo at 2021-10-05T14:00:00+02:00
reproducible: move the db_backup task to its own job

this task takes quite a long time even on an idle host, mostly in the
compression (`xz`) stage.  We are talking about hours just doing that
when the host is busy.

That precludes and delays the rest of the maintenance tasks, that are
often more relevant and somewhat time-sensitive (like updating the
apt-venv lists).

Signed-off-by: Mattia Rizzolo <mattia at debian.org>

- - - - -


3 changed files:

- + bin/reproducible_db_backup.sh
- bin/reproducible_maintenance.sh
- job-cfg/reproducible.yaml


Changes:

=====================================
bin/reproducible_db_backup.sh
=====================================
@@ -0,0 +1,50 @@
+#!/bin/bash
+# vim: set noexpandtab:
+
+# Copyright 2014-2021 Holger Levsen <holger at layer-acht.org>
+#         © 2015-2021 Mattia Rizzolo <mattia at debian.org>
+# released under the GPLv2
+
+DEBUG=false
+. /srv/jenkins/bin/common-functions.sh
+common_init "$@"
+
+# common code for tests.reproducible-builds.org
+. /srv/jenkins/bin/reproducible_common.sh
+
+#
+# we fail hard
+#
+set -euo pipefail
+
+# backup db
+if [ "$HOSTNAME" = "$MAINNODE" ] ; then
+	echo "$(date -u) - backup db and update public copy."
+	# prepare backup
+	BACKUPDIR=$BASE/debian/backups
+	mkdir -p $BACKUPDIR
+
+	# keep 30 days and the 1st of the month
+	DAY=$(date -d "30 day ago" '+%d')
+	DATE=$(date -d "30 day ago" '+%Y-%m-%d')
+	BACKUPFILE="$BACKUPDIR/reproducible_$DATE.sql.xz"
+	if [ "$DAY" != "01" ] &&  [ -f "$BACKUPFILE" ] ; then
+		rm -f "$BACKUPFILE"
+	fi
+
+	# Make a daily backup of database
+	DATE=$(date '+%Y-%m-%d')
+	BACKUPFILE="$BACKUPDIR/reproducible_$DATE.sql"
+	if [ ! -f "$BACKUPFILE.xz" ] ; then
+		# make the backup
+		DATE=$(date '+%Y-%m-%d')
+		pg_dump -x -O "$PGDATABASE" > "$BACKUPFILE"
+		xz "$BACKUPFILE"
+
+		# make the backup public
+		ln -s -f "$BACKUPFILE.xz" $BASE/reproducible.sql.xz
+
+		# recreate documentation of database
+		postgresql_autodoc -d "$PGDATABASE" -t html -f "$BASE/reproducibledb"
+	fi
+fi


=====================================
bin/reproducible_maintenance.sh
=====================================
@@ -2,7 +2,7 @@
 # vim: set noexpandtab:
 
 # Copyright 2014-2021 Holger Levsen <holger at layer-acht.org>
-#         © 2015-2018 Mattia Rizzolo <mattia at debian.org>
+#         © 2015-2021 Mattia Rizzolo <mattia at debian.org>
 # released under the GPLv2
 
 DEBUG=false
@@ -26,38 +26,6 @@ query_to_print() {
 DIRTY=false
 REP_RESULTS=/srv/reproducible-results
 
-# backup db
-if [ "$HOSTNAME" = "$MAINNODE" ] ; then
-	echo "$(date -u) - backup db and update public copy."
-	# prepare backup
-	BACKUPDIR=$BASE/debian/backups/
-	mkdir -p $BACKUPDIR
-
-	# keep 30 days and the 1st of the month
-	DAY=$(date -d "30 day ago" '+%d')
-	DATE=$(date -d "30 day ago" '+%Y-%m-%d')
-	BACKUPFILE="$BACKUPDIR/reproducible_$DATE.sql.xz"
-	if [ "$DAY" != "01" ] &&  [ -f "$BACKUPFILE" ] ; then
-		rm -f "$BACKUPFILE"
-	fi
-
-	# Make a daily backup of database
-	DATE=$(date '+%Y-%m-%d')
-	BACKUPFILE="$BACKUPDIR/reproducible_$DATE.sql"
-	if [ ! -f $BACKUPFILE.xz ] ; then
-		# make the backup
-		DATE=$(date '+%Y-%m-%d')
-		pg_dump -x -O $PGDATABASE > "$BACKUPFILE"
-		xz "$BACKUPFILE"
-
-		# make the backup public
-		ln -s -f "$BACKUPFILE.xz" $BASE/reproducible.sql.xz
-
-		# recreate documentation of database
-		postgresql_autodoc -d $PGDATABASE -t html -f "$BASE/reproducibledb"
-	fi
-fi
-
 #
 # find too large files in /var/log
 #


=====================================
job-cfg/reproducible.yaml
=====================================
@@ -531,6 +531,10 @@
 
         - '{name}_{my_task}':
             my_task:
+                - 'db_backup':
+                    my_description: Build a daily backup of the database
+                    my_timed: H H * * *
+                    my_ shellext: .sh
                 - 'scheduler':
                     my_description: 'Schedule packages to be tested for reproducibility.'
                     # dinstall runs 1|7|13|19:52 UTC - see https://salsa.debian.org/ftp-team/dak/blob/master/config/debian/crontab



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/02fccbf3b2ab930e1a9c8b12621283af1bcbaa7f

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/02fccbf3b2ab930e1a9c8b12621283af1bcbaa7f
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20211005/9ed4db16/attachment-0001.htm>