[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] jenkins nodes: fixup bc7ef0ea: use iostat and iostat_ios munin plugins on jenkins and osuosl4+5

[Holger Levsen (@holger)]

Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
bc1bd2c9 by Holger Levsen at 2023-04-09T20:38:43+02:00
jenkins nodes: fixup bc7ef0ea: use iostat and iostat_ios munin plugins on jenkins and osuosl4+5

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


2 changed files:

- TODO
- update_jdn.sh


Changes:

=====================================
TODO
=====================================
@@ -20,20 +20,15 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 
 * replace amd64 in scripts with $HOSTARCH
 * extend /etc/rc.local to do cleanup of lockfiles
-* explain in README how to write jobs, e.g. which paths are on tmpfs
-** EXECUTOR_NUMBER for X
 * run all bash scripts with set -u and set -o pipefail: http://redsymbol.net/articles/unofficial-bash-strict-mode/
-* teach bin/chroot-*.sh and bin/d-i_build.sh how to nicely deal with network problems… (as both reproducible_build.sh and schroot-create.sh do)
-* salsa stuff:
-** triggers for d-i repos are missing
 
 === 2023 things
 
 * discouple t.r-b.o from jenkins.d.n (https hosting wise), maybe even www.r-b.o
 * rename jenkins to hudson? (the hostname)
-* maint cleanup after x days
-** rm /var/lib/schroot/unpack/d-i-manual*
-** /var/lib/schroot/unpack/jenkins-*/var/cache/apt/archives not empty, rather 96GB wasted atm
+* maintenance job: rm /var/lib/schroot/unpack/d-i-manual* older than 3 days
+* /var/lib/schroot/unpack/jenkins-*/var/cache/apt/archives not empty, rather 96GB wasted atm
+* split TODO in TODO and TODO.legacy?
 
 === nodes at OSUOSL
 
@@ -59,43 +54,6 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 <Ramereth> we currently have it running all the time on all physical nodes
 ----
 
-=== Debian rebuilder 'thing'
-
-(these notes are mostly obsolete now.)
-
-* prototype only dealing with bullseye and amd64 (and arch all packages)
-** 1st step: download a specific .buildinfo file and sbuild it - done
-** 2nd step: choose a random package from bullseye and fetch the .buildinfo file from builtin-pho and build it - done
-*** done: deal with unsigned .buildinfo files, some buildds published a lot of those
-*** next: file a wishlist bug for the DIST_BASE= feature...
-*** next: file a normal bug for the unclear error messages regarding unavailable build-depends
-*** next: file a wishlist for downloading the source and verifying it against the .buildinfo file
-*** next: file a normal bug about generating a wrong cmdline for binNMU rebuilds and failing to modify changelog. (block this bug by the download sources feature)
-** 3rd step: put result in db in ionos7: search for NEXT in the _prototype.sh
-*** prelimanary db:
-**** buildinfofileurl, source, source version, binary version, binary, rebuild_status, rebuildlogurl
-** 4th step: have some scheduler job and a rebuilder job picking up tasks
-*** scheduler on ionos7
-*** rebuilds on osuosl3
-** 5th step: have a smarter scheduler
-** 6th step: export results as json and html
-* run jenkins job on ionos7 (=buildinfos.d.n), which triggers builds on osuosl3 and stores the results on ionos7
-** keep data in postgresql|sqlite on ionos7
-*** ionos7 also has the builtin-pho db so it seems natural to join them...
-** export data as .json via https
-** table with these fields:
-*** src-pkg, binary-pkg, sha1 of .deb on ftp.d.o, sha1 of .buildinfo file (signed), sha1 of .buildinfo file (unsigned or signature stripped), signature type, name of .buildinfo file (from ftp.d.o)
-*** and either architecture of .deb or filename of .deb (as that includes the architecture)
-*** new result: NBIFA - no .buildinfo file available
-* another job to import data on jenkins (into postgresql)
-* schedule/trigger rebuilds on osuosl3, using data in postgresql|sqlite on ionos7
-** scheduler unclear, we'll first try to rebuild everything in current sid (and bullseye and buster eventually) once, and then, once we've done this, we want to do this X times again. (and there will be ftbfs etc, not causing rebuilds immediately.)
-** rebuild amd64 only at first (but arm soon after), but .buildinfo files for all Debian architectures should be tracked in the db
-** also treat the base suite archive and its security, update and backports archives separately
-*** store hash of processed Packages file and only process a Packages file if it has an unknown hash -> good for suites which are not updated often
-*** ignore security archive as those .buildinfo files are not published yet (#862538)
-** good examples: piuparts (arch and any), d-e-i (udeb), d-e-c (all only), emacs (epoch) libunibreak (binNMU)
-
 
 ==== proper backup
 
@@ -110,11 +68,6 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
 * '/srv/jenkins.debian.net-scm-sync.git'
 * '/etc/.git' and '/etc'
 
-=== To be done once bugs are fixed
-
-* link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu
-* link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios
-
 === jenkins-job-builder related
 
 * yaml could be refactored, lots of duplication in there. this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them)


=====================================
update_jdn.sh
=====================================
@@ -736,7 +736,7 @@ if ! $UP2DATE || [ $BASEDIR/hosts/$HOSTNAME/etc/munin -nt $STAMP ] ; then
 		[ -e $i ] || sudo ln -s /usr/share/munin/plugins/$i $i
 	done
 	# delete some everywhere
-	sudo rm -f postfix_* open_inodes interrupts irqstats threads proc_pri vmstat if_err_* exim_* netstat fw_forwarded_local fw_packets forks open_files users nfs* iostat_ios ntp* df_abs entropy 2>/dev/null
+	sudo rm -f postfix_* open_inodes interrupts irqstats threads proc_pri vmstat if_err_* exim_* netstat fw_forwarded_local fw_packets forks open_files users nfs* ntp* df_abs entropy 2>/dev/null
 	case $HOSTNAME in
 			ionos1-a*|ionos10*|codethink16*|osuosl*) [ -L /etc/munin/plugins/squid_cache ] || for i in squid_cache squid_objectsize squid_requests squid_traffic ; do sudo ln -s /usr/share/munin/plugins/$i $i ; done ;;
 			*)	;;
@@ -747,7 +747,9 @@ if ! $UP2DATE || [ $BASEDIR/hosts/$HOSTNAME/etc/munin -nt $STAMP ] ; then
 	esac
 	case $HOSTNAME in
 		jenkins|osuosl4-amd64|osuosl5-amd64) 	: ;;
-		*)					[ ! -e /etc/munin/plugins/iostat ] || sudo rm /etc/munin/plugins/iostat ;;
+		*)					[ ! -e /etc/munin/plugins/iostat ] || sudo rm /etc/munin/plugins/iostat
+							[ ! -e /etc/munin/plugins/iostat_ios ] || sudo rm /etc/munin/plugins/iostat_ios ;;
+
 	esac
 	fi
 	if ( [ "$HOSTNAME" = "jenkins" ] || [ "$HOSTNAME" = "ionos7-amd64" ] ) && [ ! -L /etc/munin/plugins/apache_accesses ] ; then



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/bc1bd2c95fc1598083757c2ff5db2ab8d06f24d1

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/bc1bd2c95fc1598083757c2ff5db2ab8d06f24d1
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20230409/d2b95a77/attachment-0001.htm>