[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][rb-gcc-binutils-extended-timeout] 12 commits: openqa: Disable openqa-slirpvde.service
Vagrant Cascadian (@vagrant)
gitlab at salsa.debian.org
Mon Apr 10 22:01:23 BST 2023
Vagrant Cascadian pushed to branch rb-gcc-binutils-extended-timeout at Debian QA / jenkins.debian.net
Commits:
1d119713 by Roland Clobus at 2023-04-10T12:03:01+02:00
openqa: Disable openqa-slirpvde.service
The slirpvde service is not used
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
762ff4f1 by Holger Levsen at 2023-04-10T12:16:42+02:00
djm: add support for marking nodes as online again
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
a2682c74 by Holger Levsen at 2023-04-10T16:11:34+02:00
jenkins: increase amount of (most) reproducible* job logs to be kept to 150
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
245ed906 by Holger Levsen at 2023-04-10T16:19:16+02:00
jenkins: don't complain about reprotest jobs existence
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
f8595599 by Holger Levsen at 2023-04-10T16:19:56+02:00
djm: fixup 762ff4f10: add support for marking nodes as online again
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
610b2214 by Holger Levsen at 2023-04-10T18:15:39+02:00
djm: introduce djm-jenkins-parser.sh, to parse jenkins build logs for manually scheduled job runs and document them
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
b1e5a412 by Holger Levsen at 2023-04-10T19:09:48+02:00
djm-logparser: also include statistics about the jobs triggered manually
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
4cf4e2c5 by Holger Levsen at 2023-04-10T19:40:32+02:00
djm: refactoring, djm-logparser now runs djm-jenkins-parser remotely first
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
1eea05db by Holger Levsen at 2023-04-10T19:43:49+02:00
djm: actually rename djm-jenkins-parser
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
d97ccbda by Holger Levsen at 2023-04-10T20:14:45+02:00
djm: minor fixes
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
67a02dcd by Vagrant Cascadian at 2023-04-10T13:56:27-07:00
Merge remote-tracking branch 'origin/master' into rb-gcc-binutils-extended-timeout
- - - - -
04a00c7e by Vagrant Cascadian at 2023-04-10T14:00:14-07:00
Use PBUILDERTIMEOUT variable and make timeouts smaller to fix within
the 48 hour cleanup windows.
- - - - -
7 changed files:
- bin/common-functions.sh
- bin/djm
- + bin/djm-jenkins-parser
- bin/djm-logparser
- bin/reproducible_build.sh
- job-cfg/reproducible.yaml
- update_jdn.sh
Changes:
=====================================
bin/common-functions.sh
=====================================
@@ -233,7 +233,7 @@ jenkins_zombie_check() {
# related however is that I issued a reboot (via running
# /sbin/reboot) right before the zombies appeared...
#
- ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reprotest|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot' || true)"
+ ZOMBIES="$(ls -1d /var/lib/jenkins/jobs/* | grep -E 'strip-nondeterminism|reproducible_(builder_(amd64|i386|armhf|arm64)|setup_(pbuilder|schroot)_testing)|chroot-installation_wheezy|aptdpkg|stretch_install_education-thin-client-server|jessie_multiarch_versionskew|dpkg_stretch_find_trigger_cycles|sid_install_education-services|buster_install_education-services|lvc|chroot-installation_stretch_.*_upgrade_to_sid|chroot-installation_buster_.*_upgrade_to_sid|piuparts_.*_jessie|udd_stretch|d-i_pu-build|debsums-tests_stretch|debian-archive-keyring-tests_stretch|chroot-installation_jessie|chroot-installation_.*education-lang-|kirkwoot|rebootstrap_.*_gcc[5-9]($|_)|rebootstrap_.*_gcc1[01]($|_)|brcm47xx|rebootstrap_kfreebsd|diffoscope_from_git_|disorderfs_from_git_master|diffoscope_pypi|diffoscope_freebsd|diffoscope_netbsd|diffoscope_macports|diffoscope_archlinux|openwrt-target-ath97|profitbricks|pool_buildinfos_suites|g-i-installation|reproducible_compare_Debian_sha1sums|bbx15|cb3a|ff2a|ff2b|jtk1a|jtk1b|odxu4a|odxu4b|odu3a|opi2a|opi2c|p64b|p64c|ar71xx|reproducible_debian_live_build$|chroot-installation_stretch|chroot-installation_bullseye*upgrade_to_sid|rebuilder_prototype|osuosl167|osuosl168|osuosl169|osuosl170|osuosl171|osuosl172|osuosl173|osuosl174|osuosl184fakeroot-foreign|fdroid|reproducible_.*_reproducible?$|health_check_amd64_snapshot' || true)"
if [ -n "$ZOMBIES" ] ; then
DIRTY=true
figlet 'zombies!!!'
=====================================
bin/djm
=====================================
@@ -45,6 +45,7 @@ case $ACTION in
a|autoremove) ACTION=autoremove ;;
s|shell) ACTION=shell ;;
j|jenkins-ui) ACTION=jenkins-ui ;;
+ b|bring-back) ACTION=bring-back ;;
*) FAIL_REASON="${FAIL_REASON}Undefined action.\n"
;;
esac
@@ -87,7 +88,8 @@ if [ -n "$FAIL_REASON" ] ; then
echo
echo "TARGET=\$1 'all' or grepable (jenkins, amd64, ionos, osuosl3) from ./nodes/list_nodes"
echo " or special target '.'"
- echo "ACTION=\$2 r_eboot, _p_owercycle, _a_utoremove, _s_shell (for manual maintenance); _j_enkins-ui"
+ echo "ACTION=\$2 r_eboot, _p_owercycle, _a_utoremove, _s_shell (for manual maintenance),"
+ echo " _j_enkins-ui or _b_ring-back"
echo "REASON=\$3 ku=kernel update, np=no pings, dj=debug job problems, rm=regular maintenance,"
echo " cs=clock skew, hb=host back, ho=host overloaded"
# TODO
@@ -219,6 +221,11 @@ EOF
;;
jenkins-ui) echo "Documenting one manual action in the jenkins-ui."
;;
+ bring-back) TMPFILE=$(mktemp -u)
+ OFF_NODES=~jenkins/offline_nodes
+ echo "Trying to mark $SHORTNODE as online. This is the diff for $OFF_NODES on jenkins.debian.net:"
+ ssh jenkins.debian.net "grep -v $SHORTNODE $OFF_NODES > $TMPFILE ; diff $OFF_NODES $TMPFILE ; mv $TMPFILE $OFF_NODES ; chown jenkins:jenkins $OFF_NODES"
+ ;;
*) echo unsupported action, sorry, please improve.
exit 1
;;
=====================================
bin/djm-jenkins-parser
=====================================
@@ -0,0 +1,69 @@
+#!/bin/bash
+# vim: set noexpandtab:
+
+#
+# djm - documented jenkins maintenance logparser for jenkin build logs
+#
+# Copyright 2023 Holger Levsen <holger at layer-acht.org>
+# released under the GPLv2
+#
+
+set -e
+set -o pipefail # see eg http://petereisentraut.blogspot.com/2010/11/pipefail.html
+
+# check environment
+if [ -z "$DJM_USER" ] ; then
+ echo "Environment variable DJM_USER must be set. Exiting."
+ exit 1
+fi
+if [ ! -d ~jenkins/jobs ] ; then
+ echo "~jenkins/jobs is not a directory, are you running this on the jenkins host?"
+ exit 1
+fi
+cd ~jenkins/jobs
+
+export TZ="/usr/share/zoneinfo/UTC"
+MYLOG=~/.djm-jenkins-parser.log
+MYRESULTS=~/.djm-jenkins-ui.log
+
+#
+# main
+#
+
+# find recent / unparsed logfiles
+if [ ! -f $MYLOG ] ; then
+ LOGS=$(find ./reproducible_*/builds/*/log 2>/dev/null)
+ ZLOGS=$(find ./reproducible_*/builds/*/log.gz 2>/dev/null)
+ echo "Note: initial run detected, this will take a few minutes."
+ touch $MYLOG
+else
+ if [ $(cat $MYLOG | wc -l) -gt 15000 ] ; then
+ echo "Note: $MYLOG has become very big, maybe time to delete $MYLOG and $MYRESULTS?"
+ fi
+ LOGS=$(find ./reproducible_*/builds/*/log -newer $MYLOG 2>/dev/null)
+ ZLOGS=$(find ./reproducible_*/builds/*/log.gz -newer $MYLOG 2>/dev/null)
+fi
+echo "Parsing $(echo $LOGS $ZLOGS | sed 's# #\n#g' | wc -l) logfiles now."
+
+# find logfiles triggered by $DJM_USER and write correspondig djm logfile entries
+(
+for i in $LOGS $ZLOGS ; do
+ DIRNAME="$(dirname $i)"
+ if [ ! -f $i ] || grep -q "$DIRNAME/log" $MYLOG ; then
+ : # echo $i already processed, continue.
+ else
+ echo "$DIRNAME/log" >> $MYLOG
+ if [ "$(basename $i)" == "log" ] ; then
+ RESULT=$(head -1 $i | grep -E -i "Started by user.*$DJM_USER" || true)
+ else
+ RESULT=$(zcat $i | head -1 | grep -E -i "Started by user.*$DJM_USER" || true)
+ fi
+ if [ -n "$RESULT" ] ; then
+ : # echo $i
+ echo "$(stat -c %w $i | cut -d ':' -f1-2) UTC, jenkins web UI, jenkins-ui, job triggered, $(echo $i | cut -d '/' -f2)"
+ else
+ : #echo autoscheduled: $i
+ fi
+ fi
+done
+ ) | sort | tee -a $MYRESULTS
=====================================
bin/djm-logparser
=====================================
@@ -12,18 +12,34 @@ set -e
set -o pipefail # see eg http://petereisentraut.blogspot.com/2010/11/pipefail.html
#
-# define environment and parse parameters
+# define environment
#
-
-LOGFILE=~/.djm.log
+if [ -z "$DJM_USER" ] ; then
+ echo "Environment variable DJM_USER must be set. Exiting."
+ exit 1
+fi
+LOCAL_LOGFILE=~/.djm.log
+UI_LOGFILE=~/.djm-jenkins-ui.log
LOGMONTH="$(date -u '+%Y-%m')"
-THREE_C="%42s %8s %12s\n"
-TWO_C="%42s %8s\n"
-ONE_C="%42s %8s\n"
+
+#
+# parse jenkins build logs and copy the result over
+#
+ssh $DJM_USER at jenkins.debian.net "DJM_USER=$DJM_USER /srv/jenkins/bin/djm-jenkins-parser"
+scp $DJM_USER at jenkins.debian.net:$(basename $UI_LOGFILE) $UI_LOGFILE
+
+#
+# little helpers
+#
+LOGFILE=$(mktemp)
+cat $LOCAL_LOGFILE $UI_LOGFILE | grep ^$LOGMONTH | sort -u > $LOGFILE
+THREE_C="%42s %12s %12s\n"
+TWO_C="%42s %12s\n"
+ONE_C="%42s %12s\n"
seperator() {
local CHAR="${1:--}"
- for i in $(seq 1 66) ; do
+ for i in $(seq 1 79) ; do
echo -n "$CHAR"
done
echo
@@ -33,7 +49,7 @@ seperator() {
# main
#
-TOTAL=$(grep -c $LOGMONTH $LOGFILE)
+TOTAL=$(grep -c ^$LOGMONTH $LOGFILE)
HOSTS=$(cut -d ',' -f2 $LOGFILE | sort -u | grep -v "web UI"| wc -l)
HOURS=$(cut -b 1-13 $LOGFILE | sort -u | wc -l)
TEN_MIN_SLICES=$(cut -b 1-15 $LOGFILE | sort -u | wc -l)
@@ -46,7 +62,9 @@ ACTIONS=$(cut -d ',' -f3 $LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | c
REASONS=$(cut -d ',' -f4 $LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | cut -d '%' -f2| sed -s "s# #=#g")
seperator "="
-echo "Still very simple statistics for djm logs at $LOGFILE"
+echo "Still very simple statistics for djm logs:"
+echo " - $LOCAL_LOGFILE"
+echo " - $UI_LOGFILE"
echo
echo "djm = documented jenkins maintenance"
echo " jenkins.debian.net exists since 2012, djm since 2023-04."
@@ -83,8 +101,26 @@ for reason in $REASONS ; do
done
seperator "="
-# todo: sort actions and reasons by highest amount
+#
+# do statistics about the jobs triggered
+#
+printf "$ONE_C" "jobs triggered:"
+printf "$ONE_C" "---------------"
+(
+ PATTERNS="setup_schroot debian_live_build maintenance node_health_check html openwrt"
+ for i in $PATTERNS ; do
+ printf "$TWO_C" "$(grep -c reproducible_$i $LOGFILE)" "reproducible_${i}_.*"
+ done
+ PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
+ for i in $(cut -d ',' -f5 $LOGFILE | grep -v -E $PATTERNS|sort -u) ; do
+ printf "$TWO_C" "$(grep -c $i $LOGFILE)" "$i"
+ done
+) | sort -n -r
+seperator "="
+
# todo: include the number of reproducible related jenkins jobs
# todo: include notmuch mails
# - received but read
# - written
+
+rm $LOGFILE
=====================================
bin/reproducible_build.sh
=====================================
@@ -645,12 +645,12 @@ EOF
esac
fi
case "${SRCPACKAGE}" in
- gcc-*|binutils) timeout=24 ;;
- *) timeout=18 ;;
+ gcc-*|binutils) PBUILDERTTIMEOUT=19 ;;
+ *) PBUILDERTIMEOUT=18 ;;
esac
set +e
# remember to change the sudoers setting if you change the following command
- sudo timeout -k ${timeout}.1h ${timeout}h /usr/bin/ionice -c 3 /usr/bin/nice \
+ sudo timeout -k ${PBUILDERTIMEOUT}.1h ${PBUILDERTIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice \
/usr/sbin/pbuilder --build \
--configfile $TMPCFG \
--distribution "$SUITE" \
@@ -667,7 +667,7 @@ EOF
rm $TMPCFG
case $PRESULT in
124)
- echo "$(date -u) - pbuilder was killed by timeout after ${timeout}h." | tee -a b1/build.log
+ echo "$(date -u) - pbuilder was killed by timeout after ${PBUILDERTIMEOUT}h." | tee -a b1/build.log
exit 3
;;
1) # FTBFS, for whatever reason.
@@ -768,14 +768,14 @@ EOF
esac
fi
case "${SRCPACKAGE}" in
- gcc-*|binutils) timeout=28 ;;
- *) timeout=24 ;;
+ gcc-*|binutils) PBUILDERTIMEOUT=26 ;;
+ *) PBUILDERTIMEOUT=24 ;;
esac
set +e
# remember to change the sudoers setting if you change the following command
# (the 2nd build gets a longer timeout trying to make sure the first build
# aint wasted when then 2nd happens on a highly loaded node)
- sudo timeout -k ${timeout}.1h ${timeout}h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 \
+ sudo timeout -k ${PBUILDERTIMEOUT}.1h ${PBUILDERTIMEOUT}h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 \
/usr/bin/unshare --uts -- \
/usr/sbin/pbuilder --build \
--configfile $TMPCFG \
@@ -794,7 +794,7 @@ EOF
rm $TMPCFG
case $PRESULT in
124)
- echo "$(date -u) - pbuilder was killed by timeout after ${timeout}h." | tee -a b2/build.log
+ echo "$(date -u) - pbuilder was killed by timeout after ${PBUILDERTIMEOUT}h." | tee -a b2/build.log
exit 3
;;
1) # FTBFS, for whatever reason.
=====================================
job-cfg/reproducible.yaml
=====================================
@@ -23,7 +23,7 @@
- timed: '{my_timed}'
logrotate: &logrotate
daysToKeep: 31
- numToKeep: 50
+ numToKeep: 150
artifactDaysToKeep: -1
artifactNumToKeep: -1
builders:
=====================================
update_jdn.sh
=====================================
@@ -769,6 +769,9 @@ if ! $UP2DATE ; then
;;
osuosl3-amd64) sudo adduser jenkins sbuild
sudo adduser jenkins docker
+ # openqa does not use slirpvde: reset its status and disable
+ sudo systemctl reset-failed openqa-slirpvde.service
+ sudo systemctl disable openqa-slirpvde.service
;;
osuosl4-amd64) # allow vagrant to use libvirt for debugging
sudo adduser vagrant libvirt
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/c4557c597803dda4fad1ea8d47bb6019298d7c83...04a00c7e4273b1f65f184e414b47b3299ad71051
--
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/c4557c597803dda4fad1ea8d47bb6019298d7c83...04a00c7e4273b1f65f184e414b47b3299ad71051
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20230410/8a866a7e/attachment-0001.htm>
More information about the Qa-jenkins-scm
mailing list