[Qa-jenkins-scm] [Git][qa/jenkins.debian.net][master] 5 commits: djm: refactoring, merge djm-logparser into djm
Holger Levsen (@holger)
gitlab at salsa.debian.org
Mon May 15 14:41:09 BST 2023
Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net
Commits:
d54fc55c by Holger Levsen at 2023-05-15T15:12:45+02:00
djm: refactoring, merge djm-logparser into djm
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
9ca3c343 by Holger Levsen at 2023-05-15T15:19:11+02:00
djm: refactoring
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
beb89ee0 by Holger Levsen at 2023-05-15T15:25:06+02:00
djm: add --verbose option (to make --report less verbose by default)
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
ee267c9f by Holger Levsen at 2023-05-15T15:35:11+02:00
djm: automatically fetch logs if logs older than 18h
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3c3dae18 by Holger Levsen at 2023-05-15T15:40:50+02:00
djm: add --fetch option to force fetching logs
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
3 changed files:
- TODO
- bin/djm
- − bin/djm-logparser
Changes:
=====================================
TODO
=====================================
@@ -40,17 +40,16 @@ See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian
** cleanup/backup logs on the first of the month
** document running "mosh jenkins shell-monitor" processes :) (same with my_btop to be renamed and git added...)
** create osuosl tickets?
-** launch "comeback terminal" after powercycling nodes like after rebooting nodes
** document jenkins plugin updates -> manually / though my local script which calls the webpages can log this
** document other jenkins ui actions. are there any?
* djm-logparser:
-** merge into djm --report and --show-month 04
+** new feature: --show-month 04
** include hours with manual jobs triggered
** include notmuch mails, "received and read". write djm-notmuch-parser.
+** include irc. (timeslices when one said something on relevant channels)
** include amount of job runs on jenkins
-* djm-jenknis-parser:
-** should be called from djm (if local log is older than 12h) and not from djm-logparser (so that it's enough to run djm-logparser once a month or so)
-** cleanup/backup logs on the first of the month
+* djm-fetch-logs
+** backup+cleanup logs on the first of the month, also automatically
=== nodes at OSUOSL
=====================================
bin/djm
=====================================
@@ -16,6 +16,8 @@ set -o pipefail # see eg http://petereisentraut.blogspot.com/2010/11/pipefail.h
#
MODE=djm
DRY_MODE=false
+VERBOSE=false
+FETCH=false
LOGFILE=~/.djm.log
FAIL_REASON=""
CONFIRM=false
@@ -23,13 +25,34 @@ BG=""
TARGET=
ACTION=
REASON=
+# djm-logparser
+LOCAL_LOGFILE=~/.djm.log
+UI_LOGFILE=~/.djm-jenkins-ui.log
+PARSER_LOGFILE=~/.djm-jenkins-parser.log
+LOGMONTH="$(date -u '+%Y-%m')"
+JOBS=~/.djm-jobs.txt
+if [ -z "$DJM_USER" ] ; then
+ if [ -n "$DEBFULLNAME" ] ; then
+ DJM_USER="$(echo $DEBFULLNAME | cut -d ' ' -f1 | tr '[:upper:]' '[:lower:]')"
+ else
+ echo "Environment variable DJM_USER (or DEBFULLNAME) must be set. Exiting."
+ exit 1
+ fi
+fi
+
+#
+# little helpers
+#
+THREE_C="%42s %12s %12s\n"
+TWO_C="%42s %12s\n"
+ONE_C="%42s %12s\n"
#
# parse parameters
#
# thanks to /usr/share/doc/util-linux/examples/getopt-parse.bash
-TEMP=$(getopt -o 'd' --long 'dry-run' -n 'djm' -- "$@")
+TEMP=$(getopt -o 'dfrv' --long 'dry-run,fetch,report,verbose' -n 'djm' -- "$@")
if [ $? -ne 0 ]; then
echo 'Terminating...' >&2
exit 1
@@ -40,26 +63,163 @@ unset TEMP
while true; do
case "$1" in
'-d'|'--dry-run')
- echo 'Option a'
DRY_MODE=true
shift
continue
- ;;
+ ;;
+ '-f'|'--fetch')
+ FETCH=true
+ shift
+ continue
+ ;;
+ '-r'|'--report')
+ MODE=report
+ shift
+ continue
+ ;;
+ '-v'|'--verbose')
+ VERBOSE=true
+ shift
+ continue
+ ;;
'--')
shift
break
- ;;
+ ;;
*)
echo 'Internal error!' >&2
exit 1
- ;;
+ ;;
esac
done
-#
-# some helper functions
-#
+seperator() {
+ local CHAR="${1:--}"
+ for i in $(seq 1 79) ; do
+ echo -n "$CHAR"
+ done
+ echo
+}
+
+printf_if_not_zero() {
+ if [ $2 -ne 0 ] ; then
+ printf "$1" "$2" "$3"
+ fi
+}
+
+printf_heading() {
+ printf "$ONE_C" "$1"
+ printf "$ONE_C" "$(for i in $(seq 1 ${#1}) ; do echo -n '-' ; done)"
+}
+
+djm_fetch_logs() {
+ #
+ # parse jenkins build logs and copy the result over
+ #
+ ssh $DJM_USER at jenkins.debian.net "DJM_USER=$DJM_USER /srv/jenkins/bin/djm-jenkins-parser"
+ scp $DJM_USER at jenkins.debian.net:$(basename $UI_LOGFILE) $UI_LOGFILE
+ scp $DJM_USER at jenkins.debian.net:$(basename $PARSER_LOGFILE) $PARSER_LOGFILE
+ ssh $DJM_USER at jenkins.debian.net "cd ~jenkins/jobs ; ls -1d reproducible_* |wc -l" > $JOBS
+}
+
+djm_logparser() {
+ TMP_LOGFILE=$(mktemp)
+ cat $LOCAL_LOGFILE $UI_LOGFILE | grep ^$LOGMONTH | sort -u > $TMP_LOGFILE
+ TOTAL=$(grep -c ^$LOGMONTH $TMP_LOGFILE)
+ HOSTS=$(cut -d ',' -f2 $TMP_LOGFILE | sort -u | grep -v "web UI"| wc -l)
+ HOURS=$(cut -b 1-13 $TMP_LOGFILE | sort -u | wc -l)
+ TEN_MIN_SLICES=$(cut -b 1-15 $TMP_LOGFILE | sort -u | wc -l)
+ TWENTY_MIN_SLICES=$(( TEN_MIN_SLICES + TEN_MIN_SLICES ))
+ SLICES_AS_HOURS=$(($TWENTY_MIN_SLICES/3))
+ # create sorted list of actions/reasons (sorted by amount of occurances)
+ # the character '%" is solely used as a delimeter for cut later
+ # the character '=' is added so later we can use a for-loop without dealing with $IFS
+ ACTIONS=$(cut -d ',' -f3 $TMP_LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | cut -d '%' -f2| sed -s "s# #=#g")
+ REASONS=$(cut -d ',' -f4 $TMP_LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | cut -d '%' -f2| sed -s "s# #=#g")
+
+ seperator "="
+ echo "Still very simple statistics for djm logs:"
+ echo " - $LOCAL_LOGFILE"
+ echo " - $UI_LOGFILE"
+ echo
+ echo "djm = documented jenkins maintenance"
+ echo " jenkins.debian.net exists since 2012, djm since 2023-04."
+ echo " so expect changes for some time."
+ seperator "="
+ printf "$TWO_C" "month:" "$LOGMONTH"
+ seperator
+ printf "$TWO_C" "djm actions:" "$TOTAL"
+ printf "$TWO_C" "hosts maintained:" "$HOSTS"
+ printf "$TWO_C" "jenkins jobs maintained:" "$(cat $JOBS)"
+ seperator
+ printf "$TWO_C" "hours with djm usage:" "${HOURS}"
+ printf "$TWO_C" " ten minute slices with djm usage:" "$TEN_MIN_SLICES"
+ printf "$TWO_C" " twenty minute slices (because focus):" "$TWENTY_MIN_SLICES"
+ printf "$TWO_C" " thoses slices as hours:" "${SLICES_AS_HOURS}"
+ seperator "="
+
+ printf "$ONE_C" "actions:"
+ printf "$ONE_C" "--------"
+ for action in $ACTIONS ; do
+ grep_action="$(echo $action | sed -s 's#=# #g' | xargs echo)"
+ AMOUNT=$(cut -d ',' -f3 $TMP_LOGFILE |grep "$grep_action"|wc -l)
+ PERCENT=$((200*$AMOUNT/$TOTAL % 2 + 100*$AMOUNT/$TOTAL))
+ printf "$THREE_C" "$grep_action:" "$PERCENT%" "($AMOUNT / $TOTAL)"
+ done
+ seperator "="
+
+ printf "$ONE_C" "reasons:"
+ printf "$ONE_C" "--------"
+ for reason in $REASONS ; do
+ grep_reason="$(echo $reason | sed -s 's#=# #g' | xargs echo)"
+ AMOUNT=$(cut -d ',' -f4 $TMP_LOGFILE |grep "$grep_reason"|wc -l)
+ PERCENT=$((200*$AMOUNT/$TOTAL % 2 + 100*$AMOUNT/$TOTAL))
+ printf "$THREE_C" "$grep_reason:" "$PERCENT%" "($AMOUNT / $TOTAL)"
+ done
+ seperator "="
+
+ #
+ # statistics about jenkins jobs
+ #
+ PATTERNS="setup_schroot setup_pbuilder debian_live maintenance node_health_check
+ html alpine archlinux builder_archlinux openwrt .*fedora builds opensuse .*strap
+ .*diffoscope reprotest disorderfs .*lfs" # .*(json|db|lfs)"
+ PIPE_PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
+ HEADING="$(cat $TMP_LOGFILE | wc -l) jobs triggered manually:"
+ printf_heading "$HEADING"
+ (
+ for i in $PATTERNS ; do
+ printf_if_not_zero "$TWO_C" $(grep -c reproducible_$i $TMP_LOGFILE) "reproducible_${i}_.*"
+ done
+ for i in $(cut -d ',' -f5 $TMP_LOGFILE | grep -v -E $PIPE_PATTERNS|sort -u) ; do
+ printf_if_not_zero "$TWO_C" $(grep -c $i $TMP_LOGFILE) "$i"
+ done
+ ) | sort -n -r
+
+ if $VERBOSE ; then
+ seperator
+ HEADING="$(cat $PARSER_LOGFILE | wc -l) total jobs run:"
+ printf_heading "$HEADING"
+ (
+ for i in $PATTERNS ; do
+ printf_if_not_zero "$TWO_C" $(grep -c reproducible_$i $PARSER_LOGFILE) "reproducible_${i}_.*"
+ done
+ for i in $(cut -d '/' -f2 $PARSER_LOGFILE | grep -v -E $PIPE_PATTERNS|sort -u) ; do
+ printf_if_not_zero "$TWO_C" $(grep -c $i $PARSER_LOGFILE) "$i"
+ done
+ ) | sort -n -r
+ fi
+ seperator "="
+
+ rm $TMP_LOGFILE
+
+}
+
+
get_arch_color() {
+ #
+ # define terminal background color based on architecture
+ #
case "$1" in
*amd64*) BG=lightgreen ;;
*i386*) BG=lightblue ;;
@@ -152,7 +312,8 @@ verify_target_action_reason(){
show_help(){
echo
- echo problem parsing parameters, djm by default needs three:
+ echo problem parsing parameters, eithers of these are possible:
+ echo "djm -d/--dry-run -f/--fetch -r/--report -v/--verbose"
echo "djm -d/--dry-run \$TARGET \$ACTION \$REASON"
echo
echo "TARGET= 'all' or grepable (jenkins, amd64, ionos, osuosl3) from ./nodes/list_nodes"
@@ -174,7 +335,7 @@ show_help(){
echo
}
-if [ "$MODE" = "djm" ] ; then
+djm_do() {
#
# make sure we have target, action & reason
#
@@ -186,7 +347,6 @@ if [ "$MODE" = "djm" ] ; then
if [ -n "$FAIL_REASON" ] ; then
show_help $1 $2 $3 "$FAIL_REASON"
exit 1
-
fi
#
@@ -293,4 +453,25 @@ if [ "$MODE" = "djm" ] ; then
done
echo Thank you for doing documented jenkins maintenance.
+}
+
+#
+# main
+#
+DUMMY=$(mktemp)
+touch -d "$(date -u -d "18 hours ago" '+%Y-%m-%d %H:%M') UTC" $DUMMY
+if [ $FETCH ] || [ ! -f $JOBS ] || [ $DUMMY -nt $JOBS ] ; then
+ if ! $DRY_MODE ; then
+ echo "Fetching remote logs."
+ djm_fetch_logs
+ fi
+else
+ echo "Not fetching remote logs."
+fi
+rm $DUMMY
+
+if [ "$MODE" = "report" ] ; then
+ djm_logparser
+elif [ "$MODE" = "djm" ] ; then
+ djm_do $1 $2 $3
fi
=====================================
bin/djm-logparser deleted
=====================================
@@ -1,160 +0,0 @@
-#!/bin/bash
-# vim: set noexpandtab:
-
-#
-# djm - documented jenkins maintenance logparser
-#
-# Copyright 2023 Holger Levsen <holger at layer-acht.org>
-# released under the GPLv2
-#
-
-set -e
-set -o pipefail # see eg http://petereisentraut.blogspot.com/2010/11/pipefail.html
-
-#
-# define environment
-#
-if [ -z "$DJM_USER" ] ; then
- if [ -n "$DEBFULLNAME" ] ; then
- DJM_USER="$(echo $DEBFULLNAME | cut -d ' ' -f1 | tr '[:upper:]' '[:lower:]')"
- else
- echo "Environment variable DJM_USER must be set. Exiting."
- exit 1
- fi
-fi
-LOCAL_LOGFILE=~/.djm.log
-UI_LOGFILE=~/.djm-jenkins-ui.log
-PARSER_LOGFILE=~/.djm-jenkins-parser.log
-LOGMONTH="$(date -u '+%Y-%m')"
-JOBS=~/.djm-jobs.txt
-
-#
-# parse jenkins build logs and copy the result over
-#
-case $1 in
- -f|--fetch)
- ssh $DJM_USER at jenkins.debian.net "DJM_USER=$DJM_USER /srv/jenkins/bin/djm-jenkins-parser"
- scp $DJM_USER at jenkins.debian.net:$(basename $UI_LOGFILE) $UI_LOGFILE
- scp $DJM_USER at jenkins.debian.net:$(basename $PARSER_LOGFILE) $PARSER_LOGFILE
- ssh $DJM_USER at jenkins.debian.net "cd ~jenkins/jobs ; ls -1d reproducible_* |wc -l" > $JOBS
- ;;
- *) : ;;
-esac
-
-#
-# little helpers
-#
-LOGFILE=$(mktemp)
-cat $LOCAL_LOGFILE $UI_LOGFILE | grep ^$LOGMONTH | sort -u > $LOGFILE
-THREE_C="%42s %12s %12s\n"
-TWO_C="%42s %12s\n"
-ONE_C="%42s %12s\n"
-
-seperator() {
- local CHAR="${1:--}"
- for i in $(seq 1 79) ; do
- echo -n "$CHAR"
- done
- echo
-}
-
-printf_if_not_zero() {
- if [ $2 -ne 0 ] ; then
- printf "$1" "$2" "$3"
- fi
-}
-
-printf_heading() {
- printf "$ONE_C" "$1"
- printf "$ONE_C" "$(for i in $(seq 1 ${#1}) ; do echo -n '-' ; done)"
-}
-
-#
-# main
-#
-
-TOTAL=$(grep -c ^$LOGMONTH $LOGFILE)
-HOSTS=$(cut -d ',' -f2 $LOGFILE | sort -u | grep -v "web UI"| wc -l)
-HOURS=$(cut -b 1-13 $LOGFILE | sort -u | wc -l)
-TEN_MIN_SLICES=$(cut -b 1-15 $LOGFILE | sort -u | wc -l)
-TWENTY_MIN_SLICES=$(( TEN_MIN_SLICES + TEN_MIN_SLICES ))
-SLICES_AS_HOURS=$(($TWENTY_MIN_SLICES/3))
-# create sorted list of actions/reasons (sorted by amount of occurances)
-# the character '%" is solely used as a delimeter for cut later
-# the character '=' is added so later we can use a for-loop without dealing with $IFS
-ACTIONS=$(cut -d ',' -f3 $LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | cut -d '%' -f2| sed -s "s# #=#g")
-REASONS=$(cut -d ',' -f4 $LOGFILE | sort | sed 's#^#%#g'| uniq -c | sort -nr | cut -d '%' -f2| sed -s "s# #=#g")
-
-seperator "="
-echo "Still very simple statistics for djm logs:"
-echo " - $LOCAL_LOGFILE"
-echo " - $UI_LOGFILE"
-echo
-echo "djm = documented jenkins maintenance"
-echo " jenkins.debian.net exists since 2012, djm since 2023-04."
-echo " so expect changes for some time."
-seperator "="
-printf "$TWO_C" "month:" "$LOGMONTH"
-seperator
-printf "$TWO_C" "djm actions:" "$TOTAL"
-printf "$TWO_C" "hosts maintained:" "$HOSTS"
-printf "$TWO_C" "jenkins jobs maintained:" "$(cat $JOBS)"
-seperator
-printf "$TWO_C" "hours with djm usage:" "${HOURS}"
-printf "$TWO_C" " ten minute slices with djm usage:" "$TEN_MIN_SLICES"
-printf "$TWO_C" " twenty minute slices (because focus):" "$TWENTY_MIN_SLICES"
-printf "$TWO_C" " thoses slices as hours:" "${SLICES_AS_HOURS}"
-seperator "="
-
-printf "$ONE_C" "actions:"
-printf "$ONE_C" "--------"
-for action in $ACTIONS ; do
- grep_action="$(echo $action | sed -s 's#=# #g' | xargs echo)"
- AMOUNT=$(cut -d ',' -f3 $LOGFILE |grep "$grep_action"|wc -l)
- PERCENT=$((200*$AMOUNT/$TOTAL % 2 + 100*$AMOUNT/$TOTAL))
- printf "$THREE_C" "$grep_action:" "$PERCENT%" "($AMOUNT / $TOTAL)"
-done
-seperator "="
-
-printf "$ONE_C" "reasons:"
-printf "$ONE_C" "--------"
-for reason in $REASONS ; do
- grep_reason="$(echo $reason | sed -s 's#=# #g' | xargs echo)"
- AMOUNT=$(cut -d ',' -f4 $LOGFILE |grep "$grep_reason"|wc -l)
- PERCENT=$((200*$AMOUNT/$TOTAL % 2 + 100*$AMOUNT/$TOTAL))
- printf "$THREE_C" "$grep_reason:" "$PERCENT%" "($AMOUNT / $TOTAL)"
-done
-seperator "="
-
-#
-# statistics about jenkins jobs
-#
-PATTERNS="setup_schroot setup_pbuilder debian_live maintenance node_health_check
- html alpine archlinux builder_archlinux openwrt .*fedora builds opensuse .*strap
- .*diffoscope reprotest disorderfs .*lfs" # .*(json|db|lfs)"
-PIPE_PATTERNS=$(echo $PATTERNS | sed 's# #|#g')
-HEADING="$(cat $LOGFILE | wc -l) jobs triggered manually:"
-printf_heading "$HEADING"
-(
- for i in $PATTERNS ; do
- printf_if_not_zero "$TWO_C" $(grep -c reproducible_$i $LOGFILE) "reproducible_${i}_.*"
- done
- for i in $(cut -d ',' -f5 $LOGFILE | grep -v -E $PIPE_PATTERNS|sort -u) ; do
- printf_if_not_zero "$TWO_C" $(grep -c $i $LOGFILE) "$i"
- done
-) | sort -n -r
-seperator
-
-HEADING="$(cat $PARSER_LOGFILE | wc -l) total jobs run:"
-printf_heading "$HEADING"
-(
- for i in $PATTERNS ; do
- printf_if_not_zero "$TWO_C" $(grep -c reproducible_$i $PARSER_LOGFILE) "reproducible_${i}_.*"
- done
- for i in $(cut -d '/' -f2 $PARSER_LOGFILE | grep -v -E $PIPE_PATTERNS|sort -u) ; do
- printf_if_not_zero "$TWO_C" $(grep -c $i $PARSER_LOGFILE) "$i"
- done
-) | sort -n -r
-seperator "="
-
-rm $LOGFILE
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/76526cee94466d4e18f6a42c12458cba865ee750...3c3dae18a3f00b5807d252d138eb2d5bd050cd1d
--
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/compare/76526cee94466d4e18f6a42c12458cba865ee750...3c3dae18a3f00b5807d252d138eb2d5bd050cd1d
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20230515/ef6a8ee7/attachment-0001.htm>
More information about the Qa-jenkins-scm
mailing list