[Git][qa/jenkins.debian.net][master] reproducible node health: also show failed user services which require manual cleanup

[Holger Levsen (@holger)]

Holger Levsen pushed to branch master at Debian QA / jenkins.debian.net


Commits:
ad33f482 by Holger Levsen at 2024-03-26T19:16:44+01:00
reproducible node health: also show failed user services which require manual cleanup

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


3 changed files:

- bin/reproducible_node_health_check.sh
- bin/reproducible_system_health.sh
- logparse/reproducible.rules


Changes:

=====================================
bin/reproducible_node_health_check.sh
=====================================
@@ -182,7 +182,7 @@ if ! systemctl is-system-running > /dev/null; then
 	done
 	if ! systemctl is-system-running > /dev/null; then
 		systemctl status|head -5
-		echo "Warning: systemd is reporting errors:"
+		echo "Warning: systemctl is reporting errors:"
 		systemctl list-units --state=error,failed > $SERVICES
 		if ! grep -q '0 loaded units listed' $SERVICES ; then
 			cat $SERVICES
@@ -194,17 +194,21 @@ if ! systemctl is-system-running > /dev/null; then
 		echo "Manual cleanup needed."
 		DIRTY=true
 	fi
-	rm $SERVICES
+	rm -f $SERVICES
 fi
 echo "$(date -u) - checking whether all user services are running fine..."
 XDG_RUNTIME_DIR=/run/user/"$(id -u jenkins)"
 export XDG_RUNTIME_DIR
 if ! systemctl --user is-system-running > /dev/null; then
 	echo "$(date -u) - problematic user services found:"
-	echo "Warning: systemd is reporting errors:"
-	systemctl --user list-units --state=error,failed || true
-	echo "Manual cleanup needed."
+	echo "Warning: systemctl is reporting errors.
+	SERVICES=$(mktemp --tmpdir=$TEMPDIR node-health-XXXXXXX)
+	systemctl --user list-units --state=error,failed | grep " failed " > $SERVICES || true
+	echo "$(date -u) - problematic services found:"
+	cat $SERVICES
+	echo " Manual cleanup needed."
 	DIRTY=true
+	rm -f $SERVICES
 fi
 
 #


=====================================
bin/reproducible_system_health.sh
=====================================
@@ -198,9 +198,9 @@ for JOB_NAME in $(ls -1d reproducible_* | sort ) ; do
 			small_note "reproducible_build.sh orphans"
 		elif $(grep -q "State: starting" $LOG) ; then
 			small_note "node in starting state"
-		elif $(grep -q "Warning: systemd is reporting errors" $LOG) && $(grep -q "problematic user services found" $LOG) ; then
+		elif $(grep -q "Warning: systemctl is reporting errors" $LOG) && $(grep -q "problematic user services found" $LOG) ; then
 			small_note "problematic user services found, manual cleanup needed"
-		elif $(grep -q "Warning: systemd is reporting errors" $LOG) ; then
+		elif $(grep -q "Warning: systemctl is reporting errors" $LOG) ; then
 			small_note "undefined service problems"
 		elif $(grep -E -q "Warning: failed to update .* chdist." $LOG) ; then
 			small_note "failed to update Debian chdist"


=====================================
logparse/reproducible.rules
=====================================
@@ -33,7 +33,7 @@ warning /Warning: jenkins.log is larger than.+/
 warning /^Warning: running kernel needs attention/
 warning /Warning: schroots older than/
 warning /Warning: SQL query .+ failed.$/
-warning /Warning: systemd is reporting errors/
+warning /Warning: systemctl is reporting errors/
 warning /Warning: .+ has wrong MTU, please tell the jenkins admins to fix this./
 warning /Warning: Problems analysing/
 warning /Warning: .+ .xiting cleanly as this is out-of-scope./



View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/ad33f48243d699e57277dfcba1586ac4dab4e0cc

-- 
View it on GitLab: https://salsa.debian.org/qa/jenkins.debian.net/-/commit/ad33f48243d699e57277dfcba1586ac4dab4e0cc
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/qa-jenkins-scm/attachments/20240326/d34ce901/attachment-0001.htm>