[Piuparts-commits] rev 329 - in piatti: . home/piupartsm home/piupartsm/bin home/piupartss/bin

[Holger Levsen]

Author: holger
Date: 2009-04-07 15:33:52 +0000 (Tue, 07 Apr 2009)
New Revision: 329

Added:
   piatti/home/piupartsm/bin/detect_slave_problems
Modified:
   piatti/README.txt
   piatti/home/piupartsm/crontab
   piatti/home/piupartss/bin/slave_run
   piatti/home/piupartss/bin/update-piuparts-setup
Log:
new cronjob to monitor the slave and report problems

Modified: piatti/README.txt
===================================================================
--- piatti/README.txt	2009-04-07 12:04:19 UTC (rev 328)
+++ piatti/README.txt	2009-04-07 15:33:52 UTC (rev 329)
@@ -3,7 +3,6 @@
 
 == ToDo
 
-- improve slave-watcher cronjob entry, so that i checks every hour whether tail still looks the same, if so mail it, else mail the full day diff at midnight only -> less frequent mails :)
 - review sudoers and come up with (a bit) more restrictive one
 - use local mirror
 - there should be a 2nd group of piuparts-people. those who can sudo into piupartsm to process logfiles. maybe make that the qa-group

Added: piatti/home/piupartsm/bin/detect_slave_problems
===================================================================
--- piatti/home/piupartsm/bin/detect_slave_problems	                        (rev 0)
+++ piatti/home/piupartsm/bin/detect_slave_problems	2009-04-07 15:33:52 UTC (rev 329)
@@ -0,0 +1,56 @@
+#!/bin/sh
+
+# Copyright 2009 Holger Levsen (holger at layer-acht.org)
+# 
+# This program is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+# 
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
+# Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+#
+# this scripts monitors the output of piuparts-slave
+# when running in screen started by ~piupartss/bin/slave_run
+#
+
+SCREENLOG=/org/piuparts.debian.org/slave/screenlog.0
+MONITORDIR=/org/piuparts.debian.org/master/monitor-slave
+LOCKFILE="$MONITORDIR/mail-sent"
+
+mkdir -p $MONITORDIR
+cd $MONITORDIR
+
+# ~piupartss/bin/slave_run deletes screenlog.old so we know piuparts-slave has just been started 
+mv screenlog.new screenlog.old || cp cp $SCREENLOG screenlog.old
+cp $SCREENLOG screenlog.new
+
+diff screenlog.old screenlog.new >/dev/null 2>&1 && {
+	if [ ! -f $LOCKFILE ] ; then
+		{
+			echo "Either piuparts-slave hangs or is not running at all or wasn't started with ~piupartss/bin/slave_run - please investigate and take appropriate measures!"
+			echo
+			tail screenlog.new 
+		} | mail -s "problem with piuparts-slave detected" piupartsm
+		touch $LOCKFILE
+		chmod 660 $LOCKFILE
+	fi
+}
+
+#
+# send screenlog daily no matter what, but only once a day :)
+#
+HOUR=`date +%H`
+if [ $HOUR = "00" ] ; then
+	touch screenlog.yesterday
+	diff screenlog.yesterday screenlog.new | mail -s "piuparts-slave screenlog yesterday" piupartsm
+	cp screenlog.new screenlog.yesterday
+fi
+


Property changes on: piatti/home/piupartsm/bin/detect_slave_problems
___________________________________________________________________
Added: svn:executable
   + *

Modified: piatti/home/piupartsm/crontab
===================================================================
--- piatti/home/piupartsm/crontab	2009-04-07 12:04:19 UTC (rev 328)
+++ piatti/home/piupartsm/crontab	2009-04-07 15:33:52 UTC (rev 329)
@@ -9,11 +9,18 @@
 #  (dinstall runs 1|7|13|19:52, so this is long after mirror pushes...)
 #
 5 0 * * * PYTHONPATH=/org/piuparts.debian.org/lib/python2.4/site-packages:/org/piuparts.debian.org/lib/python2.5/site-packages nice python /org/piuparts.debian.org/share/piuparts/piuparts-report
+
 #
-# monitor slave session every six hours
+# monitor slave session every hour
 #
-23 0,6,12,18 * * *  tail /org/piuparts.debian.org/slave/screenlog.0
+23 * * * * /home/piupartsm/bin/detect_slave_problems
+
 #
+# make sure slave is reminded daily (and also monitored if not started with ~piupartss/bin/slave_run)
+#
+23 0 * * * rm -f /org/piuparts.debian.org/master/monitor-slave/mail-sent     
+
+#
 # monitor for problems once a day
 #
 42 7 * * * /home/piupartsm/bin/detect_stale_mounts

Modified: piatti/home/piupartss/bin/slave_run
===================================================================
--- piatti/home/piupartss/bin/slave_run	2009-04-07 12:04:19 UTC (rev 328)
+++ piatti/home/piupartss/bin/slave_run	2009-04-07 15:33:52 UTC (rev 329)
@@ -5,6 +5,9 @@
 set -e
 cd /org/piuparts.debian.org/slave
 (ps fax|grep -v grep| grep piuparts-slave ) && exit 0
-rm -f /org/piuparts.debian.org/slave/screenlog.0
+rm -f /org/piuparts.debian.org/slave/screenlog.0			# used by ~piupartsm/bin/detect_slave_problems
+rm -f /org/piuparts.debian.org/master/monitor-slave/mail-sent		# to only warn once a day about piuparts hanging
+rm -f /org/piuparts.debian.org/master/monitor-slave/screenlog.old	# and not to warn when it was just started 
+
 screen -L -d -m -S piuparts-slave-screen su - piupartss -c "cd /org/piuparts.debian.org/slave && python ../share/piuparts/piuparts-slave"
 

Modified: piatti/home/piupartss/bin/update-piuparts-setup
===================================================================
--- piatti/home/piupartss/bin/update-piuparts-setup	2009-04-07 12:04:19 UTC (rev 328)
+++ piatti/home/piupartss/bin/update-piuparts-setup	2009-04-07 15:33:52 UTC (rev 329)
@@ -35,6 +35,7 @@
 	svn co svn://svn.debian.org/svn/piuparts/piatti/home/piupartss .
 fi
 pwd
+crontab crontab
 svn up
 svn status
 grep -q PYTHONPATH .bashrc || {