[Piuparts-commits] rev 921 - piatti/home/piupartss trunk trunk/debian

Holger Levsen holger at alioth.debian.org
Sun Jul 10 13:11:19 UTC 2011


Author: holger
Date: 2011-07-10 13:11:19 +0000 (Sun, 10 Jul 2011)
New Revision: 921

Modified:
   piatti/home/piupartss/crontab
   trunk/TODO
   trunk/debian/changelog
   trunk/piuparts-slave.py
Log:
* piuparts-slave: 
  - check if chroot-tgz is older than MAX_TGZ_AGE (currently
    hardcoded to 30) and recreate it, if it is. Keep backup and put in back
    in place when debootstrapping a new chroot-tgz fails. Thanks to Scott
    Schaefer for the patch. (Closes: #632924)
  - fix bug in piuparts-slave which prevented running a single section by 
    including section name as command-line argument - thanks again to Scott
    and see 632924 too.

piatti/home/piupartss/crontab: disable deleting of chroot-tgzs


Modified: piatti/home/piupartss/crontab
===================================================================
--- piatti/home/piupartss/crontab	2011-07-10 12:58:04 UTC (rev 920)
+++ piatti/home/piupartss/crontab	2011-07-10 13:11:19 UTC (rev 921)
@@ -14,8 +14,3 @@
 #
 23 */6 * * * /home/piupartss/bin/detect_leftover_processes
 
-#
-# delete chroot-tarballs once a month, so that new ones are created automatically
-# after that, kill and restart the slave session to recreate them
-#
-42 23 1 * * sudo rm -v /org/piuparts.debian.org/slave/*/*.tar.gz ; sudo screen -r piuparts-slave-screen -X quit ; sleep 60 ; sudo /home/piupartss/bin/slave_run

Modified: trunk/TODO
===================================================================
--- trunk/TODO	2011-07-10 12:58:04 UTC (rev 920)
+++ trunk/TODO	2011-07-10 13:11:19 UTC (rev 921)
@@ -4,9 +4,12 @@
 Please also check the BTS, especially the bugs with a severity
 higher than wishlist!
 
+for 0.41
 
-for 0.41:
+- piuparts-slave: make MAX_TGZ_AGE configurable
 
+for 0.42:
+
 - piuparts-report should keep state, ie cache the results of the log 
   analysis and generated html pages. Thus, a piuparts-report run could
   take minutes instead of hours... this is now (with four distros tested 
@@ -21,7 +24,7 @@
 - fix scripts for dash as default shell
 
 
-for 0.42:
+for 0.43:
 
 - use python-debianbts to track bugginess status of packages via
   the bts and remove the "bugged" directory-status tracking
@@ -38,7 +41,7 @@
   why and fix it.
 
 
-for 0.43:
+for 0.44:
 
 - provide mail templates for bugreports (which ideally only
   should need manual logfile trimming) at
@@ -57,26 +60,19 @@
 - publish FAI classes to setup piuparts.$fqdn automatically.
 
 
-for 0.44:
+for 0.45:
 
 - PackagesDB: get rid of 50% of unique()
 
 - remove configfile option "mirror" and use the mirror from
   "packages-url".
 
-- piuparts-slave: if chroot.tgz is older than N days, regenerate
-  it
-  - currently this is done via ~piupartss/crontab, but that's a
-    hack
-  - only throw away tarballs (and recreate them) after testing
-    that they can be recreated 
-
 - master:
   - write stats about number of daily tests (-> report: create
     fancy graphs)
 
 
-for 0.45:
+for 0.46:
 
 - slave_run: cleanup stale proc mountpoints
 
@@ -90,7 +86,7 @@
   prevent start-stop-daemon from starting any daemons ;)
 
 
-for 0.46 and later:
+for 0.47 and later:
 
 - report:
   - write stats about the reasons for failures, as its done with

Modified: trunk/debian/changelog
===================================================================
--- trunk/debian/changelog	2011-07-10 12:58:04 UTC (rev 920)
+++ trunk/debian/changelog	2011-07-10 13:11:19 UTC (rev 921)
@@ -28,12 +28,20 @@
       - Thanks to Scott Schaefer for the patch. (Closes: #632046)
     - new option "--no-install-purge-test" to only do upgrade tests
       - Thanks to Andreas Bergmann for the patch (Closes: #588482)
+  * piuparts-slave: 
+    - check if chroot-tgz is older than MAX_TGZ_AGE (currently
+      hardcoded to 30) and recreate it, if it is. Keep backup and put in back
+      in place when debootstrapping a new chroot-tgz fails. Thanks to Scott
+      Schaefer for the patch. (Closes: #632924)
+    - fix bug in piuparts-slave which prevented running a single section by 
+      including section name as command-line argument - thanks again to Scott
+      and see 632924 too.
   * debian/control:
     - require python >= 2.6.6-3~, get rid of python-support build-depends.
     - add X-Python-Version: pseudo-header
   * debian/rules: use dh_python2 instead of python-support to build the 
     package
-  * Makefile: build for python 2.6 and 2.7 instead of 2.5 and 2.6 
+  * Makefile: build for python 2.6 and 2.7 instead of 2.5 and 2.6  
 
  -- Holger Levsen <holger at debian.org>  Sat, 25 Jun 2011 23:33:49 +0200
 

Modified: trunk/piuparts-slave.py
===================================================================
--- trunk/piuparts-slave.py	2011-07-10 12:58:04 UTC (rev 920)
+++ trunk/piuparts-slave.py	2011-07-10 13:11:19 UTC (rev 921)
@@ -25,17 +25,17 @@
 
 import os
 import sys
+import stat
 import time
 import logging
 import ConfigParser
-import urllib
 
-
 import piupartslib.conf
 import piupartslib.packagesdb
 
 
 CONFIG_FILE = "/etc/piuparts/piuparts.conf"
+MAX_TGZ_AGE = 60*60*24*30
 
 
 def setup_logging(log_level, log_file_name):
@@ -235,13 +235,15 @@
         if self._config["chroot-tgz"] and not self._config["distro"]:
           logging.info("The option --chroot-tgz needs --distro.")
 
-        if self._config["chroot-tgz"] and not os.path.exists(self._config["chroot-tgz"]):
-            create_chroot(self._config, self._config["chroot-tgz"], self._config["distro"])
+        tarball = self._config["chroot-tgz"]
+        if tarball:
+            create_or_replace_chroot_tgz(self._config, tarball,
+                       MAX_TGZ_AGE, self._config["distro"])
 
-        if (self._config["upgrade-test-distros"] and self._config["upgrade-test-chroot-tgz"]
-            and not os.path.exists(self._config["upgrade-test-chroot-tgz"])):
-            create_chroot(self._config, self._config["upgrade-test-chroot-tgz"], 
-                        self._config["upgrade-test-distros"].split()[0])
+        tarball = self._config["upgrade-test-chroot-tgz"]
+        if self._config["upgrade-test-distros"] and tarball: 
+            create_or_replace_chroot_tgz(self._config, tarball,
+                       MAX_TGZ_AGE, self._config["upgrade-test-distros"].split()[0])
     
         for rdir in ["new", "pass", "fail"]:
             rdir = os.path.join(self._slave_directory, rdir)
@@ -416,6 +418,21 @@
     f.close()
     os.rename(tarball + ".new", tarball)
 
+def create_or_replace_chroot_tgz(config, tgz, max_age, distro):
+    age = 0 
+    if os.path.exists(tgz):
+        age = time.time() - os.stat(tgz)[stat.ST_CTIME]
+        if age > max_age:
+            os.rename(tgz, tgz + ".old")
+            logging.info("%s too old.  Renaming to force re-creation" % tgz)
+    if not os.path.exists(tgz):
+        create_chroot(config, tgz, distro)
+        if not os.path.exists(tgz) and age > 0 and age > max_age:
+            os.rename(tgz + ".old", tgz)
+            logging.info("Failed to create ... reverting to old %s" % tgz)
+            create_chroot(config, tgz, distro)
+        else:
+            os.unlink(tgz + ".old")
 
 def fetch_packages_file(config, distro):
     mirror = config["mirror"]
@@ -448,11 +465,11 @@
     # argument referring to a section in configuration file.  
     # If no argument is given, the "global" section is assumed.
     section_names = []
+    global_config = Config(section="global")
+    global_config.read(CONFIG_FILE)
     if len(sys.argv) > 1:
         section_names = sys.argv[1:]
     else:
-        global_config = Config(section="global")
-        global_config.read(CONFIG_FILE)
         section_names = global_config["sections"].split()
 
     sections = []




More information about the Piuparts-commits mailing list