[Piuparts-commits] rev 90 - / trunk trunk/piupartslib
John Wright
jsw-guest at alioth.debian.org
Wed Sep 5 01:07:43 UTC 2007
Author: jsw-guest
Date: 2007-09-05 01:07:43 +0000 (Wed, 05 Sep 2007)
New Revision: 90
Modified:
/
trunk/
trunk/README
trunk/piuparts-master.conf.sample
trunk/piuparts-master.py
trunk/piuparts-slave.conf.sample
trunk/piuparts-slave.py
trunk/piupartslib/packagesdb.py
Log:
r99 at neptune: jswright | 2007-09-04 19:07:34 -0600
Merge in /branches/people/jsw/multi-arch-master
Property changes on:
___________________________________________________________________
Name: svk:merge
- b12c1668-1bd8-44a9-a1e9-87753b0bf707:/local/piuparts:79
+ b12c1668-1bd8-44a9-a1e9-87753b0bf707:/local/piuparts:99
Property changes on: trunk
___________________________________________________________________
Name: svk:merge
+ 58b57fa5-7c20-0410-abb5-d811a221819a:/branches/upstream:20
58b57fa5-7c20-0410-abb5-d811a221819a:/people/jsw/debian-native:22
58b57fa5-7c20-0410-abb5-d811a221819a:/trunk:70
b12c1668-1bd8-44a9-a1e9-87753b0bf707:/local/piuparts/people/jsw/multi-arch-master:98
Modified: trunk/README
===================================================================
--- trunk/README 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/README 2007-09-05 01:07:43 UTC (rev 90)
@@ -42,7 +42,7 @@
suffice:
1. Pick a machine to run the master. It cannot be a chroot,
- but sarge should be good enough.
+ but etch should be good enough.
2. Install piuparts on it.
3. Create an account for the master.
4. Configure /etc/piuparts/piuparts-master.conf
@@ -59,15 +59,15 @@
9. Create an ssh keypair for the slave. No passphrase.
10. Add the slave's public key to the master's
.ssh/authorized_keys
- 11. Configure sudo on the slave machine to allow the slave
- account become root without password (otherwise you'll
- be typing in a password all the time).
+ 11. Configure sudo on the slave machine to allow the slave
+ account run /usr/sbin/piuparts as root without password
+ (otherwise you'll be typing in a password all the time).
12. Run /usr/share/piuparts/piuparts-slave.py on the slave
- accounts. Packages that are installed want to use
- /dev/tty, so you can't do this from cron. Also, you'll
- want to keep an eye on what is happening, to catch
- runaway processes and stuff.
+ accounts. Packages that are installed want to use
+ /dev/tty, so you can't do this from cron. Also, you'll
+ want to keep an eye on what is happening, to catch
+ runaway processes and stuff.
13. The logs go into the master account, into
subdirectories.
@@ -196,10 +196,14 @@
[master]
foo = bar
- where "master" is the name of a section. There can only be
- one section in the master configuration file, but it must
- be named "master". "foo" is the name of a configuration
- item, and "bar" is the value.
+ where "master" is the name of a section. By default,
+ piuparts-master.py uses the "master" section in the master
+ configuration file, but you can tell it to use another
+ section by giving the name of that section as an argument.
+ "foo" is the name of a configuration item, and "bar" is the
+ value. If a section other than "master" is used, all the
+ piuparts logs for that section are kept in a subdirectory by
+ the name of the section.
The following configuration items are defined for the master:
@@ -230,8 +234,9 @@
piuparts-slave uses the configuration file
/etc/piuparts/piuparts-slave.conf. It has the same
general syntax as the master's configuration file. The
- section is called "slave" and the configuration items
- are as follows:
+ default section is called "slave" (again, you can use a
+ different section by specifying it on the command-line) and
+ the configuration items are as follows:
* "idle-sleep" is the length of time the slave should wait
before querying the master again if the master didn't have
@@ -251,7 +256,13 @@
* "master-command" is the command to run on master-host to
start the master. When the master has been installed from
the Debian package, the command is
- "python /usr/share/piuparts/piuparts-master.py".
+ "python /usr/share/piuparts/piuparts-master.py". If you
+ want to use a section in the master configuration file
+ other than "master", append the section name to this
+ command. For example, if the master configuration file has
+ a "sid-ia64" section that you want to use, the command
+ should be
+ "python /usr/share/piuparts/piuparts-master.py sid-ia64".
* "mirror" tells the slave which mirror it is to use. The
slave gives this to piuparts when it runs it.
@@ -272,7 +283,8 @@
* "upgrade-test-distros" is the space delimited list of
distributions the slave should use for testing upgrades
between distributions (i.e., Debian versions). Currently,
- "sarge etch sid" is a good choice.
+ "sarge etch sid" is a good choice. Make this empty if you
+ do not want to run upgrade tests.
* "upgrade-test-chroot-tgz" is the name of the file the slave
should use for the tarball to keep the chroot for the
@@ -292,6 +304,13 @@
more. For a contemporary AMD64 machine with a reasonably
fast disk subsystem the value 50 seems to work fine.
+ * "keep-sources-list" controls whether the slave runs
+ piuparts with the "--keep-sources-list" option. This
+ option does not apply to upgrade tests. The value should
+ be "yes" or "no", with the default being "no". Use this
+ option for dists that you need a custom sources.list for,
+ such as "testing-proposed-updates".
+
* "debug" tells the slave whether to log debug level
messages. The value should be "yes" or "no", with the
default being "no".
Modified: trunk/piuparts-master.conf.sample
===================================================================
--- trunk/piuparts-master.conf.sample 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/piuparts-master.conf.sample 2007-09-05 01:07:43 UTC (rev 90)
@@ -12,3 +12,15 @@
[master]
log-file = /dev/null
# packages-url = http://your.mirror.here/debian/dists/sid/main/binary-i386/Packages.bz2
+
+# You can have other sections, in order to support one master for multiple
+# dists and/or architectures. In the slave configuration file, be sure to
+# append the appropriate section name to the master-command option.
+
+## [sid-ia64]
+## log-file = /dev/null
+## packages-url = http://ftp.debian.org/debian/dists/sid/binary-ia64/Packages.bz2
+
+## [t-p-u-i386]
+## log-file = /dev/null
+## packages-url = http://ftp.debian.org/debian/dists/testing-proposed-updates/binary-i386/Packages.bz2
Modified: trunk/piuparts-master.py
===================================================================
--- trunk/piuparts-master.py 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/piuparts-master.py 2007-09-05 01:07:43 UTC (rev 90)
@@ -48,8 +48,8 @@
class Config(piupartslib.conf.Config):
- def __init__(self):
- piupartslib.conf.Config.__init__(self, "master",
+ def __init__(self, section="master"):
+ piupartslib.conf.Config.__init__(self, section,
{
"log-file": None,
"packages-url": None,
@@ -104,7 +104,7 @@
class Master(Protocol):
- def __init__(self, input, output, packages_file):
+ def __init__(self, input, output, packages_file, section=None):
Protocol.__init__(self, input, output)
self._commands = {
"reserve": self._reserve,
@@ -113,7 +113,7 @@
"fail": self._fail,
"untestable": self._untestable,
}
- self._db = piupartslib.packagesdb.PackagesDB()
+ self._db = piupartslib.packagesdb.PackagesDB(prefix=section)
self._db.create_subdirs()
self._db.read_packages_file(packages_file)
self._writeline("hello")
@@ -170,14 +170,23 @@
def main():
- config = Config()
+ # For supporting multiple architectures and suites, we take a command-line
+ # argument referring to a section in the master configuration file. For
+ # backwards compatibility, if no argument is given, the "master" section is
+ # assumed.
+ if len(sys.argv) == 2:
+ section = sys.argv[1]
+ config = Config(section=section)
+ else:
+ section = None
+ config = Config()
config.read(CONFIG_FILE)
setup_logging(logging.DEBUG, config["log-file"])
logging.info("Fetching %s" % config["packages-url"])
packages_file = piupartslib.open_packages_url(config["packages-url"])
- m = Master(sys.stdin, sys.stdout, packages_file)
+ m = Master(sys.stdin, sys.stdout, packages_file, section=section)
packages_file.close()
while m.do_transaction():
pass
Modified: trunk/piuparts-slave.conf.sample
===================================================================
--- trunk/piuparts-slave.conf.sample 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/piuparts-slave.conf.sample 2007-09-05 01:07:43 UTC (rev 90)
@@ -22,3 +22,25 @@
upgrade-test-chroot-tgz = sarge.tar.gz
max-reserved = 50
debug = no
+
+# As in piuparts-master.conf, you can have multiple sections, so that you can
+# run piuparts-slave.py for multiple dists
+
+# For testing-proposed-updates, you'll have to roll your own chroot, or modify
+# the sources.list in a generated chroot for testing, in order to include the
+# appropriate deb line for testing-proposed-updates
+## [t-p-u-i386]
+## idle-sleep = 300
+## # master-host = localhost
+## # master-user = piuparts
+## # master-directory = .
+## master-command = python /usr/share/piuparts/piuparts-master.py t-p-u-i386
+## # mirror = http://your.mirror.here/debian
+## piuparts-cmd = sudo piuparts
+## distro = testing-proposed-updates
+## chroot-tgz = t-p-u.tar.gz
+## keep-sources-list = yes
+## upgrade-test-distros =
+## upgrade-test-chroot-tgz =
+## max-reserved = 50
+## debug = no
Modified: trunk/piuparts-slave.py
===================================================================
--- trunk/piuparts-slave.py 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/piuparts-slave.py 2007-09-05 01:07:43 UTC (rev 90)
@@ -56,8 +56,8 @@
class Config(piupartslib.conf.Config):
- def __init__(self):
- piupartslib.conf.Config.__init__(self, "slave",
+ def __init__(self, section="slave"):
+ piupartslib.conf.Config.__init__(self, section,
{
"idle-sleep": "10",
"master-host": None,
@@ -72,6 +72,8 @@
"upgrade-test-chroot-tgz": None,
"max-reserved": "1",
"debug": "no",
+ "keep-sources-list": "no",
+ "arch": None,
},
["master-host", "master-user", "master-command"])
@@ -236,6 +238,8 @@
command = "%(piuparts-cmd)s -ad %(distro)s -b %(chroot-tgz)s " % \
config
+ if config["keep-sources-list"] in ["yes", "true"]:
+ command += "--keep-sources-list "
command += package["Package"]
logging.debug("Executing: %s" % command)
@@ -278,8 +282,8 @@
def create_chroot(config, tarball, distro):
logging.info("Creating new tarball %s" % tarball)
- command = "%s -ad %s -s %s.new hello" % \
- (config["piuparts-cmd"], distro, tarball)
+ command = "%s -ad %s -s %s.new -m %s hello" % \
+ (config["piuparts-cmd"], distro, tarball, config["mirror"])
logging.debug("Executing: " + command)
f = os.popen("{ %s; } 2>&1" % command, "r")
for line in f:
@@ -290,7 +294,11 @@
def fetch_packages_file(config, distro):
mirror = config["mirror"]
- arch = "i386" # FIXME to figure our arch right
+ arch = config["arch"]
+ if not arch:
+ # Try to figure it out ourselves, using dpkg
+ input, output = os.popen2("dpkg --print-architecture")
+ arch = output.read().rstrip()
packages_url = \
"%s/dists/%s/main/binary-%s/Packages.bz2" % (mirror, distro, arch)
@@ -311,7 +319,17 @@
def main():
setup_logging(logging.INFO, None)
- config = Config()
+ # For supporting multiple piuparts-slave configurations on a particular
+ # machine (e.g. for testing multiple suites), we take a command-line
+ # argument referring to a section in the slave configuration file. For
+ # backwards compatibility, if no argument is given, the "slave" section is
+ # assumed.
+ if len(sys.argv) == 2:
+ section = sys.argv[1]
+ config = Config(section=section)
+ else:
+ section = None
+ config = Config()
config.read(CONFIG_FILE)
if config["debug"] in ["yes", "true"]:
@@ -321,7 +339,8 @@
if not os.path.exists(config["chroot-tgz"]):
create_chroot(config, config["chroot-tgz"], config["distro"])
- if not os.path.exists(config["upgrade-test-chroot-tgz"]):
+ if (config["upgrade-test-distros"] and not
+ os.path.exists(config["upgrade-test-chroot-tgz"])):
create_chroot(config, config["upgrade-test-chroot-tgz"],
config["upgrade-test-distros"].split()[0])
Modified: trunk/piupartslib/packagesdb.py
===================================================================
--- trunk/piupartslib/packagesdb.py 2007-09-05 00:53:33 UTC (rev 89)
+++ trunk/piupartslib/packagesdb.py 2007-09-05 01:07:43 UTC (rev 90)
@@ -111,14 +111,14 @@
return dircache.listdir(dirname)
def exists(self, pathname):
- try:
- cache = self.exists_cache
- except AttributeError:
- self.exists_cache = {}
- cache = self.exists_cache
- if pathname not in cache:
- cache[pathname] = os.path.exists(pathname)
- return cache[pathname]
+ try:
+ cache = self.exists_cache
+ except AttributeError:
+ self.exists_cache = {}
+ cache = self.exists_cache
+ if pathname not in cache:
+ cache[pathname] = os.path.exists(pathname)
+ return cache[pathname]
def open_file(self, pathname, mode):
return file(pathname, mode)
@@ -137,17 +137,17 @@
return False
def any_log_exists(self, package, subdirs):
- try:
- cache = self.basename_cache
- except AttributeError:
- self.basename_cache = {}
- cache = self.basename_cache
+ try:
+ cache = self.basename_cache
+ except AttributeError:
+ self.basename_cache = {}
+ cache = self.basename_cache
package_name = package["Package"]
for subdir in subdirs:
for basename in self.listdir(subdir):
- if basename not in cache:
- cache[basename] = basename.split("_", 1)
- parts = cache[basename]
+ if basename not in cache:
+ cache[basename] = basename.split("_", 1)
+ parts = cache[basename]
if len(parts) == 2 and parts[0] == package_name:
return True
return False
@@ -212,7 +212,8 @@
"dependency-fix-not-yet-tested": "dependency-fix-not-yet-tested",
}
- def __init__(self, logdb=None):
+ def __init__(self, logdb=None, prefix=None):
+ self.prefix = prefix
self._packages_files = []
self._ready_for_testing = None
self._logdb = logdb or LogDB()
@@ -225,18 +226,24 @@
def set_subdirs(self, ok=None, fail=None, evil=None, reserved=None,
moreok=None, morefail=None):
+ # Prefix all the subdirs with the prefix
+ if self.prefix:
+ format = self.prefix + "/%s"
+ else:
+ format = "%s"
+
if ok:
- self._ok = ok
+ self._ok = format % ok
if fail:
- self._fail = fail
+ self._fail = format % fail
if evil:
- self._evil = evil
+ self._evil = format % evil
if reserved:
- self._reserved = reserved
+ self._reserved = format % reserved
if moreok:
- self._moreok = moreok
+ self._moreok = [format % s for s in moreok]
if morefail:
- self._morefail = morefail
+ self._morefail = [format % s for s in morefail]
self._all = [self._ok, self._fail, self._evil, self._reserved] + \
self._moreok + self._morefail
More information about the Piuparts-commits
mailing list