[Piuparts-commits] rev 296 - in trunk: . debian
Holger Levsen
holger at alioth.debian.org
Fri Mar 20 12:33:51 UTC 2009
Author: holger
Date: 2009-03-20 12:33:51 +0000 (Fri, 20 Mar 2009)
New Revision: 296
Added:
trunk/debian/NEWS
Modified:
trunk/
trunk/Makefile
trunk/README.txt
trunk/TODO
trunk/debian/
trunk/debian/changelog
trunk/piuparts-report.py
Log:
* The README also has been restructured and updated to reflect the
configuration changes in master-slave mode.
* Add NEWS file.
Property changes on: trunk
___________________________________________________________________
Name: svn:ignore
+ README.html
README.pdf
piuparts.1
piuparts
Modified: trunk/Makefile
===================================================================
--- trunk/Makefile 2009-03-20 11:35:16 UTC (rev 295)
+++ trunk/Makefile 2009-03-20 12:33:51 UTC (rev 296)
@@ -26,9 +26,7 @@
install-conf:
install -d $(etcdir)/piuparts
- for x in master slave report; do \
- install -m 0644 piuparts-$$x.conf.sample \
- $(etcdir)/piuparts/piuparts-$$x.conf; done
+ install -m 0644 piuparts.conf.sample $(etcdir)/piuparts/piuparts.conf
install-py:
install -d $(sbindir)
Modified: trunk/README.txt
===================================================================
--- trunk/README.txt 2009-03-20 11:35:16 UTC (rev 295)
+++ trunk/README.txt 2009-03-20 12:33:51 UTC (rev 296)
@@ -25,12 +25,178 @@
License, version 2.
+How to use piuparts in 5 minutes
+--------------------------------
+
+Basic Usage
+~~~~~~~~~~~
+
+Testing your packages with piuparts is as easy as typing at the console prompt:
+
+----
+ # piuparts sm_0.6-1_i386.deb
+----
+
+Note that in order to work, piuparts has to be executed as user root, so you
+need to be logged as root or use sudo.
+
+This will create a sid chroot with debootstrap, where it'll test your package.
+
+If you want to test your package in another release, for example, lenny, you can
+do so with:
+
+----
+ # piuparts sm_0.6-1_i386.deb -d lenny
+----
+
+By default, this will read the first mirror from your '/etc/apt/sources.list '
+file. If you want to specify a different mirror you can do it with the option
+'-m':
+
+----
+ # piuparts sm_0.6-1_i386.deb -m http://ftp.de.debian.org/debian
+----
+
+
+Some tips
+~~~~~~~~~
+
+If you use piuparts on a regular basis, waiting for it to create a chroot every
+time takes too much time, even if you are using a local mirror or a caching tool
+such as approx.
+
+Piuparts has the option of using a tarball as the contents of the initial chroot,
+instead of building a new one with debootstrap. A easy way to use this option
+is use a tarball created with pbuilder. If you are not a pbuilder user, you can
+create this tarball with the command (again, as root):
+
+----
+ # puilder create
+----
+
+then you only have to remember to update this tarball with:
+
+----
+ # pbuilder update.
+----
+
+To run piuparts using this tarball:
+
+----
+ # piuparts -p sm_0.6-1_i386.deb
+----
+
+If you want to use your own pre-made tarball:
+
+----
+ # piuparts --basetgz=/path/to/my/tarball.tgz sm_0.6-1_i386.deb
+----
+
+Piuparts also has the option of using a tarball as the contents of the initial
+chroot, instead of building a new one with pbuilder. You can save a tarball
+for later use with the -s (--save) piuparts option. Some people like this,
+others prefer to only have to maintain one tarball. Read the piuparts manpage
+about the -p, -b and -s options
+
+
+* Piuparts tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, piuparts does two tests:
+1 - Installation and purging test.
+2 - Installation, upgrade and purging tests.
+
+The first test installs the package in a minimal chroot, removes it and purges
+it. The second test installs the current version in the archive of the given
+packages, then upgrades to the new version (deb files given to piuparts in the
+input), removes and purges.
+
+If you only want to perfom the first test, you can use the option:
+'--no-upgrade-test'
+
+
+* Analyzing piuparts results
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When piuparts finishes all the tests satisfactorily, you will get these lines
+as final output:
+
+----
+ 0m39.5s INFO: PASS: All tests.
+ 0m39.5s INFO: piuparts run ends.
+----
+
+Anyway, it is a good idea to read the whole log in order to discover possible
+problems that did not stop the piuparts execution.
+
+If you do not get those lines, piuparts has failed during a test. The latest
+lines should give you a pointer to the problem with your package.
+
+Custom scripts with piuparts
+------------------------------
+
+You can specify several custom scripts to be run inside piuparts.
+You have to store them in a directory and give it as argument to
+piuparts: '--scriptsdir=/dir/with/the/scripts'
+
+The scripts can be run:
+
+ Before *installing* your package. The name of the script must
+ start with:
+ 'pre_install_'
+
+ After *installing* your package and its deps. In the case of the
+ upgrade test, it is after install and upgrade. The name of the
+ script must start with:
+ 'post_install_'
+
+ After *removing* your package, The name of the script must start with:
+ 'post_remove_'
+
+ After *purging* your package, The name of the script must start with:
+ 'post_purge_'
+
+ Before *upgrading* your package, once the current version in the archive
+ has been installed (this is done in the second test, "Installation,
+ upgrade and purging test"). The name of the script must start with:
+ 'pre_upgrade_'
+
+ Before upgrading the chroot to another distro and after upgrading:
+ 'pre_distupgrade_'
+ 'post_distupgrade_'
+
+You can run several scripts in every step, they are run in alphabetical
+order.
+
+The scripts are run *inside* the piuparts chroot and only can be shell
+scripts, if you want to run Python or Perl scripts, you have to install
+Python or Perl. The chroot where piuparts is run is minized and does not
+include Perl.
+
+
+It would be interesting to declare some piuparts variables like the name
+of the current testing packages, and be able to use this variable in the
+custom scripts. But this option is not available yet.
+
+Example custom script:
+~~~~~~~~~~~~~~~~~~~~~~
+
+'$cat post_install_number.sh'
+----
+#!/bin/bash
+
+number=`dpkg -l | wc -l`
+echo "There are $number packages installed."
+exit 0
+----
+
+
Distrubuted testing
-~~~~~~~~~~~~~~~~~~~
+-------------------
- As part of the quality assurance effort of Debian, I run
- piuparts on the Debian package archive. This requires a
- lot of processing power, and so the work is distributed
+ As part of the quality assurance effort of Debian,
+ piuparts is run on the Debian package archive. This requires a
+ lot of processing power, and so the work can be distributed
over several hosts.
There is one central machine, the master, and any number
@@ -46,27 +212,27 @@
but a etch system is good enough.
2. Install piuparts on it.
3. Create an account for the master.
- 4. Configure /etc/piuparts/piuparts-master.conf
- appropriately.
+ 4. Configure '/etc/piuparts/piuparts.conf' appropriately.
5. Pick one or more slaves to run the slave. You can use
the machine running the master also as a slave. Etch is
fine, it can even be in a chroot.
6. Install piuparts on it.
- 7. Configure /etc/piuparts/piuparts-slave.conf
- appropriately.
+ 7. Configure '/etc/piuparts/piuparts.conf' appropriately -
+ if master and slave share the machine, they also share
+ the config file.
8. Create an account for the slave. This must be different
from the master account.
9. Create an ssh keypair for the slave. No passphrase.
10. Add the slave's public key to the master's
- .ssh/authorized_keys
+ '.ssh/authorized_keys'
11. Configure sudo on the slave machine to allow the slave
- account run /usr/sbin/piuparts as root without password
+ account run '/usr/sbin/piuparts' as root without password
(otherwise you'll be typing in a password all the time).
- 12. Run /usr/share/piuparts/piuparts-slave.py on the slave
+ 12. Run '/usr/share/piuparts/piuparts-slave' on the slave
accounts. Packages that are installed want to use
- /dev/tty, so you can't do this from cron. Also, you'll
+ '/dev/tty', so you can't do this from cron. Also, you'll
want to keep an eye on what is happening, to catch
runaway processes and stuff.
@@ -85,7 +251,7 @@
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The slave machine and the piuparts-master program
- communicate using a simplistic line based protocol. Ssh
+ communicate using a simplistic line based protocol. SSH
takes care of authentication, so there is nothing in the
protocol for that. The protocol is transaction based:
the slave gives a command, the master program responds.
@@ -93,7 +259,7 @@
long (a status line plus additional data lines). Simple
commands and responses are of the following format:
- keyword arg1 arg2 arg3 ... argN
+ 'keyword arg1 arg2 arg3 ... argN'
The keyword is a command or status code ("ok"), and it
and the arguments are separated by spaces. An argument
@@ -110,7 +276,8 @@
A sample session (">>" indicates what the slave sends,
"<<" what the master responds with):
-
+
+----
<< hello
>> pass liwc 1.2.3-4
>> The piuparts
@@ -120,6 +287,7 @@
<< ok
>> reserve
<< ok vorbisgain 2.3-4
+----
Here the slave first reports a successful test of
package liwc, version 1.2.3-4, and sends the piuparts
@@ -131,46 +299,55 @@
spoken.
Commands and responses in this protocol:
-
+
+----
Command: reserve
Success: ok <packagename> <packageversion>
Failure: error
-
+----
Slave asks master to reserve a package (a
particular version of it) for the slave to test.
The slave may reserve any number of packages to
test. If the transaction fails, there are no
more packages to test, and the slave should
disconnect, wait some time and try again.
-
+
+----
Command: unreserve <packagename> <packageversion>
Success: ok
+----
Slave informs master it cannot test the desired
version of a package (perhaps it went away from
the mirror?).
+----
Command: pass <packagename> <packageversion>
log file contents
.
Success: ok
+----
Slave reports that it has tested a particular
version of a package and that the package passed
all tests. Master records this and stores the
log file somewhere suitable.
+----
Command: fail <packagename> <packageversion>
log file contents
.
Success: ok
+----
Same as "pass", but package failed one or more tests.
+----
Command: untestable <packagename> <packageversion>
log file contents
.
Success: ok
+----
Slave reports that a particular package is
untestable, possibly because it insists on
@@ -186,58 +363,62 @@
if the slave sends garbage, or sends too much data.
-Master configuration file
-~~~~~~~~~~~~~~~~~~~~~~~~~
+piuparts.conf configuration file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- piuparts-master uses the configuration file
- /etc/piuparts/piuparts-master.conf. The syntax is defined
- by the Python ConfigParser class, and is, briefly, like
- this:
-
- [master]
- foo = bar
-
- where "master" is the name of a section. By default,
- piuparts-master.py uses the "master" section in the master
- configuration file, but you can tell it to use another
- section by giving the name of that section as an argument.
- "foo" is the name of a configuration item, and "bar" is the
- value. If a section other than "master" is used, all the
- piuparts logs for that section are kept in a subdirectory by
- the name of the section.
-
- The following configuration items are defined for the master:
-
- * "log-file" is the name of a file to where the master should
- write its log messages. In the default configuration file
- it is "/dev/null", that is, log messages are not put in a
- file.
+piuparts-master, piuparts-slave and piuparts-report share the configuration file
+'/etc/piuparts/piuparts.conf'. The syntax is defined
+by the Python ConfigParser class, and is, briefly, like
+this:
- * "packages-url" is a URL to the Packages.bz2 file for your
- mirror. It is usually best to use the Packages.bz2 for
- sid (unstable), unless you know what you're doing. For
- example, you might use
-
- http://ftp.debian.org/debian/dists/sid/main/
- binary-i386/Packages.bz2
-
- (all on one line, of course), but you really do want
- to replace "ftp.debian.org" with the name of your local
- mirror.
+ [master]
+ foo = bar
+
+FIXME:
+#where "master" is the name of a section. By default,
+#piuparts-master.py uses the "master" section in the master
+#configuration file, but you can tell it to use another
+#section by giving the name of that section as an argument.
+#"foo" is the name of a configuration item, and "bar" is the
+#value. If a section other than "master" is used, all the
+#piuparts logs for that section are kept in a subdirectory by
+#the name of the section.
- Both configuration items must be in the configuration file,
- or master will refuse to run.
+global configuration
+^^^^^^^^^^^^^^^^^^^^
+These settings are used by piuparts-master, -slave and -report alike:
-Slave configuration file
-~~~~~~~~~~~~~~~~~~~~~~~~
+master configuration
+^^^^^^^^^^^^^^^^^^^^
- piuparts-slave uses the configuration file
- /etc/piuparts/piuparts-slave.conf. It has the same
- general syntax as the master's configuration file. The
- default section is called "slave" (again, you can use a
- different section by specifying it on the command-line) and
- the configuration items are as follows:
+the following settings are only useful for piuparts-master:
+
+* "log-file" is the name of a file to where the master should
+ write its log messages. In the default configuration file
+ it is "/dev/null", that is, log messages are not put in a
+ file.
+
+* "packages-url" is a URL to the Packages.bz2 file for your
+ mirror. It is usually best to use the Packages.bz2 for
+ sid (unstable), unless you know what you're doing. For
+ example, you might use
+
+ http://ftp.debian.org/debian/dists/sid/main/
+ binary-i386/Packages.bz2
+
+ (all on one line, of course), but you really do want
+ to replace "ftp.debian.org" with the name of your local
+ mirror.
+
+Both configuration items must be in the configuration file,
+or master will refuse to run.
+
+
+slave configuration
+^^^^^^^^^^^^^^^^^^^
+
+the following settings are only useful for piuparts-slave:
* "idle-sleep" is the length of time the slave should wait
before querying the master again if the master didn't have
@@ -320,152 +501,11 @@
is best to set them all to be sure what the configuration
actually is.
-How to use piuparts in 5 minutes
---------------------------------
+report configuration
+^^^^^^^^^^^^^^^^^^^^^
-* Basic Usage
-~~~~~~~~~~~~~
+the following settings are only useful for piuparts-report:
-Testing your packages with piuparts is as easy as typing at the console prompt:
+FIXME
- # piuparts sm_0.6-1_i386.deb
-Note that in order to work, piuparts has to be executed as user root, so you
-need to be logged as root or use sudo.
-
-This will create a sid chroot with debootstrap, where it'll test your package.
-
-If you want to test your package in another release, for example, lenny, you can
-do so with:
-
- # piuparts sm_0.6-1_i386.deb -d lenny
-
-By default, this will read the first mirror from your /etc/apt/sources.list
-file. If you want to specify a different mirror you can do it with the option
--m:
-
- # piuparts sm_0.6-1_i386.deb -m http://ftp.de.debian.org/debian
-
-
-* Some tips
-~~~~~~~~~~~
-
-If you use piuparts on a regular basis, waiting for it to create a chroot every
-time takes too much time, even if you are using a local mirror or a caching tool
-such as approx.
-
-Piuparts has the option of using a tarball as the contents of the initial chroot,
-instead of building a new one with debootstrap. A easy way to use this option
-is use a tarball created with pbuilder. If you are not a pbuilder user, you can
-create this tarball with the command (again, as root):
-
- # puilder create
-
-then you only have to remember to update this tarball with:
-
- # pbuilder update.
-
-To run piuparts using this tarball:
-
- # piuparts -p sm_0.6-1_i386.deb
-
-If you want to use your own pre-made tarball:
-
- # piuparts --basetgz=/path/to/my/tarball.tgz sm_0.6-1_i386.deb
-
-Piuparts also has the option of using a tarball as the contents of the initial
-chroot, instead of building a new one with pbuilder. You can save a tarball
-for later use with the -s (--save) piuparts option. Some people like this,
-others prefer to only have to maintain one tarball. Read the piuparts manpage
-about the -p, -b and -s options
-
-
-* Piuparts tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-By default, piuparts does two tests:
-1 - Installation and purging test.
-2 - Installation, upgrade and purging tests.
-
-The first test installs the package in a minimal chroot, removes it and purges
-it. The second test installs the current version in the archive of the given
-packages, then upgrades to the new version (deb files given to piuparts in the
-input), removes and purges.
-
-If you only want to perfom the first test, you can use the option:
---no-upgrade-test
-
-
-* Analyzing piuparts results
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When piuparts finishes all the tests satisfactorily, you will get these lines
-as final output:
-
- 0m39.5s INFO: PASS: All tests.
- 0m39.5s INFO: piuparts run ends.
-
-Anyway, it is a good idea to read the whole log in order to discover possible
-problems that did not stop the piuparts execution.
-
-If you do not get those lines, piuparts has failed during a test. The latest
-lines should give you a pointer to the problem with your package.
-
-Custom scripts with piuparts
-------------------------------
-
-You can specify several custom scripts to be run inside piuparts.
-You have to store them in a directory and give it as argument to
-piuparts: --scriptsdir=/dir/with/the/scripts
-
-The scripts can be run:
-
- Before *installing* your package. The name of the script must
- start with:
- pre_install_
-
- After *installing* your package and its deps. In the case of the
- upgrade test, it is after install and upgrade. The name of the
- script must start with:
- post_install_
-
- After *removing* your package, The name of the script must start with:
- post_remove_
-
- After *purging* your package, The name of the script must start with:
- post_purge_
-
- Before *upgrading* your package, once the current version in the archive
- has been installed (this is done in the second test, "Installation,
- upgrade and purging test"). The name of the script must start with:
- pre_upgrade_
-
- Before upgrading the chroot to another distro and after upgrading:
- pre_distupgrade_
- post_distupgrade_
-
-You can run several scripts in every step, they are run in alphabetical
-order.
-
-The scripts are run *inside* the piuparts chroot and only can be shell
-scripts, if you want to run Python or Perl scripts, you have to install
-Python or Perl. The chroot where piuparts is run is minized and does not
-include Perl.
-
-
-It would be interesting to declare some piuparts variables like the name
-of the current testing packages, and be able to use this variable in the
-custom scripts. But this option is not available yet.
-
-Example script:
-
-'$cat post_install_number.sh'
-----
-#!/bin/bash
-
-number=`dpkg -l | wc -l`
-echo "There are $number packages installed."
-exit 0
-----
-
-
Modified: trunk/TODO
===================================================================
--- trunk/TODO 2009-03-20 11:35:16 UTC (rev 295)
+++ trunk/TODO 2009-03-20 12:33:51 UTC (rev 296)
@@ -3,7 +3,6 @@
must for 0.36
-- convert to NEWS Debian system and introduce breakge of conf file backwards compatibility...
- globalize more config variables.
- document this in README
- document sections for piuparts-slave in README.
@@ -16,6 +15,9 @@
- create $section_packages.txt with into about all packages
- better header (link to FAQ, point out to read the bottom first..) for piuparts-logs
- debug is on, even though its off in the config
+- take care of old conf files on upgrades
+- take care of old pyc files from python-central
+- scripts in /usr/share/piuparts/ probably a FHS violation
for 0.37 and on
Property changes on: trunk/debian
___________________________________________________________________
Name: svn:ignore
+ piuparts.debhelper.log
piuparts.postinst.debhelper
piuparts.substvars
files
piuparts.prerm.debhelper
piuparts
Added: trunk/debian/NEWS
===================================================================
--- trunk/debian/NEWS (rev 0)
+++ trunk/debian/NEWS 2009-03-20 12:33:51 UTC (rev 296)
@@ -0,0 +1,12 @@
+piuparts (0.36) unstable; urgency=low
+
+ * piuparts 0.36 introduces many changes to master-slave mode, please read
+ debian/changelog carefully.
+ * Backwards compatibility of the configuration files for master-slave-mode
+ has been broken, as the three config files got merged into one:
+ /etc/piuparts/piuparts/piuparts.conf
+ * piuparts-reports has been re-added and improved.
+ * The documentation files have been merged and converted to asciidoc.
+
+ -- Holger Levsen <holger at debian.org> Tue, 10 Mar 2009 15:23:59 +0100
+
Modified: trunk/debian/changelog
===================================================================
--- trunk/debian/changelog 2009-03-20 11:35:16 UTC (rev 295)
+++ trunk/debian/changelog 2009-03-20 12:33:51 UTC (rev 296)
@@ -43,7 +43,10 @@
unversioned and add build-dependencies for debhelper and asciidoc.
* Rewrite debian/rules using debhelper.
* Merge README, how-to-use-piuparts.txt and custom-scripts.txt into
- README.txt and build pdf and html versions of it.
+ README.txt and build pdf and html versions of it. The README also has been
+ restructured and updated to reflect the configuration changes in
+ master-slave mode.
+ * Add NEWS file.
-- Holger Levsen <holger at debian.org> Tue, 10 Mar 2009 15:23:59 +0100
Modified: trunk/piuparts-report.py
===================================================================
--- trunk/piuparts-report.py 2009-03-20 11:35:16 UTC (rev 295)
+++ trunk/piuparts-report.py 2009-03-20 12:33:51 UTC (rev 296)
@@ -274,7 +274,7 @@
</tr>
<tr class="normalrow">
<td class="contentcell2">
- 2009-03-19: lenny2squeeze is not needed, so all logs for squeeze (as well as lenny2squeeze) were deleted. (As squezze now includes two kinds of tests: installation and removal in squeeze, and installation in lenny, upgrade to squeeze, removal in squeeze.
+ 2009-03-19: lenny2squeeze is not needed, so all logs for squeeze (as well as lenny2squeeze) were deleted. (As squeeze now includes two kinds of tests: installation and removal in squeeze, and installation in lenny, upgrade to squeeze, removal in squeeze.
</td>
</tr>
</table>
@@ -327,7 +327,7 @@
"packages-url": None,
"master-directory": ".",
},
- ["output-directory", "packages-url", "master-directory"])
+ ["output-directory"])
def setup_logging(log_level, log_file_name):
@@ -357,7 +357,7 @@
def emphasize_reason(str):
if str == "unknown-package" or str == "failed-testing":
- str = "<em" + str + "</em>"
+ str = "<em>" + str + "</em>"
return str
More information about the Piuparts-commits
mailing list