[Piuparts-commits] [SCM] piuparts git repository branch, develop, updated. 0.52-19-g9617a30
Holger Levsen
holger at layer-acht.org
Tue May 28 12:49:42 UTC 2013
The following commit has been merged in the develop branch:
commit 9617a308b2a7d818bbdb5bc3e1b8ca0968b001fc
Author: Holger Levsen <holger at layer-acht.org>
Date: Tue May 28 14:49:20 2013 +0200
Move master-slave related documentation from README.txt to README_server.txt.
* Move master-slave related documentation from README.txt to
README_server.txt and leave a pointer in the former.
* piuparts-master.deb: Ship README_server.html.
* piuparts-report.py: Add a link to README_server.html.
diff --git a/Makefile b/Makefile
index f5ae690..50ebc40 100644
--- a/Makefile
+++ b/Makefile
@@ -23,7 +23,7 @@ endif
SCRIPTS_TEMPLATES = $(wildcard *.in master-bin/*.in slave-bin/*.in conf/*.in)
SCRIPTS_PYTHON_BINARY = $(wildcard *.py master-bin/*.py slave-bin/*.py)
SCRIPTS_GENERATED = $(SCRIPTS_TEMPLATES:.in=) $(SCRIPTS_PYTHON_BINARY:.py=)
-DOCS_GENERATED = piuparts.1 piuparts.1.html piuparts_slave_run.8 piuparts_slave_join.8 README.html
+DOCS_GENERATED = piuparts.1 piuparts.1.html piuparts_slave_run.8 piuparts_slave_join.8 README.html README_server.html
define placeholder_substitution
sed -r \
@@ -57,6 +57,9 @@ build-doc: $(DOCS_GENERATED)
README.html: README.txt
a2x --copy -a toc -a toclevels=3 -f xhtml -r /etc/asciidoc/ README.txt
+README_server.html: README_server.txt
+ a2x --copy -a toc -a toclevels=3 -f xhtml -r /etc/asciidoc/ README_server.txt
+
piuparts.1: piuparts.1.txt
a2x -f manpage piuparts.1.txt
@@ -71,7 +74,7 @@ piuparts.1.html: piuparts.1.txt
install-doc: build-stamp
install -d $(DESTDIR)$(docdir)/
- install -m 0644 README.txt README.html docbook-xsl.css $(DESTDIR)$(docdir)/
+ install -m 0644 README.txt README.html README_server.html docbook-xsl.css $(DESTDIR)$(docdir)/
install -d $(DESTDIR)$(man1dir)
install -m 0644 piuparts.1 $(DESTDIR)$(man1dir)/
install -d $(DESTDIR)$(man8dir)
@@ -161,7 +164,7 @@ check:
clean:
rm -f build-stamp
rm -f $(DOCS_GENERATED)
- rm -f piuparts.1.xml README.xml docbook-xsl.css piuparts.html
+ rm -f piuparts.1.xml README.xml README_server.xml docbook-xsl.css piuparts.html
rm -f *.pyc piupartslib/*.pyc master-bin/*.pyc slave-bin/*.pyc
rm -f $(SCRIPTS_GENERATED)
diff --git a/README.txt b/README.txt
index 8ca0553..f3c7f9f 100644
--- a/README.txt
+++ b/README.txt
@@ -257,484 +257,6 @@ exit 0
== Distributed testing
-As part of the quality assurance effort of Debian, piuparts is
-run on the Debian package archive. This requires a lot of
-processing power, and so the work can be distributed over several
-hosts.
-
-There is one central machine, the master, and any number of slave
-machines. Each slave machine connects to the master, via ssh, and
-runs the piuparts-master program to report results of packages it
-has tested already, and to get more work.
-
-To set this up for yourself, the following steps should suffice:
-
-. Pick a machine to run the master. It cannot be a chroot, but
- basically any real (or properly virtualized) Debian system is good
- enough.
-. Install the package piuparts-master on it.
-. Create an account for the master (the package creates piupartsm).
-. Configure '/etc/piuparts/piuparts.conf' appropriately.
-. Pick one or more slaves to run the slave. You can use the machine
- running the master also as a slave. Etch is fine, it can even be
- in a chroot.
-. Install the package piuparts-slave on it.
-. Configure '/etc/piuparts/piuparts.conf' appropriately - if master
- and slave share the machine, they also share the config file.
-. Create an account for the slave. This must be different from the
- master account. (The package creates piupartss.)
-. Create an ssh keypair for the slave. No passphrase.
-. Add the slave's public key to the master's '.ssh/authorized_keys'
- The key should be restricted to only allow running
- 'piuparts-master' by prefixing it with
- 'command="/usr/share/piuparts/piuparts-master",no-pty,no-port-forwarding '
-. Configure sudo on the slave machine to allow the slave account
- run '/usr/sbin/piuparts' as root without password (otherwise
- you'll be typing in a password all the time).
-. Run '/usr/share/piuparts/piuparts-slave' on the slave accounts.
- Packages that are installed want to use '/dev/tty', so you can't
- do this from cron. Also, you'll want to keep an eye on what is
- happening, to catch runaway processes and stuff.
-. The logs go into the master account, into subdirectories.
-
-Please note that running piuparts this way is somewhat risky, to
-say the least. There are security implications that you want to
-consider. It's best to do it on machines that you don't mind
-wiping clean at a moment's notice, and preferably so that they
-don't have direct network access.
-
-
-=== Distributed piuparts testing protocol
-
-The slave machine and the piuparts-master program communicate
-using a simplistic line based protocol. SSH takes care of
-authentication, so there is nothing in the protocol for that. The
-protocol is transaction based: the slave gives a command, the
-master program responds. Commands and responses can be simple (a
-single line) or long (a status line plus additional data lines).
-Simple commands and responses are of the following format:
-
- 'keyword arg1 arg2 arg3 ... argN'
-
-The keyword is a command or status code ("ok"), and it and the
-arguments are separated by spaces. An argument may not contain a
-space.
-
-A long command or response is deduced from the context: certain
-commands always include additional data, and certain commands
-always get a long response, if successful (error responses are
-always simple). The first line of a long command or response is
-the same as for a simple one, the additional lines are prefixed
-with a space, and followed by a line containing only a period.
-
-A sample session (">>" indicates what the slave sends, "<<" what
-the master responds with):
-
-----
-<< hello
->> section sid
-<< ok
->> pass liwc 1.2.3-4
->> The piuparts
->> log file comes
->> here
->> .
-<< ok
->> reserve
-<< ok vorbisgain 2.3-4
-----
-
-Here the slave first reports a successful test of package liwc,
-version 1.2.3-4, and sends the piuparts log file for it. Then it
-reserves a new package to test and the master gives it
-vorbisgain, version 2.3-4.
-
-The communication always starts with the master saying "hello".
-The slave shall not speak until the master has spoken.
-
-Commands and responses in this protocol:
-
-----
-Command: section <string>
-Success: ok
-Failure: error
-Failure: busy
-----
-Slave asks master to select the given section.
-This must be the very first command sent by the slave, but may
-be repeated later on to switch between sections.
-It will return "error" if the section is unknown and "busy" if
-it is currently processed by another master instance. If the
-section command fails, no other commands than "section" will be
-allowed until one succeeds.
-
-----
-Command: recycle
-Success: ok
-Failure: error
-----
-Slave asks master to enable logfile recycling mode. In this mode
-logfiles that have been marked for rechecking will be deleted
-and reissued in subsequent "reserve" commands. The "recycle"
-command must be issued before the first "reserve" (or "status")
-command. It will return "error" if no more logfiles are marked
-for rechecking or the command is issued too late.
-
-----
-Command: idle
-Success: ok <int>
-----
-Slave asks master whether it remembers having no packages
-available at a previous "reserve" command. Returns 0 (not known
-to be idle or timeout expired) or the number of seconds until
-the master wants to recompute the package state. This command
-should be given after "recycle" and logfile submission, but
-before "reserve" or "status" commands. If the slave closes the
-connection without issuing a "reserve" or "status" command, the
-expensive Packages file parsing and status computation will be
-skipped.
-
-----
-Command: reserve
-Success: ok <packagename> <packageversion>
-Failure: error
-----
-Slave asks master to reserve a package (a particular version of
-it) for the slave to test. The slave may reserve any number of
-packages to test. If the transaction fails, there are no more
-packages to test, and the slave should disconnect, wait some time
-and try again.
-
-----
-Command: unreserve <packagename> <packageversion>
-Success: ok
-----
-
-Slave informs master it cannot test the desired version of a
-package and the package should be rescheduled by the master.
-
-----
-Command: pass <packagename> <packageversion>
- log file contents
- .
-Success: ok
-----
-
-Slave reports that it has tested a particular version of a
-package and that the package passed all tests. Master records
-this and stores the log file somewhere suitable.
-
-----
-Command: fail <packagename> <packageversion>
- log file contents
- .
-Success: ok
-----
-
-Same as "pass", but package failed one or more tests.
-
-----
-Command: untestable <packagename> <packageversion>
- log file contents
- .
-Success: ok
-----
-
-Slave informs master it cannot test the desired version of a
-package (perhaps it went away from the mirror?).
-
-----
-Command: status
-Success: ok <package-state>=<count> <package-state>=<count>...
-----
-Slave asks master to report the number of packages in all
-different states. The "status" command should only be issued
-after all logs have been transmitted ("pass", "fail", and
-"untestable" commands).
-
-In all cases, if the master cannot respond with "ok" (e.g.,
-because of a disk error storing a log file), it aborts and the
-connection fails. The slave may only assume the command has
-succeeded if the master responds with "ok".
-
-The master may likewise abort, without an error message, if the
-slave sends garbage, or sends too much data.
-
-
-=== piuparts.conf configuration file
-
-piuparts-master, piuparts-slave and piuparts-report share the
-configuration file '/etc/piuparts/piuparts.conf'. The syntax is
-defined by the Python ConfigParser class, and is, briefly, like
-this:
-
-----
- [master]
- foo = bar
-----
-
-==== global configuration
-
-These settings have to be placed in the [global] section and are
-used for all further sections.
-
-* "sections" defaults to sid and defines which sections should be
- processed in master-slave mode. Each section defined here has to
- have a section with the section specific settings explained below.
- The first section defined should always be sid, because the data
- from first section a package is in is used for the source package
- html report.
-
-* "master-host" is the host where the master exists. The slave will
- give this host to ssh. This option is mandatory.
-
-* "master-user" is the username of the master. The slave will log in
- using this username. This option is mandatory.
-
-* "master-directory" is the directory where the master keeps its
- files. Can be relative to the master's home directory.
-
-* "slave-directory" is the directory where the slave keeps its
- files. Can be relative to the slave's home directory.
-
-* "slave-count" is the number of concurrent slaves to start.
- Default: "1".
-
-* "output-directory" is the directory where piuparts-report places
- the logfiles, generated html files, charts, ... that can be
- served by a webserver.
-
-* "backup-directory" is the directory where the prepare_backup
- script will place copies of the history data needed to generate the
- plots. This directory should be included in system backups while
- the logfiles and html pages in 'master-directory' and
- 'output-directory' (several GB of data) are regeneratable with some
- effort and can be excluded from backups. By default this is
- undefined meaning that no backups of the history data will be made.
-
-* "doc-root" is the location where the webserver will serve the
- piuparts report from. Default: "/".
-
-* "slave-load-max" specifies the system load limit when
- piuparts-slave will enter sleep mode. Operation will be resumed
- after load drops below 'slave-load-max - 1.0'. Floating point
- value. Defaults to 0 (= disabled).
-
-* "proxy" sets the http_proxy that will be used for fetching
- Packages files etc. (by master/slave/report) and .debs etc. (by
- piuparts). This will override a http_proxy setting in the
- environment. By default (no value being set) the http_proxy
- variable from the environment will be used (and no proxy will be
- used if this is not set). It is highly recommended to use a proxy
- running on localhost (e.g. installing squid and using a setting of
- "http://localhost:3128") due to the high bandwidth consumption of
- piuparts and repeated downloading of the same files.
-
-==== section specific configuration
-
-The section specific settings will be reloaded each time a section
-is being run. All these keys can be specified in the [global]
-section, too, and will serve as defaults for all other sections
-(overriding the builtin defaults).
-
-* "master-command" is the command to run on master-host to start
- the master. When the master has been installed from the Debian
- package, the command is '/usr/share/piuparts/piuparts-master'.
- This does not need to be set here if it is already set in
- '~piupartsm/.ssh/authorized_keys' to limit ssh access to that
- single command.
-
-* "idle-sleep" is the length of time the slave should wait before
- querying the master again if the master didn't have any new
- packages to test. In seconds, so a value of 300 would mean five
- minutes, and that seems to be a good value when there are fairly
- few slaves per master. The default is 300 seconds.
-
-* "max-tgz-age" is used to specify the maximum age (in seconds)
- after which basesystem tarballs will be recreated. If recreation
- fails, the old tarball will be used again. The default is 2592000
- seconds, which is 30 days. A value of 0 disables recreation.
-
-* "min-tgz-retry-delay" is used to specify the minimum time (in
- seconds) between attempts to recreate a tarball which was created
- more than "max-tgz-age" seconds ago. The default is 21600 seconds,
- which is 6h.
-
-* "log-file" is the name of a file to where the master should write
- its log messages. In the default configuration file it is
- "$SECTION/master.log". To disable logging, set it to "/dev/null".
- The global "log-file" setting (defaulting to master-error.log) is
- used for logging stderr output from piuparts-master. This logfile
- will be placed in the 'master-directory' and has the PID appended.
-
-* "piuparts-command" is the command the slave uses to start
- piuparts. It should include 'sudo' if necessary so that piuparts
- runs with sufficient priviledges to do its testing (and that
- means root priviledges). This command should be given in the
- [global] section and include all flags that are common for all
- sections.
-
-* "piuparts-flags" are appended to "piuparts-command" and should
- contain the section-specific flags.
-
-* "tmpdir" is the scratch area where piuparts will create the
- chroots. Note: the filesystem where this is located must not be
- mounted with the nodev or nosuid options. This is a mandatory
- setting with no default. The scripts that are monitoring this
- directory for leftover mountpoints and chroots only evaluate the
- [global] setting.
-
-* "description" is a synopsis of the test used in the report. A
- default description will be generated if this is not set or will
- be prepended (appended) if the description starts (ends) with
- '+'.
-
-* "mirror" tells the slave which mirror it is to use. The slave
- gives this to piuparts when it runs it. The URLs for Packages and
- Sources files will be generated from this setting, too. Default
- (for fetching Packages/Sources): "http://cdn.debian.net/debian".
-
-* "distro" is the distribution the slave should tell piuparts to
- use for basic install/purge testing. It is also possible to use a
- "partial" distribution as defined in distros.conf. No default.
- If 'upgrade-test-distros' is set, this selects the distribution
- that will be used for getting the packages to be tested. Defaults
- to the last entry in 'upgrade-test-distros', but other useful
- settings are the first entry (to test upgrades of "disappearing"
- packages) or the restricted set in a partial distribution (e.g.
- stable to backports to testing).
-
-* "area" is the archive area to use, set to one of main, contrib,
- non-free. Default: "main".
-
-* "arch" is the architecture to use. Default: dpkg
- --print-architecture.
-
-* "chroot-tgz" is the name of the file the slave should use for
- the tarball containing the base chroot. The default name is
- generated automatically from the "distro" or "upgrade-test-distros"
- setting. If the tarball doesn't exist, the slave creates it.
-
-* "basetgz-directory" is the directory where "chroot-tgz" (or the
- automatically selected default name) is located. The default is
- '.'.
-
-* "upgrade-test-distros" is the space delimited list of
- distributions the slave should use for testing upgrades
- between distributions (i.e., Debian versions). Using "partial"
- distributions as defined in distros.conf is possible. Currently,
- "squeeze wheezy sid" is a good choice.
- Setting this switches from doing install/purge tests to
- dist-upgrade tests. Not set by default.
-
-* "max-reserved" is the maximum number of packages the slave will
- reserve at once. It should be large enough that the host that
- runs master is not unduly stressed by frequent ssh logins and
- running master (both of which take quite a bit of CPU cycles),
- yet at the same time it should not be so large that one slave
- grabs so many packages all other slaves just sit idle. The number
- obviously depends on the speed of the slave. A good value seems
- to be enough to let the slave test packages for about an hour
- before reporting results and reserving more. For a contemporary
- AMD64 machine with a reasonably fast disk subsystem the value 50
- seems to work fine. To disable a section set this to 0.
-
-* "keep-sources-list" controls whether the slave runs piuparts
- with the '--keep-sources-list' option. This option does not
- apply to upgrade tests. The value should be "yes" or "no", with
- the default being "no". Use this option for dists that you need
- a custom sources.list for, such as "stable-proposed-updates".
-
-* "precedence" controls the order the sections are being processed
- by the slave. Sections with a larger precedence value will be run
- only if all sections with a smaller precedence value are idle,
- i.e. master does not have any packages that this slave could
- test. Sections with the same precedence value will be processed
- round-robin until they are all idle (or a more important section
- has packages to be tested). The default is 1.
-
-* "depends-sections" lists additional sections that will be
- searched for dependencies that are not available in the current
- section if that describes a partial distro.
-
-* "known-problem-directory" is the path to the directory containing
- definitions of known problems.
- Default: "${prefix}/share/piuparts/known_problems"
-
-* "debug" tells the slave whether to log debug level messages. The
- value should be "yes" or "no", with the default being "no".
- piuparts itself currently always produces debug output and there
- is no way to disable that.
-
-* "PYTHONPATH" (global) sets the search path to the piupartslib
- python modules if they are not installed in their default location
- in /usr.
-
-* "reschedule-untestable-days" (global) sets the rescheduling
- delay for untestable packages (e.g. due to unsatisfied
- dependencies). This is handled by the 'report_untestable_packages'
- script and the default is "7" days.
-
-* "reschedule-old-days" (global, section) and the following five
- settings define the rescheduling scheme that it performed by the
- 'reschedule_oldest_logs' script. Passed/failed logs that are
- older than reschedule-(old|fail)-days will be marked for
- rechecking (limited to reschedule-(old|fail)-count). Only packages
- that are actually testable will be reissued by piuparts-master (and
- the "old" log will be deleted at that time). Logs that are marked
- for recycling but have not been rechecked due to missing/failing
- dependecies will be deleted anyway if they are older than
- expire-(old|fail)-days.
-
-* "reschedule-old-count" (global, section) is the maximum number of
- passed logs that will be marked for recycling. Set to 0 to disable
- rescheduling passed logs.
-
-* "expire-old-days" (global, section) can be set to a value larger
- than 'reschedule-old-days' to delete logs older than the setting
- that are marked for recycling but haven't been rechecked due to
- failing or missing dependecies. Disabled by default ("0").
-
-* "reschedule-fail-days" (global, section) sets the minimum age of
- failing logs (fail/*.log or affected/*.log) before they will be
- rechecked.
-
-* "reschedule-fail-count" (global, section) is the maximum number
- of failed logs that will be marked for recycling. Set to 0 to
- disable rescheduling failed logs.
-
-* "expire-fail-days" (global, section) can be set to a value larger
- than 'reschedule-fail-days' to delete logs older than the setting
- that are marked for recycling but haven't been rechecked due to
- failing or missing dependecies. Disabled by default ("0").
-
-* "auto-reschedule" (section) can be set to "no" to disable
- rescheduling of passed and failed packages. To disable only
- rescheduling one of passed or failed logs, set the corresponding
- -count variable to zero.
-
-Some of the configuration items are not required, but it is best
-to set them all to be sure what the configuration actually is.
-
-==== piuparts.debian.org specific configuration
-
-In addition to some of the above settings the following
-configuration settings are used by the scripts in '~piuparts?/bin/'
-used to run piuparts.debian.org. They are all optional, default
-values are set in the scripts.
-
-* "urlbase" (global) is the base url of the webserver serving this
- piuparts instance. Used to provide links to logfiles in email
- reports.
-
-
-=== Running piuparts in master-slave mode, piuparts-report and the setup on piuparts.debian.org
-
-If you want to run piuparts-report (which is only+very useful if
-you run piuparts in master-slave mode), you need to 'apt-get
-install python-rpy r-recommended r-base-dev'. For more
-information see
-link:http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_server.txt[http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_server.txt] and
-link:http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt[http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt].
+This is described in README_server.txt.
// vim: set filetype=asciidoc:
diff --git a/README_server.txt b/README_server.txt
index 3a6e8d0..fbf428f 100644
--- a/README_server.txt
+++ b/README_server.txt
@@ -1,3 +1,53 @@
+piuparts README_server
+----------------------
+
+Author: Lars Wirzenius, Holger Levsen and Andreas Beckmann
+Email: <liw at iki.fi>
+
+== piuparts in master/slave mode
+
+As part of the quality assurance effort of Debian, piuparts is
+run on the Debian package archive. This requires a lot of
+processing power, and so the work can be distributed over several
+hosts.
+
+There is one central machine, the master, and any number of slave
+machines. Each slave machine connects to the master, via ssh, and
+runs the piuparts-master program to report results of packages it
+has tested already, and to get more work.
+
+To set this up for yourself, the following steps should suffice:
+
+. Pick a machine to run the master. It cannot be a chroot, but
+ basically any real (or properly virtualized) Debian system is good
+ enough.
+. Install the package piuparts-master on it.
+. Create an account for the master (the package creates piupartsm).
+. Configure '/etc/piuparts/piuparts.conf' appropriately.
+. Pick one or more slaves to run the slave. You can use the machine
+ running the master also as a slave. Etch is fine, it can even be
+ in a chroot.
+. Install the package piuparts-slave on it.
+. Configure '/etc/piuparts/piuparts.conf' appropriately - if master
+ and slave share the machine, they also share the config file.
+. Create an account for the slave. This must be different from the
+ master account. (The package creates piupartss.)
+. Create an ssh keypair for the slave. No passphrase.
+. Add the slave's public key to the master's '.ssh/authorized_keys'
+ The key should be restricted to only allow running
+ 'piuparts-master' by prefixing it with
+ 'command="/usr/share/piuparts/piuparts-master",no-pty,no-port-forwarding'
+. Configure sudo on the slave machine to allow the slave account
+ run '/usr/sbin/piuparts' as root without password (otherwise
+ you'll be typing in a password all the time).
+. Run '/usr/share/piuparts/piuparts-slave' on the slave accounts.
+ Packages that are installed want to use '/dev/tty', so you can't
+ do this from cron. Also, you'll want to keep an eye on what is
+ happening, to catch runaway processes and stuff.
+. The logs go into the master account, into subdirectories.
+
+=== Setup from piuparts-master and piuparts-slaves packages
+
The piuparts-server package installs a piuparts server along the lines of
http://piuparts.debian.org/.
@@ -24,3 +74,450 @@ particular, piuparts-report will create a web summary, defaulting to
http://localhost/piuparts, served by Apache. Uncomment the lines in the cron
file to enable the jobs.
+=== Setup from GIT
+
+http://piuparts.debian.org has been set up directly from GIT, this is
+described in '/usr/share/doc/piuparts-master/README_pejacevic.txt'.
+
+== Distributed testing
+
+WARNING: Please note that running piuparts this way is somewhat risky, to
+say the least. There are security implications that you want to
+consider. It's best to do it on machines that you don't mind
+wiping clean at a moment's notice, and preferably so that they
+don't have direct network access.
+
+=== Distributed piuparts testing protocol
+
+The slave machine and the piuparts-master program communicate
+using a simplistic line based protocol. SSH takes care of
+authentication, so there is nothing in the protocol for that. The
+protocol is transaction based: the slave gives a command, the
+master program responds. Commands and responses can be simple (a
+single line) or long (a status line plus additional data lines).
+Simple commands and responses are of the following format:
+
+ 'keyword arg1 arg2 arg3 ... argN'
+
+The keyword is a command or status code ("ok"), and it and the
+arguments are separated by spaces. An argument may not contain a
+space.
+
+A long command or response is deduced from the context: certain
+commands always include additional data, and certain commands
+always get a long response, if successful (error responses are
+always simple). The first line of a long command or response is
+the same as for a simple one, the additional lines are prefixed
+with a space, and followed by a line containing only a period.
+
+A sample session (">>" indicates what the slave sends, "<<" what
+the master responds with):
+
+----
+<< hello
+>> section sid
+<< ok
+>> pass liwc 1.2.3-4
+>> The piuparts
+>> log file comes
+>> here
+>> .
+<< ok
+>> reserve
+<< ok vorbisgain 2.3-4
+----
+
+Here the slave first reports a successful test of package liwc,
+version 1.2.3-4, and sends the piuparts log file for it. Then it
+reserves a new package to test and the master gives it
+vorbisgain, version 2.3-4.
+
+The communication always starts with the master saying "hello".
+The slave shall not speak until the master has spoken.
+
+Commands and responses in this protocol:
+
+----
+Command: section <string>
+Success: ok
+Failure: error
+Failure: busy
+----
+Slave asks master to select the given section.
+This must be the very first command sent by the slave, but may
+be repeated later on to switch between sections.
+It will return "error" if the section is unknown and "busy" if
+it is currently processed by another master instance. If the
+section command fails, no other commands than "section" will be
+allowed until one succeeds.
+
+----
+Command: recycle
+Success: ok
+Failure: error
+----
+Slave asks master to enable logfile recycling mode. In this mode
+logfiles that have been marked for rechecking will be deleted
+and reissued in subsequent "reserve" commands. The "recycle"
+command must be issued before the first "reserve" (or "status")
+command. It will return "error" if no more logfiles are marked
+for rechecking or the command is issued too late.
+
+----
+Command: idle
+Success: ok <int>
+----
+Slave asks master whether it remembers having no packages
+available at a previous "reserve" command. Returns 0 (not known
+to be idle or timeout expired) or the number of seconds until
+the master wants to recompute the package state. This command
+should be given after "recycle" and logfile submission, but
+before "reserve" or "status" commands. If the slave closes the
+connection without issuing a "reserve" or "status" command, the
+expensive Packages file parsing and status computation will be
+skipped.
+
+----
+Command: reserve
+Success: ok <packagename> <packageversion>
+Failure: error
+----
+Slave asks master to reserve a package (a particular version of
+it) for the slave to test. The slave may reserve any number of
+packages to test. If the transaction fails, there are no more
+packages to test, and the slave should disconnect, wait some time
+and try again.
+
+----
+Command: unreserve <packagename> <packageversion>
+Success: ok
+----
+
+Slave informs master it cannot test the desired version of a
+package and the package should be rescheduled by the master.
+
+----
+Command: pass <packagename> <packageversion>
+ log file contents
+ .
+Success: ok
+----
+
+Slave reports that it has tested a particular version of a
+package and that the package passed all tests. Master records
+this and stores the log file somewhere suitable.
+
+----
+Command: fail <packagename> <packageversion>
+ log file contents
+ .
+Success: ok
+----
+
+Same as "pass", but package failed one or more tests.
+
+----
+Command: untestable <packagename> <packageversion>
+ log file contents
+ .
+Success: ok
+----
+
+Slave informs master it cannot test the desired version of a
+package (perhaps it went away from the mirror?).
+
+----
+Command: status
+Success: ok <package-state>=<count> <package-state>=<count>...
+----
+Slave asks master to report the number of packages in all
+different states. The "status" command should only be issued
+after all logs have been transmitted ("pass", "fail", and
+"untestable" commands).
+
+In all cases, if the master cannot respond with "ok" (e.g.,
+because of a disk error storing a log file), it aborts and the
+connection fails. The slave may only assume the command has
+succeeded if the master responds with "ok".
+
+The master may likewise abort, without an error message, if the
+slave sends garbage, or sends too much data.
+
+
+=== piuparts.conf configuration file
+
+piuparts-master, piuparts-slave and piuparts-report share the
+configuration file '/etc/piuparts/piuparts.conf'. The syntax is
+defined by the Python ConfigParser class, and is, briefly, like
+this:
+
+----
+ [master]
+ foo = bar
+----
+
+==== global configuration
+
+These settings have to be placed in the [global] section and are
+used for all further sections.
+
+* "sections" defaults to sid and defines which sections should be
+ processed in master-slave mode. Each section defined here has to
+ have a section with the section specific settings explained below.
+ The first section defined should always be sid, because the data
+ from first section a package is in is used for the source package
+ html report.
+
+* "master-host" is the host where the master exists. The slave will
+ give this host to ssh. This option is mandatory.
+
+* "master-user" is the username of the master. The slave will log in
+ using this username. This option is mandatory.
+
+* "master-directory" is the directory where the master keeps its
+ files. Can be relative to the master's home directory.
+
+* "slave-directory" is the directory where the slave keeps its
+ files. Can be relative to the slave's home directory.
+
+* "slave-count" is the number of concurrent slaves to start.
+ Default: "1".
+
+* "output-directory" is the directory where piuparts-report places
+ the logfiles, generated html files, charts, ... that can be
+ served by a webserver.
+
+* "backup-directory" is the directory where the prepare_backup
+ script will place copies of the history data needed to generate the
+ plots. This directory should be included in system backups while
+ the logfiles and html pages in 'master-directory' and
+ 'output-directory' (several GB of data) are regeneratable with some
+ effort and can be excluded from backups. By default this is
+ undefined meaning that no backups of the history data will be made.
+
+* "doc-root" is the location where the webserver will serve the
+ piuparts report from. Default: "/".
+
+* "slave-load-max" specifies the system load limit when
+ piuparts-slave will enter sleep mode. Operation will be resumed
+ after load drops below 'slave-load-max - 1.0'. Floating point
+ value. Defaults to 0 (= disabled).
+
+* "proxy" sets the http_proxy that will be used for fetching
+ Packages files etc. (by master/slave/report) and .debs etc. (by
+ piuparts). This will override a http_proxy setting in the
+ environment. By default (no value being set) the http_proxy
+ variable from the environment will be used (and no proxy will be
+ used if this is not set). It is highly recommended to use a proxy
+ running on localhost (e.g. installing squid and using a setting of
+ "http://localhost:3128") due to the high bandwidth consumption of
+ piuparts and repeated downloading of the same files.
+
+==== section specific configuration
+
+The section specific settings will be reloaded each time a section
+is being run. All these keys can be specified in the [global]
+section, too, and will serve as defaults for all other sections
+(overriding the builtin defaults).
+
+* "master-command" is the command to run on master-host to start
+ the master. When the master has been installed from the Debian
+ package, the command is '/usr/share/piuparts/piuparts-master'.
+ This does not need to be set here if it is already set in
+ '~piupartsm/.ssh/authorized_keys' to limit ssh access to that
+ single command.
+
+* "idle-sleep" is the length of time the slave should wait before
+ querying the master again if the master didn't have any new
+ packages to test. In seconds, so a value of 300 would mean five
+ minutes, and that seems to be a good value when there are fairly
+ few slaves per master. The default is 300 seconds.
+
+* "max-tgz-age" is used to specify the maximum age (in seconds)
+ after which basesystem tarballs will be recreated. If recreation
+ fails, the old tarball will be used again. The default is 2592000
+ seconds, which is 30 days. A value of 0 disables recreation.
+
+* "min-tgz-retry-delay" is used to specify the minimum time (in
+ seconds) between attempts to recreate a tarball which was created
+ more than "max-tgz-age" seconds ago. The default is 21600 seconds,
+ which is 6h.
+
+* "log-file" is the name of a file to where the master should write
+ its log messages. In the default configuration file it is
+ "$SECTION/master.log". To disable logging, set it to "/dev/null".
+ The global "log-file" setting (defaulting to master-error.log) is
+ used for logging stderr output from piuparts-master. This logfile
+ will be placed in the 'master-directory' and has the PID appended.
+
+* "piuparts-command" is the command the slave uses to start
+ piuparts. It should include 'sudo' if necessary so that piuparts
+ runs with sufficient priviledges to do its testing (and that
+ means root priviledges). This command should be given in the
+ [global] section and include all flags that are common for all
+ sections.
+
+* "piuparts-flags" are appended to "piuparts-command" and should
+ contain the section-specific flags.
+
+* "tmpdir" is the scratch area where piuparts will create the
+ chroots. Note: the filesystem where this is located must not be
+ mounted with the nodev or nosuid options. This is a mandatory
+ setting with no default. The scripts that are monitoring this
+ directory for leftover mountpoints and chroots only evaluate the
+ [global] setting.
+
+* "description" is a synopsis of the test used in the report. A
+ default description will be generated if this is not set or will
+ be prepended (appended) if the description starts (ends) with
+ '+'.
+
+* "mirror" tells the slave which mirror it is to use. The slave
+ gives this to piuparts when it runs it. The URLs for Packages and
+ Sources files will be generated from this setting, too. Default
+ (for fetching Packages/Sources): "http://cdn.debian.net/debian".
+
+* "distro" is the distribution the slave should tell piuparts to
+ use for basic install/purge testing. It is also possible to use a
+ "partial" distribution as defined in distros.conf. No default.
+ If 'upgrade-test-distros' is set, this selects the distribution
+ that will be used for getting the packages to be tested. Defaults
+ to the last entry in 'upgrade-test-distros', but other useful
+ settings are the first entry (to test upgrades of "disappearing"
+ packages) or the restricted set in a partial distribution (e.g.
+ stable to backports to testing).
+
+* "area" is the archive area to use, set to one of main, contrib,
+ non-free. Default: "main".
+
+* "arch" is the architecture to use. Default: dpkg
+ --print-architecture.
+
+* "chroot-tgz" is the name of the file the slave should use for
+ the tarball containing the base chroot. The default name is
+ generated automatically from the "distro" or "upgrade-test-distros"
+ setting. If the tarball doesn't exist, the slave creates it.
+
+* "basetgz-directory" is the directory where "chroot-tgz" (or the
+ automatically selected default name) is located. The default is
+ '.'.
+
+* "upgrade-test-distros" is the space delimited list of
+ distributions the slave should use for testing upgrades
+ between distributions (i.e., Debian versions). Using "partial"
+ distributions as defined in distros.conf is possible. Currently,
+ "squeeze wheezy sid" is a good choice.
+ Setting this switches from doing install/purge tests to
+ dist-upgrade tests. Not set by default.
+
+* "max-reserved" is the maximum number of packages the slave will
+ reserve at once. It should be large enough that the host that
+ runs master is not unduly stressed by frequent ssh logins and
+ running master (both of which take quite a bit of CPU cycles),
+ yet at the same time it should not be so large that one slave
+ grabs so many packages all other slaves just sit idle. The number
+ obviously depends on the speed of the slave. A good value seems
+ to be enough to let the slave test packages for about an hour
+ before reporting results and reserving more. For a contemporary
+ AMD64 machine with a reasonably fast disk subsystem the value 50
+ seems to work fine. To disable a section set this to 0.
+
+* "keep-sources-list" controls whether the slave runs piuparts
+ with the '--keep-sources-list' option. This option does not
+ apply to upgrade tests. The value should be "yes" or "no", with
+ the default being "no". Use this option for dists that you need
+ a custom sources.list for, such as "stable-proposed-updates".
+
+* "precedence" controls the order the sections are being processed
+ by the slave. Sections with a larger precedence value will be run
+ only if all sections with a smaller precedence value are idle,
+ i.e. master does not have any packages that this slave could
+ test. Sections with the same precedence value will be processed
+ round-robin until they are all idle (or a more important section
+ has packages to be tested). The default is 1.
+
+* "depends-sections" lists additional sections that will be
+ searched for dependencies that are not available in the current
+ section if that describes a partial distro.
+
+* "known-problem-directory" is the path to the directory containing
+ definitions of known problems.
+ Default: "${prefix}/share/piuparts/known_problems"
+
+* "debug" tells the slave whether to log debug level messages. The
+ value should be "yes" or "no", with the default being "no".
+ piuparts itself currently always produces debug output and there
+ is no way to disable that.
+
+* "PYTHONPATH" (global) sets the search path to the piupartslib
+ python modules if they are not installed in their default location
+ in /usr.
+
+* "reschedule-untestable-days" (global) sets the rescheduling
+ delay for untestable packages (e.g. due to unsatisfied
+ dependencies). This is handled by the 'report_untestable_packages'
+ script and the default is "7" days.
+
+* "reschedule-old-days" (global, section) and the following five
+ settings define the rescheduling scheme that it performed by the
+ 'reschedule_oldest_logs' script. Passed/failed logs that are
+ older than reschedule-(old|fail)-days will be marked for
+ rechecking (limited to reschedule-(old|fail)-count). Only packages
+ that are actually testable will be reissued by piuparts-master (and
+ the "old" log will be deleted at that time). Logs that are marked
+ for recycling but have not been rechecked due to missing/failing
+ dependecies will be deleted anyway if they are older than
+ expire-(old|fail)-days.
+
+* "reschedule-old-count" (global, section) is the maximum number of
+ passed logs that will be marked for recycling. Set to 0 to disable
+ rescheduling passed logs.
+
+* "expire-old-days" (global, section) can be set to a value larger
+ than 'reschedule-old-days' to delete logs older than the setting
+ that are marked for recycling but haven't been rechecked due to
+ failing or missing dependecies. Disabled by default ("0").
+
+* "reschedule-fail-days" (global, section) sets the minimum age of
+ failing logs (fail/*.log or affected/*.log) before they will be
+ rechecked.
+
+* "reschedule-fail-count" (global, section) is the maximum number
+ of failed logs that will be marked for recycling. Set to 0 to
+ disable rescheduling failed logs.
+
+* "expire-fail-days" (global, section) can be set to a value larger
+ than 'reschedule-fail-days' to delete logs older than the setting
+ that are marked for recycling but haven't been rechecked due to
+ failing or missing dependecies. Disabled by default ("0").
+
+* "auto-reschedule" (section) can be set to "no" to disable
+ rescheduling of passed and failed packages. To disable only
+ rescheduling one of passed or failed logs, set the corresponding
+ -count variable to zero.
+
+Some of the configuration items are not required, but it is best
+to set them all to be sure what the configuration actually is.
+
+==== piuparts.debian.org specific configuration
+
+In addition to some of the above settings the following
+configuration settings are used by the scripts in '~piuparts?/bin/'
+used to run piuparts.debian.org. They are all optional, default
+values are set in the scripts.
+
+* "urlbase" (global) is the base url of the webserver serving this
+ piuparts instance. Used to provide links to logfiles in email
+ reports.
+
+
+=== Running piuparts in master-slave mode, piuparts-report and the setup on piuparts.debian.org
+
+If you want to run piuparts-report (which is only+very useful if
+you run piuparts in master-slave mode), you need to 'apt-get
+install python-rpy r-recommended r-base-dev'. For more
+information see
+link:http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_server.txt[http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_server.txt] and
+link:http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt[http://anonscm.debian.org/gitweb/?p=piuparts/piuparts.git;hb=master;a=blob;f=README_pejacevic.txt].
+
+// vim: set filetype=asciidoc:
diff --git a/TODO b/TODO
index 08e8b3e..2e874ae 100644
--- a/TODO
+++ b/TODO
@@ -24,9 +24,8 @@ for 0.5x:
- maybe do this manually again as well?
- Documentation related:
- - Move master-slave related documentation from README.txt to README_server.txt
- and leave a pointer in the former. Link README_server.html on piuparts.d.o.
- examples are duplicated in piuparts.1.txt and README.txt - only keep one copy.
+ - README_server has some duplication information on configuration as well.
- in maintainer view:
- sort buggy states up again
diff --git a/debian/changelog b/debian/changelog
index 2d0f41d..3081cdf 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -31,6 +31,10 @@ piuparts (0.53) UNRELEASED; urgency=low
[ Holger Levsen ]
* piuparts.1.txt, README.txt and README_pejacevic.txt: proof-read and
cleaned up.
+ * Move master-slave related documentation from README.txt to
+ README_server.txt and leave a pointer in the former.
+ * piuparts-master.deb: Ship README_server.html.
+ * piuparts-report.py: Add a link to README_server.html.
-- Andreas Beckmann <anbe at debian.org> Mon, 27 May 2013 18:24:27 +0200
diff --git a/debian/piuparts-master.docs b/debian/piuparts-master.docs
index c4cb52b..695daae 100644
--- a/debian/piuparts-master.docs
+++ b/debian/piuparts-master.docs
@@ -1,3 +1,4 @@
+README_server.html
README_server.txt
README_pejacevic.txt
conf/piuparts.conf.sample
diff --git a/piuparts-report.py b/piuparts-report.py
index 668f9eb..2af62b0 100644
--- a/piuparts-report.py
+++ b/piuparts-report.py
@@ -116,12 +116,14 @@ HTML_HEADER = """
</tr>
<tr class="normalrow">
<td class="contentcell">
- <a href="$doc_root/doc/README.html" target="_blank">piuparts README</a>
+ <a href="$doc_root/doc/piuparts.1.html" target="_blank">piuparts manpage</a>
</td>
</tr>
<tr class="normalrow">
<td class="contentcell">
- <a href="$doc_root/doc/piuparts.1.html" target="_blank">piuparts manpage</a>
+ <a href="$doc_root/doc/README.html" target="_blank">piuparts README</a>,
+ <br>
+ <a href="$doc_root/doc/README_server.html" target="_blank">README_server</a>
</td>
</tr>
<tr class="normalrow">
--
piuparts git repository
More information about the Piuparts-commits
mailing list