[Piuparts-devel] Bug#670150: Candidate retest manager

[Dave Steele]

tags 670150 patch
severity 670150 normal
thanks

I have a candidate implementation of a package retest manager
available in my retest-manager branch:

    https://github.com/davesteele/piuparts/tree/retest-manager

The architecture breaks down as follows:

- The master gains idle support by section, with state stored in an
idle.stamp file.
    - The section is idle when there are no packages to be tested,
including those in reserved.
- reschedule_oldest_logs is modified to monitor master idle state
across sections, and delete logs if the system is idle overall (and if
locks can be applied to all active sections).
    - Logs are deleted in section priority order.
    - The master section idle state is cleared if logs are deleted.
    - cron is modified to run reschedule-oldest-logs often.
- A light-weight 'idle' login response is added to the master-slave
protocol, piggy-backing on the existing 'busy' response from master
locking.
    - The slave responds to idle by closing the connection, and moving on.

To better support the feature, I made a few other changes:
- Add 'default section definition' support in piuparts.conf. If a
variable is not defined in a section, conf will check for a default
definition in 'global'. This reduces the number of variable
definitions needed per section, while supporting by-section retest
definitions to, for instance, use reschedule-old-days to cause a
section to be opportunistically re-run.
- Eliminate section idle processing from the slave. The slave relies
on the master to handle the section idle state.
- While I was in there I made multiple slaves operating in one section
directory tree an error condition. This is recommended, to better
support maintaining and reporting slave state. I consider this
necessary for creating a future piuparts service.
- Unit tests are added to unittest, for 'make check' support.

The changes to master, slave, and the protocol between them are
minimal. Slave processing is actually simpler. Additional
responsibility is focused on the recurring reschedule_oldest_logs
task.


Here is sample output from the new reschedule_oldest_logs (there is no
output if logs are not recycled):

>From piupartsm at foo Mon Jun 11 20:35:02 2012
From: root at foo (Cron Daemon)
To: piupartsm at foo
Subject: Cron <piupartsm at foo> /usr/share/piuparts/master/reschedule_oldest_logs
Date: Mon, 11 Jun 2012 20:35:01 -0400

reschedule_oldest_logs - Logs scheduled for retest

sid
    fail - 2, from 06/10/12 19:24 to 06/10/12 19:24
        rsyslog-mysql_5.8.11-1+b1.log
        rsyslog-pgsql_5.8.11-1+b1.log

    pass - 131, from 06/04/12 11:30 to 06/04/12 20:08
        cairo-dock-plug-ins_3.0.0-1+b1.log
        cairo-dock_3.0.0-2.log
        csound-gui_1:5.17.6~dfsg-3.log
        default-jdk_1:1.6-47.log
        electric_8.10-2.log
        festvox-kdlpc8k_1.4.0-5.log
...

The branch is divided as follows:

$ git log --pretty=oneline
ec68c5d8192b1fc36d36823d962a6ba149377c4c Reschedule cron to run
reschedule_oldest_logs often.
69818e9768e98d3cb73ea18f4cd066ae16934958 Add unit tests for
reschedule_oldest_logs to unittest
8edd0b2ff1fe6d4d3b2df4dc870817548533c326 Update conf file for new
reschedule_oldest_logs
80314b6dcba3941f24053434de22a619f8fa1d35 Update README.txt for
reschedule options.
dd6e43359999a1cd95790a525c4d7b4e21163278 Replace
reschedule_oldest_logs with an idle-savvy Python script
71244bdaa5472077b414804e80a8edba4da416c6 Modify sample config file to
use global defaults.
dfb702cd96d07c2d32cbda12de7a417f18b97525 Defaults for Section
variables can be defined in the global section.
3b03645f6f2c3918207f86143ebedac4698376c4 Add 'idle' as an option for
the master-to-slave greeting.
5b6f01949b78b39ef195c5a7d26ed0dd911493f2 Add idle-stamp support to
piuparts-master   497780ad4669488b89f8eeba2325dfc8e49b2205
piuparts-master - use relative path for the master directory.
cee20331d1dc64810bcdd2d950814b4ee5438ec2 Remove piuparts-slave.py
Section sleep logic
430bf25e9cf4664f5aa70c2f8816e2136c578a05 Multiple slaves in the same
directory tree is a failure condition.
be3bd7201fc4a6d047a41c6f02675fac64211214 Remove slave Section sleep
states on error conditions