[Piuparts-devel] setting priority of sections

[Dave Steele]

On Wed, Nov 23, 2011 at 9:19 AM, Andreas Beckmann <debian at abeckmann.de> wrote:
> hi,
>
> I was just thinking about a small priority system
>
> * each section in piuparts.conf has a priority value (default: 1)
> * sections with equal priority will be run round-robin as it is done now
> * sections will be run in the order of increasing priority value
> * sections with a higher priority value will only send log files but not
> request new work or run pending work as long as a section with lower
> priority value has done some work
>
> (s/priority/some better name/ e.g. rank, precedence)
>
> example:
>
> 1: some-urgent-tests
> 5: stable, stable-updates, stable-security, stable2stable-updates,
> stable2stable-security
> 10: main (sid, wheezy)
> 20: contrib, non-free (sid, wheezy)
> 25: experimental
> 30: stable2testing oldstable2stable
> 40: testing2unstable
> 50: full-rebuild-test
>
>
>
> Andreas
>
> _______________________________________________
> Piuparts-devel mailing list
> Piuparts-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/piuparts-devel
>

Some thoughts:

I'm wondering how automated retesting would work into the system -
perhaps by setting failure retests lower than any of these priorities
(with retests of successful packages under that).

Under normal scenarios, there should be little discernible difference
as a result of implementing priorities. Given the (typically) small
number of packages to be processed in each queue, at least one of the
slaves will get around to the high priority task fairly quickly.

Your 'do this package now' and 'full rebuild' scenarios poke some
holes in the last paragraph.

The startup calculation that the 'master' process performs is fairly
expensive. This would require that process to be run many times over
for each pass at lower priority tasks.

You could emulate priority processing by throwing more slaves at the
problem, giving them tailored lists of sections to process. 'Do it
now' gets a dedicated slave in addition to the round-robin processing.
'Full rebuild' only gets built on a pre-defined schedule out of cron,
etc.

To do this cleanly would perhaps call for a dedicated master daemon.
That would make the queue preparation expense independent of the
number of slaves, and support event-based (re)test kickoff.