[Piuparts-devel] setting priority of sections

[Dave Steele]

On Sat, Nov 26, 2011 at 9:33 AM, Holger Levsen <holger at layer-acht.org> wrote:
> Hi Dave,
>
> On Samstag, 26. November 2011, Dave Steele wrote:
>> Here is a candidate patch that does this, if you replace the word
>> 'section' with 'state'.
>
> while I applaud your patch in principle, I dont like this. I really would like
> to be able to absolutly prioritize main over non-free no matter how well non-
> free is tested. And then, main over bpo over contrib over non-free... :)
>
> And then the states...

Fair enough. I didn't address section priorities at all - that came up
(for me) after I started.

It would be a simple matter to add section priority logic to the inner
loop in piuparts-lave main(), replacing "for section in sections:".
That would allow it to behave as you describe.

>
>> This version will test only 'waiting' packages if there are any.
>> Otherwise, it will process failed logs over a threshold age. After
>> that, 'successful' logs are rerun.
>
> Does it handle replacement of logs? I have to admit I didnt look at the diff
> closely yet :)

reserve( ^ waiting ) transitions packages from failed/successful to
'waiting' by removing the log file and adjusting the _in_state lists
(which btw need to be unique'd more). The subsequent test puts a new
log file in place.

>
>> 1) There is much downloading/parsing of the Packages file, and
>> recalculating package states going on. Both are expensive.
>
> For the first problem, we should fix really fix proxy support :) (It's broken
> currently, try it... not sure if there is a bug.)

Yes, this part calls for caching and IMS support.

>
> For the 2nd, well, it's slightly inevitable and then I also dont think those
> 1-2min hurt that much.
>

The point here is that the number of times this calculation must be
performed per run may explode as priority logic is implemented
(particularly if priorities are multi-dimensional, e.g. section and
state).

>> 2) The master process doesn't like package state changes it didn't
>> cause. There are opportunities for race conditions with multiple
>> slaves/master processes, which would need to be worked out.
>
> There should always only be one master...

This is not enforced. If you have two slaves, you can have two active
masters working in the same section tree. I assumed that is why there
is a shuffle() call in reserve_package().

Bug? I don't think so, currently. Reserve() handles races in creating
the reserve log file.

>
> How do you cause rescheduling? The best is really to just delete the log.
>

That is essentially what the patch is doing. The trick is to do that
automatically, while maintaining a quick response to new submissions.
If master could provide visibility into the size of his prioritized
work queues, a fairly efficient compromise could be implemented in a
cron job (delete log files to get the queue up to a threshold number
of entries).

On the other hand, such a job would wreak havoc on the priority
strategy as Andreas described it (every section but the lowest would
be starved for resources). Maybe retry support needs to remain
embedded along the lines of this patch.

> Probably best to move this to a wishlist bug in the BTS.
>

It's probably better off just hanging out in the ether. The value for
me is learning the challenge involved in maintaining responsiveness to
new submissions. This code took 7 minutes to get back to looking at
the 'waiting' queue, despite the prioritization placed on that queue,
and having no other work to do.

That, and it doesn't address your priorities (double entendre intended).

It's just something to think about as you work on adding priority
support. The best bang-for-the-buck may come from a different angle of
attack, such as tailored/schedule-able slaves, or a persistent master
daemon.



The Piuparts sid statistics over the last couple of days show piatti
running way below the '3 day sid' benchmark mentioned in the wiki. The
summary suggests that it would take more like 3 months to work off the
"broken symlink" warnings. Do we know why it is taking so long to
rerun the failed queue?