[Piuparts-devel] for preview: preview/submit-after-processing (was: Re: submitting logs after section has finished?)

[Dave Steele]

On Thu, Jun 21, 2012 at 3:57 AM, Andreas Beckmann <debian at abeckmann.de>wrote:

>
> On 2012-06-20 16:22, Dave Steele wrote:
> > I'd consider a couple of steps further. Submitting completed work across
> > sections should be job #1, above processing sections. And it could be
> good
>
> That would get much more invasive, but I'll look into it.


I'm not sure. Try 1) adding a 'push only' option to the slave connect
(needed for both of our strategies),  2) getting a list of sections with
finished work at the top of the loop, sorted by priority, and 3) if the
list has more than one entry, doing a 'push only' for all but the first.
(2) should be the most invasive part, and it doesn't look bad.


> We need to get
> experience it its worth the effort with the current patches in place.


Ok, but I haven't seen anything in the patches which should affect
performance here.


> So
> whenever the master is busy for that section, the slave won't be able to
> send the stuff ...
>

Yes - this logic represents a retry strategy. I think it is equivalent to
your 'connect at the end' strategy in the normal case, without needing the
second connect.

... There is a small race. Change (3) to flush all sections with work, to
fix the race. This restores the need for the second connect in the normal
case. ... or maybe look at the push after a section has been successfully
worked (i.e. at the end of the loop). This needs some thought. It is worth
trying to avoid the second connect in the typical case - the invasive
option here is getting master to not parse 'Packages' in the 'second
connect push' case.


> > for slaves to submit logs in batches much smaller than max_slaves ...
>
> If the slave is buffering a sensible amount of packages, e.g. 1 hour,
> would we really need this?
>
>
This is about the ... 'starving slowdown' situation I mentioned in the
timeout defaults thread. Maybe not a big deal, but the timing is currently
large relative to my patience watching it. If the slave work queue is an
hour, then the number of slaves working on a section will be non-optimal
for a multiple of that, as the blocks are cleared.

BTW, I've set the work queue to 20 minutes, based on a target of 5% master
processing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/piuparts-devel/attachments/20120621/97a46bc8/attachment.html>