[Piuparts-devel] Bug#670150: Bug#670150: piuparts: rescheduling old logs by piuparts-master

[Dave Steele]

> Hi,
>
> there were thoughts to move rescheduling of old logs into the master,
> here come a few more.
>

I'm working on a flexible-retest candidate that works closer to the
current design, while maintaining priority for new packages, and
aggressively retesting the rest of the time. It keeps the current
offline log deletion mechanism and master-slave protocol, but adds
state to the master to identify overall idle periods.

The work builds on the master-idle branch currently in github. This
branch implements the master 'idle.stamp' you mention, and returns
'busy' to slaves accessing that section during the idle timeout
period. Master does not compute package states in this case.
idle.stamp is only created in response to a slave 'reserve' call, and
is deleted when stale.

The code is untested. Applicable commit at:

    https://github.com/davesteele/piuparts/commit/c44051892ce3c285583135d1eaa8f817f5c977ca

The offline retest manager would monitor for all managed sections to
be in the idle state, and respond by proportionally culling old logs
in each of the sections - possibly throttling the rate to keep the
retest load steady across the log lifetime. If logs are deleted, then
the idle.stamp is deleted as well. There is no 'recycle' mode for the
slave - it has no knowledge of the process.

I considered adding a "can I sleep" message for the slave to call
before entering idle in the main loop, allowing the master to work
retests on the fly. But, that proved to be a bit invasive, since the
master's realm currently is limited to a single section.

Support for your directory of hard links (or soft links?) could be
added to support arbitrary rescheduling of logs.

> Having some efficient way to minimize master communication is important
> for me: > 50 sections, 4-7 slaves running in parallel (slaves sharing a
> slave tree (for concurrent processing of different sections) as well as
> slaves on different slave trees (for concurrent processing on the same
> section, e.g. on full recheck).

Wow. I'm going to have to start running a second section someday.

The current priority mechanism can't possibly be holding up well with
that many sections (OTOH, not everything is as busy as sid). Do we
need to talk about the separate manager process again (remove package
state processing from the slave path, largely by adding a
waiting-to-be-tested directory to master sections)?

'Slaves sharing a slave tree' is another discussion. I thought you
would want to be able to have all of those slave commit to the same
section simultaneously, if necessary (without having to reconfigure
the slave layout). Note that I made this a failure condition in one of
the commits in master-idle. That commit is not critical to the branch
- it could be changed to just delete the sleep_until line.