[Piuparts-devel] Thoughts on masterd

[Dave Steele]

On Sun, Dec 4, 2011 at 5:55 AM, Holger Levsen <holger at layer-acht.org> wrote:
> Hi Dave,
>

> of course you're free to come to this conclusion, but let me assure you that
> I'm all for making piuparts-master faster. And maybe your proposal is the way
> to go! I just don't know...
>

Ok

> The problem is, your plans are quite intrusive and we have a reliably working
> master atm.

Ok

> So if you want me to replace that code with something else, you
> need to show me the code, not just a mere proposal.

I'm asking for feedback before investing a fair amount of work in the
change. That can lead to a more efficient use of available resources.

>
> As I see it, improving startup speed of master would gain/free 30-60 cpu
> minutes on piatti per day, at the maximum.

... for work done on the 'most important' priority section. You need
to multiply this times the number of priority levels for work done on
the lesser priorities. And there are latency implications.

> While the bottleneck really is disk I/O...

Yes, there would be significant gains from freeing the slaves from
spinning disk IO.

>
> All this said, having a masterd running all the time sounds like a good idea
>

I was proposing a cron process. The main feature was a masterd load
that is much more independent of the number of slaves and the number
of priority levels.

>
> I'm unsure whether to run a second slave on piatti (it has a dualcore cpu),

Ah, here is the disconnect. The code and web summaries suggested that
piatti was running a single slave, but I didn't believe it. If the
main piuparts server doesn't test in parallel, there is not much point
to the code changes I suggested.

A single slave means that the master process and the piuparts test
never run concurrently. Sounds like a RAM disk for the piuparts work
area may be viable (+ squid?).

With a significantly faster (single) slave, all of this talk about
implementing priorities and the cost of retesting would become largely
mute.

>
> (And piuparts-report should not take a quarter of the day...
>

I had not run piuparts-report. That does sound like a more obvious
target for work.

>
> But as said: give me patches which are proven to work, and I'll be thankful
> and gladly merge them! :-)
>

There's a higher bar to merging than that. You've convinced me that
this is not the way to go, at least for the time being. I didn't
understand piatti well enough to see the big picture.


Thanks for the feedback.


Dave