[Piuparts-devel] concerning master-slave packaging

[Dave Steele]

On Tue, Jun 12, 2012 at 7:46 PM, Andreas Beckmann <debian at abeckmann.de> wrote:
> Hi,
>
> IMHO piuparts-master and piuparts-slave need to be packaged
> independently. That makes the whole thing much more difficult and
> requires manual configuration (at least if a standalone slave is being
> set up). But it is the only useful solution for the use case: there is
> one master (and one reporting process + webserver), but many slaves on
> different (virtual) machines, serving possibly different architectures
> (maybe some in qemu).
>

I challenge the "only useful solution" part. It is not difficult to
set this up with a piuparts-server. If you are provisioning the VMs,
you have a recipe that pulls the slave code (piuparts-server) and the
correct configuration file. Add to that recipe a delete of the master
cron, apache conf, and authorized keys, and disable (or delete)
apache. That should give you effective standalone slaves. There is a
cost associated with carrying along apache and the master files, but
it is small relative to launching a separate VM in the first place.

I question whether there is substantial enough benefit there to
justify the cost.

> Even if it's today still unclear how multiple architectures should be
> handled in -master and -report ... we should prepare work with these in
> mind.

I haven't thought through multiple architecture considerations.

My second objection to master + slave is the idea that we (I, anyway)
don't know enough yet about how this should be set up.

Consider security. The slave runs, as root, code signed by any one of
1000+ keys. It may be prudent to sandbox it, and anything it touches,
from any public-facing interfaces. That implies grouping master and
slave together logically, and peeling off generate_daily_report and
its ilk into the separate package. I don't know yet if that is a
better way to group things. piuparts-server keeps the options open.

I just read your 173MB comment in another reply... Whoa. ... But, put
it in perspective. We were throwing away far more RAM than that for
hours a day before a two-line change in piuparts-report. This is still
a small percentage increase on VM size (I think).

Should all that cruft be separated from slave via piuparts-master, or
piuparts-public-display-thingie? Or is it even enough to matter? I
don't think we know the answer yet.

> And for the single architecture case that we do currently multiple
> *distributed* slaves will give nice speedups.
>

I'm on a new server (power supply failure). I had intended to go hog
wild with remote slaves, but I don't see the point now. The standalone
server went through sid in a day the first time through. It doesn't
seem worth it to distribute slaves for performance. Piatti should be
fine with a hardware upgrade. ... I don't know how I would feel with
50 sections.