[Soc-coordination] Aptitude GTK Remote Host Management Proposal

[Daniel Burrows]

On Wed, Apr 01, 2009 at 09:38:35PM -0400, Sagie <sagiea at bgu.ac.il> was heard to say:
> Bellow is the gist of my proposal. Any and all feedback much
> appreciated. 

  No, not another great proposal! :-P

> Add remote host package management through Aptitude GTK. The current
> local interaction with apt will be extended to span across SSH
> connections to remote hosts. In addition, the UI will be extended to
> incorporate a comparison view similar in nature to SVN code merge views.
> This will list differences in package configurations & versions between
> hosts while allowing to selectively synchronize differences. Design will
> aim at allowing future non GTK front-end developers to reuse
> functionality.

  Sagie,

  This is something I really have wanted to do for a long time.
However, getting aptitude to work over a network could be really, really
hard; I think you'll have to use some sort of a nasty hack to make it
even tractable at all.  Before I even began to consider this project I
would need to be convinced that you've thought through how to approach
this aspect of it and that you can produce something basically
functional in three months.  (more on this below)

  Also, we already have a lot of proposals for aptitude projects, and I
only feel comfortable trying to mentor one of them.  I'm just saying
that to warn you that there's a lot of competition here; if your main
goal is to participate in SOC, I would recommend filing a second
proposal alongside this one in case I end up picking a different
aptitude project.  (note that the deadline is *Friday*)

  However, your proposal sounds very well thought through and I am going
to take it seriously.  If you aren't picked, I would encourage you to
work on this anyway; I would be happy to work with you to integrate
patches that work in this direction.

> --- Sought Features & Implementation ---
> Basically the idea is simple : abstract the fact we are administering
> apt remotely - this will be achieved by generating an interface
> containing a most of the functionality present in 'apt.cc' which
> 'local_apt' & 'remote_apt' classes will implement. All remote apt
> functions will be piped through the SSH channel.

  This paragraph makes me wonder if you've contemplated the full scope
of what you need to do.  apt.cc is only the tip of the iceberg: it
handles initializing the standard apt structures and loading them into
memory.  It does **NOT** provide access to the apt database; that is
done in aptcache.cc, which extends some core apt classes found in
/usr/include/apt-pkg.  The apt database class, in turn, is linked to a
number of other classes (such as the raw package cache structures, as
well as package and version iterators) which expose the contents of the
cache.

  In short: you would have to essentially write a fully abstracted
apt-like library on top of apt for this to work.  And that would be
terrible for the performance of a local aptitude (many things that are
simple inline lookups would turn into function calls), while not really
working over the network (because there are just too many calls to apt
functions to tunnel efficiently over the network).

  I think a more pragmatic, albeit also uglier, approach is to simply
lock the package database and cache on the remote side, then *copy* all
the apt state structures over the network, copying them back when you're
done (of course, you can use rsync for this instead).




  The advantage of this is that then you only have to deal with making
most of aptitude stop using the global apt cache pointer (which it
shouldn't be using anyway) and start using some wrapper class instead.
It can crawl all over the package cache however much it wants without
worrying that it's a remote cache.  You may also need to fix some stuff
in apt; I think it tends to assume that there's only one copy of
everything and that it's in the standard places (but my RootDir patches,
which are integrated in sid, should help a bit here).

> [-] modular design
> All functionality related to package configuration comparison will be
> extracted to a separate module, possible compiled as a standalone
> library. This may be used by network management tools such as nagios.

  I'm not quite sure what this means in practical terms.

> [-] SSH communication channels with remote apt hosts
> SSH interaction may use pssh, openssh or similar. Although ultimately
> apt would adopt a full client-server model this is beyond the scope of
> this project. I am still investigating the applicability of using RPC to
> remotely invoke libapt. Output parsing is another option, though one
> which should be avoided if possible. Establishing exactly which files
> should be imported via the network is to be established after consulting
> the Debian community.

  The thought of trying to run apt calls through RPC makes my hair stand
on end.  aptitude probably makes, oh, a few million or so calls to apt
methods before it even shows you its main interface window.  I would be
shocked if the number is below several hundred thousand.  Can you tunnel
that over the network as synchronous RPC calls without having the
latency kill you?  (I'm pretty sure the answer is "no", but I haven't
used RPC lately)

  The other stuff is easy once you solve these problems; I won't comment
on it.

>    1. SSH based communication with remote apt based hosts, preferable as
> a standalone library
>    2. An abstraction layer between local & remote apt instances

  If you could get these two working reliably and not implement anything
else, I would consider your project a huge success (although in reality
you would have to write at least a little bit of frontend code to test
the backend).  The other stuff is not that hard once these two are
implemented, and if you submit a proposal I would recommend marking 3-6
as "reach" goals, to be dealt with if you manage to complete the meat of
your proposal ahead of schedule.  Obviously it would be even better to
have a strong UI for this, but these two are the hard sticking points.

  Right now I think your proposal is probably too much for you to bite
off over the summer, particularly since you don't seem to be very
familiar with apt's guts (something that you'll have to deal closely
with in order to carry the project out) but I am willing to be convinced
otherwise.

  Daniel