[Soc-coordination] [GSoC] Using a messaging system to interface the different parts of Debian infrastructure

[Nicolas Dandrimont]

Hi!

Adding soc-coordination at l.a.d.o in the loop and keeping context for them.

* Simon Chopin <chopin.simon at gmail.com> [2013-04-09 15:12:56 +0200]:

> Hi!
> 
> I'm a prospective GSoC student as well as a Debian user and contributor,
> and was looking for project ideas to submit for this year's GSoC when
> Nicolas Dandrimont (olasd) mentionned to me fedmsg[1] and the current
> state of interfaces throughout the Debian infrastructure.

Well, you beat me to writing that mail. Great!

> As far as I understand it, there is no standard in the infrastructure
> for providing interfaces. Most of what I heard and read was ad hoc
> interfacing and heavy use of cronjobs to keep it all in sync.
> 
> Fedora is in the process of implementing a message bus architecture,
> fedmsg, that addresses the exact same issues. I was wondering if there
> was people interested in using the same technology for our
> infrastructure. I've sent this mail to the teams I assume would be
> interested.

One of the key outcomes of getting such a system in place, is that everyone,
everywhere, can start listening to the messages and using them, opening up lots
of doors for people to make amazing services based on Debian.

A few ideas:
 - getting a signal from the archive on an accepted package (I'm confusing
   binaries and sources for the sake of brevity):
   → Trigger a piuparts run
   → Trigger lintian checks
   → Let any derivative intent a rebuild
   → Signal ports to rebuild
   → Trigger a jenkins job on specific package uploads
   → Post to pump.io/identi.ca/twitter
   → get a notification on your desktop
   → ...
 - one of your pet packages gets a git commit
   → try a rebuild
   → run QA checks
   → ...

(boy, that escalated quickly)

I think the possibilities are quite nice, and, as the fedmsg webpage says, that
"gives new meaning to open infrastructure".

> Assuming interest, I was thinking of the following plan for the GSoC
> project:
> 
>   * Packaging fedmsg

I actually think this packaging work should start now. There are a bunch of
dependencies (of which only one seems fairly "big", moksha.hub), and the sooner
the better.

>   * Make dak (or whoever else wanting to be the guinea pig) send out
>     messages on diverse events (NEW processing, testing migration...)

Starting with dak might be a tad ambitious, but that's because I have no idea
whatsoever how pluggable its message sending system is. Maybe the -dak people
could chime in?

I think a reasonable approach could be:

- setting up the bus, writing the code to adapt it to our way of life

For instance, we're heavy GPG users, whereas fedmsg uses SSL for its message
authentication, we need to think about the PKI bits or adapt fedmsg to our web
of trust.

This doesn't sound so bad, but actually I think settling on something right
will be the challenging part.

- wiring some services as a proof of concept.

What easy services am I thinking about? The wiki (setting a test moinmoin
instance up is a piece of cake, plus it's Python), the planet, and commit hooks
on Alioth (which is non-DD-accessible, making it easy-ish to try stuff out for
real). The idea is to play it realistic, proving that the thing works and
enticing service maintainers to plug themselves to the bus.

>   * Make the QA clients (and others if they volunteer) of dak use this
>     new interface instead of the old one.

One thing that should be clear, is that while I think that fedmsg is great for
streaming updates, nothing beats a good ol' sync from time to time. AFAICT,
nothing tells you that you have received all the messages.

Furthermore, answering Raphaël's question, there is no replay service as of
yet. I talked a bit to the fedora people about this, and currently, there is a
service (datanommer) that sinks the fedmsg bus into a database. They intend to
give it an API, but it doesn't exist yet, so as of now you'd need database access.

>   * Lather, rince, repeat with another piece of the infrastructure.

If time permits :)

> About myself:
> I'm a 23 year old CS student, in second year at the University of Rennes
> 1, in France. In Debian, I've mostly done Python packaging within the
> Debian Python Modules Team and Python Applications Packaging Team, but
> I'm also remotely interested in QA and am curious about the whole
> infrastructure aspect of the distribution.
> 
> So, anyone interested in the project, esp. for mentoring ? :-)

I'd be glad to. It'd be interesting to get a willing service maintainer or two
onboard, so that real services can get wired into the PoC. Also, maybe someone
from Fedora could be interested in joining (I haven't talked about that yet, so
it's just an idea).

> Cheers,
> Simon
> 
> [1] http://fedmsg.com/

Thanks,
-- 
Nicolas Dandrimont

BOFH excuse #89:
Electromagnetic energy loss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/soc-coordination/attachments/20130409/5f482de0/attachment.pgp>