[Soc-coordination] Third report: Provide some metrics in Debile

[Clément Schreiner]

Hi,

this is third first weekly report on my Summer of Code project 'Provide
some metrics in Debile'[1][2].

clang build
-----------

I have finally succeeded in making debile build packages with clang. The
bug in my code that I was stuck on last week was a stupid typo, but then
I had another problem. In order to make sbuild use clang instead of
gcc/g++, we use a (hideous) hack[3], courtesy of Sylvestre: with a
chroot hook, the gcc and g++ binaries are replaced by clang's before the
build starts. However, sbuild has not been able to run chroot hooks as
root for a few years:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840.

Fortunately, someone has posted a patch[3] on that thread allowing just
that, which I applied[4]. That taught me how to use quilt for
adding patches to a debian package.

clanganalyzer/scan-build
------------------------

This is still work in progress. I ported the existing debile plugin
(developed by Léo Cavaillé during his GSoC last year) to the current
debile-slave API. It took me some time to realize the plugin was using
an option not available in sbuild vanilla, and I had to patch it again,
this time using Léo's[5].

At the moment, while clanganalyzer jobs seem to run correctly, they
don't actually use scan-build, for some reason. I have to look into
that.

debile-incoming (huge) memory consumption and bad performances
--------------------------------------------------------------

Yesterday evening, tried to login on ursae (the serveur hosting
debile-master and http://debile.debian.net/), and the ssh connection would
timeout, or I would get a "broken pipe" error. I suspected the swap was
full, which was confirmed when I finally got a prompt. A few hours
later, I managed to login again and type pkill debile, and the system
was responsive again.

The culprit was debile-incoming, the script that scans debile-master's
upload queue for importing new packages or job results into the
database.

A cron was executing debile-incoming every minute, which did not
terminate in that time: dozens of processes were running at the same
time. Worse: for some reason, it uses about 2GiB of ram (or even 5GiB in
some cases), and now takes at least a hour to finish its work.

I run a profiler on debile-incoming[6], which confirmed my suspicion that
it was because of firewoes[7]' analysis of firehose XML files.

Nicolas Dandrimont gave me some hints to reduce python's memory use[8],
but I'll also need to improve the algorithm, since firewoes seems unable
to process in a reasonable time the huge files that some checks are
generating.

loofah/ricky
------------

In order to test debile in a more realistic scale, I used loofah[9] to
send dozens of packages to debile. This made me aware of
debile-incoming's ram/performance problems above.


That's it for this week. Thanks for reading.


---------
Clément


[1] [https://wiki.debian.org/SummerOfCode2014/StudentApplications/ClementSchreiner]
[2]
[http://lists.alioth.debian.org/pipermail/soc-coordination/2014-May/002037.html]
[3] [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840#27]
[4] [http://www.mux.me/debile/]
[5]
[http://leo.cavaille.net/public/gsoc/sbuild-patches/0001-Added-support-for-schroot-sessions.patch]
[6] [http://paste.debian.net/103741/]
[7] [https://github.com/Debian/firewoes/]
[8]
[http://guillaume.segu.in/blog/code/487/optimizing-memory-usage-in-python-a-case-study/]
[9] [https://github.com/sylvestre/loofah]