[Soc-coordination] Fifth report: Provide some metrics in Debile

[Clément Schreiner]

Hi,

this is the fifth weekly report on my Summer of Code project 'Provide
some metrics in Debile'[1]. 

(Previous report:
[http://lists.alioth.debian.org/pipermail/soc-coordination/2014-June/002108.html])

scan-build/clanganalyzer
------------------------

Although the clanganalyzer was running correctly, its results did not
appear on debile-web. Indeed, the firehose wrapper[2] was not used at
all. I fixed that, and we now have firehose reports for clanganalyzer
builds on debile.d.n[3].

However, those are not very useful, especially because firewoes[4], the web
interface for firehose reports, is not yet used by debile-web.

Thus I have modified debile-slave jobs' API to allow uploading files in
addition to the dud (.changes-like files used internally by debile), log
and firehose files, and then made the clanganalyzer plugin upload clanganalyzer's
beautiful HTML reports:
[http://ursae.inria.fr/debile/files/default/openssl_1.0.1h-3/clanganalyzer_source/453/index.html]

debile-incoming performance
---------------------------

As I previously explained, debile-incoming is really slow (and RAM
consuming) when processing large firehose files. I have started looking
into that: I will try using raw SQL instead of the orm in the 'uniquify'
function in firewoes.lib.hash[5].

In the meantime, I found a workaround. Using flock, a nice tool in
util-linux, I make sure that debile-incoming is run only once: 
  flock -n /srv/debile/debile-incoming.lock debile-incoming .

At least the server won't be unreachable because the swap is full again.


rebuilding the debian archive, and building NEW packages
--------------------------------------------------------

In order to be able to rebuild existing and NEW packages, I have
packaged ricky[6] and loofah[7]. Sylvestre sponsored me, and they are
waiting in NEW[8][9]. fred[10] will soon follow.

firewoes integration
--------------------

I have setup an instance of firewoes on my debile-development VM, and
can confirm that it works fine with debile's database[11].

My next task will be to integrate firewoes to debile. I'll discuss that
with Sylvestre and the Tanglu[12] people, since they are using debile in
production and we don't want to break anything, as well as Matthieu
Caneill and Stefano Zacchiroli (firewoes developers) if needed.


Thanks for reading,

-------
Clément


[1]
[https://wiki.debian.org/SummerOfCode2014/StudentApplications/ClementSchreiner]
[2]
[https://github.com/fedora-static-analysis/firehose/blob/master/firehose/parsers/clanganalyzer.py]
[3]
[http://ursae.inria.fr/debile/files/default/openssl_1.0.1h-3/clanganalyzer_source/453/openssl_1.0.1h-3_source.603.firehose.xml]
[4] [http://git.upsilon.cc/?p=firewoes.git]
[5]
[https://github.com/Debian/firewoes/blob/master/firewoes/lib/hash.py]
[6] [https://github.com/paultag/ricky]
[7] [https://github.com/paultag/loofah]
[8] [https://ftp-master.debian.org/new/ricky_0.1-1.html]
[9] [https://ftp-master.debian.org/new/loofah_0.1-1.html]
[10] [https://github.com/paultag/fred]
[11] [http://www.mux.me/debile/debile-firewoes.png]
[12] [http://tanglu.org/en/]