[Soc-coordination] Report 3 : improving the online package review interface for mentors.d.n

[Baptiste Mouterde]

Dear Mailing list,

Here is the third report on improving the online package review
interface for mentors.debian.net (debexpo),

_So what's done yet?_

__Code Browser__

___source viewer___
  The first work and the most interesting work was on creating the
code browser for the source browser, I started by looking at what
exists online, I saw the github code browser... uh. a lot of JS,
HTML5 features, sometimes not valid, I found there a lot of Ideas
and a lot of errors to avoid.
   The first try was a table with one column for line number
and one other for code but.. when you try to select code you select
lines number too. Bad Idea.. I try to fix that using magic JS but that
wasn't
working.
   I looked again at git hub and I use the same trick, a table with
2 column and only 1 lines and then <pre> will make the work for line

___files viewer___
   The tree viewer uses the same engine as the source viewer.
The backend just generates a special file with links to the contents
of the directory to be rendered. Currently, it uses the absolute path
to the file (like /var/www/foo) it may not be the best thing
to show but it will be fixed soon.
   Comments will work here too, if someone has the strange idea of
commenting the name
of a file.. (this could also be used for general comments about a file)

___syntax Highlighting___
   table was done, I needed to find some code and have some syntax
Highlighting, I first tried to imagine how I could make it, using a lot
of regex and string parsers and also switch-that-doesn't-exist-in-python.
I spent 10 minutes on that and I asked on IRC, they told me to look
at python-pygments, which is a nice python syntax highlighting library that
produces clear html. You can choose almost all the options you need, it's
perfect.
    Pygments works fine, the .html file is ready, so now was time to
integrate it
with pylons. I make a template from the .html file and I add pylons and
mako requirement,
I create something to read pygments' work. A lot of troubles began here:
pygments creates a string with html tags,  I write it into a file, then a
controller
opens this file, sends it to mako that prints a file, and... I got
the weirdest rendering possible, pygments has safety checks that protects
from
malicious code executing, pylons/mako have some too. Moreover
I would like to have utf-8 files. all those stuff mixed give a weird result.
I lost here a lot of time to fix it, decoding, encoding utf-8, tell mako
that your file is safe, that the file contains html tags etc...
    I finally succeeded and I got my code browser working without color.
Because syntax highlighting without color is a little bit stupid,
I asked pygments to create the css from its "emacs" color style, I changed
it to a
dark version that I found fancy, but it wasn't the global feeling, sadly.
I created a white version with colors that fit to the rest of the project.
    I added a few stuff like anchors on lines, line coloration on hover
and I started to create the comments panel, it's yet not really the best
one but
with minor css / js adjustments it may be good. it's a panel hidden and that
comes when you click on the comments button, I tried to make something that
won't hide or break the rest of the source code (as github does it [1]).
It will allow discussion too. I made minor changes to allow code browsing
with
a mobile phone (apparently some people uses it from mobile phone...).
   a preview can be seen here [2]

__Git Storage__

    A huge amount of work has been done to deal with the git storage, first
of all there
 is some hard requirement, that it will use python-dulwich[3], a python lib
that adds
the support of git to python but opposite of python-git which uses
 the high level git commands, it recreates the whole low-level storage
model of git in pure python.

In order to follow this last point, we wanted to avoid the use of git shell
commands in my work.
and this was a major difficulty, I spent a lot of time in the dulwich doc,
I tried
to practice on my server, creating a repository, commiting, playing with
trees and blobs.

Following the basic how to was pretty easy but when you want to go deeper,
docs are
harder to find. I asked a few questions on the dulwich channel on IRC about
basic stuff
in order to start with good knowledge and then I started creating the
so-called-plugins
(plugins in debexpo can't really be unplugged), it more like a lib than a
plugin. I created a
commit function, an init one, and other basic functions without any
troubles.

   But I wanted the git storage for two main reasons, first of all I wanted
to have
an easy access to the source code from packages, and I wanted to have an
history of
changes. Perfect git shell can handle both natively but not dulwich, there
is no
simple way to create files that have changed between two commits. And the
only tool I found to do it
was a write_tree_diff that makes the same thing than a git diff, a patch
file with --- and +++.
I searched for a magic lib to use that patch without trouble but apparently
such a thing doesn't exist.
That's why I created a buildTreeDiff function that apply the patch.
Yet it's working pretty fine and creates, where you want, the files that
have changed between two trees.
   The second part of the work was to have all the sources, not only files
that have changed,
I looked for the shutil.copytree function, that can copy files from a place
to an other excluding some files
and can make links instead of files copy. It was too perfect to work fine,
as you can copyTree if the dest folder already exist.
I first used a trick that makes a temp folder then moves all the files and
clears temp. It is now working fine.

   But I was not proud at all of my last works, the buildTreeDiff parses
the string using
a dirty function, it's only a for loop and "if foo in" stuff and it was
obviously not the
good way to do it. Moreover the copyTree trick was dirty too, making a temp
folder is
not a good way to do it. But the BuildTreeDiff may be a really useful
function
(and it may interested some people that use the git shell) so I want to
keep it, maybe in a cleaner version.

   In the last few days I cleaned that and changed almost everything, now
it's building blobs
from the object store, by walking the tree from the commit parent. It's
cleaner than string parsing.
It allows me to uses the copytree too because instead of writing inside a
file, I create a
list of blob ids and a list of files that didn't change at the same time.
Using this trick I can apply
first the copyTree then the blob builder.
   I added an other useful function that will help me retreive the history
of a file,
which gets all the previous version of a file.

__msg_service__
   this stuff is done and working but I have to fix the ajax feature.
I still have trouble to send data from js to python. But it's working
in the "classical" pylons work.

_what is missing?_

  History on files and folders
  ajax on msg_service
  comments (on database and the engine)
  blog post on dulwich (pending)

_what were the main difficulties?_

  On work on Dulwich, I lost a lot of time in the doc,
as well as in practicing..
  On the last part (git storage) I think I didn't take
enough time talking about it. That's why I didn't succeed
at making clean code
  Managing time was a bit difficult, because I got
suddenly a lot of time and I was always saying that
I can do it later because of this. Now it's over because
I'm less lost in my project schedule too (I think I wrote
it a little be too fast).

_What is next?

  test the gitStorage with real packages
  Work on history and version tracker
  Work on DataBase for comments
  end the post on blog (I have almost 3 or 4 drafts waiting...)
  comments engine will require a lot of work because it will
upgrade the existing comments engine and implement a lot of the
will-come-soon statistic engine
  this statistic engine is the karma and activity tracker that will be
used for the sort interface, and it will be the next huge piece of cake.
  I'm preparing also a "web saga" on my blog that deals with how I see
the web and stuff around since I'm on the internet.

Thanks for reading!
Again Feel free to query me on Irc for any question, idea, or information!
Good luck!
ko_lo

[1]
https://github.com/rails/rails/commit/0495736339df48a625fd58f4304b7b497b02d859#L0R33
[2] http://kolo.fr/codebrowser.html
[3]http://www.samba.org/~jelmer/dulwich/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/soc-coordination/attachments/20120701/78771ea3/attachment-0001.html>