[Soc-coordination] Backend Tools and Infrastructure for DEX - Report 4

[Nathan Handler]

Hello everyone,

Due to the midterm evaluation, this report will cover a span of 4
weeks. As a result, it will be quite a bit longer than my previous
reports.

One of the first things I did after my last report was create a FAQ
for the project. At first, the FAQ was a plain text file living only
in the git repository. Then, it moved to being maintained on the
Debian wiki with a cronjob pulling it into my local repository for me
to commit and push. Finally, the cronjob moved to wagner. The FAQ is
not actually committed to the git repository, but it is available in
the dextools instance running on wagner [1] or on the wiki [2]. There
is also now a basic README file that is part of the git repository
that simply points people to the FAQ on the wiki (so people who branch
the repository can easily find the documentation).

For those of you who have been reading my other reports and/or testing
the dashboard, you have probably noticed the problems I was having
getting the text boxes sized properly. After spending countless hours
researching and testing different solutions, we finally decided to
give up on this for the time being. Therefore, all of the 'size'
attributes have been removed from the text inputs, and the background
color has been reset to its default value. This means that we will no
longer experience the issue of very wide text boxes and a horizontally
stretched table. However, it also means that long strings will get
cut-off when displayed in a text input.

Since my initial mockup design at the start of the summer, I have had
the rows of the table zebra-striped. Based on some feedback from Paul
Wise (pabs), I changed this so that the row colors are based on the
status of the task. Open tasks are red, closed tasks are green, and in
progress tasks are yellow. This makes it much easier to quickly see
the progress of the project and find tasks you are interested in.

My original dashboard consisted of one main dex.html file. This file
used some javascript to pull in each project's task list. We felt that
a much more common use case would involve a user only interested in
one particular project. As a result, they would not be likely to use
the select boxes to change projects. This ultimately led to me
creating standalone dex.html dashboards for each project. These
per-project dashboards were identical to the original dex.html, except
they did not include the select boxes at the top for changing
projects. Eventually, I decided to completely get rid of the original
dex.html dashboard. That file now consists of a simple list of links
to all of the projects sorted by distribution. Another advantage of
this change is that it is much easier to link to a specific project.
For example, you might now link to
/projects/dex-ubuntu-ancient-patches/dex.html (you can get the URL by
simply copying it from the address bar). Another advantage is that you
can now link directly to a specific project's graphs (just append
#graph).

Another feature that I am still working on is per-project
documentation. From the start, we wanted to make it easy for new
contributors to get involved in DEX. One way we can do this is by
having clear documentation about what the project is and how to deal
with tasks. I decided that the easiest approach would be to have this
documentation live on the wiki. This allows users to use the familiar
wiki markup language to format their text, and it gives us all of the
other benefits a wiki provides. Documentation will live at
/DEX/<distribution>/<project>. The dashboard will then take care of
pulling in this documentation and displaying it at the top of the
proper page. For a while, I was thinking that it would be necessary to
filter the HTML produced by the wiki to prevent malicious code from
ending up on the dashboard. However, after playing around for a bit, I
think that it should be pretty safe to just use the HTML as-is (if
someone spams the wiki, we can always revert the change). I am
currently finishing up some of the code to handle this, and it should
be live sometime this week (a partially working version is live on
wagner).

There are also two new checkboxes at the top of each dashboard. These
allow users to add completed tasks and/or tasks with completed bugs.
This is useful for projects that are close to being complete, as it
allows users to quickly view the few remaining tasks. This feature was
requested by several people, and I am glad to finally be able to
implement it for them.

There is also a second graph on each page. This graph is a bar graph
that shows the number of tasks each person has completed. It gets the
people from the 'assignee' column for the project, and it only counts
tasks with a status of 'Complete'. The graph is updated whenever the
getbugs.py script runs to update the task listing (currently every 10
minutes). The bars are sorted from tallest to shortest. The idea
behind this graph is to recognize the people working on a project.

The form processing script has not worked on wagner. This was mainly
due to a permission issue. Thanks to a hint from Raphaël Hertzog, I
was able to use setfacl to fix this. The form can now be submitted,
and it should successfully save all changes made via the web form.
This involved some updates to support the per-project dex.html files.
The form processing script will also ignore any status that is not
'complete', 'in progress', or 'incomplete', which prevents people from
messing up the dashboard by submitting invalid information. Finally,
the script will now try to use the HTTP_REFERER header to send the
user back to the correct page. If this header is not set, the user
will be sent back to the main dex.html index page.

Sorting the table has been partially broken since I started including
text inputs. I finally got this issue resolved, and I should now be
able to sort by text inputs, select boxes, or anything else I might
end up including in the table. You can test this by clicking on any
header in the table. I will probably add a tiny image to make it more
clear which header is being used to sort the table. By default, it is
sorted based on the status and then the task name.

The status column now has a select box for setting the status. Since
the row coloring is done via javascript, the minute you change the
status in the select box, the row color will change to reflect the new
status. Keep in mind, for the time being, any change you make via the
web form will be lost if you do not hit the 'Submit' button at the top
of the form.

That covers most of the dashboard changes that have taken place since
my last report. However, there were also some other changes that are
worth noting.

Matt Zimmerman, my original mentor, settled down from his traveling
and resumed serving as my mentor for the project. I would like to
thank Stefano Zacchiroli for all of his help this summer. I look
forward to working with him more in the future.

I have also started using the issue tracker feature on alioth [3] for
dextools. Right now, it is mainly being used to help me keep track of
bugs and features that I need to deal with. I am hoping that as
dextools starts to be used by more people that they will also help
report issues on the tracker. This issue tracker is a good source of
information for people interested in knowing what I will be working on
in the future. I will briefly talk about some of the more important
features that I will be working on.

The first issue is one that I have already touched upon in this
report. I want to sort out the per-project documentation. This is
rather important, as it will most likely be utilized by all projects
using the dashboard. It is also essential if we are to get new
contributors involved in DEX projects.

Second, I want to setup some method for backing up the projects/
directory on wagner. This directory is not maintained in the git
repository, and it contains all of the information submitted via the
web form and generated by the scripts. My current plan is to have this
directory maintained in its own VCS. Every time getbugs.py runs or the
form is submitted, a new commit will be made. This will allow us to
easily revert any changes (spam) made to the dashboard. Since we do
not foresee the dashboard being the target of much abuse, any user
will be able to modify data via the web form.

I also want to add the ability for users to create new projects and
task via the web. I envision this as a separate form where they enter
the distribution name, project name, a list of BTS bug numbers, and a
list of any additional tasks. The script will then take care of
applying the necessary usertag to the bugs and/or creating a 'tasks'
file in the projects directory for the non-bts tasks. Doing this will
make starting a new project trivial, and will eliminate the need for
VCS access to create projects that do not utilize BTS bugs.

Finally, I want to eliminate the need for the user to hit the 'Submit'
button to save their changes. Any change the user makes should be
submitted and saved instantly using AJAX. This will also help
eliminate some confusion that was caused by the row color instantly
changing when the status is changed. Some users thought that the color
change meant that the change was saved when it really was not.

Matt asked me the other day what features I think need to get added
before the dashboard is really ready for public use. While the
features and issues I mentioned above will help make the dashboard
easier to use, the dashboard is still perfectly usable without them.
There are only a few more weeks left in the Google Summer of Code
2011. By that time, all of these features and bugs should be sorted
out, and we should be able to properly announce the dashboard and make
it available for use on all DEX projects. That does not mean that the
dashboard will be "done". Instead, it will probably result in many new
bugs and feature requests being submitted. However, at that point, I
will make a very strong effort to keep the dashboard stable and to
prevent data loss.

I am looking forward to the finally finishing up this project and
seeing it used by the DEX team. As always, if you are interested in
helping out with DEX or dextools, my code is available in a git
repository [4] on alioth. There is also the new issue tracker [3]
where you can report bugs and request new features. Finally, you can
join the debian-derivatives mailing list, join #debian-derivatives on
OFTC, or contact me directly via email or on IRC (nhandler).

Thanks,
Nathan Handler (nhandler)

[1] http://dex.alioth.debian.org/gsoc2011/docs/FAQ
[2] http://wiki.debian.org/DEX/dextools
[3] https://alioth.debian.org/tracker/index.php?group_id=100600&atid=413120
[4] http://anonscm.debian.org/gitweb/?p=dex/gsoc2011.git