[Soc-coordination] [debmetrics] week 9 status update

[Joseph Bisch]

Hi all,

This is my status update for week 9. I have updated the instance at
http://metrics.debian.net so it is up-to-date.

This week
=======

This week I fixed debmetrics.wsgi for the new directory structure. I
added debian.css and added html to match the style and layout of
debian websites. I was able to use debian.css almost unmodified,
except for the path to the images. I was able to remove the reset.css
file, since debian.css has its own reset styles. I removed the tabbed
interface from the dynamic interface, so that the graph and table are
together. I added a csv download link to the dynamic interface. I
fixed the path to pull_runner.py in the crontab, since it changed due
to the directory structure change. I moved create_all.py to the root
directory, since running it from inside the debmetrics dir caused
problems with imports. I removed the quote function so that strings
are no longer quoted before putting them in the database. It was
necessary to do so for psycopg2, but is not necessary for sqlalchemy.

I swapped the metrics list and metric add button on the dynamic
interface so that it would have the more logical order of list then
button from left to right. I added some css so that the up/down arrows
in the table header would not overlap the header text. I added an
options panel to the dynamic interface that allows the user to adjust
the graph size and the min and max for the xaxis. I have the options
be populated with default values. I added links in the footer to
readthedocs and to a thank you page that thanks us and other
contributors, linking to the authors file in the repo.

I added jquery.dragtable.js, which is a plugin that allows table
columns to be drag and dropped to change the order of the columns. I
fixed the paths to modules in the docs, because the new directory
structure caused them to change. I made the options on the dynamic
interface an "accordion dropdown" so that the options would be hidden
by default and the options heading would need to be clicked to reveal
the options. I allowed multiple metrics to be added to the graph, so
that they can be compared. I made the columns for a metrics show/hide
on the graph when you click on the metric name in the legend. I added
a remove all metrics button that removes the data from the graph and
table. I added delete buttons to the legend. These can be used to
delete an individual column or all the columns for a metric.

I modified the code so that a metric can only be added once. Before it
allowed you to add the metric to the graph multiple times. I found a
bug in runner_helper.py where db_insert() was only adding the last row
to the db. I fixed it so that it adds all rows to the db. That means
package_stats now has historical data too, since the data feed
includes all the data. I added delete_on_insert to the manifests. It
means that the existing data in the db will be deleted from the db
prior to inserting the new data. It is for metrics like releases where
there would be multiple copies of the data if you don't delete the
existing data first. Releases uses an autoincrementing id so it won't
rollback even if the rest of the data is the same. Delete_on_insert
isn't necessary for something like package_stats because package stats
uses ts for its primary key, so if it sees a ts that already exists,
it will rollback. If there is a better way of fixing this, let me
know.

I added a grab handle to the table. Before the grab handle, dragtable
was interfering with tablesorter, because it didn't allow
tablesorter's up/down arrows to be clicked. I corrected the path to
the graphs in graph_helper.py. I added Session.commit() to
db_delete_all() to ensure that the deletion doesn't get rolled back.

Next week
========

For the static interface, I need to add a download link for the csv
file. The tabbed interface doesn't quite work. It requires setting a
height on the container. If the height of the table is greater than
that of the container then it looks weird. So I will make it display
the table below the graph without any tabs like the dynamic interface
does.

For the dynamic interface, I will make it possible to delete columns
from the table. I will also make it possible to add multiple metrics
to the table. Right now only the last metric added will display.

The combination of stickyheaders and dragtable cause the header to
appear twice when dragging a column. I will look into a fix or
alternative plugins that play together well.

If you add multiple metrics to the graph and they have different start
or end dates, they are hard to compare. I was thinking about possibly
only displaying the portion of the graph that is common to all the
metrics.

I fixed the releases metric, but not the releases_count metric, so I
will continue to look into why the data from releases_count is not
being inserted into the db. The error with releases was due to the
error with db_insert() only inserting the last row of data.

Tasks from last week to work on next week
===============================

There are some tasks that I did not finish.

I still need to add more parsers besides csv.

I still need to add a custom graph for releases_count. It doesn't make
sense to work on this until I have the releases_count metric working
again.

I still have to switch to python3 and do all internal operations in
unicode. I was waiting on python3-flask to be included in stable, but
if I have time I might go ahead and setup a jessie chroot next week.

Zack mentioned something about reworking the landing interface and
urls. If you expand on what you mean, I can work on that next week.

Cheers,
Joseph