[Teammetrics-discuss] Phase I: Statistics for mailing lists on Alioth

[Andreas Tille]

On Tue, May 31, 2011 at 03:24:26PM +0530, Sukhbir Singh wrote:
> Please update your local 'teammetrics' Git repository as I have pushed
> some new changes.

Done.
 
> Here is what works now:
> 
> + You can create a configuration file and specify the lists to be
> parsed. Run the script and it will guide you on how to create the
> file.

Remarks:

 - You are relaying on $HOME of a user which runs liststat.py.
   I do not consider this as the best choice because the script finally
   should run as a service.  So I would rather move the mboxes to
   either
     /var/lib/teammetrics/liststats
   or
     /var/cache/teammetrics/liststats
   and the configuration file should be rather reside in
     /etc/teammetrics
   For the time of testing this is no problem because you used
   variables - however the variables relay on having config and
   data in the same directory which is not flexible enough.

 - I would prefer a better readable config file.  There is no need
   for xml, but some kind of key-value syntax (perhaps RFC 822 as used
   in many Debian related config files) comes to mind.  There should
   also be the option to add comments.

 - I tried with

$ cat ~/.teammetrics/listinfo 
http://lists.alioth.debian.org/pipermail/
blends-commit
cdd-commits
teammetrics-discuss

and got

$ python liststat.py 
Base URL is 'http://lists.alioth.debian.org/pipermail/'.

[1 of 3]
Reading: http://lists.alioth.debian.org/pipermail/blends-commit
Downloading 28 mbox archives...

[2 of 3]
Reading: http://lists.alioth.debian.org/pipermail/cdd-commits
Downloading 60 mbox archives...
Traceback (most recent call last):
  File "liststat.py", line 148, in <module>
    main()
  File "liststat.py", line 89, in main
    mbox = urllib2.urlopen(mbox_url)
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found


This is most probably because the list cdd-commits was stopped however,
I also got this error, when replacing this list by something else - so
something is broken here.

> Note that when the downloading of mbox archives takes place, it will
> just say 'Downloading...' and show no progress bar. If you want an
> indication of progress, please let me know.

Nono, this thingy should run silently in a cron job or something like
this.  For the purpose of debugging / developing this is fine.  Perhaps
you might use a DEBUG variable which controls the output and enables
switching it to complete silence if everything works as expected.

> If you want to keep it
> simple, we can probably leave it like this or output the progress to a
> log file (perhaps?).

Logfile would be probably a good idea.
 
> Comments on code are most welcome and I intend to cleanup a little you
> give the green signal about the working/ design.
> 
> I hope you like it!

Yes, thanks.

Kind regards

        Andreas.

PS: I hope list admins will be able to fix the subscription issue.  I just
    do not get the confirmation mail. :-(

-- 
http://fam-tille.de