[Teammetrics-discuss] Please check VCS commits of Debian Med team

[Andreas Tille]

Hi,

more news about this.  After I restarted the job I was watching

  SELECT count(*) from commitstat ; 

from time to time and learned that it did not changed for more than 20h.
Now it started changing *very* slowly.  It seems commitstat.py is
re-evaluating all the repositories from scratch rather than starting
from the last commit that was inspected in the previous run.  It also
seems to do this in a very unstructured and slow manner.  I think the
main part of the job is to connect and disconnect to alioth for every
single bit of work.  I think this is no acceptable approach to solve the
problem of answering the question who commited when to what repository.

If I imagine a very simply approach to checkout every single repository
and inspect the log this should take not more than say 10 hours.
Considering that this is a quite brute force idea to solve the problem I
wonder whether some more clever job running on alioth to prepare some
kind of preprocessed data file and fetching only this file for
postprocessing and feeding the database would be *way* more performant
that we are currently doing.

Sukhbir, since you have not answered a single mail here, could you
confirm that you

   a) are busy right now and come back later to this
   b) are engaged in other fields and has droped the teammetrics topic

This would help me to decide how much time I need to spend into this
in the next couple of monthes.

Kind regards

        Andreas.

On Tue, Nov 04, 2014 at 08:44:12PM +0100, Andreas Tille wrote:
> Hi again,
> 
> after more than 30 hours processing it ended in:
> 
> $ commitstat.py -u tille
> Traceback (most recent call last):
>   File "./commitstat.py", line 256, in <module>
>     get_stats()
>   File "./commitstat.py", line 224, in get_stats
>     svnstat.fetch_logs(ssh, conn, cur, svn)
>   File "/home/tille/debian-maintain/alioth/teammetrics/repository/svnstat.py", line 34, in fetch_logs
>     ftp.put(LOCAL_PATH, 'revisions.hash')                                                   
>   File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 565, in put
>     fr = self.file(remotepath, 'wb')
>   File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 245, in open
>     t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
>   File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 635, in _request    
>     return self._read_response(num)
>   File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 682, in _read_response
>     self._convert_status(msg)
>   File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 710, in _convert_status
>     raise IOError(errno.EACCES, text) 
> IOError: [Errno 13] Permission denied
> Exception in thread Thread-1 (most likely raised during interpreter shutdown):
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
>   File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1574, in run
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
> 
> 
> I restart the job (really hoping that some commit statement was injected
> to not loose all those data ...)
> 
> Kind regards
> 
>        Andreas.
> 
> On Mon, Nov 03, 2014 at 02:58:24PM +0100, Andreas Tille wrote:
> > Hi Sukhbir,
> > 
> > this problem was fixed in dd89476b6c332bf7d5c6c8ab97013fc1752686fc.
> > 
> > However, the sukhbir-guest account is not working on moszumanska.d.o.
> > It would be great if you could fix this to enable easily running
> > update-data.
> > 
> > Kind regards
> > 
> >            Andreas.
> > 
> > On Mon, Nov 03, 2014 at 08:43:12AM +0100, Andreas Tille wrote:
> > > Hi Sukhbir,
> > > 
> > > according to the graph the Debian Med team has *way* less commits than
> > > the last two years which is definitely wrong.  Moreover the stat did not
> > > changed for November compared to the October stats.  This is definitely
> > > not the case.
> > > 
> > > Could you please check this?
> > > 
> > > Thanks a lot
> > > 
> > >          Andreas.
> > > 
> > > -- 
> > > http://fam-tille.de
> > > 
> > > _______________________________________________
> > > Teammetrics-discuss mailing list
> > > Teammetrics-discuss at lists.alioth.debian.org
> > > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/teammetrics-discuss
> > > 
> > 
> > -- 
> > http://fam-tille.de
> > 
> > _______________________________________________
> > Teammetrics-discuss mailing list
> > Teammetrics-discuss at lists.alioth.debian.org
> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/teammetrics-discuss
> > 
> 
> -- 
> http://fam-tille.de
> 
> _______________________________________________
> Teammetrics-discuss mailing list
> Teammetrics-discuss at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/teammetrics-discuss
> 

-- 
http://fam-tille.de