[Teammetrics-discuss] MemoryError Observations

Sukhbir Singh sukhbir.in at gmail.com
Mon Sep 19 11:28:41 UTC 2011


Hi,

Yes, we will have a MemoryError almost surely the first time the code
runs *only* for SVN repositories. This is because each time Paramiko
fetches data, it creates a new connection object. These keep on
increasing as the code continues and so does the memory consumption. I
started with 300 MB and now I am at 1 GB after about seven hours of
operation. Not cool.

Possible solutions for this problem:

1. I am not sure, but if we perform all the operations on Alioth and
then fetch everything all at once (like you said :), it might work.
But we need to test this because the size of the data is large given
that pkg-perl itself has 77977 revisions. Whether this works or not,
this is to be tested.

2. What if we populate the 'teammetrics' database on blends.d.n from
vasks _directly_ without having to go through our script? Then we
eliminate the need for getting it through Paramiko and running it into
memory problems. So we have a script on vasks that does this for us.

If we finish then we can present teams with some statistics :)

-- 
Sukhbir



More information about the Teammetrics-discuss mailing list