[Teammetrics-discuss] MemoryError Observations
Sukhbir Singh
sukhbir.in at gmail.com
Mon Sep 19 11:28:41 UTC 2011
Hi,
Yes, we will have a MemoryError almost surely the first time the code
runs *only* for SVN repositories. This is because each time Paramiko
fetches data, it creates a new connection object. These keep on
increasing as the code continues and so does the memory consumption. I
started with 300 MB and now I am at 1 GB after about seven hours of
operation. Not cool.
Possible solutions for this problem:
1. I am not sure, but if we perform all the operations on Alioth and
then fetch everything all at once (like you said :), it might work.
But we need to test this because the size of the data is large given
that pkg-perl itself has 77977 revisions. Whether this works or not,
this is to be tested.
2. What if we populate the 'teammetrics' database on blends.d.n from
vasks _directly_ without having to go through our script? Then we
eliminate the need for getting it through Paramiko and running it into
memory problems. So we have a script on vasks that does this for us.
If we finish then we can present teams with some statistics :)
--
Sukhbir
More information about the Teammetrics-discuss
mailing list