[pymvpa] GroupClusterThreshold memory usage

[Bill Broderick]

That worked! I assumed I would need to do something else, but just
adding n_blocks=1000 brought down my memory usage more than enough. It
took about 3 times as long, but I think that will be fine.

Thanks!
Bill

On Thu, Sep 3, 2015 at 12:28 PM, Nick Oosterhof
<n.n.oosterhof at googlemail.com> wrote:
>
>> On 03 Sep 2015, at 18:04, Bill Broderick <billbrod at gmail.com> wrote:
>>
>> I'm trying to run group cluster thresholding using the defaults of
>> GroupClusterThreshold (100000 bootstraps) and I'm running into memory
>> issues. In the documentation, it looks like either n_proc (to split
>> the load across several nodes on our cluster) or n_blocks would help,
>> but it's not clear to me how to use these parameters.
>
> Peak memory usage is in the order of (n_bootstrap * n_features / n_blocks), where n_features is the number of features of the dataset.
> For example, if you set n_blocks=1000, then memory consumption will be reduced by about a factor of 1,000 compared to n_blocks=1.
>
> I'm not sure how the Parallel module behaves, but it may be the case that using n_proc processes will actually multiply memory demands by a factor of n_proc. If you want to keep memory consumption low, my suggestion would be to start with n_proc=1 and try higher values for n_blocks.
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa