<div dir="ltr"><div>Hey All, <br><br>I'm using some structural data, 133 subjects with a
size of about 350 meg, on a machine with 32gig of ram. I'm trying to
set up a basic algorithm to make sure that everything works okay. I have
chosen ridge regression and I'm attempting to use a HalfPartitioner and
I'm running all of the code from ipython notebook.<br><br> from mvpa2.clfs.ridge import RidgeReg<br> a = HalfPartitioner(attr='runtype')<br> clf = RidgeReg()<br> cv = CrossValidation(clf, a)<br> cv_results = cv(ds) <br><br></div>However, I get the following memory error, even when I use a very minimal subset such as 2 subjects.<br><br><pre><span>---------------------------------------------------------------------------</span>
<span>MemoryError</span> Traceback (most recent call last)
<span><ipython-input-205-fafb290e8469></span> in <span><module></span><span>()</span>
<span>----> 1</span><span> </span>cv_results <span>=</span> cv<span>(</span>ds<span>)</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 257</span> <span>"used and auto training is disabled."</span><span></span>
<span> 258</span> % str(self))
<span>--> 259</span><span> </span><span>return</span> super<span>(</span>Learner<span>,</span> self<span>)</span><span>.</span>__call__<span>(</span>ds<span>)</span><span></span>
<span> 260</span> <span></span>
<span> 261</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 119</span> <span></span>
<span> 120</span> self<span>.</span>_precall<span>(</span>ds<span>)</span><span></span>
<span>--> 121</span><span> </span>result <span>=</span> self<span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 122</span> result <span>=</span> self<span>.</span>_postcall<span>(</span>ds<span>,</span> result<span>)</span><span></span>
<span> 123</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 495</span> <span># always untrain to wipe out previous stats</span><span></span><span></span>
<span> 496</span> self<span>.</span>untrain<span>(</span><span>)</span><span></span>
<span>--> 497</span><span> </span><span>return</span> super<span>(</span>CrossValidation<span>,</span> self<span>)</span><span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 498</span> <span></span>
<span> 499</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 324</span> ca<span>.</span>datasets<span>.</span>append<span>(</span>sds<span>)</span><span></span>
<span> 325</span> <span># run the beast</span><span></span><span></span>
<span>--> 326</span><span> </span>result <span>=</span> node<span>(</span>sds<span>)</span><span></span>
<span> 327</span> <span># callback</span><span></span><span></span>
<span> 328</span> <span>if</span> <span>not</span> self<span>.</span>_callback <span>is</span> None<span>:</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 257</span> <span>"used and auto training is disabled."</span><span></span>
<span> 258</span> % str(self))
<span>--> 259</span><span> </span><span>return</span> super<span>(</span>Learner<span>,</span> self<span>)</span><span>.</span>__call__<span>(</span>ds<span>)</span><span></span>
<span> 260</span> <span></span>
<span> 261</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 119</span> <span></span>
<span> 120</span> self<span>.</span>_precall<span>(</span>ds<span>)</span><span></span>
<span>--> 121</span><span> </span>result <span>=</span> self<span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 122</span> result <span>=</span> self<span>.</span>_postcall<span>(</span>ds<span>,</span> result<span>)</span><span></span>
<span> 123</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 598</span> for i in dstrain.get_attr(splitter.get_space())[0].unique])
<span> 599</span> <span># ask splitter for first part</span><span></span><span></span>
<span>--> 600</span><span> </span>measure<span>.</span>train<span>(</span>dstrain<span>)</span><span></span>
<span> 601</span> <span># cleanup to free memory</span><span></span><span></span>
<span> 602</span> <span>del</span> dstrain<span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>train</span><span>(self, ds)</span>
<span> 130</span> <span># things might have happened during pretraining</span><span></span><span></span>
<span> 131</span> <span>if</span> ds<span>.</span>nfeatures <span>></span> <span>0</span><span>:</span><span></span>
<span>--> 132</span><span> </span>result <span>=</span> self<span>.</span>_train<span>(</span>ds<span>)</span><span></span>
<span> 133</span> <span>else</span><span>:</span><span></span>
<span> 134</span> warning<span>(</span><span>"Trying to train on dataset with no features present"</span><span>)</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/clfs/ridge.pyc</span> in <span>_train</span><span>(self, data)</span>
<span> 76</span> <span>if</span> self<span>.</span>__lm <span>is</span> None<span>:</span><span></span>
<span> 77</span> <span># Not specified, so calculate based on .05*nfeatures</span><span></span><span></span>
<span>---> 78</span><span> </span>Lambda <span>=</span> <span>.05</span><span>*</span>data<span>.</span>nfeatures<span>*</span>np<span>.</span>eye<span>(</span>data<span>.</span>nfeatures<span>)</span><span></span>
<span> 79</span> <span>else</span><span>:</span><span></span>
<span> 80</span> <span># use the provided penalty</span><span></span><span></span>
<span>/usr/lib64/python2.6/site-packages/numpy/lib/twodim_base.pyc</span> in <span>eye</span><span>(N, M, k, dtype)</span>
<span> 208</span> <span>if</span> M <span>is</span> None<span>:</span><span></span>
<span> 209</span> M <span>=</span> N<span></span>
<span>--> 210</span><span> </span>m <span>=</span> zeros<span>(</span><span>(</span>N<span>,</span> M<span>)</span><span>,</span> dtype<span>=</span>dtype<span>)</span><span></span>
<span> 211</span> <span>if</span> k <span>>=</span> M<span>:</span><span></span>
<span> 212</span> <span>return</span> m<span></span>
<span>MemoryError</span>: <br><br><br></pre><pre>Surely, this can't really be because I'm out of memory can it? The 32gig machine should be <br></pre><pre>more than able to load the a small portion of the 350 meg dataset.<br><br></pre><pre>Is there something I'm missing or have I configured something wrongly?<br><br></pre><pre>Also, I'm not really sure how the halfpartitioner works. Does it just generate the complement <br></pre><pre>of the sets I determine or can it split the dataset in two for me based on the <br></pre><pre>groups? If that's possible, how can I do that?<br><br></pre><pre>Thanks,<br><br>Tom </pre></div>