[pymvpa] Memory Error

[Michael Hanke]

Hi,

could you please show the code that assembles the dataset? Also, the output
of

print ds.summary()

could help to find the problem.

Michael




On Mon, Nov 24, 2014 at 4:35 PM, Thomas Nickson <thomas.nickson at gmail.com>
wrote:

> Hey All,
>
> I'm using some structural data, 133 subjects with a size of about 350 meg,
> on a machine with 32gig of ram. I'm trying to set up a basic algorithm to
> make sure that everything works okay. I have chosen ridge regression and
> I'm attempting to use a HalfPartitioner and I'm running all of the code
> from ipython notebook.
>
>   from mvpa2.clfs.ridge import RidgeReg
>   a = HalfPartitioner(attr='runtype')
>   clf = RidgeReg()
>   cv = CrossValidation(clf, a)
>   cv_results = cv(ds)
>
> However, I get the following memory error, even when I use a very minimal
> subset such as 2 subjects.
>
> ---------------------------------------------------------------------------MemoryError                               Traceback (most recent call last)<ipython-input-205-fafb290e8469> in <module>()----> 1 cv_results = cv(ds)
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc in __call__(self, ds)    257                                    "used and auto training is disabled."    258                                    % str(self))--> 259         return super(Learner, self).__call__(ds)    260     261
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc in __call__(self, ds)    119     120         self._precall(ds)--> 121         result = self._call(ds)    122         result = self._postcall(ds, result)    123
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc in _call(self, ds)    495         # always untrain to wipe out previous stats    496         self.untrain()--> 497         return super(CrossValidation, self)._call(ds)    498     499
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc in _call(self, ds)    324                 ca.datasets.append(sds)    325             # run the beast--> 326             result = node(sds)    327             # callback    328             if not self._callback is None:
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc in __call__(self, ds)    257                                    "used and auto training is disabled."    258                                    % str(self))--> 259         return super(Learner, self).__call__(ds)    260     261
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc in __call__(self, ds)    119     120         self._precall(ds)--> 121         result = self._call(ds)    122         result = self._postcall(ds, result)    123
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc in _call(self, ds)    598                     for i in dstrain.get_attr(splitter.get_space())[0].unique])    599         # ask splitter for first part--> 600         measure.train(dstrain)    601         # cleanup to free memory    602         del dstrain
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc in train(self, ds)    130             # things might have happened during pretraining    131             if ds.nfeatures > 0:--> 132                 result = self._train(ds)    133             else:    134                 warning("Trying to train on dataset with no features present")
> /home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/clfs/ridge.pyc in _train(self, data)     76             if self.__lm is None:     77                 # Not specified, so calculate based on .05*nfeatures---> 78                 Lambda = .05*data.nfeatures*np.eye(data.nfeatures)     79             else:     80                 # use the provided penalty
> /usr/lib64/python2.6/site-packages/numpy/lib/twodim_base.pyc in eye(N, M, k, dtype)    208     if M is None:    209         M = N--> 210     m = zeros((N, M), dtype=dtype)    211     if k >= M:    212         return m
> MemoryError:
>
>
> Surely, this can't really be because I'm out of memory can it? The 32gig machine should be
>
> more than able to load the a small portion of the 350 meg dataset.
>
> Is there something I'm missing or have I configured something wrongly?
>
> Also, I'm not really sure how the halfpartitioner works. Does it just generate the complement
>
> of the sets I determine or can it split the dataset in two for me based on the
>
> groups? If that's possible, how can I do that?
>
> Thanks,
>
> Tom
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>



-- 
Michael Hanke
http://mih.voxindeserto.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20141124/a4c8e549/attachment-0001.html>