[pymvpa] Memory Error

[Thomas Nickson]

Hey All,

I'm using some structural data, 133 subjects with a size of about 350 meg,
on a machine with 32gig of ram. I'm trying to set up a basic algorithm to
make sure that everything works okay. I have chosen ridge regression and
I'm attempting to use a HalfPartitioner and I'm running all of the code
from ipython notebook.

  from mvpa2.clfs.ridge import RidgeReg
  a = HalfPartitioner(attr='runtype')
  clf = RidgeReg()
  cv = CrossValidation(clf, a)
  cv_results = cv(ds)

However, I get the following memory error, even when I use a very minimal
subset such as 2 subjects.

---------------------------------------------------------------------------MemoryError
                              Traceback (most recent call
last)<ipython-input-205-fafb290e8469> in <module>()----> 1 cv_results
= cv(ds)
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc
in __call__(self, ds)    257                                    "used
and auto training is disabled."    258
   % str(self))--> 259         return super(Learner,
self).__call__(ds)    260     261
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc
in __call__(self, ds)    119     120         self._precall(ds)--> 121
       result = self._call(ds)    122         result =
self._postcall(ds, result)    123
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc
in _call(self, ds)    495         # always untrain to wipe out
previous stats    496         self.untrain()--> 497         return
super(CrossValidation, self)._call(ds)    498     499
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc
in _call(self, ds)    324                 ca.datasets.append(sds)
325             # run the beast--> 326             result = node(sds)
  327             # callback    328             if not self._callback
is None:
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc
in __call__(self, ds)    257                                    "used
and auto training is disabled."    258
   % str(self))--> 259         return super(Learner,
self).__call__(ds)    260     261
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc
in __call__(self, ds)    119     120         self._precall(ds)--> 121
       result = self._call(ds)    122         result =
self._postcall(ds, result)    123
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc
in _call(self, ds)    598                     for i in
dstrain.get_attr(splitter.get_space())[0].unique])    599         #
ask splitter for first part--> 600         measure.train(dstrain)
601         # cleanup to free memory    602         del dstrain
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc
in train(self, ds)    130             # things might have happened
during pretraining    131             if ds.nfeatures > 0:--> 132
           result = self._train(ds)    133             else:    134
             warning("Trying to train on dataset with no features
present")
/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/clfs/ridge.pyc
in _train(self, data)     76             if self.__lm is None:     77
               # Not specified, so calculate based on
.05*nfeatures---> 78                 Lambda =
.05*data.nfeatures*np.eye(data.nfeatures)     79             else:
80                 # use the provided penalty
/usr/lib64/python2.6/site-packages/numpy/lib/twodim_base.pyc in eye(N,
M, k, dtype)    208     if M is None:    209         M = N--> 210
m = zeros((N, M), dtype=dtype)    211     if k >= M:    212
return m
MemoryError:


Surely, this can't really be because I'm out of memory can it? The
32gig machine should be

more than able to load the a small portion of the 350 meg dataset.

Is there something I'm missing or have I configured something wrongly?

Also, I'm not really sure how the halfpartitioner works. Does it just
generate the complement

of the sets I determine or can it split the dataset in two for me based on the

groups? If that's possible, how can I do that?

Thanks,

Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20141124/bcae4ca4/attachment.html>