[pymvpa] Searchlight Memory Error

[Yaroslav Halchenko]

How many cpu cores do you have? There is nproc parameter (and nblock?) to searchlight - try setting to some smaller number

Given the simplicity of the within searchlight function I wonder if there is something else which causes the blow up though

On May 14, 2020 7:03:00 PM EDT, Jacob Elder <jacobelder52 at gmail.com> wrote:
>Hi,
>
>I am trying to run a whole-brain searchlight RSA (on a cluster with a
>lot
>of processing cores available) but every time I do it returns a
>memoryError. However, when I split it into hemispheres or run as an
>ROI, it
>successfully runs without MemoryError. I have tried following some
>other
>suggestions I found online but nothing has resolved the issue. Memory
>shouldn't be an issue as the cluster has plenty of RAM available. We've
>also tried to run this on another cluster at a separate institution
>using a
>high mem option and it still didn't work. I am pretty stumped and was
>wondering if anyone can help or has any potential solutions in mind?
>
>Here is what the code looks like:
>
>    subject = subjectData(inputs.rootDir, inputs.ID)
>
>
>    *# load the trait similarity data*
>
>    traitData = subject.importSimilarity()
>
>
>    *# load the neural data*
>
>    neuralData = subject.importNeural()
>
>
>
>    neuralData = remove_invariant_features(neuralData)
>
>
>    *print* ("Complete loading neural data")
>
>
>    *# create an object from the neuroCorrelation class*
>
>    matrixCorr = neuroCorrelation(traitData)
>
>
>    *# set up searchlight parameters*
>
>    *# set the radius based on the input*
>
>    rad = inputs.rad
>
>
>    *# Fisher's z-transformation for correlation coefficient*
>
>FisherTransform = FxMapper('features', *lambda* r: 0.5 * np.log((1 + r)
>/ (1 - r)))
>
>
>    *# create the search light*
>
>    sl = sphere_searchlight(matrixCorr.correlate, rad, results_backend=
>'hdf5', postproc=FisherTransform)
>
>
>    *print* ("Complete search light set-up")
>
>
>    *# apply the search light function to the neural data*
>
>    sl_output = sl(neuralData)
>
>
>    *print* ("Search light completed")
>
>
>    *# transform the search light output to an image data*
>
>    sl_image = map2nifti(data=sl_output, dataset=neuralData)
>
>
>    *# save the output*
>
>   outputDir = os.path.join(inputs.rootDir, "RSA", "searchLightResult",
>"Subject{n}".format(n=inputs.ID),
>
>
>"sub{n}_sl_results_{v}{h}_P_R{r}.nii.gz".format(n=inputs.ID,
>v=inputs.valence, h=inputs.hemisphere, r=inputs.rad))
>
>    sl_image.to_filename(outputDir)
>
>
>    *print* ("Complete with subject{n}".format(n=inputs.ID))
>
>
>Here is the error:
>
>/home/user/anaconda2/lib/python2.7/site-packages/numpy/lib/type_check.py:546:
>DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use
>a.item() instead
>
>  'a.item() instead', DeprecationWarning, stacklevel=1)
>
>Traceback (most recent call last):
>
>  File "./script/RSA_rad_FB_test.py", line 54, in <module>
>
>    sl_output = sl(neuralData)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/learner.py",
>line 258, in __call__
>
>    return super(Learner, self).__call__(ds)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/node.py",
>line
>138, in __call__
>
>    result = self._postcall(ds, result)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/measures/base.py",
>line 128, in _postcall
>
>    result = super(Measure, self)._postcall(dataset, result)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/node.py",
>line
>179, in _postcall
>
>    result = self._apply_postproc(ds, result)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/node.py",
>line
>253, in _apply_postproc
>
>    result = self.__postproc(result)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/learner.py",
>line 258, in __call__
>
>    return super(Learner, self).__call__(ds)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/node.py",
>line
>137, in __call__
>
> result = self._call(ds, **(_call_kwargs or self._get_call_kwargs(ds)))
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/mappers/base.py",
>line 291, in _call
>
>    return self.forward(ds)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/mappers/base.py",
>line 215, in forward
>
>    return self._forward_dataset(data)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/mappers/fx.py",
>line 204, in _forward_dataset
>
>    col[attr] = a
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/collections.py",
>line 590, in __setitem__
>
>    value = ArrayCollectable(value)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/collections.py",
>line 185, in __init__
>
>    Collectable.__init__(self, value=value, name=name, doc=doc)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/collections.py",
>line 67, in __init__
>
>    self._set(value)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/mvpa2/base/collections.py",
>line 291, in _set
>
>    val = np.asanyarray(val)
>
>  File
>"/home/user/anaconda2/lib/python2.7/site-packages/numpy/core/numeric.py",
>line 591, in asanyarray
>
>    return array(a, dtype, copy=False, order=order, subok=True)
>
>MemoryError
>
>
>Any help would be greatly appreciated! Thank you.
>
>-- 
>Jacob Elder
>Ph.D. Student, Dept. of Psychology
>University of California, Riverside
>The UCR Social Neuroscience Lab <https://www.hugheslab.org/>

-- 
Yaroslav O. Halchenko (mobile version)
Center for Open Neuroscience   http://centerforopenneuroscience.org
Dartmouth College, NH, USA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20200514/5c4ad358/attachment-0001.html>