[pymvpa] how to do multiprocessing in pymvpa?

[Jonas Kaplan]

Yeah, pprocess.get_number_of_cores() looks to find info about the cores in /proc/cpuinfo which does not exist on OSX. 


----
Jonas Kaplan, Ph.D.
Research Assistant Professor
Brain & Creativity Institute
University of Southern California

On Feb 4, 2013, at 10:40 AM, Meng Liang <meng.liang at hotmail.co.uk> wrote:

> Hi Michael and Jonas,
> 
> Thanks very much for your prompt reply! Jonas's suggestion solved the problem - If I do not put 'nproc = 3' explicitly in the searchlight command, I always get the error "IOError: [Errno 2] No such file file or directory: '/proc/cpuinfo'".
> 
> Thanks again!
> 
> Best,
> Meng
> 
> From: jtkaplan at usc.edu
> Date: Mon, 4 Feb 2013 10:14:45 -0800
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: Re: [pymvpa] how to do multiprocessing in pymvpa?
> 
> I have pprocess working on Mac OSX for searchlights. The trick is that it fails when checking the number of cores, so you have to specify nproc. 
> 
> Jonas
> 
> On Feb 4, 2013, at 9:57 AM, Meng Liang <meng.liang at hotmail.co.uk> wrote:
> 
> Hi Nick,
> 
> Thanks so much for your reply! Following your suggestion, I downloaded and installed the pprocess on Neurodebian virtual machine on a Windows 7 PC and now the script runs much faster! 
> 
> However, it does not work on Mac. I googled but didn't find much relevant info about it - I got the feeling that pprocess is not supported in Mac. Is it true? If so, doesn't that mean running pymvpa on Mac machines cannot utilize multi-core computation?
> 
> Finally, an irrelevant question: is the value of 'radius' parameter in the Searchlight command in millimeter? It says so in Pymvpa documentation ( http://www.pymvpa.org/modref/mvpa.measures.searchlight.html ). But in a given example on Pymvpa website, it seems to suggest that the value of 'radius' means the number of voxels ( http://www.pymvpa.org/examples/searchlight_minimal.html ): "# setup searchlight with 2 voxels radius" and "sl = sphere_searchlight(cv, radius=2, space='myspace',...". It's a bit confusing to me...
> 
> Also, do you know if multi-core support is also available for other commands other than searchlight? I find that cross-validation with large number of permutations is also quite slow - it would be good if multi-core support is also implemented in the permutation scenario. 
> 
> Best,
> Meng
> 
> 
> 
> > Date: Mon, 4 Feb 2013 14:17:10 +0100
> > From: nikolaas.oosterhof at unitn.it
> > To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> > Subject: Re: [pymvpa] how to do multiprocessing in pymvpa?
> > 
> > On 4 February 2013 14:09, Meng Liang <meng.liang at hotmail.co.uk> wrote:
> > > I'm trying to run a searchlight analysis using the whole brain mask but I
> > > found it impossible to finish within a reasonable time - it has been running
> > > three days for one two-way classification task in one subject, and it still
> > > hasn't finished. To make the computing faster, would multiprocessing be an
> > > option? I didn't see any part of the tutorial mentioning multiprocessing -
> > > is it because it is already embedded/implemented in pymvpa commands?
> > 
> > Yes, multi-core support is already part of pyMVPA's searchlight
> > implementation. It requires, however, the presence of pprocess [1].
> > The Searchlight constructor has an optional parameter 'nproc' that
> > controls the number of parallel processes to use. By default
> > (nproc=None) it uses the number of cores of the machine - unless
> > pprocess is not available, in which case nproc=1.
> > 
> > As per the example (doc/examples/searchlight.py) it is recommended to
> > enable the searchlight debug messages - these include messages related
> > to parallelization.
> > 
> > In my experience it's easy to use pprocess with neurodebian. On a mac
> > (OS X Lion) I did not get it to work, although I have to admit I did
> > not try very hard.
> > 
> > [1] http://pypi.python.org/pypi/pprocess
> > 
> > 
> > 
> > >
> > > Best,
> > > Meng
> > >
> > > _______________________________________________
> > > Pkg-ExpPsy-PyMVPA mailing list
> > > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> > > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> > 
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/8a6301af/attachment-0001.html>