[pymvpa] optimal way of loading the whole-brain data

[Dmitry Smirnov]

Hi Nick,
Thanks for advice!

I'm using a server with 256gb of RAM, while the data in 5 runs would be
something 12.5gb altogether.
Each run is 2.5gb, dimensions: 91x109x91x350

The problem is still there: I've adjusted the code after your reply and ran
it immediately, and it is still running.

-Dima



2014-05-06 14:57 GMT+03:00 Nick Oosterhof <nikolaas.oosterhof at unitn.it>:

>
> On May 6, 2014, at 1:19 PM, Dmitry Smirnov <dmi.smirnov07 at gmail.com>
> wrote:
>
> > I was wondering, what would be the most optimal solution for loading
> massive data in PyMVPA.
> > [...]
> > # Trim a number of slices in the end of the file
> > def trimImage(filename,cutoff):
> >     tmp = nib.load(filename)
> >     return
> nib.Nifti1Image(tmp.get_data()[:,:,:,0:cutoff],tmp.get_affine())
> > [...]
> > fds = fmri_dataset(samples=[trimImage(('run%i/epi.nii' % (r+1)),346) for
> r in range(runs)],
> > targets=targets,
> > chunks=selector,
> >
> mask='/triton/becs/scratch/braindata/DSmirnov/HarvardOxford/MNI152_T1_2mm_brain_mask.nii')
>
> I don't think you would need the trimImage function, and it is possible
> that using it increases the execution time significantly. You can slice the
> samples to get the same effect:
>
> run_ds=[]
> for r in range(runs)
>   ds=fmri_dataset('run%i/epi.nii' % (r+1),...)
>   ds=ds[:cutoff,:]
>   run_ds.append(ds)
>
> all_ds=hstack(run_ds,True)
>
> Also, are you sure you are not constrained by RAM? How big are the images?
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>



-- 

Dmitry Smirnov (MSc.)
PhD Candidate, Brain & Mind Laboratory <http://becs.aalto.fi/bml/>
BECS, Aalto University School of Science
00076 AALTO, FINLAND
mobile: +358 50 3015072
email: dmitry.smirnov at aalto.fi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20140506/0167a7ee/attachment.html>