<div dir="ltr"><div>Alright! <br>Below are the outputs, but speaking of the solution, I would still try to find some way which avoids making more copies of the data.<br></div><div>This is not urgent for me, so for example if suggested modification to allow user to load uncompressed .nii in memory will appear in some nearby future - that is enough for me :)<br>
</div><div><br><br>Here goes the time from uncompressed .nii, took again some enormous time:<br>start = time.time()<br>fds = fmri_dataset(samples=['%s/run%i/epi.nii' % (dataroot,(r+1)) for r in range(runs)], <br>targets=targets, <br>
chunks=selector, <br>mask='/MNI152_T1_2mm_brain_mask.nii')<br>end = time.time()<br>print end - start<br><br>[DS_] DBG{0.000 sec}: Duplicating samples shaped (1750, 91, 109, 91)<br>[DS_] DBG{0.001 sec}: Create new dataset instance for copy<br>
[DS_] DBG{0.000 sec}: Return dataset copy (ID: 103268432) of source (ID: 45836304)<br>[DS_] DBG{5.597 sec}: Selecting feature/samples of (1750, 902629)<br>[DS_] DBG{14.003 sec}: Selected feature/samples (1750, 902629)<br>
>>> 14120.6634622<br><br></div>Call to mvpa2.wtf():<br><br>Current date: 2014-05-08 14:51<br>PyMVPA:<br> Version: 2.2.0<br> Hash: ad955620e460965ce83c652bc690bea4dc2e21eb<br> Path: /usr/lib/pymodules/python2.7/mvpa2/__init__.pyc<br>
Version control (GIT):<br> GIT information could not be obtained due "/usr/lib/pymodules/python2.7/mvpa2/.. is not under GIT"<br>SYSTEM:<br> OS: posix Linux 2.6.32-46-generic #107-Ubuntu SMP Fri Mar 22 20:15:42 UTC 2013<br>
Distribution: Ubuntu/12.04/precise<br>EXTERNALS:<br> Present: atlas_fsl, cPickle, ctypes, good scipy.stats.rdist, good scipy.stats.rv_continuous._reduce_func(floc,fscale), good scipy.stats.rv_discrete.ppf, griddata, gzip, h5py, ipython, liblapack.so, libsvm, libsvm verbosity control, lxml, matplotlib, mdp, mdp ge 2.4, nibabel, nipy, nose, numpy, numpy_correct_unique, openopt, pylab, pylab plottable, pywt, pywt wp reconstruct, reportlab, scipy, sg ge 0.6.4, sg ge 0.6.5, sg_fixedcachesize, shogun, shogun.krr, shogun.mpd, shogun.svmocas, skl, statsmodels, weave<br>
Absent: atlas_pymvpa, cran-energy, elasticnet, glmnet, hcluster, lars, mass, nipy.neurospin, pprocess, pywt wp reconstruct fixed, rpy2, running ipython env, shogun.lightsvm, shogun.svrlight<br> Versions of critical externals:<br>
shogun:full : 1.1.0_02ce3cd_2011-12-12_08:17_<br> shogun:rev : 2941901<br> reportlab : 2.5<br> shogun : 1.1.0<br> nibabel : 1.3.0<br> matplotlib : 1.1.1rc<br> scipy : 0.9.0<br> nipy : 0.3.0<br>
ipython : 0.12.1<br> skl : 0.14.1<br> mdp : 3.4<br> numpy : 1.6.1<br> ctypes : 1.1.0<br> matplotlib : 1.1.1rc<br> lxml : 2.3.2<br> nifti : failed to query due to "nifti is not a known dependency key."<br>
numpy : 1.6.1<br> openopt : 0.38<br> openopt : failed to query due to "No module named scikits.openopt"<br> pywt : 0.2.0<br> shogun : v1.1.0_02ce3cd_2011-12-12_08:17_<br> Matplotlib backend: Qt4Agg<br>
RUNTIME:<br> PyMVPA Environment Variables:<br> PYTHONPATH : ":/usr/lib/python2.7/lib-old:/usr/local/lib/python2.7/dist-packages:/usr/lib/pymodules/python2.7/openopt/:/u/smirnod1/kernel:/home/smirnod1/.local/lib/python2.7/site-packages:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-dynload:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/dist-packages/gtk-2.0:/home/smirnod1/.python27_compiled:/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode:/usr/lib/python2.7/dist-packages:/usr/lib/pymodules/python2.7:/usr/lib/python2.7/dist-packages/gst-0.10:/usr/lib/python2.7:/usr/lib/python2.7/dist-packages/spyderlib/utils/external:/u/smirnod1:/usr/lib/python2.7/dist-packages/PIL:/usr/lib/pymodules/python2.7/openopt/kernel"<br>
PYTHONSTARTUP : "/usr/lib/python2.7/dist-packages/spyderlib/scientific_startup.py"<br> PyMVPA Runtime Configuration:<br> [general]<br> verbose = 1<br> <br> [externals]<br> have running ipython env = no<br>
have numpy = yes<br> have scipy = yes<br> have matplotlib = yes<br> have h5py = yes<br> have reportlab = yes<br> have weave = yes<br> have good scipy.stats.rdist = yes<br> have good scipy.stats.rv_discrete.ppf = yes<br>
have good scipy.stats.rv_continuous._reduce_func(floc,fscale) = yes<br> have pylab = yes<br> have lars = no<br> have elasticnet = no<br> have glmnet = no<br> have skl = yes<br> have ctypes = yes<br> have libsvm = yes<br>
have shogun = yes<br> have sg ge 0.6.5 = yes<br> have shogun.mpd = yes<br> have shogun.lightsvm = no<br> have shogun.svrlight = no<br> have shogun.krr = yes<br> have shogun.svmocas = yes<br> have sg_fixedcachesize = yes<br>
have openopt = yes<br> have nibabel = yes<br> have mdp = yes<br> have mdp ge 2.4 = yes<br> have statsmodels = yes<br> have pywt = yes<br> have cpickle = yes<br> have gzip = yes<br> have cran-energy = no<br> have griddata = yes<br>
have nipy.neurospin = no<br> have lxml = yes<br> have atlas_fsl = yes<br> have atlas_pymvpa = no<br> have hcluster = no<br> have ipython = yes<br> have liblapack.so = yes<br> have libsvm verbosity control = yes<br>
have mass = no<br> have nipy = yes<br> have nose = yes<br> have numpy_correct_unique = yes<br> have pprocess = no<br> have pylab plottable = yes<br> have pywt wp reconstruct = yes<br> have pywt wp reconstruct fixed = no<br>
have rpy2 = no<br> have sg ge 0.6.4 = yes<br> Process Information:<br> Name: /usr/bin/python<br> State: R (running)<br> Tgid: 31307<br> Pid: 31307<br> PPid: 31273<br> TracerPid: 0<br> Uid: 1021772 1021772 1021772 1021772<br>
Gid: 310001 310001 310001 310001<br> FDSize: 128<br> Groups: 310001 310002 310020 310044 <br> VmPeak: 38029860 kB<br> VmSize: 4351092 kB<br> VmLck: 0 kB<br> VmHWM: 29355884 kB<br>
VmRSS: 3333720 kB<br> VmData: 3606424 kB<br> VmStk: 272 kB<br> VmExe: 2504 kB<br> VmLib: 137680 kB<br> VmPTE: 7916 kB<br> Threads: 19<br> SigQ: 0/1162705<br> SigPnd: 0000000000000000<br>
ShdPnd: 0000000000000000<br> SigBlk: 0000000000000000<br> SigIgn: 0000000001001000<br> SigCgt: 0000000180010002<br> CapInh: 0000000000000000<br> CapPrm: 0000000000000000<br> CapEff: 0000000000000000<br>
CapBnd: ffffffffffffffff<br> Cpus_allowed: ffffffff<br> Cpus_allowed_list: 0-31<br> Mems_allowed: 00000000,00000003<br> Mems_allowed_list: 0-1<br> voluntary_ctxt_switches: 3086144<br> nonvoluntary_ctxt_switches: 53383<br>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-07 16:22 GMT+03:00 Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
On Wed, 07 May 2014, Dmitry Smirnov wrote:<br>
> Indeed, when I gzipped the images, it took a few minutes to load the whole<br>
> thing:<br>
<br>
> [DS_] DBG{0.000 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
> [DS_] DBG{0.001 sec}: Create new dataset instance for copy<br>
> [DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source<br>
> (ID: 59680656)<br>
> [DS_] DBG{4.083 sec}: Selecting feature/samples of (350, 902629)<br>
> [DS_] DBG{2.064 sec}: Selected feature/samples (350, 902629)<br>
> [DS_] DBG{0.423 sec}: Selecting feature/samples of (350, 228483)<br>
> [DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>
> [DS_] DBG{59.208 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
> [DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>
> [DS_] DBG{0.000 sec}: Return dataset copy (ID: 121150032) of source<br>
> (ID: 59680656)<br>
> [DS_] DBG{4.123 sec}: Selecting feature/samples of (350, 902629)<br>
> [DS_] DBG{1.985 sec}: Selected feature/samples (350, 902629)<br>
> [DS_] DBG{0.309 sec}: Selecting feature/samples of (350, 228483)<br>
> [DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>
> [DS_] DBG{57.703 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
> [DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>
> [DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source<br>
> (ID: 121093840)<br>
> [DS_] DBG{4.384 sec}: Selecting feature/samples of (350, 902629)<br>
> [DS_] DBG{2.056 sec}: Selected feature/samples (350, 902629)<br>
> [DS_] DBG{0.293 sec}: Selecting feature/samples of (350, 228483)<br>
> [DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>
> [DS_] DBG{57.575 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
> [DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>
> [DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source<br>
> (ID: 121093840)<br>
> [DS_] DBG{4.094 sec}: Selecting feature/samples of (350, 902629)<br>
> [DS_] DBG{2.273 sec}: Selected feature/samples (350, 902629)<br>
> [DS_] DBG{0.384 sec}: Selecting feature/samples of (350, 228483)<br>
> [DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>
> [DS_] DBG{62.976 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
> [DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>
> [DS_] DBG{0.000 sec}: Return dataset copy (ID: 121150032) of source<br>
> (ID: 59680656)<br>
> [DS_] DBG{4.122 sec}: Selecting feature/samples of (350, 902629)<br>
> [DS_] DBG{2.143 sec}: Selected feature/samples (350, 902629)<br>
> [DS_] DBG{0.353 sec}: Selecting feature/samples of (350, 228483)<br>
> [DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>
<br>
</div></div>For the record filed<br>
<a href="https://github.com/nipy/nibabel/issues/238" target="_blank">https://github.com/nipy/nibabel/issues/238</a><br>
could you please also share output (cut/paste) of mvpa2.wtf() call?<br>
<br>
could you also time analogously the run directly on uncompressed .nii?<br>
<div class=""><br>
> A follow up question - if nibabel by default memmaps the uncompressed<br>
> images, can you guide me how I can possibly go around that behavior?<br>
> I don't want to gzip all my data, and in my current context, I'd like to be<br>
> able to load the whole dataset into memory at once. Is there some simple<br>
> way to do it?<br>
<br>
</div>well -- the simplest way would be to sacrifice those few hours of<br>
loading as you did, and then dump those datasets into hdf5 files<br>
(h5save) and then in the script load/use those instead of reconstructing<br>
all the way again from nifti's.... let me know if this would not work<br>
for you -- then we could come up with other alternatives ;-)<br>
<div class="HOEnZb"><div class="h5"><br>
--<br>
Yaroslav O. Halchenko, Ph.D.<br>
<a href="http://neuro.debian.net" target="_blank">http://neuro.debian.net</a> <a href="http://www.pymvpa.org" target="_blank">http://www.pymvpa.org</a> <a href="http://www.fail2ban.org" target="_blank">http://www.fail2ban.org</a><br>
Research Scientist, Psychological and Brain Sciences Dept.<br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a></div></div></blockquote></div><br><br clear="all">
<br>-- <br> <br>Dmitry Smirnov (MSc.)<br>PhD Candidate, <a href="http://becs.aalto.fi/bml/" target="_blank">Brain & Mind Laboratory</a><div>BECS, Aalto University School of Science<br>00076 AALTO, FINLAND<br>mobile: +358 50 3015072<br>
email: <a href="mailto:dmitry.smirnov@aalto.fi" target="_blank">dmitry.smirnov@aalto.fi</a></div>
</div>