<div dir="ltr"><div><div><div><div><br></div>Hi and thanks for the tip, Yaroslav!<br><br></div>Indeed, when I gzipped the images, it took a few minutes to load the whole thing:<br><br>[DS_] DBG{0.000 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
[DS_] DBG{0.001 sec}: Create new dataset instance for copy<br>[DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source (ID: 59680656)<br>[DS_] DBG{4.083 sec}: Selecting feature/samples of (350, 902629)<br>
[DS_] DBG{2.064 sec}: Selected feature/samples (350, 902629)<br>[DS_] DBG{0.423 sec}: Selecting feature/samples of (350, 228483)<br>[DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>[DS_] DBG{59.208 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
[DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>[DS_] DBG{0.000 sec}: Return dataset copy (ID: 121150032) of source (ID: 59680656)<br>[DS_] DBG{4.123 sec}: Selecting feature/samples of (350, 902629)<br>
[DS_] DBG{1.985 sec}: Selected feature/samples (350, 902629)<br>[DS_] DBG{0.309 sec}: Selecting feature/samples of (350, 228483)<br>[DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>[DS_] DBG{57.703 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
[DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>[DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source (ID: 121093840)<br>[DS_] DBG{4.384 sec}: Selecting feature/samples of (350, 902629)<br>
[DS_] DBG{2.056 sec}: Selected feature/samples (350, 902629)<br>[DS_] DBG{0.293 sec}: Selecting feature/samples of (350, 228483)<br>[DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>[DS_] DBG{57.575 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
[DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>[DS_] DBG{0.000 sec}: Return dataset copy (ID: 150670160) of source (ID: 121093840)<br>[DS_] DBG{4.094 sec}: Selecting feature/samples of (350, 902629)<br>
[DS_] DBG{2.273 sec}: Selected feature/samples (350, 902629)<br>[DS_] DBG{0.384 sec}: Selecting feature/samples of (350, 228483)<br>[DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br>[DS_] DBG{62.976 sec}: Duplicating samples shaped (350, 91, 109, 91)<br>
[DS_] DBG{0.000 sec}: Create new dataset instance for copy<br>[DS_] DBG{0.000 sec}: Return dataset copy (ID: 121150032) of source (ID: 59680656)<br>[DS_] DBG{4.122 sec}: Selecting feature/samples of (350, 902629)<br>
[DS_] DBG{2.143 sec}: Selected feature/samples (350, 902629)<br>[DS_] DBG{0.353 sec}: Selecting feature/samples of (350, 228483)<br>[DS_] DBG{0.000 sec}: Selected feature/samples (350, 228483)<br><br></div>A follow up question - if nibabel by default memmaps the uncompressed images, can you guide me how I can possibly go around that behavior? I don't want to gzip all my data, and in my current context, I'd like to be able to load the whole dataset into memory at once. Is there some simple way to do it?<br>
<br></div><div>Thank you in advance for any suggestion!<br><br></div><div>Best, Dima<br></div><div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-06 18:08 GMT+03:00 Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=""><br>
On Tue, 06 May 2014, Dmitry Smirnov wrote:<br>
<br>
> Hi Nick,<br>
> Thanks for advice!<br>
<br>
> I'm using a server with 256gb of RAM, while the data in 5 runs would be<br>
> something 12.5gb altogether.<br>
> Each run is 2.5gb, dimensions: 91x109x91x350<br>
> The problem is still there: I've adjusted the code after your reply and<br>
> ran it immediately, and it is still running.<br>
<br>
</div>1. I wonder if that is an effect of memory mapping which happens (by<br>
nibabel) if original files are uncompressed .nii<br>
<br>
2. it might be worth timing such a run in bigger detail, e.g.:<br>
<br>
$> MVPA_DEBUG=DS.* MVPA_DEBUG_METRICS=reltime nosetests -s -v mvpa2/tests/test_niftidataset.py<br>
[DS_ ] DBG{0.000 sec}: Binding function save to AttrDataset class<br>
T: MVPA_SEED=948286987<br>
T: Skipping testing of all dependencies since verbosity (MVPA_TESTS_VERBOSITY) is too low<br>
Basic testing of NiftiDataset ... [DS_ ] DBG{0.971 sec}: Duplicating samples shaped (2, 128, 96, 24)<br>
[DS_ ] DBG{0.006 sec}: Create new dataset instance for copy<br>
[DS_ ] DBG{0.001 sec}: Return dataset copy #70317136 of source #68617232<br>
[DS_ ] DBG{0.706 sec}: Duplicating samples shaped (2, 294912)<br>
[DS_ ] DBG{0.001 sec}: Create new dataset instance for copy<br>
[DS_ ] DBG{0.007 sec}: Return dataset copy #70479632 of source #70317136<br>
[DS_ ] DBG{0.045 sec}: Duplicating samples shaped (2, 128, 96, 24)<br>
[DS_ ] DBG{0.001 sec}: Create new dataset instance for copy<br>
[DS_ ] DBG{0.001 sec}: Return dataset copy #68616784 of source #70355792<br>
[DS_ ] DBG{1.264 sec}: Selecting feature/samples of (2, 294912)<br>
[DS_ ] DBG{0.006 sec}: Selected feature/samples (2, 294912)<br>
...<br>
<br>
that reltime is time to took from previous debug message being printed<br>
<br>
in code you could enable it by<br>
<br>
mvpa2.debug.active += ['DS_']<br>
mvpa2.debug.metrics += ['reltime']<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Yaroslav O. Halchenko, Ph.D.<br>
<a href="http://neuro.debian.net" target="_blank">http://neuro.debian.net</a> <a href="http://www.pymvpa.org" target="_blank">http://www.pymvpa.org</a> <a href="http://www.fail2ban.org" target="_blank">http://www.fail2ban.org</a><br>
Research Scientist, Psychological and Brain Sciences Dept.<br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" target="_blank">http://www.linkedin.com/in/yarik</a><br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br> <br>Dmitry Smirnov (MSc.)<br>PhD Candidate, <a href="http://becs.aalto.fi/bml/" target="_blank">Brain & Mind Laboratory</a><div>BECS, Aalto University School of Science<br>
00076 AALTO, FINLAND<br>mobile: +358 50 3015072<br>email: <a href="mailto:dmitry.smirnov@aalto.fi" target="_blank">dmitry.smirnov@aalto.fi</a></div>
</div>