[pymvpa] optimal way of loading the whole-brain data

[Yaroslav Halchenko]

On Tue, 06 May 2014, Dmitry Smirnov wrote:

>    Hi Nick,
>    Thanks for advice!

>    I'm using a server with 256gb of RAM, while the data in 5 runs would be
>    something 12.5gb altogether.
>    Each run is 2.5gb, dimensions: 91x109x91x350
>    The problem is still there: I've adjusted the code after your reply and
>    ran it immediately, and it is still running.

1. I wonder if that is an effect of memory mapping which happens (by
nibabel) if original files are uncompressed .nii 

2. it might be worth timing such a run in bigger detail, e.g.:

$> MVPA_DEBUG=DS.* MVPA_DEBUG_METRICS=reltime nosetests -s -v mvpa2/tests/test_niftidataset.py 
[DS_     ] DBG{0.000 sec}:            Binding function save to AttrDataset class
T: MVPA_SEED=948286987
T: Skipping testing of all dependencies since verbosity (MVPA_TESTS_VERBOSITY) is too low
Basic testing of NiftiDataset ... [DS_             ] DBG{0.971 sec}:                     Duplicating samples shaped (2, 128, 96, 24)
[DS_             ] DBG{0.006 sec}:                     Create new dataset instance for copy
[DS_             ] DBG{0.001 sec}:                     Return dataset copy #70317136 of source #68617232
[DS_             ] DBG{0.706 sec}:                Duplicating samples shaped (2, 294912)
[DS_             ] DBG{0.001 sec}:                Create new dataset instance for copy
[DS_             ] DBG{0.007 sec}:                Return dataset copy #70479632 of source #70317136
[DS_             ] DBG{0.045 sec}:                     Duplicating samples shaped (2, 128, 96, 24)
[DS_             ] DBG{0.001 sec}:                     Create new dataset instance for copy
[DS_             ] DBG{0.001 sec}:                     Return dataset copy #68616784 of source #70355792
[DS_             ] DBG{1.264 sec}:                  Selecting feature/samples of (2, 294912)
[DS_             ] DBG{0.006 sec}:                  Selected feature/samples (2, 294912)
...

that reltime is time to took from previous debug message being printed

in code you could enable it by

mvpa2.debug.active += ['DS_']
mvpa2.debug.metrics += ['reltime']

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik