[pymvpa] MemoryError

[McKell Carter]

On Fri, Jun 25, 2010 at 5:28 PM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
>> Hi, All,
>
>>    I've been trying to load large datasets into pymvpa (15000 samples,
>> 32000 features) on a 64bit Centos machine with 32GB of ram.
> 32k after masking, right?

Yes, 32K after masking. The full image size is 48X58X49.

>
>> NiftiDataset builds a data set but only fills the first 7065 data
>> elements (the remaining elements are zeros). It doesn't return an
> by elements, do you mean samples (0th dimension of .samples) or features (1st
> dimension)?  or do you really mean first 7065 elements in unraveled
> array of 480000000 elements?
>

The last non-zero sample is 7064 (0 indexed), this is true for all of
features I randomly tested.


>> error message. I've verified that the mvpa, nifticlib, pynifti and
>> numpy packages that we have installed are _64. Does anyone have any
>> guesses for potential solutions?
> what is the output of
>
> mvpa.wtf()
>
> ?
>

Nice, that is handy. Here is the output:

Current date:   2010-06-26 11:39
PyMVPA:
 Version:       0.4.4
 Path:          /usr/lib64/python2.4/site-packages/mvpa/__init__.pyc
 Version control (GIT):
 GIT information could not be obtained due
"/usr/lib64/python2.4/site-packages/mvpa/.. is not under GIT"
SYSTEM:
 OS:            posix Linux 2.6.18-194.3.1.el5 #1 SMP Thu May 13
13:08:30 EDT 2010
 Distribution:  redhat/5.5/Final
EXTERNALS:
 Present:       cPickle, ctypes, gzip, libsvm, libsvm verbosity
control, matplotlib, nifti, nifti ge 0.20090205.1, nose, numpy, pylab,
pylab plottable, running ipython env, scipy, weave
 Absent:        atlas_fsl, atlas_pymvpa, elasticnet, glmnet, good
scipy.stats.rdist, good scipy.stats.rv_discrete.ppf, griddata,
hcluster, lars, lxml, mdp, mdp ge 2.4, openopt, pywt, pywt wp
reconstruct, pywt wp reconstruct fixed, reportlab, rpy, sg ge 0.6.4,
sg_fixedcachesize, shogun, shogun.krr, shogun.lightsvm, shogun.mpd,
shogun.svrlight
 Versions of critical externals:
  ctypes      : 1.0.2
  matplotlib  : 0.91.2
  nifti       : 0.20100412.1
  numpy       : 1.2.1
  scipy       : 0.6.0
 Matplotlib backend: GTKAgg
RUNTIME:
 PyMVPA Environment Variables:
 PyMVPA Runtime Configuration:
  [externals]
  have griddata = no
  have good scipy.stats.rdist = no
  have pylab plottable = yes
  have pywt wp reconstruct = no
  have pywt = no
  have lxml = no
  have running ipython env = yes
  have ctypes = yes
  have sg_fixedcachesize = no
  have elasticnet = no
  have shogun.mpd = no
  have matplotlib = yes
  have pywt wp reconstruct fixed = no
  have scipy = yes
  have reportlab = no
  have openopt = no
  have libsvm = yes
  have shogun.krr = no
  have nifti ge 0.20090205.1 = yes
  have nose = yes
  have weave = yes
  have atlas_fsl = no
  have mdp = no
  have hcluster = no
  have sg ge 0.6.4 = no
  have good scipy.stats.rv_discrete.ppf = no
  have libsvm verbosity control = yes
  have mdp ge 2.4 = no
  have shogun.svrlight = no
  have rpy = no
  have shogun = no
  have glmnet = no
  have lars = no
  have atlas_pymvpa = no
  have nifti = yes
  have cpickle = yes
  have numpy = yes
  have pylab = yes
  have shogun.lightsvm = no
  have gzip = yes

  [general]
  verbose = 1
 Process Information:
  Name:	ipython
  State:	R (running)
  SleepAVG:	97%
  Tgid:	11921
  Pid:	11921
  PPid:	11772
  TracerPid:	0
  Uid:	2131	2131	2131	2131
  Gid:	100	100	100	100
  FDSize:	256
  Groups:	100
  VmPeak:	10330204 kB
  VmSize:	 2381948 kB
  VmLck:	       0 kB
  VmHWM:	 8299536 kB
  VmRSS:	 1953540 kB
  VmData:	 1938696 kB
  VmStk:	      96 kB
  VmExe:	       4 kB
  VmLib:	   98900 kB
  VmPTE:	    4704 kB
  StaBrk:	10a51000 kB
  Brk:	128a3000 kB
  StaStk:	7fffe6658960 kB
  Threads:	1
  SigQ:	0/71680
  SigPnd:	0000000000000000
  ShdPnd:	0000000000000000
  SigBlk:	0000000000000000
  SigIgn:	0000000001001000
  SigCgt:	0000000180000002
  CapInh:	0000000000000000
  CapPrm:	0000000000000000
  CapEff:	0000000000000000
  Cpus_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff
  Mems_allowed:	00000000,00000001


> have you tried load with plain pynifti and seeing either all data looks
> ok?
>

It looks like it is messed up when loaded in pynifti too. I'm not
aware of a pynifti list, any guesses where to look for errors there?

Cheers,
McKell

> --
>                                  .-.
> =------------------------------   /v\  ----------------------------=
> Keep in touch                    // \\     (yoh@|www.)onerussian.com
> Yaroslav Halchenko              /(   )\               ICQ#: 60653192
>                   Linux User    ^^-^^    [175555]
>
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>