<div dir="ltr">Hi Yaroslav,<br><br>Thanks for your suggestions. Sorry that I am replying late, I was busy in some other stuff, so din't get time to check on your suggestions.<br><br>here is some additional info about my dataset (named mdata):<br>
<br>>>> mdata.samples.shape<br>(41, 8673536)<br>>>> mdata.samples.dtype<br>dtype('float32')<br>>>> mdata.samples.max()<br>23559.484<br>>>> mdata.samples.min()<br>-1214.7351<br><br>
print mdata.summary() goes into infinite loop!<br><br>following is the output of print mvpa.wtf() (before going into infinite loop)<br><br>Current date: 2010-03-26 18:52<br>PyMVPA:<br> Version: 0.4.4<br> Path: /var/lib/python-support/python2.5/mvpa/__init__.pyc<br>
Version control (GIT):<br> GIT information could not be obtained due "/var/lib/python-support/python2.5/mvpa/.. is not under GIT"<br>SYSTEM:<br> OS: posix Linux 2.6.26-1-amd64 #1 SMP Mon Dec 15 17:25:36 UTC 2008<br>
Distribution: debian/5.0<br>EXTERNALS:<br> Present: cPickle, ctypes, gzip, libsvm, libsvm verbosity control, matplotlib, mdp, mdp ge 2.4, nifti, nifti ge 0.20090205.1, numpy, pylab, pylab plottable, pywt, reportlab, scipy, sg_fixedcachesize, shogun, shogun.krr, shogun.mpd<br>
Absent: atlas_fsl, atlas_pymvpa, elasticnet, glmnet, good scipy.stats.rdist, good scipy.stats.rv_discrete.ppf, griddata, hcluster, lars, lxml, nose, openopt, pywt wp reconstruct, pywt wp reconstruct fixed, rpy, running ipython env, sg ge 0.6.4, shogun.lightsvm, shogun.svrlight, weave<br>
Versions of critical externals:<br> ctypes : 1.0.3<br> matplotlib : 0.98.1<br> nifti : 0.20090303.1<br> numpy : 1.1.0<br> pywt : 0.1.7<br> scipy : 0.6.0<br> shogun : v0.6.3_r3165_2008-06-13_11:03_<br>
Matplotlib backend: GTKAgg<br>RUNTIME:<br> PyMVPA Environment Variables:<br> PyMVPA Runtime Configuration:<br> [externals]<br> have griddata = no<br> have atlas_pymvpa = no<br> have good scipy.stats.rdist = no<br> have pylab plottable = yes<br>
have pywt wp reconstruct = no<br> have mdp = yes<br> have lxml = no<br> have running ipython env = no<br> have sg_fixedcachesize = yes<br> have elasticnet = no<br> have shogun.mpd = yes<br> have matplotlib = yes<br>
have pywt wp reconstruct fixed = no<br> have scipy = yes<br> have reportlab = yes<br> have openopt = no<br> have libsvm = yes<br> have nifti ge 0.20090205.1 = yes<br> have nose = no<br> have weave = no<br> have atlas_fsl = no<br>
have ctypes = yes<br> have hcluster = no<br> have sg ge 0.6.4 = no<br> have good scipy.stats.rv_discrete.ppf = no<br> have libsvm verbosity control = yes<br> have mdp ge 2.4 = yes<br> have shogun.svrlight = no<br>
have rpy = no<br> have shogun = yes<br> have glmnet = no<br> have lars = no<br> have nifti = yes<br> have shogun.krr = yes<br> have cpickle = yes<br> have numpy = yes<br> have pylab = yes<br> have shogun.lightsvm = no<br>
have pywt = yes<br> have gzip = yes<br> <br> [general]<br> verbose = 1<br> Process Information:<br> Name: python<br> State: R (running)<br> Tgid: 6316<br> Pid: 6316<br> PPid: 6152<br> TracerPid: 0<br>
Uid: 1001 1001 1001 1001<br> Gid: 1001 1001 1001 1001<br> FDSize: 256<br> Groups: 1001 <br> VmPeak: 692040 kB<br> VmSize: 630276 kB<br> VmLck: 0 kB<br> VmHWM: 73408 kB<br>
VmRSS: 73408 kB<br> VmData: 125616 kB<br> VmStk: 228 kB<br> VmExe: 1172 kB<br> VmLib: 71500 kB<br> VmPTE: 1080 kB<br> Threads: 2<br> SigQ: 0/71680<br> SigPnd: 0000000000000000<br>
ShdPnd: 0000000000000000<br> SigBlk: 0000000000000000<br> SigIgn: 0000000001001000<br> SigCgt: 0000000180000002<br> CapInh: 0000000000000000<br> CapPrm: 0000000000000000<br> CapEff: 0000000000000000<br>
CapBnd: ffffffffffffffff<br> Cpus_allowed: 000000ff<br> Cpus_allowed_list: 0-7<br> Mems_allowed: 00000000,00000001<br> Mems_allowed_list: 0<br> voluntary_ctxt_switches: 1482<br> nonvoluntary_ctxt_switches: 101<br>
<br><br><br>In order to check for improper scaling, I rescaled the whole data so that it lies in [0,1]^n but the problem of going into infinite loop still persists.<br>Also, you asked about C=1 (I assume you meant the cvtype in Nfoldsplitter) When I try with any other C value > 1, it throws back the following error:<br>
<br><br>Traceback (most recent call last):<br> File "<stdin>", line 2, in <module><br> File "/var/lib/python-support/python2.5/mvpa/datasets/splitters.py", line 199, in __call__<br> cfgs = self.splitcfg(dataset)<br>
File "/var/lib/python-support/python2.5/mvpa/datasets/splitters.py", line 388, in splitcfg<br> return self._getSplitConfig(eval('dataset.unique' + self.__splitattr))<br> File "/var/lib/python-support/python2.5/mvpa/datasets/splitters.py", line 626, in _getSplitConfig<br>
self.__cvtype)]<br> File "/var/lib/python-support/python2.5/mvpa/misc/support.py", line 265, in getUniqueLengthNCombinations<br> return _getUniqueLengthNCombinations_binary(L, n, sort=True)<br> File "/var/lib/python-support/python2.5/mvpa/misc/support.py", line 236, in _getUniqueLengthNCombinations_binary<br>
for X in range(2**N):<br>OverflowError: range() result has too many items<br><br>Has anybody worked on the dataset of this size successfully before? My question is not related to whether SVM successfully classifies or not, its related to whether it gives back output within reasonable time on such large scale dataset?<br>
<br>Thanks in advance,<br>Alok<br><br><div class="gmail_quote">On Mon, Mar 22, 2010 at 6:58 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com">debian@onerussian.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Alok,<br>
<br>
My guess is that you are using C=1 (or some other rigid value positive<br>
value) and/or improperly scaled data... then libsvm's optimizer might<br>
get into such loop.<br>
<br>
For best troubleshooting provide<br>
<br>
print mvpa.wtf()<br>
<br>
print dataset.summary()<br>
for your dataset which you try to process<br>
<br>
and actual classifier you are using<br>
<div class="im"><br>
On Mon, 22 Mar 2010, Alok Deshpande wrote:<br>
<br>
> Hi all,<br>
> I am currently working on basic classification problems on resting<br>
> state fMRI datasets (like gender differences, for example) The<br>
> dimensionality of feature vector is pretty large (128 time points X 33<br>
> X 64 X 64) I am working on a linux server with 8 cores and suff RAM.<br>
> Following is the output of command 'free -m'<br>
> free -m<br>
> total used free shared buffers<br>
> cached<br>
> Mem: 8006 1254 6752 0 100<br>
> 854<br>
> -/+ buffers/cache: 299 7706<br>
> Swap: 23454 209 23244<br>
> When I try to train a simple linear SVM classifier on the dataset (I<br>
</div>--<br>
.-.<br>
=------------------------------ /v\ ----------------------------=<br>
Keep in touch // \\ (yoh@|www.)<a href="http://onerussian.com" target="_blank">onerussian.com</a><br>
Yaroslav Halchenko /( )\ ICQ#: 60653192<br>
Linux User ^^-^^ [175555]<br>
<br>
<br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Alok S. Deshpande<br>Graduate Student<br>Electrical & Computer Engineering Department<br>University of Wisconsin Madison<br>
</div>