[pymvpa] Sensitivity map with RFE?
marco tettamanti
mrctttmnt at gmail.com
Wed Jul 17 21:16:09 UTC 2013
Dear Yaroslav,
thank you a lot for your help!
I was convinced it was my fault because of crappy snippet, I did not suspect
this could be a bug.
I also confirm the problem with sensvm producing the same result for each fold:
this is something I had noticed, but I again thought this was conventional
absolute beginner's experience...
I think that what you are suggesting is that I should go to GitHub and build a
new pymvpa version from the master. I never did that, but I can try to find out
how. There should be a way to build a .deb package from git, right?
Anyway, below is the info about my system.
Thank you again and all the best,
Marco
In [3]: mvpa2.wtf()
Out[3]:
Current date: 2013-07-17 23:01
PyMVPA:
Version: 2.2.0
Hash: ad955620e460965ce83c652bc690bea4dc2e21eb
Path: /usr/lib/pymodules/python2.7/mvpa2/__init__.pyc
Version control (GIT):
GIT information could not be obtained due
"/usr/lib/pymodules/python2.7/mvpa2/.. is not under GIT"
SYSTEM:
OS: posix Linux 3.9-1-amd64 #1 SMP Debian 3.9.8-1
Distribution: debian/jessie/sid
EXTERNALS:
Present: atlas_fsl, cPickle, ctypes, good scipy.stats.rdist, good
scipy.stats.rv_continuous._reduce_func(floc,fscale), good
scipy.stats.rv_discrete.ppf, griddata, gzip, h5py, ipython, liblapack.so,
libsvm, libsvm verbosity control, lxml, matplotlib, mdp, mdp ge 2.4, nibabel,
nose, numpy, numpy_correct_unique, pprocess, pylab, pylab plottable, pywt, pywt
wp reconstruct, reportlab, running ipython env, scipy, skl, weave
Absent: atlas_pymvpa, cran-energy, elasticnet, glmnet, hcluster, lars,
mass, nipy, nipy.neurospin, openopt, pywt wp reconstruct fixed, rpy2, sg ge
0.6.4, sg ge 0.6.5, sg_fixedcachesize, shogun, shogun.krr, shogun.lightsvm,
shogun.mpd, shogun.svmocas, shogun.svrlight, statsmodels
Versions of critical externals:
reportlab : 2.5
nibabel : 1.3.0
matplotlib : 1.1.1rc2
scipy : 0.12.0
pprocess : 0.5
ipython : 0.13.2
skl : 0.13.1
mdp : 3.4
numpy : 1.7.1
ctypes : 1.1.0
matplotlib : 1.1.1rc2
lxml : 3.2.0
nifti : failed to query due to "nifti is not a known dependency key."
numpy : 1.7.1
pywt : 0.2.0
Matplotlib backend: TkAgg
RUNTIME:
PyMVPA Environment Variables:
PYTHONPATH :
":/usr/lib/python2.7/lib-old:/home/marco/data/bll/abstrasomat_pymvpa/abstrasomat_pymvpa_36s:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-dynload:/usr/bin:.:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PIL:/usr/lib/pymodules/python2.7:/usr/lib/python2.7/dist-packages/IPython/extensions:/usr/lib/python2.7:/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode:/home/marco/.python27_compiled:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/local/lib/python2.7/dist-packages"
PyMVPA Runtime Configuration:
[general]
verbose = 1
[externals]
have running ipython env = yes
have numpy = yes
have scipy = yes
have matplotlib = yes
have h5py = yes
have reportlab = yes
have weave = yes
have good scipy.stats.rdist = yes
have good scipy.stats.rv_discrete.ppf = yes
have good scipy.stats.rv_continuous._reduce_func(floc,fscale) = yes
have pylab = yes
have lars = no
have elasticnet = no
have glmnet = no
have skl = yes
have ctypes = yes
have libsvm = yes
have shogun = no
have openopt = no
have nibabel = yes
have mdp = yes
have mdp ge 2.4 = yes
have statsmodels = no
have pywt = yes
have cpickle = yes
have gzip = yes
have cran-energy = no
have griddata = yes
have nipy.neurospin = no
have lxml = yes
have atlas_fsl = yes
have atlas_pymvpa = no
have hcluster = no
have ipython = yes
have liblapack.so = yes
have libsvm verbosity control = yes
have mass = no
have nipy = no
have nose = yes
have numpy_correct_unique = yes
have pprocess = yes
have pylab plottable = yes
have pywt wp reconstruct = yes
have pywt wp reconstruct fixed = no
have rpy2 = no
have sg ge 0.6.4 = no
have sg ge 0.6.5 = no
have sg_fixedcachesize = no
have shogun.krr = no
have shogun.lightsvm = no
have shogun.mpd = no
have shogun.svmocas = no
have shogun.svrlight = no
Process Information:
Name: ipython
State: R (running)
Tgid: 12965
Pid: 12965
PPid: 12923
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 6 20 24 25 27 29 30 44 46 100 104 113 114 116 1000 1002
VmPeak: 719900 kB
VmSize: 712644 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 101504 kB
VmRSS: 100268 kB
VmData: 233488 kB
VmStk: 136 kB
VmExe: 2280 kB
VmLib: 60212 kB
VmPTE: 1096 kB
VmSwap: 0 kB
Threads: 3
SigQ: 0/254508
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180000002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1338
nonvoluntary_ctxt_switches: 171
> [pymvpa] Sensitivity map with RFE?
> Yaroslav Halchenko debian at onerussian.com
> Wed Jul 17 18:49:41 UTC 2013
>
> Previous message: [pymvpa] Sensitivity map with RFE?
> Next message: [pymvpa] how to train/test on a single partition?
> Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
>
> sorry about the delay -- and thanks for buzzing back.
>
> I have tried your snippet with 2.2.0 version as available in Debian --
> reproduced your failure. But it works with the current master of PyMVPA, so we
> fixed something relevant since the release -- could you give it a try with your
> data? let us know if you need instructions (but then tell us about your system
> -- output of mvpa2.wtf() should be sufficient ;) )
>
> another BUT:
> it seems that your construct hits previously reported and presumably
> "fixed" issue:
> https://github.com/PyMVPA/PyMVPA/pull/53
>
> so whenever you check your code, please check -- is your senssvm
> "degenerate" in that all of its samples are the same, e.g. in my case on
> a dummy dataset
>
> *(Pydb) print senssvm.samples
> [[ 2.01502448 2.77870027 0.73660355 2.43673417 0.83915333 0.30925885]
> [ 2.01502448 2.77870027 0.73660355 2.43673417 0.83915333 0.30925885]
> [ 2.01502448 2.77870027 0.73660355 2.43673417 0.83915333 0.30925885]
> [ 2.01502448 2.77870027 0.73660355 2.43673417 0.83915333 0.30925885]]
>
>
>
> On Wed, 17 Jul 2013, marco tettamanti wrote:
>
>> Dear all, I apologize for the repost, but I didn't get any reply and I am
>> pretty stuck with this issue. Is there anybody that could kindly provide any
>> further advice?
>
>> Thank you and very best wishes, Marco
> --
> Yaroslav O. Halchenko, Ph.D.
> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
> Senior Research Associate, Psychological and Brain Sciences Dept.
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
> WWW: http://www.linkedin.com/in/yarik
>
-------- Original Message --------
Subject: Re: Re: Sensitivity map with RFE?
Date: Fri, 12 Jul 2013 15:13:07 +0200
From: marco tettamanti <tettamanti.marco at hsr.it>
To: pkg-exppsy-pymvpa at lists.alioth.debian.org
<pkg-exppsy-pymvpa at lists.alioth.debian.org>
On 07/17/2013 10:28 AM, marco tettamanti wrote:
> Dear all, I apologize for the repost, but I didn't get any reply and I am
> pretty stuck with this issue.
> Is there anybody that could kindly provide any further advice?
>
> Thank you and very best wishes,
> Marco
>
> -------- Original Message --------
> Subject: Re: Re: Sensitivity map with RFE?
> Date: Fri, 12 Jul 2013 15:13:07 +0200
> From: marco tettamanti<tettamanti.marco at hsr.it>
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> <pkg-exppsy-pymvpa at lists.alioth.debian.org>
>
> Dear Roberto,
> thank you very much for your reply!
> I have tried adding a line as you suggested, but nothing changed.
>
> Actually, I thought from the PyMVPA manual that it is the
> 'RepeatedMeasure(sensanasvm, NFoldPartitioner())' that takes care of the
> cross-validation.
>
> I have now tried modifying my code to make it more more similar to the example
> in the RFE help documentation, with two different outcomes:
>
> 1) The following is basically equivalent to the snippet I sent previously and,
> at least in my intentions, modelled upon the sensitivity analysis with feature
> selection.
> With this, I still get the error message "RuntimeError: Cannot reverse-map data
> since the original data shape is unknown. Either set `dshape` in the
> constructor, or call train()."
>
> #-----------------------------
> clfsvm = SplitClassifier(LinearCSVMC(), NFoldPartitioner())
>
> rfesvm = RFE(clfsvm.get_sensitivity_analyzer(postproc=maxofabs_sample()),
> ConfusionBasedError(clfsvm, confusion_state='stats'), Repeater(2),
> fselector=FractionTailSelector(0.30, mode='select', tail='upper'),
> stopping_criterion=NBackHistoryStopCrit(BestDetector(), 10),
> train_pmeasure=False, update_sensitivity=True)
>
> fclfsvm = FeatureSelectionClassifier(clfsvm, rfesvm)
>
> sensanasvm = fclfsvm.get_sensitivity_analyzer(postproc=maxofabs_sample())
>
> cv_sensana_svm = RepeatedMeasure(sensanasvm, NFoldPartitioner())
>
> senssvm = cv_sensana_svm(fds)
>
> print senssvm.shape
> #-----------------------------
>
>
> 2) I have also tried a different solution, which however I do not think that is
> suited to produce a sensitivity map. The following does not yield any errors and
> produces the correct map dimensionality. However, not suprisingly, the result
> does not make any sense, as I get a uniform value spread across all brain mask
> voxels.
>
> #-----------------------------
> clfsvm = SplitClassifier(LinearCSVMC(), NFoldPartitioner())
>
> rfesvm = RFE(clfsvm.get_sensitivity_analyzer(postproc=maxofabs_sample()),
> ConfusionBasedError(clfsvm, confusion_state='stats'), Repeater(2),
> fselector=FractionTailSelector(0.30, mode='select', tail='upper'),
> stopping_criterion=NBackHistoryStopCrit(BestDetector(), 10),
> train_pmeasure=False, update_sensitivity=True)
>
> fclfsvm = FeatureSelectionClassifier(clfsvm, rfesvm)
>
> cvtesvm = CrossValidation(fclfsvm, NFoldPartitioner(), errorfx=lambda p,
> t:np.mean(p == t), postproc=maxofabs_sample(), enable_ca=['confusion', 'stats'])
>
> cv_sensana_svm = RepeatedMeasure(cvtesvm, NFoldPartitioner())
>
> senssvm = cv_sensana_svm(fds)
> #-----------------------------
>
>
>
> Thank you all and very best wishes,
> Marco
>
>> Date: Fri, 12 Jul 2013 11:36:21 +0200
>> From: Roberto Guidotti<robbenson18 at gmail.com>
>> To: Development and support of PyMVPA
>> <pkg-exppsy-pymvpa at lists.alioth.debian.org>
>> Subject: Re: [pymvpa] Sensitivity map with RFE?
>> Message-ID:
>> <CAGj93cHSQy-SqXE7H7YwP4tGnCEvBHW7ypSX8ENNF6FWLrk_jQ at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Dear Marco,
>>
>>> From your snippet I notice that you get the sensitivity analyzer without
>> run any CrossValidation/Classification.
>>
>> So try to run
>> -------
>> err = fclfsvm(fds) #That let you to cross validate/classify/select features
>> on your dataset
>> -------
>> This let you to train your object and then you can get the sensitivity of
>> your classifier.
>>
>> I've never used RFE, but this is what I get by your snippet.
>>
>> Ciao
>> Roberto
>>
>>>> Dear all,
>>>> is there any manner to obtain a sensitivity map with RFE, similar to what
>>>> can be done with feature selection?
>>>>
>>>> I am trying to use the following code:
>>>>
>>>> #-----------------------------**--------------------
>>>> clfsvm = LinearCSVMC()
>>>>
>>>> rfesvm = RFE(clfsvm.get_sensitivity_**analyzer(postproc=maxofabs_**sample()),
>>>> CrossValidation(clfsvm, NFoldPartitioner(), errorfx=mean_mismatch_error,
>>>> postproc=mean_sample()), Repeater(2), fselector=**FractionTailSelector(0.30,
>>>> mode='select', tail='upper'), stopping_criterion=**NBackHistoryStopCrit(**BestDetector(),
>>>> 10), update_sensitivity=True)
>>>>
>>>> fclfsvm = FeatureSelectionClassifier(**clfsvm, rfesvm)
>>>>
>>>> sensanasvm = fclfsvm.get_sensitivity_**analyzer(postproc=maxofabs_**
>>>> sample())
>>>>
>>>> cv_sensana_svm = RepeatedMeasure(sensanasvm, NFoldPartitioner())
>>>>
>>>> senssvm = cv_sensana_svm(fds)
>>>> #-----------------------------**--------------------
>>>>
>>>> However, after a while I get the following error:
>>>>
>>>> RuntimeError: Cannot reverse-map data since the original data shape is
>>>> unknown. Either set `dshape` in the constructor, or call train().
>>>>
>>>>
>>>> Thank you in advance for any help!
>>>> Best wishes,
>>>> Marco
>
--
Marco Tettamanti, Ph.D.
Nuclear Medicine Department & Division of Neuroscience
San Raffaele Scientific Institute
Via Olgettina 58
I-20132 Milano, Italy
Phone ++39-02-26434888
Fax ++39-02-26434892
Email: tettamanti.marco at hsr.it
Skype: mtettamanti
More information about the Pkg-ExpPsy-PyMVPA
mailing list