[pymvpa] RFE & Permutation

[Yaroslav Halchenko]

On Thu, 28 Jan 2010, Matthias Ekman wrote:
> > how does your cv.states.null_dist.dist_samples would look?

> looks pretty 'okay' with mean about .5 -- so I guess everything is fine.
gaussianish? ;)

> mmh... but still wondering why p-values are so low for RFE (compared to
> feature selection with fixed number of elements).
as you have mentioned, it does perform better, so you
can expect lower p values.  If you look at the distribution of random
performances with so many samples you should get it quite narrow, so
tails would be very light and probability of getting 0.8 just by chance
-- reeealy small, as it should.

as for how much RFE could influence generalization - see for instance
figure 1 of
http://psychology.rutgers.edu/~jose/hanson_halchenko_inpress.pdf

you can see how some subjects went from random (on full brain data) to
low error (or even to 0).  So, do not undervalue the importance/effect
of feature selection and congrats on obtaining good results ;)

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]