[pymvpa] Sensitivity analysis

[Yaroslav Halchenko]

On Tue, 02 Sep 2014, Dai Xijian wrote:

>    Dear all,
>        I faced with an embarrassing situation. I have completed some
>    sensitivity analysis. When I used the script like below, I achieved a
>    discouraging accuracy rate:0.51267.
>             clf = LinearCSVMC()
>               cv = CrossValidation(clf, NFoldPartitioner(), errorfx = lambda
>    p,t:np.mean(p==t), enable_ca=['stats'],postproc = mean_sample())
>               sensana = clf.get_sensitivity_analyzer()
>               cv_sensana = RepeatedMeasure(sensana,
>    ChainNode((NFoldPartitioner(), Splitter('partitions', attr_values=(1,)))))
>               acc = cv(ds)
>               sensmap_cv = cv_sensana(ds)
>       Then I want to improve the accuracy. So I firstly select 20% features
>    which contributed most for the predicted accuracy. However, I achieved a
>    lower accuracy rate:0.40268.
>           fsel =
>    SensitivityBasedFeatureSelection(OneWayAnova(),FractionTailSelector(0.2,mode='select',tail='upper'))
>             fclf = FeatureSelectionClassifier(clf, fsel)
>             cv_sel = CrossValidation(fclf,NFoldPartitioner(), errorfx =
>    lambda p,t:np.mean(p==t),enable_ca=['stats'],postproc = mean_sample())
>             sensana_sel = fclf.get_sensitivity_analyzer()
>             cv_sensana_sel = RepeatedMeasure(sensana_sel,
>    ChainNode((NFoldPartitioner(), Splitter('partitions', attr_values=(1,)))))
>             acc_sel = cv_sel(ds)
>             sensmap_cv_sel = cv_sensana_sel(ds)
>      What's wrong? Would you tell me?

most probably nothing is wrong -- just either that your data doesn't
have information about your targets, or information is strongly
non-univariate so univariate ANOVA based feature selection doesn't
improve results.

(if you are doing binary classification, 0.51 accuracy can well be just
due to a chance, so a slight augmentation of classifier just lead to
another chance level performance)

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik