[pymvpa] extracting individual sample predictions from CV

[Yaroslav Halchenko]

On Thu, 02 Jun 2011, Brian Murphy wrote:
> My question is, in a cross-validated training/testing and is there a
> built-in way to access the whole list of individual predictions for
> each sample, in the original sample order (rather getting the
> predictions separately for each fold, and recombining them by hand)?

> ... such that float(sum(ds.targets==predictions))/len(ds.targets)
> would give you the accuracy rate ...

well, depending on the splitting strategy it might happen that not every
sample has a prediction or there might be samples with multiple
predictions (coming from different folds)...  and we do not have
explicit mechanism for providing such corresponding pairs  of "sample
id": "prediction"

but indeed:

 in case of simple splits with no additional balancing or
 bootstrapping every sample is used for prediction just once

 we store all pairs of (targets, predictions, estimates)'s for each fold in
 summary statistics (i.e.  confusion matrices for classification), e.g.   for
 OddEvenPartitioner you would get

 (Pydb) print cv.ca.stats.sets
[(array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',
       'L1'], 
      dtype='|S2'), array(['L0', 'L0', 'L1', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',
       'L1'], 
      dtype='|S2'), array([[  8.49918954e-01,   1.50081046e-01],
       [  9.95318187e-01,   4.68181306e-03],
       [  2.84213008e-01,   7.15786992e-01],
       [  9.78226382e-01,   2.17736175e-02],
       [  9.27104786e-01,   7.28952142e-02],
       [  8.70968397e-01,   1.29031603e-01],
       [  6.78895948e-03,   9.93211041e-01],
       [  2.00886368e-03,   9.97991136e-01],
       [  6.28513847e-03,   9.93714862e-01],
       [  2.67472391e-04,   9.99732528e-01],
       [  1.67885133e-03,   9.98321149e-01],
       [  2.02054194e-02,   9.79794581e-01]])), (array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',
       'L1'], 
      dtype='|S2'), array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',
       'L1'], 
      dtype='|S2'), array([[  9.96597171e-01,   3.40282861e-03],
       [  9.98160008e-01,   1.83999250e-03],
       [  9.99357359e-01,   6.42640695e-04],
       [  9.99372062e-01,   6.27937749e-04],
       [  9.96979892e-01,   3.02010788e-03],
       [  9.96291817e-01,   3.70818327e-03],
       [  2.86752720e-02,   9.71324728e-01],
       [  5.49101939e-03,   9.94508981e-01],
       [  1.74193623e-02,   9.82580638e-01],
       [  1.43911055e-02,   9.85608894e-01],
       [  3.71274795e-03,   9.96287252e-01],
       [  4.69420643e-01,   5.30579357e-01]]))]
(Pydb) print clf
<SMLR(lm=0.1)>

from those pairs you could theoretically reconstruct indeed original sequences,
which would be easy especially in the case of NFoldSplitter if samples were
grouped consecutively into incrementing chunk indices.    Alternatively you
would indeed to implement the cross-validation loop manually thus keeping a
direct knowledge of which samples go into testing.

> And is there a standard way of getting the measure of confidence of
> the classifier in each prediction (binary, in my current case). This
> must be classifier specific, but I guess that GNB should give a
> probability for each of it's classifications.

well... confidence != probability (e.g. what would be the confidence if
classifier for both classes provides equivalently high-or-low
conditional probability; or think about sample which is quite far from
both sample distributions thus has low probabilities, but
substantially higher for one class over another).    

As for conditional probabilities (p(sample|class_x)) -- as you mentioned
some classifiers natively (GNB, LDA, SMLR, ... -- look into "estimates"
conditional attribute for those classifiers).  Some classifiers'
toolkits provide assessment inline with Francisco's description, e.g. in
case of libsvm in PyMVPA you could enable conditional attribute
"probabilities" which would store them for you per each sample/label:
e.g.

(Pydb) print clf.ca.probabilities
[(0.0, {0: 0.83965509604172051, 1: 0.16034490395827955}), (0.0, {0: 0.75574290235606412, 1: 0.24425709764393591}), (0.0, {0: 0.76459426891746685, 1: 0.23540573108253321}), (0.0, {0: 0.83685634150525479, 1: 0.16314365849474513}), (0.0, {0: 0.73097404850894054, 1: 0.26902595149105951}), (0.0, {0: 0.7744902806334385, 1: 0.22550971936656142}), (1.0, {0: 0.039502942964738059, 1: 0.960497057035262}), (1.0, {0: 0.15652025179037818, 1: 0.8434797482096219}), (1.0, {0: 0.1288409909607377, 1: 0.87115900903926224}), (1.0, {0: 0.15670378161345633, 1: 0.84329621838654378}), (1.0, {0: 0.14797101226503162, 1: 0.8520289877349686}), (1.0, {0: 0.19679085118644754, 1: 0.80320914881355254})]

-- 
=------------------------------------------------------------------=
Keep in touch                                     www.onerussian.com
Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic