[pymvpa] GNBSearchlight below/above chance accuracy ... again

[Yaroslav Halchenko]

On Fri, 22 Jul 2016, basile pinsard wrote:

> Hi PyMVPA community,

> I wanted to have an advice on a problem I have using PyMVPA.
> My pipeline includes a Searchlight on BOLD data, for which I used the
> optimized GNBSearchlight because I plan to run ~100 permutations to perform
> statistical testing and it is the only one offering reasonable processing
> time (or maybe the optimized KNN).

> I have 2 classes x 8 samples for each (1 sample per chunk), the partitioner
> (thanks @Yaroslav) I use is:
> prtnr_2fold_factpart = FactorialPartitioner(
>     NFoldPartitioner(cvtype=2,attr='chunks'),
>     attr='targets',
>     selection_strategy='equidistant',
>     count=32)
> this way I repeatedly take out 2 samples of each of the 2 classes for
> testing and train on the remaining 2x6 samples, 'equidistant' allows all
> the samples to be tested approximately the same number of time, thus being
> equally represented in the final accuracy score.

> The problem is that the distribution of accuracy in searchlight map is very
> wide with significantly below-chance classification, and the results are
> very variable across scans/subjects.

first let's make sure that we we looking at the correct thing -- default
errorfx=mean_mismatch_error i.e. it is an error (lower is better ;) ),
not accuracy

> So what I did to check if there was any problem in the analysis was to
> replace my BOLD signal with random data from normal distribution, thus
> removing any potential temporal dependency (even if the design was using
> DeBruijn cycles for balancing carry-over effects) that could also result
> from GLM (GLM-LSS, Mumford 2012), detrending or else.

> Results, I get some accuracy from ~10% to ~90%, far below above chance
> expected by normal approximation to binomial distribution (25-75%).

as already mentioned, parametric assumption might not apply here since
trials might be all dependent and depending on what distribution you are
looking at.  e.g. if looking at distribution of results across
searchlights, not across random data samples for the same dataset/roi,
you should expect lots of dependencies across different spatial
locations.  Even if all data is randomly generated, neighboring
searchlights would use the same voxels... although should matter less in
case of random data.

> It seems that either from the design, pipeline or algorithm the information
> is found by chance in the random data.

> I took the neighborhood of where I got these results and ran a
> cross-validation using the same partitioner but with GNB, LinearCSVMC, LDA.
> GNB gives the same accuracy, so this is not the optimized GNBSearchlight
> that causes this
> LinearCSVMC and LDA gives about chance (50%) accuracy for the same
> neighborhood.

that is on real data, right?  how did you pre-process it?

could you share the plots of distributions (just curious)

> This can be reproduced by creating a random dataset from scratch with 2
> classes and randomly selecting some features:
> ds_rand2=dataset_wizard(
>      np.random.normal(size=(16,10000)),
>      targets=[0,1]*8,
>      chunks=np.arange(16))
> cvte=CrossValidation(
>     GNB(common_variance=True),
>     prtnr_2fold_factpart,
>     errorfx=mean_match_accuracy)
> np.max([cvte(ds_rand2[:,np.random.randint(0,ds_rand2.nfeatures,size=64)]).samples.mean()
> for i in range(1000)])
> 0.8828125
> np.min([cvte(ds_rand2[:,np.random.randint(0,ds_rand2.nfeatures,size=64)]).samples.mean()
> for i in range(1000)])
> 0.1484375

> So is there something specific to GNB that gives this kind of lucky
> overfitting of random data when use many times as in Searchlight?
> Also as this lucky features are included in multiple overlapping
> neighborhood it results in nice blobs in the searchlight which sizes
> depends on radius.
> I tried the GNB with and without common_variance (thus piecewise quadratic
> or linear) and it is quite similar.
> Does anybody have been using it to produce sensible results?
> Maybe it work better with more that 2 classes.

> LDA when applied to more features than samples is incredibly slow, thus is
> unrealistic for searchlight and even more with permutation testing, but I
> have seen it used in many papers (maybe not with permutation though), so i
> wonder if it is PyMVPA algorithm, or my python setup.
> Do you think an optimized LDA searchlight would be possible or there is
> lengthy computation (eg: matrix inversion) that cannot be factorized?

try first that m1nn as Richard recommended.  It is simple but powerful
if your data distribution matches its assumptions which are really close
to the one behind LDA ;)

> Otherwise what kind of classifier would you recommend, that would not be
> too computationally intensive? Or maybe I have to deal with that?

> Many thanks for any idea about that.
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik