[pymvpa] Number of data points per condition: what are your guidelines?

[Emanuele Olivetti]

On 05/08/2012 11:17 PM, Vadim Axel wrote:
> Thank you a lot, Emanuele!
>
> The definition that permutation test is reliable when "the data distribution is adequately
> represented by the sample data" sounds a little bit puzzling for me :) As far as I 
> looked, the authors do not discuss this point too much. How can I know when it is indeed 
> adequate? For example, with small number of data points the permutation chance level is 
> frequently high and can be even 0.6. The distribution of predictions from reshuffling  
> looks like normal, so it is not something completely pathological. Not surprisingly, for 
> some subjects it might be difficult if not impossible to reach significance with such a 
> high chance level. Is permutation test appropriate is such scenario? In other words, can 
> it be that result is non-significant with permutation test, but the information is still 
> present.
>
>


As far as I know the results of the resampling approaches (permutation test, bootstrap, etc.)
provides correct answers only "asymptotically", i.e. when the sample size grows big. The
theoretical work in this field is concerned with finding new smarter estimators that
converge to the true value faster than previous methods. I am not aware of results
about the reliability of the permutation test on small samples that give you (even
probabilistic) bounds on how far you are from the asymptotical regime. I mean besides
toy examples.

About your example I guess you could make a simulation, like creating a simple
dataset of two classes (N(0,1) for class 1 and N(delta,1) for class 2), train
classifier and then observe the behaviour of the permutation test while changing the
number of examples and the overlap between the two classes (delta). Maybe you
would see that with low sample size and high overlap, i.e. a small effect size, the
permutation test might not be too reliable. This would not be surprising: in that
situation you lower the probability of having an adequate representation
of the underlying distribution of the data from your dataset. Any deduction following
from the data *alone* would suffer.

About your last question "can it be that result is non-significant with permutation test, 
but the
information is still present": surely it could. Your question asks how to interpret a 
non-rejected
null hypothesis (of the classifier predicting at random). The classical null-hypothesis 
testing
framework - that you are using - is asymmetric and can only reject the null hypothesis, never
accept it. For this reason a non-significant test does not provide insights on the data.

Best,

Emanuele

PS: here is the code implementing the proposed method of the paper I mentioned in my
last email. Just in case you wanted to try. In Python, of course ;-)
https://github.com/emanuele/Bayes-factor-multi-subject


>
>     Emanuele Olivetti, Sriharsha Veeramachaneni, Ewa Nowakowska, Bayesian hypothesis
>     testing for pattern discrimination in brain decoding, Pattern Recognition, 45, 2012.
>     http://dx.doi.org/10.1016/j.patcog.2011.04.025
>     I know self citations suck, still I haven't found a more convincing one.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120509/3f0e1644/attachment.html>