[pymvpa] sphere_gnbsearchlight & Monte Carlo Testing

[wolf zinke]

Hi,

This really sounds like a nasty and painful bug to identify.
> quick resolution for you:  set nproc=1
Setting 'nproc=1' gave me indeed a more variable distribution. This bug 
and its solution is only affecting the sphere_searchlight, right? It is 
not affecting the constant values of the gnb_searchlight, because it 
does not allow the nproc option. There, I still have the constant 
results, that you explained before with the assumption of constant 
targets across partitioning schemes.

cheers,
wolf

On 10/24/2012 05:09 AM, Yaroslav Halchenko wrote:
> On Tue, 23 Oct 2012, wolf zinke wrote:
>>> Meanwhile, on your data, could you confirm that there is some variance
>>> in the obtained performances in a "good/lengthy" code, i.e. that values
>>> np.std(distr_est.ca.dist_samples.samples[0], axis=1)
>> Thanks for the reply. Indeed, using a sphere_searchlight with the
>> Monte Carlo Testing, this gives me zeros only.
> This is a really nice one, i.e. the bug which is there "by design" so
> not that obvious from a first sight ;)  Also good that it should not be
> relevant for 99% of other users since the constructs in this case
> are elaborate enough to not be too common ;)
>
> The reason for the observed abnormal behavior is the nproc > 1,
> when searchlights are parallelized.  Then every child process acquires
> "a copy of numpy", including the state of its random number generator
> (RNG).  That results in identical initial states of RNG in every one of
> those processes, leading to the same permutations, thus degenerate
> results in your case.
>
> quick resolution for you:  set nproc=1
>
> longer resolution for us:
>
>    we would need to sacrifice "exact reproducibility of results with a
>    single global seeding of RNG", i.e. mvpa2.seed() (or via MVPA_SEED
>    environment variable).  Every Permutator in this case should get
>    its RNG randomly seeded (i.e. we should just use a new
>    numpy.random.RandomState() in every one of them in _call, unless
>    instructed explicitly otherwise).
>
>    I have pushed an initial fix (only for Permutator, which is what
>    important for your case here).  More (I see at least a few of other
>    similar spots) to follow + feeding rng in some of the tests
>