[pymvpa] Pkg-ExpPsy-PyMVPA Digest, Vol 66, Issue 4

[J.A. Etzel]

Yes, I'm thinking of the usual assumption that adjacent runs may be more 
similar because of temporal dependencies (scanner drifts, subject 
fatigue, etc.).

My suggestion for combining temporally adjacent runs is partly motivated 
by a desire to keep things simple: people often figure that 
leave-one-run-out is a reasonably sensible and conservative 
cross-validation scheme. But when we can't do that (such as in my 
example, when we have a lot of short runs and massive imbalance in the 
number of trials/run), the next best (simplest) thing seems combining 
adjacent runs. Pragmatically, there are a lot fewer ways to combine 
adjacent runs than random run subsets, also making the choice more 
appealing (I'd be wondering why if I saw a single random 
cross-validation scheme was used).

But I don't want to claim that combining temporally adjacent runs is the 
only legitimate technique, or even to be preferred on strict theoretical 
(as opposed to pragmatic) grounds! I suggested it as something 
relatively straightforward (and hopefully safe) to consider.


Jo

Also, to cite myself, a few years ago I played with different 
cross-validation schemes systematically, including suggestions for 
justifying partitioning other than on the runs, in "The impact of 
certain methodological choices on multivariate analysis of fMRI data 
with support vector machines". 
http://dx.doi.org/10.1016/j.neuroimage.2010.08.050



On 8/7/2013 10:47 AM, jason gors wrote:
> Could you explain the reasoning here about combining temporally adjacent
> runs to make sure to not inflate accuracy scores.  First, I'm assuming
> that what you're claiming is that adjacent runs would likely be more
> similar to one another than farther apart runs -- so, like you mention,
> leaving out adjacent runs might give you lower accuracy scores for those
> left out runs, but why is something like this more desirable than, say,
> just leaving out a random split of 1/3 of the data?  Unless you had some
> kind of reason to look at the temporal nature of classification with
> previous and/or subsequent runs, it's not clear to me why this is
> needed.  Maybe I'm a bit too unsure as to what the pipeline was in doing
> this, but I don't fully understand the reasoning.  Also, it seems that
> leaving out runs 1-3 or 7-9 (with 9 being the last run) could fulfill
> that assumption nicely, but if 4-6 were used as the left out split, then
> this seems less likely to fit that assumption -- run 4 would be just as
> similar (i.e. temporally close) to run 3 as to run 5, and run 6 would be
> just as similar to run 5 as to run 7; likewise, if you used any set of 3
> adjacent runs that weren't the first 3 runs or the last three runs, then
> you'd have the same issue for every triple, such that 2 out of 3 of the
> triple would be just as similar to non-left-out runs as to the 3 runs in
> that left-out triple anyways.  Whereas the first 3 runs would have only
> 1 run, run 3, as temporally close to non-left-out runs; likewise for run
> 7 in the last triple of runs.  Why not just use a random split of 1/3 of
> the data or do split-halves (or something similar) for hold data?