[pymvpa] custom cross-validation procedure: train on individual blocks, test on averaged blocks?

[e c]

Thanks Yarik!

I don't pretend to understand the code (yet), but it runs without error, except for this warning:

WARNING: There were no samples for combination {'targets': 'HI', 'custom': 10}. It might be a sign of a disbalanced dataset <Dataset: 207x310 at float64, <sa: blocks,censor,TR,custom,chunks,TR2,targets,partitions>, <a: mapper,lastpartitionset,partitions_set,lastsplit>>.
So I'll take your word for it that it is doing what I think it is supposed to do, for now? :)

-Edmund



________________________________
interesting question...

quick answer: we don't have 1 liner pre-crafted solution but I see few
possible resolutions for you ;-)  you are hitting a tiny problem though
(which was recently brought up by M.Casey email) that output number of
predictions from the classifier cannot be different from # of samples of
input data... so it can't be a MappedClassifier, but if your goal is
just to assess such a cross-validation then you could do it with just a
bit of coding... let's discuss imho the easiest approach

I. creating custom sample-attribute based on partitioning and targets
   followed by mean_group_sample

so here would be the code for you to test (and report back) either it
does what you want:

class TestTogetherTrainAlone(Mapper):
    def _forward_dataset(self, ds):
        out = ds.copy()
        out.sa['custom'] = ds.sa.partitions.copy()
        # 1 is the "training" and 2 is the "testing" we would like to mean
        # so let's enforce separate partitions instead of 1
        partition1 = ds.sa.partitions == 1
        # 10 is just a large enough number > 2 ;)
        out.sa.custom[partition1] = 10 + np.arange(np.sum(partition1))
        return out

cv = CrossValidation(ChainMapper(
                          [TestTogetherTrainAlone(),
                           mean_group_sample(['targets', 'custom']),
                           CLASSIFIER], space='targets'),
                     NFoldPartitioner(1),
                     descr='custom-CV')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120307/551567e7/attachment.html>