<div dir="ltr">Dear all,<div><br></div><div>I am trying to train a classifier on classes A vs. B and test on classes C vs. D from the same dataset. My dataset consists of 4 conditions (A, B, C, D) and 4 chunks. I wish to use a cross-validation to approach to train the classifier on A vs. B for chunks 1-3 and test on C vs.D for chunk 4 and so on. I found some emails in the archive that have addressed this issue (<a href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2013q1/002369.html">http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2013q1/002369.html</a>), but I am unsure if I am implementing them correctly in my current analysis. </div><div><br></div><div>The attached script appears to work and provides sensible outputs. However, to ensure the script is partitioning the files correctly I attempted to do a sanity check by modifying the same script to do three things:</div><div>1) Train on (A vs. B) and test on (A vs. B) - sanity check </div><div>2) Train on (C vs. D) and test on (C vs. D) - sanity check</div><div>3) Train on (A vs. B) and test on (C vs. D) - classifier of interest</div><div><br></div><div>The part of the script I modified is the chainnode (highlighted below). </div><div>For (1) I ran </div><div> MyFilter((train, train), </div><div> partitioner.get_space(),</div><div> 'targets').</div><div><br></div><div>For (2) I ran</div><div>MyFilter((test, test), </div><div> partitioner.get_space(),</div><div> 'targets'). </div><div><br></div><div>For (3) I ran</div><div>MyFilter((train, test), </div><div> partitioner.get_space(),</div><div> 'targets').</div><div><br></div><div>However, I get identical outputs for all three scripts. Could any point me in the right direction. I'm using pyMVPA 2. </div><div><br></div><div>Cheers,<br>Charlotte<br></div><div><br></div><div><br></div><div>##################I have attached the relevant part of my script below ################<br></div><div><div><div><br></div><div>train = ['A', 'B']</div><div>test = ['C', 'D']</div><div><br></div><div>labelsList = train + test</div></div><div><br></div><div># Load in dataset</div><div>ds = fmri_dataset(samples = os.path.join(path, infile), mask = os.path.join(mask_path, mask_file), chunks = sa['chunks'], targets = sa['targets'])</div><div><br></div><div># preprocessing<br>(...)</div><div><br></div><div># Selecting labels</div><div>ds = ds[np.array([l in labelsList for l in ds.sa.targets], dtype="bool")]</div><div><br></div><div>print ds.sa.targets</div><div>print train</div><div>print test</div><div>print labelsList</div><div><br></div><div>partitioner = NFoldPartitioner(cvtype=1)<br></div><div><br></div><div>for p in partitioner.generate(ds):</div><div> # convert to string to let it align in the output</div><div> print 'PA', p.sa.partitions.astype('str')</div><div><br></div><div><br></div><div>class MyFilter(Node):</div><div> </div><div> def __init__(self, target_groups, part_attr, target_attr,</div><div> space='filtered_partitions', **kwargs):</div><div> self._target_groups = target_groups</div><div> self._target_attr = target_attr</div><div> self._part_attr = part_attr</div><div> Node.__init__(self, space=space, **kwargs)</div><div> </div><div> def generate(self, ds):</div><div> </div><div> # binary mask for training and testing portion</div><div> train_part = <a href="http://ds.sa">ds.sa</a>[self._part_attr].value == 1</div><div> test_part = <a href="http://ds.sa">ds.sa</a>[self._part_attr].value == 2</div><div> </div><div> # binary mask for the first and second target group</div><div> match_1st_group = [t in self._target_groups[0] for t in</div><div><a href="http://ds.sa">ds.sa</a>[self._target_attr].value]</div><div> match_2nd_group = [t in self._target_groups[1] for t in</div><div><a href="http://ds.sa">ds.sa</a>[self._target_attr].value]</div><div> </div><div> # removed group1 in the training set and group2 in the testing set</div><div> # as I'm only interested in the variant below</div><div><br></div><div> # in the second to-be-returned dataset we will blank out</div><div> # group2 in the training set and group1 in the testing set</div><div> new_part = <a href="http://ds.sa">ds.sa</a>[self._part_attr].value.copy()</div><div> new_part[np.logical_and(train_part, match_2nd_group)] = 0</div><div> new_part[np.logical_and(test_part, match_1st_group)] = 0</div><div> <a href="http://ds.sa">ds.sa</a>[self.get_space()] = new_part</div><div> yield ds</div><div><br></div><div><b>chain = ChainNode([partitioner,</b></div><div><b> MyFilter((train, test), </b></div><div><b> partitioner.get_space(),</b></div><div><b> 'targets')</b></div><div><b> ])</b></div><div><br></div><div>for p in chain.generate(ds):</div><div> # convert to string to let it align in the output</div><div> print 'PA', p.sa.filtered_partitions.astype('str')</div><div> </div><div># Set up classifier</div><div>clf = LinearCSVMC(C=1)</div><div><br></div><div>cv = CrossValidation(clf, partitioner, errorfx = mean_match_accuracy, enable_ca=["stats"])</div><div> </div><div># Set up searchlight</div><div>sl = sphere_searchlight(cv, radius= 3,</div><div> postproc=mean_sample(), space='voxel_indices')</div><div><br></div><div>#Results of searchlight analysis</div><div>sl_results = sl(ds)</div><div><br></div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><font color="#888888">Charlotte Murphy</font><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">PhD Student</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">Department of Psychology</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">University of York,</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">Heslington, York, YO10 5DD, UK</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">Email: cem552</span><a href="mailto:fjmr500@york.ac.uk" style="color:rgb(17,85,204)" target="_blank"><span style="color:rgb(34,34,34);background:rgb(255,255,204)">@york.ac.uk</span></a><br style="color:rgb(136,136,136)"></div></div>
</div></div>