[pymvpa] Searchlight hyperalignment crashes: "featuresBus error"

[Michael Bannert]

Dear Yaroslav,

Thanks and sorry for my slow response. Our computing cluster underwent 
major maintenance recently and I then wanted to do some testing to 
provide some more detailed information.

I haven't encountered the error again. I'm usually testing my 
hyperalignment-searchlight analysis only on a subset of subjects and 
only on few voxel ("roi_ids" and "mask_node_ids" arguments, 
respectively), which yielded no problems. This error occurred only when 
running the analysis on all subjects and all voxels.

Is it hence possible that the "Bus error" occurred because of memory 
limitations?

I am running hyperalignment searchlights in parallel on a cluster. I am 
using the in-built parallelization via the "nproc" arguments. I think 
the problem with using nproc=1 but accordingly parallelizing across more 
numerious nodes is that each process will load the whole brain data 
(taking up lots of memory) and then process only its small set of voxels 
as specified in "mask_node_ids". This redundancy probably crashed my 
jobs on the clusters. The error I mentioned in my previous email 
occurred when test-running the code on a cluster node, which may have 
happened to be too busy to load all whole-brain datasets (and which is 
only meant for "light-weight" computing).

I now use fewer cluster nodes (5-10) with the maximum number of 
processors (24) and let PyMVPA take care of the parallelization within 
the specified subset of voxels. This seems to work.

So to cut a long story short and to answer your question, Yaroslav: I 
don't think that choosing "nproc > 1" was the issue here. More likely 
memory (?). It could also have something to do with the maintenance work 
they have been undertaking at the cluster.

Thanks for your help!

Best,
Michael


On 05/10/17 15:40, Yaroslav Halchenko wrote:
> oh
> 
> could you try please with nproc=1 (for the searchlight)  and see if that
> avoids it (although would become slower)
> 
> On Thu, 05 Oct 2017, Michael Bannert wrote:
> 
>> Dear Swaroop, dear PyMVPA users,
> 
>> I'm running a couple of searchlight hyperalignments but some of them crash.
>> The error message is "featuresBus error". What does this mean? What could be
>> causing this error?
> 
>> Thanks and best,
>> Michael
> 
>> PS: Here is the output of wtf()
>> Current date:   2017-10-05 13:35
>> PyMVPA:
>>   Version:       2.6.0
> 
> just FYI, there is 2.6.1 already, so if you can (e.g. not in the middle of
> the study), update
>