[pymvpa] Using pprocess with Python 3.X

[Alex Murphy]

Hi all,

I am currently using Anaconda 3.6 with pyMVPA and everything has been fine
until I came to performing some computationally demanding full-brain
searchlight analyses. I am trying to get this to work on the institution's
cluster by qsub'ing a file that generates the  event-related dataset from
the NIFTY files and then performs a searchlight analysis.

I am running into an annoying issue and I hope someone has already figured
out a way to solve it. Basically, to have any chance of being able to solve
this without going over the 12GB RAM limit imposed by the servers (which
automatically kill any processes that exceed that amount) I need to use the
multiple threads / cores and for that, I need the pprocess module. The
version I downloaded is 0.5.1 which is apparently the latest, but the first
issue I had is that it is written in python 2.7. So, I ran 2to3 on it and
that seemed to solve one issue but then I had amother problem in socket.py

The error that I get is:

  File "/users/alex/anaconda3/lib/python3.6/socket.py", line 245, in
makefile
    raise ValueError("unbuffered streams must be binary")
ValueError: unbuffered streams must be binary

The problem seems to be that "pprocess" (which sphere_searchlight relies
on) is written in Python 2.7 and incompatible with Python 3.X. I know it is
possible to make a new virtual environment and recreate everything in 2.7
and then I *hope* that would solve the problem (but I am not sure and would
like to see if I can get it to work with Python 3.X first).

Basically, I cannot make use of the server's multi-threading procedures to
allow me to do the searchlight analysis efficiently because of this. I do
plan on using the GNB searchlight solution at some point but I would like
to not be wholly restricted to using it because the server cannot work with
"pprocess".

Has anyone run into a similar problem and knows how to figure it out?

** Summary of question **

I run into errors using Python 3.X when doing searchlight analyses on my
institution's servers because using only a single core (by default if
pprocess isn't present) means I run over the RAM limit. The servers can
handle really intensive workloads via parallel processing but I can't get
it to work because "pprocess" gives me a few errors. The first I solved by
convering pprocess.py to Python 3.X but there is an issue coming from
socket.py about unbuffered streams must be binary. Is this a problem anyone
has had or might know a solution to?

Kind regards
Alex M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170409/cd0894fb/attachment.html>