[pymvpa] Using pprocess with Python 3.X

[Yaroslav Halchenko]

On April 8, 2017 6:08:50 PM EDT, Alex Murphy <murphyalex at gmail.com> wrote:
>Hi all,
>
>I am currently using Anaconda 3.6 with pyMVPA and everything has been
>fine
>until I came to performing some computationally demanding full-brain
>searchlight analyses. I am trying to get this to work on the
>institution's
>cluster by qsub'ing a file that generates the  event-related dataset
>from
>the NIFTY files and then performs a searchlight analysis.
>
>I am running into an annoying issue and I hope someone has already
>figured
>out a way to solve it. Basically, to have any chance of being able to
>solve
>this without going over the 12GB RAM limit imposed by the servers
>(which
>automatically kill any processes that exceed that amount) I need to use
>the
>multiple threads / cores and for that, I need the pprocess module. The
>version I downloaded is 0.5.1 which is apparently the latest, but the
>first
>issue I had is that it is written in python 2.7. So, I ran 2to3 on it
>and
>that seemed to solve one issue but then I had amother problem in
>socket.py
>
>The error that I get is:
>
>  File "/users/alex/anaconda3/lib/python3.6/socket.py", line 245, in
>makefile
>    raise ValueError("unbuffered streams must be binary")
>ValueError: unbuffered streams must be binary
>
>The problem seems to be that "pprocess" (which sphere_searchlight
>relies
>on) is written in Python 2.7 and incompatible with Python 3.X. I know
>it is
>possible to make a new virtual environment and recreate everything in
>2.7
>and then I *hope* that would solve the problem (but I am not sure and
>would
>like to see if I can get it to work with Python 3.X first).
>
>Basically, I cannot make use of the server's multi-threading procedures
>to
>allow me to do the searchlight analysis efficiently because of this. I
>do
>plan on using the GNB searchlight solution at some point but I would
>like
>to not be wholly restricted to using it because the server cannot work
>with
>"pprocess".
>
>Has anyone run into a similar problem and knows how to figure it out?
>
>** Summary of question **
>
>I run into errors using Python 3.X when doing searchlight analyses on
>my
>institution's servers because using only a single core (by default if
>pprocess isn't present) means I run over the RAM limit. The servers can
>handle really intensive workloads via parallel processing but I can't
>get
>it to work because "pprocess" gives me a few errors. The first I solved
>by
>convering pprocess.py to Python 3.X but there is an issue coming from
>socket.py about unbuffered streams must be binary. Is this a problem
>anyone
>has had or might know a solution to?
>
>Kind regards
>Alex M

Fwiw
Indeed there might be issues with pprocess and python 3...  We will need to look into it. Must work correctly with python 2, so if you could, please just install that version of anaconda

gnbsearchlight isn't using it iirc and will be fast regardless
-- 
Sent from a phone which beats iPhone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20170408/ecc8a056/attachment.html>