[Neurodebian-users] Condor and Neurodebian

[Michael Hanke]

Hi,

On Sun, Dec 29, 2013 at 09:47:35AM +0330, Arman Eshaghi wrote:
> We are in the process of migration from CentOs servers to Debian
> servers (to include Neurodebian in our lab). We have been previously
> using SGE as grid engine that worked fine, although it was painful for
> the admin. I can see a nice tutorial on Neurodebian website
> http://neuro.debian.net/blog/2012/2012-03-09_parallelize_fsl_with_condor.html
> that works great on one machine. The tutorial, however, does not
> mention anything regarding a typical shared lab server and multiple
> nodes to work together. To further consider Condor on our Neurodebian
> servers, I have the following issues:
> 
> 1- Does Condor work with other softwares such as Freesurfer and more
> generally with any shell script (so users can submit custom shell
> scripts, or even Nipype python parallelization)? 

Yes, it does. Nipype has not one, but two dedicated execution plugins
for Condor -- one include the submission of entire processing graphs to
Condor's DAGMan utility -- so Condor manages the workflow execution
entirely. Beyond nipype and generic executables/scripts Condor has
dedicated support for jobs wrapped in virtual machines and jobs that
need to be submitted to cloud computing facilities.

Condor works well on sinlge machines (although pretty much a toy use
case), clusters, grids, clouds and a mixtures of all the above -- may it
be dedicated or part-time execution hosts.

> 2. I would be much grateful if anyone can point me to a tutorial
> webpage for configuration of servers (admin host in SGE terminology)
> and other nodes (execution hosts in SGE terminology) so that multiple
> nodes can work together. 

Condor's manual has pretty much all you need. It comes with the package
and is available online for all relevant versions.

Feel free to outline you plans, if you are looking for specific advise.

Cheers,

Michael

-- 
Michael Hanke
http://mih.voxindeserto.de