[Python-modules-commits] r4747 - in packages/parallelpython/trunk/debian/patches (1 file)
tviehmann at users.alioth.debian.org
tviehmann at users.alioth.debian.org
Sat Mar 8 19:22:21 UTC 2008
Date: Saturday, March 8, 2008 @ 19:22:20
Author: tviehmann
Revision: 4747
spelling suggestions
Modified:
packages/parallelpython/trunk/debian/patches/10_remove_default_password.dpatch
Modified: packages/parallelpython/trunk/debian/patches/10_remove_default_password.dpatch
===================================================================
--- packages/parallelpython/trunk/debian/patches/10_remove_default_password.dpatch 2008-03-08 19:11:53 UTC (rev 4746)
+++ packages/parallelpython/trunk/debian/patches/10_remove_default_password.dpatch 2008-03-08 19:22:20 UTC (rev 4747)
@@ -21,7 +21,7 @@
ppservers - list of active parallel python execution servers
to connect with
- secret - passphrase for network connections, if omitted a default
-- passphrase will be used. It's highly recommended to use a
+- passphrase will be used. It's highly recommended to use a
- custom passphrase for all network connections.
+ secret - passphrase for network connections; it can be set via
+ command-line or configuration file
@@ -45,7 +45,7 @@
<tr>
<td valign="top" colspan="2">
- <p> <a href="#API">Module API</a> <br /> <a href="#QUICKSMP">Quick start guide, SMP</a><br /> <a href="#QUICKCLUSTERS">Quick start guide, clusters<br /> </a><a href="#QUICKCLUSTERSAUTO">Quick start guide, clusters with auto-discovery</a><br /> <a href="#ADVANCEDCLUSTERS">Advanced guide, clusters</a><br /> <a href="#COMMANDLINE">Command line arguments, ppserver.py</a><br /> <a href="#SECURITY">Security and secret key</a> </p><hr /> <p> </p><h1 id="API"> pp 1.5.3 module API</h1> <p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Server" title="Server"></a>class <strong>Server</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2">Parallel Python SMP execution server class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Server-__init__" title="Server-__init__"></a><strong>__init__</strong>(self, ncpus<font color="#909090">='autodetect'</font>, ppservers<font color="#909090">=()</font>, secret<font color="#909090">=None</font>, loglevel<font color="#909090">=30</font>, logstream<font color="#909090">=<open file '<stderr>', mode 'w'></font>)</dt><dd>Creates <a href="#Server">Server</a> instance<br /> <br /> ncpus - the number of worker processes to start on the local <br /> computer, if parameter is omitted it will be set to <br /> the number of processors in the system<br /> ppservers - list of active parallel python execution servers <br /> to connect with<br /> secret - passphrase for network connections, if omitted a default<br /> passphrase will be used. It's highly recommended to use a <br /> custom passphrase for all network connections.<br /> loglevel - logging level<br /> logstream - log stream destination<br /> <br /> With ncpus = 1 all tasks are executed consequently<br /> For the best performance either use the default "autodetect" value<br /> or set ncpus to the total number of processors in the system</dd></dl><dl><dt><a name="Server-destroy" title="Server-destroy"></a><strong>destroy</strong>(self)</dt><dd>Kills ppworkers and closes open files</dd></dl> <dl><dt><a name="Server-get_active_nodes" title="Server-get_active_nodes"></a><strong>get_active_nodes</strong>(self)</dt><dd>Returns active nodes as a dictionary <br /> [keys - nodes, values - ncpus]</dd></dl> <dl><dt><a name="Server-get_ncpus" title="Server-get_ncpus"></a><strong>get_ncpus</strong>(self)</dt><dd>Returns the number of local worker processes (ppworkers)</dd></dl> <dl><dt><a name="Server-get_stats" title="Server-get_stats"></a><strong>get_stats</strong>(self)</dt><dd>Returns job execution statistics as a dictionary</dd></dl><dl><dt><a name="Server-print_stats" title="Server-print_stats"></a><strong>print_stats</strong>(self)</dt><dd>Prints job execution statistics. Useful for benchmarking on <br /> clusters</dd></dl> <dl><dt><a name="Server-set_ncpus" title="Server-set_ncpus"></a><strong>set_ncpus</strong>(self, ncpus<font color="#909090">='autodetect'</font>)</dt><dd>Sets the number of local worker processes (ppworkers)<br /> <br /> ncpus - the number of worker processes, if parammeter is omitted<br /> it will be set to the number of processors in the system</dd></dl> <dl><dt><a name="Server-submit" title="Server-submit"></a><strong>submit</strong>(self, func, args<font color="#909090">=()</font>, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Submits function to the execution queue<br /> <br /> func - function to be executed<br /> args - tuple with arguments of the 'func'<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when <a href="#Server-wait">wait</a>(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Server-wait" title="Server-wait"></a><strong>wait</strong>(self, group<font color="#909090">=None</font>)</dt><dd>Waits for all jobs in a given group to finish.<br /> If group is omitted waits for all jobs to finish</dd></dl> <dl><dt><strong>default_port</strong> = 60000</dt></dl> <dl><dt><strong>default_secret</strong> = 'epo20pdosl;dksldkmm'</dt></dl> </td></tr></tbody></table> </p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Template" title="Template"></a>class <strong>Template</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2"><a href="#Template">Template</a> class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Template-__init__" title="Template-__init__"></a><strong>__init__</strong>(self, job_server, func, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Creates <a href="#Template">Template</a> instance<br /> <br /> jobs_server - pp server for submitting jobs<br /> func - function to be executed<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when wait(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Template-submit" title="Template-submit"></a><strong>submit</strong>(self, *args)</dt><dd>Submits function with *arg arguments to the execution queue</dd></dl> </td></tr></tbody></table></p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#55aa55"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#ffffff"><strong>Data</strong></font></td></tr> <tr><td bgcolor="#55aa55"> </td><td> </td> <td width="100%"><strong>copyright</strong> = 'Copyright (c) 2005-2008 Vitalii Vanovschi. All rights reserved'<br /> <strong>version</strong> = '1.5.1'</td></tr></tbody></table> </p><hr /><h1 id="QUICKSMP"> Quick start guide, SMP<br /></h1> <p>1) Import pp module:</p><p><strong> import pp</strong></p><p>2) Start pp execution server with the number of workers set to the number of processors in the system </p><p><strong> job_server = pp.Server() </strong></p><p>3) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>4) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p> <hr /><h1 id="QUICKCLUSTERS"> Quick start guide, clusters </h1><p><em><strong>On the nodes</strong></em> <br /></p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver.py </strong></p><p><strong> node-2> ./ppserver.py</strong></p><p><strong> node-3> ./ppserver.py</strong></p><p><em><strong>On the client</strong></em> <br /></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver.py) </p><p><strong> ppservers=("node-1", "node-2", "node-3")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a></p> <hr /><h1 id="QUICKCLUSTERSAUTO"> Quick start guide, clusters with autodiscovery<br /> </h1><p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver.py -a<br /> </strong></p><p><strong> node-2> ./ppserver.py -a</strong></p><p><strong> node-3> ./ppserver.py -a<br /></strong></p><p><em><strong>On the client</strong></em></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Set ppservers list to auto-discovery: </p><p><strong> ppservers=("*",)</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="ADVANCEDCLUSTERS"> Advanced guide, clusters </h1> <p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes (listen to a given port 35000,<br /> and local network interface only, accept only connections which know correct secret):</p><p><strong> node-1> ./ppserver.py -p 35000 -i 192.168.0.101 -s "mysecret"<br /></strong></p><p><strong> node-2> ./ppserver.py -p 35000 -i 192.168.0.102</strong><strong> -s "mysecret"</strong></p><p><strong> node-3> ./ppserver.py -p 35000 -i 192.168.0.103</strong><strong> -s "mysecret"</strong></p><p><em><strong>On the client</strong></em> <br /></p> <p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver.py) </p><p><strong> ppservers=("node-1:35000", "node-2:</strong><strong>35000</strong><strong>", "node-3:</strong><strong>35000</strong><strong>")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system, <br />list of ppservers to connect with and secret key to authorize the connection:</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>, secret="</strong><strong>mysecret</strong><strong>") </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> 7) Print the execution statistics:<br /></p><p><strong> job_server.print_stats()</strong></p><p>To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="COMMANDLINE"> Command line options, ppserver.py </h1> <pre>Usage: ppserver.py [-hda] [-i interface] [-b broadcast] [-p port] [-w nworkers] [-s secret] [-t seconds]<br /> Options:<br /> -h : this help message<br /> -d : debug<br /> -a : enable auto-discovery service<br /> -i interface : interface to listen<br /> -b broadcast : broadcast address for auto-discovery service<br /> -p port : port to listen<br /> -w nworkers : number of workers to start<br /> -s secret : secret for authentication<br /> -t seconds : timeout to exit if no connections with clients exist<br /> </pre><hr /><h1 id="COMMANDLINE"> Security and secret key<a name="SECURITY" title="SECURITY"></a></h1><p> Due to the security concerns it is highly recommended to run ppserver.py with an non-trivial secret key (-s command line argument) which should be paired with the matching <em>secret</em> keyword of PP Server class constructor. Since PP 1.5.3 it is possible to set secret key by assigning <strong>pp_secret</strong> variable in the configuration file <strong>.pythonrc.py</strong> which should be located in the user home directory (please make this file readable and writable only by user). The secret key set in .pythonrc.py could be overridden by command line argument (for ppserver.py) and <em>secret</em> keyword (for PP Server class constructor). </p>
-+ <p> <a href="#API">Module API</a> <br /> <a href="#QUICKSMP">Quick start guide, SMP</a><br /> <a href="#QUICKCLUSTERS">Quick start guide, clusters<br /> </a><a href="#QUICKCLUSTERSAUTO">Quick start guide, clusters with auto-discovery</a><br /> <a href="#ADVANCEDCLUSTERS">Advanced guide, clusters</a><br /> <a href="#COMMANDLINE">Command line arguments, ppserver</a><br /> <a href="#SECURITY">Security and secret key</a> </p><hr /> <p> </p><h1 id="API"> pp 1.5.3 module API</h1> <p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Server" title="Server"></a>class <strong>Server</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2">Parallel Python SMP execution server class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Server-__init__" title="Server-__init__"></a><strong>__init__</strong>(self, ncpus<font color="#909090">='autodetect'</font>, ppservers<font color="#909090">=()</font>, secret<font color="#909090">=None</font>, loglevel<font color="#909090">=30</font>, logstream<font color="#909090">=<open file '<stderr>', mode 'w'></font>)</dt><dd>Creates <a href="#Server">Server</a> instance<br /> <br /> ncpus - the number of worker processes to start on the local <br /> computer, if parameter is omitted it will be set to <br /> the number of processors in the system<br /> ppservers - list of active parallel python execution servers <br /> to connect with<br /> secret - passphrase for network connections, if omitted a default<br /> passphrase will be used. It's highly recommended to use a <br /> custom passphrase for all network connections.<br /> loglevel - logging level<br /> logstream - log stream destination<br /> <br /> With ncpus = 1 all tasks are executed consequently<br /> For the best performance either use the default "autodetect" value<br /> or set ncpus to the total number of processors in the system</dd></dl><dl><dt><a name="Server-destroy" title="Server-destroy"></a><strong>destroy</strong>(self)</dt><dd>Kills ppworkers and closes open files</dd></dl> <dl><dt><a name="Server-get_active_nodes" title="Server-get_active_nodes"></a><strong>get_active_nodes</strong>(self)</dt><dd>Returns active nodes as a dictionary <br /> [keys - nodes, values - ncpus]</dd></dl> <dl><dt><a name="Server-get_ncpus" title="Server-get_ncpus"></a><strong>get_ncpus</strong>(self)</dt><dd>Returns the number of local worker processes (ppworkers)</dd></dl> <dl><dt><a name="Server-get_stats" title="Server-get_stats"></a><strong>get_stats</strong>(self)</dt><dd>Returns job execution statistics as a dictionary</dd></dl><dl><dt><a name="Server-print_stats" title="Server-print_stats"></a><strong>print_stats</strong>(self)</dt><dd>Prints job execution statistics. Useful for benchmarking on <br /> clusters</dd></dl> <dl><dt><a name="Server-set_ncpus" title="Server-set_ncpus"></a><strong>set_ncpus</strong>(self, ncpus<font color="#909090">='autodetect'</font>)</dt><dd>Sets the number of local worker processes (ppworkers)<br /> <br /> ncpus - the number of worker processes, if parammeter is omitted<br /> it will be set to the number of processors in the system</dd></dl> <dl><dt><a name="Server-submit" title="Server-submit"></a><strong>submit</strong>(self, func, args<font color="#909090">=()</font>, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Submits function to the execution queue<br /> <br /> func - function to be executed<br /> args - tuple with arguments of the 'func'<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when <a href="#Server-wait">wait</a>(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Server-wait" title="Server-wait"></a><strong>wait</strong>(self, group<font color="#909090">=None</font>)</dt><dd>Waits for all jobs in a given group to finish.<br /> If group is omitted waits for all jobs to finish</dd></dl> <dl><dt><strong>default_port</strong> = 60000</dt></dl> <dl><dt><strong>default_secret</strong> = 'epo20pdosl;dksldkmm'</dt></dl> </td></tr></tbody></table> </p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Template" title="Template"></a>class <strong>Template</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2"><a href="#Template">Template</a> class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Template-__init__" title="Template-__init__"></a><strong>__init__</strong>(self, job_server, func, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Creates <a href="#Template">Template</a> instance<br /> <br /> jobs_server - pp server for submitting jobs<br /> func - function to be executed<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when wait(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Template-submit" title="Template-submit"></a><strong>submit</strong>(self, *args)</dt><dd>Submits function with *arg arguments to the execution queue</dd></dl> </td></tr></tbody></table></p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#55aa55"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#ffffff"><strong>Data</strong></font></td></tr> <tr><td bgcolor="#55aa55"> </td><td> </td> <td width="100%"><strong>copyright</strong> = 'Copyright (c) 2005-2008 Vitalii Vanovschi. All rights reserved'<br /> <strong>version</strong> = '1.5.1'</td></tr></tbody></table> </p><hr /><h1 id="QUICKSMP"> Quick start guide, SMP<br /></h1> <p>1) Import pp module:</p><p><strong> import pp</strong></p><p>2) Start pp execution server with the number of workers set to the number of processors in the system </p><p><strong> job_server = pp.Server() </strong></p><p>3) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>4) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p> <hr /><h1 id="QUICKCLUSTERS"> Quick start guide, clusters </h1><p><em><strong>On the nodes</strong></em> <br /></p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver </strong></p><p><strong> node-2> ./ppserver</strong></p><p><strong> node-3> ./ppserver</strong></p><p><em><strong>On the client</strong></em> <br /></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver) </p><p><strong> ppservers=("node-1", "node-2", "node-3")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a></p> <hr /><h1 id="QUICKCLUSTERSAUTO"> Quick start guide, clusters with autodiscovery<br /> </h1><p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver -a<br /> </strong></p><p><strong> node-2> ./ppserver -a</strong></p><p><strong> node-3> ./ppserver -a<br /></strong></p><p><em><strong>On the client</strong></em></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Set ppservers list to auto-discovery: </p><p><strong> ppservers=("*",)</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="ADVANCEDCLUSTERS"> Advanced guide, clusters </h1> <p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes (listen to a given port 35000,<br /> and local network interface only, accept only connections which know correct secret):</p><p><strong> node-1> ./ppserver -p 35000 -i 192.168.0.101 -s "mysecret"<br /></strong></p><p><strong> node-2> ./ppserver -p 35000 -i 192.168.0.102</strong><strong> -s "mysecret"</strong></p><p><strong> node-3> ./ppserver -p 35000 -i 192.168.0.103</strong><strong> -s "mysecret"</strong></p><p><em><strong>On the client</strong></em> <br /></p> <p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver) </p><p><strong> ppservers=("node-1:35000", "node-2:</strong><strong>35000</strong><strong>", "node-3:</strong><strong>35000</strong><strong>")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system, <br />list of ppservers to connect with and secret key to authorize the connection:</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>, secret="</strong><strong>mysecret</strong><strong>") </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> 7) Print the execution statistics:<br /></p><p><strong> job_server.print_stats()</strong></p><p>To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="COMMANDLINE"> Command line options, ppserver</h1> <pre>Usage: ppserver [-hda] [-i interface] [-b broadcast] [-p port] [-w nworkers] [-s secret] [-t seconds]<br /> Options:<br /> -h : this help message<br /> -d : debug<br /> -a : enable auto-discovery service<br /> -i interface : interface to listen<br /> -b broadcast : broadcast address for auto-discovery service<br /> -p port : port to listen<br /> -w nworkers : number of workers to start<br /> -s secret : secret for authentication<br /> -t seconds : timeout to exit if no connections with clients exist<br /> </pre><hr /><h1 id="COMMANDLINE"> Security and secret key<a name="SECURITY" title="SECURITY"></a></h1><p> Due to the security concerns, in Debian we decided to disable default password authentication, and require to run ppserver with an non-trivial secret key (-s command line argument) which should be paired with the matching <em>secret</em> keyword of PP Server class constructor. An alternative way to set a secret key is by assigning <strong>pp_secret</strong> variable in the configuration file <strong>.pythonrc.py</strong> which should be located in the user home directory (please make this file readable and writable only by user). The secret key set in .pythonrc.py could be overridden by command line argument (for ppserver) and <em>secret</em> keyword (for PP Server class constructor). Note that passing the password on the command line, allows every user to see it (using ps tool, for example), and that running it on an open/untrusted network can be a security problem as traffic, including the password, is not encrypted. </p>
++ <p> <a href="#API">Module API</a> <br /> <a href="#QUICKSMP">Quick start guide, SMP</a><br /> <a href="#QUICKCLUSTERS">Quick start guide, clusters<br /> </a><a href="#QUICKCLUSTERSAUTO">Quick start guide, clusters with auto-discovery</a><br /> <a href="#ADVANCEDCLUSTERS">Advanced guide, clusters</a><br /> <a href="#COMMANDLINE">Command line arguments, ppserver</a><br /> <a href="#SECURITY">Security and secret key</a> </p><hr /> <p> </p><h1 id="API"> pp 1.5.3 module API</h1> <p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Server" title="Server"></a>class <strong>Server</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2">Parallel Python SMP execution server class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Server-__init__" title="Server-__init__"></a><strong>__init__</strong>(self, ncpus<font color="#909090">='autodetect'</font>, ppservers<font color="#909090">=()</font>, secret<font color="#909090">=None</font>, loglevel<font color="#909090">=30</font>, logstream<font color="#909090">=<open file '<stderr>', mode 'w'></font>)</dt><dd>Creates <a href="#Server">Server</a> instance<br /> <br /> ncpus - the number of worker processes to start on the local <br /> computer, if parameter is omitted it will be set to <br /> the number of processors in the system<br /> ppservers - list of active parallel python execution servers <br /> to connect with<br /> secret - passphrase for network connections, if omitted a default<br /> passphrase will be used. It's highly recommended to use a <br /> custom passphrase for all network connections.<br /> loglevel - logging level<br /> logstream - log stream destination<br /> <br /> With ncpus = 1 all tasks are executed consequently<br /> For the best performance either use the default "autodetect" value<br /> or set ncpus to the total number of processors in the system</dd></dl><dl><dt><a name="Server-destroy" title="Server-destroy"></a><strong>destroy</strong>(self)</dt><dd>Kills ppworkers and closes open files</dd></dl> <dl><dt><a name="Server-get_active_nodes" title="Server-get_active_nodes"></a><strong>get_active_nodes</strong>(self)</dt><dd>Returns active nodes as a dictionary <br /> [keys - nodes, values - ncpus]</dd></dl> <dl><dt><a name="Server-get_ncpus" title="Server-get_ncpus"></a><strong>get_ncpus</strong>(self)</dt><dd>Returns the number of local worker processes (ppworkers)</dd></dl> <dl><dt><a name="Server-get_stats" title="Server-get_stats"></a><strong>get_stats</strong>(self)</dt><dd>Returns job execution statistics as a dictionary</dd></dl><dl><dt><a name="Server-print_stats" title="Server-print_stats"></a><strong>print_stats</strong>(self)</dt><dd>Prints job execution statistics. Useful for benchmarking on <br /> clusters</dd></dl> <dl><dt><a name="Server-set_ncpus" title="Server-set_ncpus"></a><strong>set_ncpus</strong>(self, ncpus<font color="#909090">='autodetect'</font>)</dt><dd>Sets the number of local worker processes (ppworkers)<br /> <br /> ncpus - the number of worker processes, if parammeter is omitted<br /> it will be set to the number of processors in the system</dd></dl> <dl><dt><a name="Server-submit" title="Server-submit"></a><strong>submit</strong>(self, func, args<font color="#909090">=()</font>, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Submits function to the execution queue<br /> <br /> func - function to be executed<br /> args - tuple with arguments of the 'func'<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when <a href="#Server-wait">wait</a>(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Server-wait" title="Server-wait"></a><strong>wait</strong>(self, group<font color="#909090">=None</font>)</dt><dd>Waits for all jobs in a given group to finish.<br /> If group is omitted waits for all jobs to finish</dd></dl> <dl><dt><strong>default_port</strong> = 60000</dt></dl> <dl><dt><strong>default_secret</strong> = 'epo20pdosl;dksldkmm'</dt></dl> </td></tr></tbody></table> </p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#ffc8d8"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#000000"><a name="Template" title="Template"></a>class <strong>Template</strong></font></td></tr> <tr bgcolor="#ffc8d8"><td rowspan="2"> </td> <td colspan="2"><a href="#Template">Template</a> class<br /> </td></tr> <tr><td> </td> <td width="100%">Methods defined here:<br /> <dl><dt><a name="Template-__init__" title="Template-__init__"></a><strong>__init__</strong>(self, job_server, func, depfuncs<font color="#909090">=()</font>, modules<font color="#909090">=()</font>, callback<font color="#909090">=None</font>, callbackargs<font color="#909090">=()</font>, group<font color="#909090">='default'</font>, globals<font color="#909090">=None</font>)</dt><dd>Creates <a href="#Template">Template</a> instance<br /> <br /> jobs_server - pp server for submitting jobs<br /> func - function to be executed<br /> depfuncs - tuple with functions which might be called from 'func'<br /> modules - tuple with module names to import<br /> callback - callback function which will be called with argument <br /> list equal to callbackargs+(result,) <br /> as soon as calculation is done<br /> callbackargs - additional arguments for callback function<br /> group - job group, is used when wait(group) is called to wait for<br /> jobs in a given group to finish<br /> globals - dictionary from which all modules, functions and classes<br /> will be imported, for instance: globals=globals()</dd></dl> <dl><dt><a name="Template-submit" title="Template-submit"></a><strong>submit</strong>(self, *args)</dt><dd>Submits function with *arg arguments to the execution queue</dd></dl> </td></tr></tbody></table></p><p> <table border="0" cellspacing="0" cellpadding="2" width="100%" summary="section"> <tbody><tr bgcolor="#55aa55"> <td colspan="3" valign="bottom"> <br /> <font face="helvetica, arial" color="#ffffff"><strong>Data</strong></font></td></tr> <tr><td bgcolor="#55aa55"> </td><td> </td> <td width="100%"><strong>copyright</strong> = 'Copyright (c) 2005-2008 Vitalii Vanovschi. All rights reserved'<br /> <strong>version</strong> = '1.5.1'</td></tr></tbody></table> </p><hr /><h1 id="QUICKSMP"> Quick start guide, SMP<br /></h1> <p>1) Import pp module:</p><p><strong> import pp</strong></p><p>2) Start pp execution server with the number of workers set to the number of processors in the system </p><p><strong> job_server = pp.Server() </strong></p><p>3) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>4) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p> <hr /><h1 id="QUICKCLUSTERS"> Quick start guide, clusters </h1><p><em><strong>On the nodes</strong></em> <br /></p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver </strong></p><p><strong> node-2> ./ppserver</strong></p><p><strong> node-3> ./ppserver</strong></p><p><em><strong>On the client</strong></em> <br /></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver) </p><p><strong> ppservers=("node-1", "node-2", "node-3")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a></p> <hr /><h1 id="QUICKCLUSTERSAUTO"> Quick start guide, clusters with autodiscovery<br /> </h1><p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes:</p><p><strong> node-1> ./ppserver -a<br /> </strong></p><p><strong> node-2> ./ppserver -a</strong></p><p><strong> node-3> ./ppserver -a<br /></strong></p><p><em><strong>On the client</strong></em></p><p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Set ppservers list to auto-discovery: </p><p><strong> ppservers=("*",)</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system and list of ppservers to connect with :</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>) </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="ADVANCEDCLUSTERS"> Advanced guide, clusters </h1> <p><em><strong>On the nodes</strong></em> </p><p>1) Start parallel python execution server on all your remote computational nodes (listen to a given port 35000,<br /> and local network interface only, accept only connections which know correct secret):</p><p><strong> node-1> ./ppserver -p 35000 -i 192.168.0.101 -s "mysecret"<br /></strong></p><p><strong> node-2> ./ppserver -p 35000 -i 192.168.0.102</strong><strong> -s "mysecret"</strong></p><p><strong> node-3> ./ppserver -p 35000 -i 192.168.0.103</strong><strong> -s "mysecret"</strong></p><p><em><strong>On the client</strong></em> <br /></p> <p>2) Import pp module:</p><p><strong> import pp</strong></p><p>3) Create a list of all the nodes in your cluster (computers where you've run ppserver) </p><p><strong> ppservers=("node-1:35000", "node-2:</strong><strong>35000</strong><strong>", "node-3:</strong><strong>35000</strong><strong>")</strong><br /></p><p>4) Start pp execution server with the number of workers set to the number of processors in the system, <br />list of ppservers to connect with and secret key to authorize the connection:</p><p><strong> job_server = pp.Server(</strong><strong>ppservers=</strong><strong>ppservers</strong><strong>, secret="</strong><strong>mysecret</strong><strong>") </strong></p><p>5) Submit all the tasks for parallel execution:</p><p><strong> f1 = job_server.submit(func1, args1, depfuncs1, modules1)</strong></p><p><strong> f2 = job_server.submit(func1, args2, depfuncs1, modules1) </strong></p><p><strong> f3 = job_server.submit(func2, args3, depfuncs2, modules2) </strong><br /> </p><p> ...etc...<br /></p><p>6) Retrieve the results as needed:</p><p><strong> r1 = f1()</strong></p><p><strong> r2 = f2()</strong></p><p><strong> r3 = f3() </strong> </p><p> ...etc...</p><p> 7) Print the execution statistics:<br /></p><p><strong> job_server.print_stats()</strong></p><p>To find out how to achieve efficient parallelization with pp please take a look at <a href="http://www.parallelpython.com/content/view/17/31/" title="Parallel Python Implementation Examples">examples</a> </p><hr /><h1 id="COMMANDLINE"> Command line options, ppserver</h1> <pre>Usage: ppserver [-hda] [-i interface] [-b broadcast] [-p port] [-w nworkers] [-s secret] [-t seconds]<br /> Options:<br /> -h : this help message<br /> -d : debug<br /> -a : enable auto-discovery service<br /> -i interface : interface to listen<br /> -b broadcast : broadcast address for auto-discovery service<br /> -p port : port to listen<br /> -w nworkers : number of workers to start<br /> -s secret : secret for authentication<br /> -t seconds : timeout to exit if no connections with clients exist<br /> </pre><hr /><h1 id="COMMANDLINE"> Security and secret key<a name="SECURITY" title="SECURITY"></a></h1><p> Due to the security concerns, in Debian we decided to disable default password authentication, and require to run ppserver with an non-trivial secret key (-s command line argument) which should be paired with the matching <em>secret</em> keyword of PP Server class constructor. An alternative way to set a secret key is by assigning <strong>pp_secret</strong> variable in the configuration file <strong>.pythonrc.py</strong> which should be located in the user home directory (please make this file readable and writable only by user). The secret key set in .pythonrc.py could be overridden by command line argument (for ppserver) and <em>secret</em> keyword (for PP Server class constructor). Note that passing the password on the command line allows every user to see it (e.g. using ps(1)), and that running it on an open/untrusted network can be a security problem as traffic, including the password, is not encrypted.</p>
</td>
</tr>
@@ -59,7 +59,7 @@
-Due to the security concerns it is highly recommended to run ppserver.py with an non-trivial secret key (-s command line argument) which should be paired with the matching secret keyword of PP Server class constructor. An alternative way to set a secret key is by assigning
+Due to the security concerns, in Debian we decided to disable default password authentication, and require to run
+.B ppserver
-+with an non-trivial secret key (-s command line argument) which should be paired with the matching secret keyword of PP Server class constructor. An alternative way to set a secret key is by assigning
++with an non-trivial secret key (-s command line argument) which should be paired with the matching secret keyword of PP Server class constructor. An alternative way to set a secret key is by assigning
.B pp_secret
variable in the configuration file
-.B .pythonrc.py
@@ -67,9 +67,9 @@
which should be located in the user home directory (please make this file readable and writable only by user).
-The secret key set in .pythonrc.py could be overridden by command line argument (for ppserver.py) and secret keyword (for PP Server class constructor).
+The secret key set in .pythonrc.py could be overridden by command line argument (for ppserver) and secret keyword (for PP Server class constructor).
-+Note that passing the password on the command line, allows every user to see it (using
-+.B ps
-+tool, for example), and that running it on an open/untrusted network can be a security problem as traffic, including the password, is not encrypted.
++Note that passing the password on the command line allows every user to see it (e.g. using
++.B ps(1)
++) and that running it on an open/untrusted network can be a security problem as traffic, including the password, is not encrypted.
.SH AUTHOR
This manual page was written by Sandro Tosi <matrixhasu at gmail.com>,
for the Debian project (but may be used by others).
More information about the Python-modules-commits
mailing list