[med-svn] [Git][med-team/python-ruffus][master] 6 commits: routine-update: New upstream version
Steffen Möller
gitlab at salsa.debian.org
Wed Jun 10 01:05:26 BST 2020
Steffen Möller pushed to branch master at Debian Med / python-ruffus
Commits:
1227e611 by Steffen Moeller at 2020-06-10T01:49:34+02:00
routine-update: New upstream version
- - - - -
33017e96 by Steffen Moeller at 2020-06-10T01:49:35+02:00
New upstream version 2.8.4
- - - - -
524f7b89 by Steffen Moeller at 2020-06-10T01:50:01+02:00
Update upstream source from tag 'upstream/2.8.4'
Update to upstream version '2.8.4'
with Debian dir 3d1e70f29798d47e8ffce04e2dbd05c2cd0ca6bb
- - - - -
86707c1a by Steffen Moeller at 2020-06-10T01:50:01+02:00
routine-update: Standards-Version: 4.5.0
- - - - -
8ec04769 by Steffen Moeller at 2020-06-10T01:50:01+02:00
routine-update: debhelper-compat 13
- - - - -
50e7a92a by Steffen Moeller at 2020-06-10T01:59:25+02:00
Upload.
- - - - -
22 changed files:
- .gitignore
- README.rst
- debian/changelog
- debian/control
- − debian/patches/privacy.patch
- debian/patches/series
- debian/rules
- doc/_templates/index.html
- doc/conf.py
- doc/contents.rst
- + doc/contributing.rst
- doc/global.inc
- doc/history.rst
- doc/installation.rst
- doc/release.rst
- ruffus/file_name_parameters.py
- ruffus/ruffus_utility.py
- ruffus/ruffus_version.py
- ruffus/task.py
- ruffus/test/Makefile
- ruffus/test/test_drmaa_wrapper_run_job_locally.py
- setup.py
Changes:
=====================================
.gitignore
=====================================
@@ -6,4 +6,10 @@ build
_build
doc/hosted_site.cmd
.ruffus_history.sqlite
-ruffus_development.*
\ No newline at end of file
+ruffus_development.*
+
+# Vim auxiliary files
+*~
+*.swp
+
+*.time
=====================================
README.rst
=====================================
@@ -1,3 +1,8 @@
+===========
+CGAT-ruffus
+===========
+
+
***************************************
Overview
***************************************
@@ -14,15 +19,20 @@ Each python function can be called in parallel to run multiple **jobs**.
Ruffus was originally designed for use in bioinformatics to analyse multiple genome
data sets.
+More recently, we have extended the functionality of CGAT-ruffus to incude cluster integration (Currently
+support SGE, SLURM and PBS-pro/Torque), paramaterisation, logging, database integration
+and conda environment switching. `CGAT-core <https://github.com/cgat-developers/cgat-core>`_ code and `documentation <https://cgat-core.readthedocs.io/en/latest/>`_.
+
***************************************
Documentation
***************************************
-Ruffus documentation can be found `here <http://www.ruffus.org.uk/>`_ ,
-with `installation notes <http://www.ruffus.org.uk/installation.html>`_ ,
-a `short tutorial <http://www.ruffus.org.uk/tutorials/simple_tutorial/simple_tutorial.html>`_ and
-an `in-depth manual <http://www.ruffus.org.uk/tutorials/manual/manual_introduction.html>`_ .
+Ruffus documentation can be found `here <https://cgat-ruffus.readthedocs.io/en/latest/>`_ ,
+with `installation notes <https://cgat-ruffus.readthedocs.io/en/latest/installation.html>`_ , and
+an `in-depth manual <https://cgat-ruffus.readthedocs.io/en/latest/tutorials/new_tutorial/manual_contents.html>`_ .
+However, to utilise the full power of this workflow management system we recomend
+using `CGAT-core <https://github.com/cgat-developers/cgat-core>`_ (`documentation <https://cgat-core.readthedocs.io/en/latest/>`_).
***************************************
Background
=====================================
debian/changelog
=====================================
@@ -1,3 +1,12 @@
+python-ruffus (2.8.4-1) unstable; urgency=medium
+
+ * Team upload.
+ * New upstream version
+ * Standards-Version: 4.5.0 (routine-update)
+ * debhelper-compat 13 (routine-update)
+
+ -- Steffen Moeller <moeller at debian.org> Wed, 10 Jun 2020 01:49:34 +0200
+
python-ruffus (2.8.1-6) unstable; urgency=medium
* Test-Depend: s/python-pytest/python3-pytest/
=====================================
debian/control
=====================================
@@ -4,7 +4,7 @@ Uploaders: Andreas Tille <tille at debian.org>
Section: python
Testsuite: autopkgtest-pkg-python
Priority: optional
-Build-Depends: debhelper-compat (= 12),
+Build-Depends: debhelper-compat (= 13),
dh-python,
python3-all,
python3-setuptools,
@@ -12,15 +12,14 @@ Build-Depends: debhelper-compat (= 12),
python3-sphinx-rtd-theme,
python3-pytest,
graphviz
-Standards-Version: 4.4.1
+Standards-Version: 4.5.0
Vcs-Browser: https://salsa.debian.org/med-team/python-ruffus
Vcs-Git: https://salsa.debian.org/med-team/python-ruffus.git
Homepage: http://www.ruffus.org.uk/
Package: python3-ruffus
Architecture: all
-Depends: ${shlibs:Depends},
- ${misc:Depends},
+Depends: ${misc:Depends},
${python3:Depends},
graphviz
Description: Python3 computation pipeline library widely used in bioinformatics
=====================================
debian/patches/privacy.patch deleted
=====================================
@@ -1,16 +0,0 @@
-Author: Andreas Tille <tille at debian.org>
-Last-Update: Wed, 11 Jul 2018 11:21:00 +0200
-Description: Avoid privacy issue
-
---- a/doc/_templates/index.html
-+++ b/doc/_templates/index.html
-@@ -153,8 +153,7 @@ The very latest (in development) code ca
- </li>
-
- <li>
-- <img src="https://groups.google.com/forum/my-groups-color.png"
-- height=30 alt="Google Groups"> <b>Subscribe to the <a href="https://groups.google.com/forum/#!forum/ruffus_discuss"><i>ruffus_discuss</i> mailing list </a></b>
-+ <b>Subscribe to the <a href="https://groups.google.com/forum/#!forum/ruffus_discuss"><i>ruffus_discuss</i> mailing list </a></b>
- <br>
- <form action="http://groups.google.com/group/ruffus_discuss/boxsubscribe">
- Email: <input type=text name=email>
=====================================
debian/patches/series
=====================================
@@ -1,4 +1,3 @@
use_libjs-mathjax.patch
#use_png_instead_of_jpg.patch
#fix_test.patch
-privacy.patch
=====================================
debian/rules
=====================================
@@ -54,6 +54,8 @@ override_dh_auto_clean:
doc/_build/doctrees \
ruffus/test/*.log \
.ruffus_history.sqlite
+ rm -f ruffus.egg-info/PKG-INFO ruffus.egg-info/SOURCES.txt ruffus.egg-info/dependency_links.txt ruffus.egg-info/top_level.txt
+
override_dh_install:
dh_install
=====================================
doc/_templates/index.html
=====================================
@@ -11,7 +11,7 @@
<meta content='index, follow' name='robots'/>
<meta content='en-uk' name='language'/>
<meta content='United Kingdom' name='country'/>
-<meta name='copyright' content='Leo Goodstadt'/>
+<meta name='copyright' content='cgat-developers'/>
{% endblock %}
@@ -120,62 +120,36 @@ Please cite <I>Ruffus</I> as:<br/>
-<h2>Get <I>Ruffus</I></h2>
+<h2>Install <I>Ruffus</I></h2>
<p>
- <i>Ruffus</i> is available as an <a
- href="http://peak.telecommunity.com/DevCenter/EasyInstall">easy-install</a>able package on the <a href="http://pypi.python.org/pypi/ruffus">Python Package
+ <i>Ruffus</i> is available as a <a
+ href="">conda package</a>able package on the <a href="http://pypi.python.org/pypi/ruffus">Python Package
Index</a>.
</p>
Just run:
-<div style="margin-left: 30px" class="highlight-python"><pre>sudo pip install ruffus --upgrade</pre> or
+<div style="margin-left: 30px" class="highlight-python"><pre>conda install -c bioconda ruffus</pre>
</div>
+or
-
-<div style="margin-left: 30px" class="highlight-python"><pre>easy_install -U ruffus</pre>
+<div style="margin-left: 30px" class="highlight-python"><pre>pip install ruffus</pre>
</div>
<br>
<p>
-The very latest (in development) code can be obtained via <a href="https://code.google.com/p/ruffus/source/checkout"> git </a>:
- <div style="color: #ff0000" >
- <pre>git clone https://bunbun68@code.google.com/p/ruffus/</pre>
+The very latest (in development) code can be obtained via <a href="https://github.com/cgat-developers/ruffus"> git </a>:
+<div style="margin-left: 30px" class="highlight-python"><pre>git clone https://github.com/cgat-developers/ruffus.git</pre>
</div>
</p>
<h1>Feedback and Getting Involved:</h1>
-<ul>
- <li>
- The <I>Ruffus</I> project is hosted with <b><a
- href="http://code.google.com/p/ruffus">Google
- Code here.</a></b><br><br>
- </li>
-
- <li>
- <img src="https://groups.google.com/forum/my-groups-color.png"
- height=30 alt="Google Groups"> <b>Subscribe to the <a href="https://groups.google.com/forum/#!forum/ruffus_discuss"><i>ruffus_discuss</i> mailing list </a></b>
- <br>
- <form action="http://groups.google.com/group/ruffus_discuss/boxsubscribe">
- Email: <input type=text name=email>
- <input type=submit name="sub" value="Subscribe">
- </form><br>
- </li>
-
-
- <li>
- <a href="http://groups.google.com/group/ruffus_discuss">Check out the mailing list without subscribing here.</a>
- <br>
- </li>
-
- <li>
- <a href="http://code.google.com/p/ruffus/issues/list">Bugs or feature requests can be posted here.</a>
- <br>
- </li>
-
-
-</ul>
+<p>We would love your input and involvment. If you find a bug or would like to request an improvement
+please raise a github issue. for more information on how to contribute code please see the
+ <a href="contributing.html"> contributing </a> section for more information on how to
+contribute to the project</p>
+
{% endblock %}
=====================================
doc/conf.py
=====================================
@@ -1,4 +1,4 @@
-# -*- coding: utf-8 -*-
+# -*- coding: utf-8 -*-
#
# ruffus documentation build configuration file, created by
# sphinx-quickstart on Tue Apr 21 16:55:56 2009.
@@ -10,10 +10,11 @@
#
# All configuration values have a default; values that are commented out
# serve to show the default.
+from __future__ import print_function
import sys, os
sys.path.insert(0, os.path.abspath(os.path.join("..")))
-print sys.path
+print(sys.path)
import ruffus, ruffus.task, ruffus.ruffus_version
@@ -52,7 +53,7 @@ master_doc = 'contents'
# General information about the project.
project = u'ruffus'
-copyright = u'2009-2015 Leo Goodstadt'
+copyright = u'2009-2019 cgat-developers'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -63,7 +64,7 @@ version=ruffus.ruffus_version.__version #major.minor[.patch[.sub]]
# The full version, without betaincluding alpha/beta/rc tags.
import re
release = re.subn("([\d\.]+).*", r"\1", version)[0]
-print version, release
+print(version, release)
# The language for content autogenerated by Sphinx. Refer to documentation
=====================================
doc/contents.rst
=====================================
@@ -12,14 +12,15 @@ Start Here:
:caption: Introduction
:name: introduction_toc
+ index.html
installation.rst
history.rst
+ tutorials/new_tutorial/manual_contents.rst
tutorials/new_syntax.rst
- Future plans <todo>
+ Future plans
faq.rst
gallery.rst
why_ruffus.rst
- tutorials/new_tutorial/manual_contents.rst
release.rst
@@ -61,11 +62,15 @@ Quick Reference:
print_dependencies.rst
adjacent_pairs_iterate.rst
+.. toctree::
+ :maxdepth: 1
+ :caption: Contributing
+ :name: contributing_toc
-=====================
-Indices and tables
-=====================
+ contributing.rst
+
+=======
+Indices
+=======
* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
=====================================
doc/contributing.rst
=====================================
@@ -0,0 +1,61 @@
+.. _contribute:
+
+============
+Contributing
+============
+
+Contributions are very much encouraged and we greatly appreciate the time and effort people make to help maintain and support out tools. Every contribution helps, please dont be shy, we dont bite.
+
+You can contribute to the development of our software in a number of different ways:
+
+-------------------
+Reporting bug fixes
+-------------------
+
+Bugs are annoying and reporting them will help us to fix your issue.
+
+Bugs can be reported using the issue section in `github <https://github.com/cgat-developers/ruffus/issues>`_
+
+When reporting issues, please include:
+
+- Steps in your code/command that led to the bug so it can be reproduced..
+- The error message from the log message.
+- Any other helpful info, such as the system/cluster engine or version information.
+
+-----------------------------------
+Proposing a new feature/enhancement
+-----------------------------------
+
+If you wish to contribute a new feature to the CGAT-ruffus repository then the best way is to raise this as an issue and label it as an enhancement in `github <https://github.com/cgat-developers/ruffus/issues>`_
+
+If you propose a new feature then please:
+
+- Explain how your enhancement will work
+- Describe as best as you can how you plan to implement this.
+- If you dont think you have the necessary skills to implement this on your own then please say and we will try our best to help (or implement this for you). However, please be aware that this is a community developed software and our volunteers have other jobs. Therefore, we may not be able to work as fast as you hoped.
+
+-----------------------
+Pull Request Guidelines
+-----------------------
+
+Why not contribute to our project, its a great way of making the project better, your help is always welcome. We follow the fork/pull request `model <https://guides.github.com/activities/forking>`_. To update our documentation, fix bugs or add extra enhancements you will need to create a pull request through github.
+
+To create a pull request perform these steps:
+
+1. Create a github account.
+2. Create a personal fork of the project on github.
+3. Clone the fork onto your local machine. Your remote repo on github is called ``origin``.
+4. Add the orginal repository as a remote called ``upstream``.
+5. If you made the fork a while ago then please make sure you ``git pull upstream`` to keep your repository up to date
+6. Create a new branch to work on! We usually name our branches with capital first and last followed by a dash and something unique. For example: ``git checkout -b AC-new_doc``.
+7. Impliment your fix/enhancement and make sure your code is effectively documented.
+8. Our code has tests and these will be ran when a pull request is submitted, however you can run our tests before you make the pull request, we have a number written in the ``ruffus/test/`` directory. To run all test run: `cd ruffus/test && /bin/bash run_all_unit_tests.cmd`.
+9. Add or change our documentation in the ``docs/`` directory.
+10. Squash all of your commits into a single commit with git `interactive rebase <https://help.github.com/articles/about-git-rebase/>`_.
+11. Push your branch to your fork on github ``git push origin``
+12. From your fork in github.com, open a pull request in the correct branch.
+13. ... This is where someone will review your changes and modify them or approve them ...
+14. Once the pull request is approved and merged you can pull the changes from the ``upstream`` to your local repo and delete your branch.
+
+.. note:: Always write your commit messages in the present tense. Your commit messages should describe what the commit does to the code and not what you did to the code.
+
=====================================
doc/global.inc
=====================================
@@ -6,8 +6,7 @@
:keywords: python, pipeline, computation, cluster, Ruffus, bioinformatics, parallel' name='keywords
:Author: Leo Goodstadt
:Publisher: Leo Goodstadt
- :Email: ruffus at llew.org.uk
:robots: index, follow
:language: en-uk
:country: United Kingdom
- :copyright: Leo Goodstadt 2009-2014
+ :copyright: cgat-developers 2009-2019
=====================================
doc/history.rst
=====================================
@@ -18,6 +18,20 @@ Major Features added to Ruffus
See :ref:`To do list <todo>` for future enhancements to Ruffus
+
+********************************************************************
+version 2.8.1
+********************************************************************
+
+* compatibility with gevent >= 1.2
+
+********************************************************************
+version 2.8.1
+********************************************************************
+
+* Ctrl-C will kill drmaa jobs
+* python3.7 compatibility, thanks to @jbarlow83, @QuLogic
+
********************************************************************
version 2.6.3
********************************************************************
=====================================
doc/installation.rst
=====================================
@@ -9,99 +9,45 @@ Installation
.. note ::
- Ruffus requires Python 2.6 or higher or Python 3.0 or higher
+ Ruffus requires Python 3.0 or higher, python 2 is no longer supported but older versions will still be python 2 compatible.
==============================================================================
-The easy way
+conda installation
==============================================================================
- *Ruffus* is available as an
- `easy-install <http://peak.telecommunity.com/DevCenter/EasyInstall>`_ -able package
- on the `Python Package Index <http://pypi.python.org/pypi/Sphinx>`_.
+The recomended method for installing CGAT-ruffus is to install using `conda <https://conda.io/en/latest/>`_ through
+the `bioconda <https://anaconda.org/bioconda/ruffus>`_ channel. The reson for conda being the prefferred method
+is that the dependancied are taken care of.
::
- sudo pip install ruffus --upgrade
-
- This may also work for older installations::
-
- easy_install -U ruffus
-
-
- See below if ``eady_install`` is missing
-
-
-==============================================================================
-The most up-to-date code:
-==============================================================================
- * `Download the latest sources <https://pypi.python.org/pypi/ruffus>`_ or
-
- * Check out the latest code from Google using git::
-
- git clone https://bunbun68@code.google.com/p/ruffus/ .
-
- * Bleeding edge Ruffus development takes place on github::
-
- git clone git at github.com:bunbun/ruffus.git .
-
-
- * To install after downloading, change to the , type::
-
- python ./setup.py install
-
-======================
-Prequisites
-======================
-==============================================================================
-Installing easy_install
-==============================================================================
-
- If your system doesn't have ``easy_install``, you can `install <ubuntu/linux mint>`__ one using a package manager, for example::
-
- # ubuntu/linux mint
- $ sudo apt-get install python-setuptools
- $ or sudo yum install python-setuptools
-
- or manually::
-
- sudo curl http://peak.telecommunity.com/dist/ez_setup.py | python
-
- or manually::
-
- wget peak.telecommunity.com/dist/ez_setup.py
- sudo python ez_setup.py
-
-
+ conda install -c bioconda ruffus
==============================================================================
-Installing pip
+pip installation
==============================================================================
- If Pip is missing::
+ *Ruffus* is also available on `pypi <https://pypi.org/project/ruffus/>`_.
- $ sudo easy_install -U pip
+ ::
+ pip install ruffus
==============================================================================
- Graphical flowcharts The most up-to-date code:
+Manual installation
==============================================================================
- **Ruffus** relies on the ``dot`` programme from `Graphviz <http://www.graphviz.org/>`_
- ("Graph visualisation") to make pretty flowchart representations of your pipelines in multiple
- graphical formats (e.g. ``png``, ``jpg``). The crossplatform Graphviz package can be
- `downloaded here <http://www.graphviz.org/Download.php>`_ for Windows,
+To obtain the latest code, check it out from `github <https://github.com/cgat-developers/ruffus>`_ and activate it.
- Linux, Macs and Solaris. For Fedora, try
- ::
+ ::
- yum list 'graphviz*'
+ git clone https://github.com/cgat-developers/ruffus.git
+ cd ruffus
+ python setup.py install
- For ubuntu / Debian, try
- ::
- sudo apt-get install graphviz
=====================================
doc/release.rst
=====================================
@@ -2,6 +2,18 @@
Release notes
*************
+=============
+Release 2.8.3
+=============
+
+* add missing import
+
+=============
+Release 2.8.2
+=============
+
+* implement retry behaviour for input file check.
+
=============
Release 2.8.1
=============
@@ -21,3 +33,23 @@ Release 2.8.0
* [#86] use pytest for testing
* python3.7 compatibility, thanks to @jbarlow83, @QuLogic
+
+=============
+Release 2.6.3
+=============
+
+* @transform works even when the ouput has more than one file.
+* @subdivide works in exactly the same way as @transform.
+* ruffus.drmaa_wrapper.run_job() works with python3 `(github) Fixed issue with byte and text streams.
+* ruffus.drmaa.wrapper.run_job() allows env (environment) to be set for jobs run locally as well as those on the cluster.
+* New object-orientated style syntax works seamlessly with Ruffus command line support.
+
+=============
+Release 2.6.2
+=============
+
+* pipeline_printout_graph()` incompatibility with python3 fixed.
+* checkpointing did not work correctly with @split and @subdivide.
+* @transoform has easier to understand syntax and takes care of most common use cases of ruffus (Thanks Milan Simonovic).
+* @transform takes an optional output_dir.
+* Decorators can take named parameters.
=====================================
ruffus/file_name_parameters.py
=====================================
@@ -2,7 +2,7 @@ from __future__ import print_function
import re
from . import dbdict
from .ruffus_utility import *
-from .ruffus_utility import shorten_filenames_encoder
+from .ruffus_utility import shorten_filenames_encoder, FILE_CHECK_RETRY, FILE_CHECK_SLEEP
from .ruffus_exceptions import *
################################################################################
@@ -55,6 +55,7 @@ from .ruffus_exceptions import *
# 88888888888888888888888888888888888888888888888888888888888888888888888888888888888888888
import os
import sys
+import time
import glob
from itertools import groupby
import itertools
@@ -354,39 +355,43 @@ def needs_update_check_directory_missing(*params, **kwargs):
return True, ": Directories %r is missing" % (missing_directories[0])
return False, "All directories exist"
-# _________________________________________________________________________________________
-# check_input_files_exist
+def check_input_files_exist(*params):
+ """If inputs are missing then there is no way a job can run
+ successful. Must throw exception.
-# _________________________________________________________________________________________
+ This extra function is a hack to make sure input files exists
+ right before job is called for better error messages, and to save
+ things from blowing up inside the task function.
+ In practice, we have observed a sporadic time-lag between a task
+ completing on a remote node and an output file appearing in the
+ expected location on the host running the ruffus pipeline. This
+ causes a subsequent task to fail with missing input file even
+ though the file will appear a few seconds later. It is not clear
+ if this is an issue of a poorly configured storage, but a retry
+ behaviour has been implemented to work around such issues.
-def check_input_files_exist(*params):
- """
- If inputs are missing then there is no way a job can run successful.
- Must throw exception.
- This extra function is a hack to make sure input files exists right before
- job is called for better error messages, and to save things from blowing
- up inside the task function
"""
if len(params):
input_files = params[0]
for f in get_strings_in_flattened_sequence(input_files):
- if not os.path.exists(f):
- if os.path.lexists(f):
- raise MissingInputFileError("No way to run job: " +
- "Input file '%s' is a broken symbolic link." % f)
- else:
- raise MissingInputFileError("No way to run job: " +
- "Input file '%s' does not exist" % f)
-
-
-# _________________________________________________________________________________________
+ tries = FILE_CHECK_RETRY
+ while tries > 0:
+ if not os.path.exists(f):
+ if os.path.lexists(f):
+ raise MissingInputFileError("No way to run job: " +
+ "Input file '%s' is a broken symbolic link." % f)
+ tries -= 1
+ time.sleep(FILE_CHECK_SLEEP)
+ continue
+ break
+ if tries <= 0:
+ raise MissingInputFileError("No way to run job: " +
+ "Input file '%s' does not exist" % f)
-# needs_update_check_exist
-# _________________________________________________________________________________________
def needs_update_check_exist(*params, **kwargs):
"""
Given input and output files, see if all exist
@@ -1108,7 +1113,17 @@ def yield_io_params_per_job(input_params,
else:
input_param = orig_input_param
- # extras
+ # extras - this statement applies transformations on the
+ # extra parameters. It traverses nested data
+ # structures. The transformation forces a copy of each
+ # parameter which causes memory duplication. The memory use
+ # can be significant if extras is large.
+ #
+ # For example, if extras contains a list of 10000 files,
+ # there will be one copy for each input parameter. If
+ # there are 1000 input files, this means there will be
+ # 1000 * 100000 filenames (instead of 1000 references to
+ # the same list containing 10000 strings).
extra_params = tuple(file_names_transform.substitute(
filenames, p) for p in extra_specs)
=====================================
ruffus/ruffus_utility.py
=====================================
@@ -105,6 +105,11 @@ CHECKSUM_FUNCTIONS_AND_PARAMS = 3
CHECKSUM_REGENERATE = 2 # regenerate checksums
+
+# number of times to check if an input file exists
+FILE_CHECK_RETRY = 5
+# number of seconds to sleep before retrying a file check
+FILE_CHECK_SLEEP = 10
# _________________________________________________________________________________________
# t_extra_inputs
@@ -873,20 +878,29 @@ SUFFIX_SUBSTITUTE = 1
#
-def regex_replace(filename, regex_str, compiled_regex, substitution_patterns, regex_or_suffix=REGEX_SUBSTITUTE):
- return apply_func_to_sequence(substitution_patterns, t_regex_replace(filename, regex_str, compiled_regex, regex_or_suffix))
+def regex_replace(filename, regex_str, compiled_regex, substitution_patterns,
+ regex_or_suffix=REGEX_SUBSTITUTE):
+ return apply_func_to_sequence(
+ substitution_patterns,
+ t_regex_replace(filename, regex_str, compiled_regex, regex_or_suffix))
def formatter_replace(filenames, regex_str, compiled_regex, substitution_patterns):
- return apply_func_to_sequence(substitution_patterns, t_formatter_replace(filenames, regex_str, compiled_regex))
+ return apply_func_to_sequence(
+ substitution_patterns,
+ t_formatter_replace(filenames, regex_str, compiled_regex))
def nested_formatter_replace(filenames, regex_strings, compiled_regexes, substitution_patterns):
- return apply_func_to_sequence(substitution_patterns, t_nested_formatter_replace(filenames, regex_strings, compiled_regexes))
+ return apply_func_to_sequence(
+ substitution_patterns,
+ t_nested_formatter_replace(filenames, regex_strings, compiled_regexes))
def nested_string_replace(prev_str, new_str, substitution_patterns):
- return apply_func_to_sequence(substitution_patterns, t_nested_string_replace(prev_str, new_str))
+ return apply_func_to_sequence(
+ substitution_patterns,
+ t_nested_string_replace(prev_str, new_str))
# _________________________________________________________________________________________
=====================================
ruffus/ruffus_version.py
=====================================
@@ -23,4 +23,4 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#################################################################################
-__version = '2.8.1'
+__version = '2.8.4'
=====================================
ruffus/task.py
=====================================
@@ -456,7 +456,8 @@ def generic_job_descriptor(unglobbed_params, verbose_abbreviated_path, runtime_d
def io_files_job_descriptor(unglobbed_params, verbose_abbreviated_path, runtime_data):
- extra_param = ", " + shorten_filenames_encoder(unglobbed_params[2:], verbose_abbreviated_path)[1:-1] \
+ extra_param = ", " + shorten_filenames_encoder(map(str, unglobbed_params[2:]),
+ verbose_abbreviated_path)[1:-1] \
if len(unglobbed_params) > 2 else ""
out_param = shorten_filenames_encoder(unglobbed_params[1], verbose_abbreviated_path) \
if len(unglobbed_params) > 1 else "??"
@@ -873,10 +874,11 @@ class Pipeline(dict):
if task_name not in self.task_names:
break
else:
- raise ruffus_exceptions.error_duplicate_task_name("The task string '%s' is ambiguous for "
- "Pipeline '%s'. You must disambiguate "
- "explicitly with different task names "
- % (task_str, self.name))
+ raise ruffus_exceptions.error_duplicate_task_name(
+ "The task string '{}' is ambiguous for "
+ "Pipeline '{}'. You must disambiguate "
+ "explicitly with different task names ".format(
+ task_str, self.name))
return Task(task_func, task_name, self)
#
@@ -2342,7 +2344,7 @@ class Task(node):
#
# rethrow exception after adding task name
#
- if exceptionType == error_task:
+ if exceptionType == ruffus_exceptions.error_task:
exceptionValue.specify
inst.specify_task(self, "Exceptions in dependency checking")
raise
@@ -5028,7 +5030,10 @@ def pipeline_run(target_tasks=[],
import gevent.event
import gevent.queue
import gevent.pool
- import gevent.signal
+ try:
+ from gevent.signal import signal as gevent_signal
+ except:
+ import gevent.signal as gevent_signal
try:
import gevent.lock as gevent_lock
except:
@@ -5038,9 +5043,9 @@ def pipeline_run(target_tasks=[],
pool_t = gevent.pool.Pool
pool = pool_t(parallelism)
queue_t = gevent.queue.Queue
- gevent.signal(signal.SIGINT, functools.partial(handle_sigint, pool=pool, pipeline=pipeline))
- gevent.signal(signal.SIGUSR1, functools.partial(handle_sigusr1, pool=pool, pipeline=pipeline))
- gevent.signal(signal.SIGUSR2, functools.partial(handle_sigusr2, pool=pool, pipeline=pipeline))
+ gevent_signal(signal.SIGINT, functools.partial(handle_sigint, pool=pool, pipeline=pipeline))
+ gevent_signal(signal.SIGUSR1, functools.partial(handle_sigusr1, pool=pool, pipeline=pipeline))
+ gevent_signal(signal.SIGUSR2, functools.partial(handle_sigusr2, pool=pool, pipeline=pipeline))
else:
raise ValueError("unknown pool manager '{}'".format(pool_manager))
=====================================
ruffus/test/Makefile
=====================================
@@ -17,7 +17,7 @@ NOTEST_RUN=$(NOTEST:%.py=%.run)
all: $(PYTEST_RUN) $(NOTEST_RUN)
%.pytest: %.py
- pytest $(PYTEST_OPTIONS) $<
+ /usr/bin/time --output=$*.time -v pytest $(PYTEST_OPTIONS) $<
%.run: %.py
- python $<
+ /usr/bin/time --output=$*.time -v python $<
=====================================
ruffus/test/test_drmaa_wrapper_run_job_locally.py
=====================================
@@ -114,14 +114,14 @@ class Test_ruffus(unittest.TestCase):
environ = {"RUFFUS_HEEHEE": "what?"}
home_dir = os.path.expanduser("~")
sys.stderr.write(" Run echoing to screen...\n")
- stdout, stderr = ruffus.drmaa_wrapper.run_job(cmd_str="python %s/slow_process_for_testing.py" % script_dir,
+ stdout, stderr = ruffus.drmaa_wrapper.run_job(cmd_str="%s %s/slow_process_for_testing.py" % (sys.executable, script_dir),
job_environment=environ,
working_directory=home_dir,
run_locally=True,
verbose=1,
local_echo=True)
sys.stderr.write(" Run silently...\n")
- stdout, stderr = ruffus.drmaa_wrapper.run_job(cmd_str="python %s/slow_process_for_testing.py" % script_dir,
+ stdout, stderr = ruffus.drmaa_wrapper.run_job(cmd_str="%s %s/slow_process_for_testing.py" % (sys.executable, script_dir),
job_environment=environ,
working_directory=home_dir,
run_locally=True,
=====================================
setup.py
=====================================
@@ -152,7 +152,7 @@ Usage
""",
url='http://www.ruffus.org.uk',
- download_url = "https://pypi.python.org/pypi/ruffus",
+ download_url = "https://github.com/cgat-developers/ruffus",
install_requires = module_dependencies, #['multiprocessing>=1.0', 'json' ], #, 'python>=2.5'],
setup_requires = module_dependencies, #['multiprocessing>=1.0', 'json'], #, 'python>=2.5'],
View it on GitLab: https://salsa.debian.org/med-team/python-ruffus/-/compare/aa0328fe78135954f0c81d6724ab696f51ce8db6...50e7a92a2cd70b0c7dca55fcfb58319c02911993
--
View it on GitLab: https://salsa.debian.org/med-team/python-ruffus/-/compare/aa0328fe78135954f0c81d6724ab696f51ce8db6...50e7a92a2cd70b0c7dca55fcfb58319c02911993
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20200610/9b02f96f/attachment-0001.html>
More information about the debian-med-commit
mailing list