[med-svn] [snakemake] 01/03: New upstream version 4.3.1
Andreas Tille
tille at debian.org
Wed Dec 6 16:29:36 UTC 2017
This is an automated email from the git hooks/post-receive script.
tille pushed a commit to branch master
in repository snakemake.
commit 06c7c9eff5f247b90796b51e52fbfd2f1856d968
Author: Andreas Tille <tille at debian.org>
Date: Wed Dec 6 17:04:44 2017 +0100
New upstream version 4.3.1
---
CHANGELOG.md | 7 +++++++
docs/index.rst | 21 ++++++++++++++-------
docs/snakefiles/deployment.rst | 20 +++++++++++---------
docs/snakefiles/remote_files.rst | 13 +++++++++++++
snakemake/__init__.py | 11 ++++++++++-
snakemake/conda.py | 9 +++++++++
snakemake/dag.py | 5 +++--
snakemake/script.py | 7 +++++--
snakemake/version.py | 2 +-
snakemake/workflow.py | 11 ++++++++++-
snakemake/wrapper.py | 37 ++++++++++++++++++++++++++++++++-----
test-environment.yml | 4 ++--
tests/test_r_wrapper/Snakefile | 5 +++++
tests/test_r_wrapper/wrapper.R | 3 +++
14 files changed, 125 insertions(+), 30 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e096e4f..91b6ae1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
# Change Log
+## [4.3.1] - 2017-11-16
+### Added
+- List all conda environments with their location on disk via --list-conda-envs.
+### Changed
+- Do not clean up shadow on dry-run.
+- Allow R wrappers.
+
## [4.3.0] - 2017-10-27
### Added
- GridFTP remote provider. This is a specialization of the GFAL remote provider that uses globus-url-copy to download or upload files.
diff --git a/docs/index.rst b/docs/index.rst
index c073c18..2fbdec7 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,8 +1,8 @@
.. _manual-main:
-=====================================
-Welcome to Snakemake's documentation!
-=====================================
+=========
+Snakemake
+=========
.. image:: https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg
:target: https://bioconda.github.io/recipes/snakemake/README.html
@@ -22,8 +22,10 @@ Welcome to Snakemake's documentation!
.. image:: https://img.shields.io/badge/stack-overflow-orange.svg
:target: http://stackoverflow.com/questions/tagged/snakemake
+.. image:: https://img.shields.io/twitter/follow/johanneskoester.svg?style=social&label=Follow
+ :target: https://twitter.com/johanneskoester
-The Snakemake workflow management system is a tool to create reproducible and scalable data analyses.
+The Snakemake workflow management system is a tool to create **reproducible and scalable** data analyses.
Workflows are described via a human readable, Python based language.
They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow definition.
Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment.
@@ -77,8 +79,9 @@ To get started, consider the :ref:`tutorial`, the `introductory slides <http://s
Support
-------
+* First, check the :ref:`FAQ <project_info-faq>`.
* In case of questions, please post on `stack overflow <http://stackoverflow.com/questions/tagged/snakemake>`_.
-* To discuss with other Snakemake users, you can use the `mailing list <https://groups.google.com/forum/#!forum/snakemake>`_.
+* To discuss with other Snakemake users, you can use the `mailing list <https://groups.google.com/forum/#!forum/snakemake>`_. **Please do not post questions there. Use stack overflow for questions.**
* For bugs and feature requests, please use the `issue tracker <https://bitbucket.org/snakemake/snakemake/issues>`_.
* For contributions, visit Snakemake on `bitbucket <https://bitbucket.org/snakemake/snakemake>`_ and read the :ref:`guidelines <project_info-contributing>`.
@@ -95,15 +98,19 @@ Resources
---------
`Snakemake Wrappers Repository <https://snakemake-wrappers.readthedocs.org>`_
- The Snakemake Wrapper Repository is a collection of reusable wrappers that allow to quickly use popular command line tools from Snakemake rules and workflows.
+ The Snakemake Wrapper Repository is a collection of reusable wrappers that allow to quickly use popular tools from Snakemake rules and workflows.
`Snakemake Workflows Project <https://github.com/snakemake-workflows/docs>`_
This project provides a collection of high quality modularized and re-usable workflows.
The provided code should also serve as a best-practices of how to build production ready workflows with Snakemake.
Everybody is invited to contribute.
+`Snakemake Profiles Project <https://github.com/snakemake-profiles/doc`_
+ This project provides Snakemake configuration profiles for various execution environments.
+ Please consider contributing your own if it is still missing.
+
`Bioconda <https://bioconda.github.io/>`_
- Bioconda can be used from Snakemake for creating completely reproducible workflows by pin pointing the used software version and providing binaries.
+ Bioconda can be used from Snakemake for creating completely reproducible workflows by defining the used software versions and providing binaries.
.. project_info-publications_using:
diff --git a/docs/snakefiles/deployment.rst b/docs/snakefiles/deployment.rst
index c251b14..11d50ed 100644
--- a/docs/snakefiles/deployment.rst
+++ b/docs/snakefiles/deployment.rst
@@ -13,11 +13,11 @@ following structure:
├── README.md
├── LICENSE.md
├── config.yaml
- ├── environment.yaml
├── scripts
- │ ├── __init__.py
│ ├── script1.py
│ └── script2.R
+ ├── envs
+ │ └── myenv.yaml
└── Snakefile
Then, a workflow can be deployed to a new system via the following steps
@@ -31,16 +31,18 @@ Then, a workflow can be deployed to a new system via the following steps
# edit config and workflow as needed
vim config.yaml
- # install dependencies into isolated environment
- conda env create -n myworkflow --file environment.yaml
+ # execute workflow, deploy software dependencies via conda
+ snakemake -n --use-conda
- # activate environment
- source activate myworkflow
+Importantly, git branching and pull requests can be used to modify and possibly re-integrate workflows.
+A `cookiecutter <https://github.com/audreyr/cookiecutter>`_ template for creating this structure can be found `here <https://github.com/snakemake-workflows/cookiecutter-snakemake-workflow>`_.
+Given that cookiecutter is installed, you can use it via:
- # execute workflow
- snakemake -n
+.. code-block:: bash
-Importantly, git branching and pull requests can be used to modify and possibly re-integrate workflows.
+ cookiecutter gh:snakemake-workflows/cookiecutter-snakemake-workflow
+
+Visit the `Snakemake Workflows Project <https://github.com/snakemake-workflows/docs>`_ for best-practice workflows.
.. _integrated_package_management:
diff --git a/docs/snakefiles/remote_files.rst b/docs/snakefiles/remote_files.rst
index a7dbfd1..b45492c 100644
--- a/docs/snakefiles/remote_files.rst
+++ b/docs/snakefiles/remote_files.rst
@@ -110,6 +110,19 @@ The remote provider also supports a new ``glob_wildcards()`` (see :ref:`glob-wil
# (result looks just like as if the local glob_wildcards() function were used on a locally with a folder called "bucket-name")
+If the AWS CLI is installed it is possible to configure your keys globally. This removes the necessity of hardcoding the keys in the Snakefile. The interactive AWS credentials setup can be done using the following command:
+
+.. code-block:: python
+
+ aws configure
+
+S3 then can be used without the keys.
+
+.. code-block:: python
+
+ from snakemake.remote.S3 import RemoteProvider as S3RemoteProvider
+ S3 = S3RemoteProvider()
+
Google Cloud Storage (GS)
=========================
diff --git a/snakemake/__init__.py b/snakemake/__init__.py
index 12f7562..4c212a2 100644
--- a/snakemake/__init__.py
+++ b/snakemake/__init__.py
@@ -108,6 +108,7 @@ def snakemake(snakefile,
use_singularity=False,
singularity_args="",
conda_prefix=None,
+ list_conda_envs=False,
singularity_prefix=None,
create_envs_only=False,
mode=Mode.default,
@@ -199,7 +200,8 @@ def snakemake(snakefile,
singularity_args (str): additional arguments to pass to singularity
conda_prefix (str): the directory in which conda environments will be created (default None)
singularity_prefix (str): the directory to which singularity images will be pulled (default None)
- create_envs_only (bool): If specified, only builds the conda environments specified for each job, then exits.
+ create_envs_only (bool): If specified, only builds the conda environments specified for each job, then exits.
+ list_conda_envs (bool): List conda environments and their location on disk.
mode (snakemake.common.Mode): Execution mode
wrapper_prefix (str): Prefix for wrapper script URLs (default None)
kubernetes (str): Submit jobs to kubernetes, using the given namespace.
@@ -440,6 +442,7 @@ def snakemake(snakefile,
conda_prefix=conda_prefix,
singularity_prefix=singularity_prefix,
singularity_args=singularity_args,
+ list_conda_envs=list_conda_envs,
kubernetes=kubernetes,
kubernetes_envvars=kubernetes_envvars,
container_image=container_image,
@@ -490,6 +493,7 @@ def snakemake(snakefile,
list_code_changes=list_code_changes,
list_input_changes=list_input_changes,
list_params_changes=list_params_changes,
+ list_conda_envs=list_conda_envs,
summary=summary,
archive=archive,
latency_wait=latency_wait,
@@ -1198,6 +1202,10 @@ def get_argument_parser(profile=None):
help="If specified, only creates the job-specific "
"conda environments then exits. The `--use-conda` "
"flag must also be set.")
+ parser.add_argument("--list-conda-envs",
+ action="store_true",
+ help="List all conda environments and their location on "
+ "disk.")
parser.add_argument(
"--use-singularity",
action="store_true",
@@ -1452,6 +1460,7 @@ def main(argv=None):
force_use_threads=args.force_use_threads,
use_conda=args.use_conda,
conda_prefix=args.conda_prefix,
+ list_conda_envs=args.list_conda_envs,
use_singularity=args.use_singularity,
singularity_prefix=args.singularity_prefix,
singularity_args=args.singularity_args,
diff --git a/snakemake/conda.py b/snakemake/conda.py
index a839be8..8b8da5b 100644
--- a/snakemake/conda.py
+++ b/snakemake/conda.py
@@ -175,6 +175,15 @@ class Env:
return env_path
+ def __hash__(self):
+ # this hash is only for object comparison, not for env paths
+ return hash(self.file)
+
+ def __eq__(self, other):
+ if isinstance(other, Env):
+ return self.file == other.file
+ return False
+
def shellcmd(env_path):
return "source activate {};".format(env_path)
diff --git a/snakemake/dag.py b/snakemake/dag.py
index 50c5a95..e6b5e14 100644
--- a/snakemake/dag.py
+++ b/snakemake/dag.py
@@ -154,7 +154,7 @@ class DAG:
except KeyError:
pass
- def create_conda_envs(self, dryrun=False, forceall=False):
+ def create_conda_envs(self, dryrun=False, forceall=False, init_only=False):
conda.check_conda()
# First deduplicate based on job.conda_env_file
jobs = self.jobs if forceall else self.needrun_jobs
@@ -168,7 +168,8 @@ class DAG:
hash = env.hash
self.conda_envs[env_file] = env
if hash not in hash_set:
- env.create(dryrun)
+ if not init_only:
+ env.create(dryrun)
hash_set.add(hash)
def pull_singularity_imgs(self, dryrun=False, forceall=False):
diff --git a/snakemake/script.py b/snakemake/script.py
index c5f0502..30c7445 100644
--- a/snakemake/script.py
+++ b/snakemake/script.py
@@ -275,9 +275,12 @@ def script(path, basedir, input, output, params, wildcards, threads, resources,
py_exec = "python"
else:
logger.warning("Conda environment defines Python "
- "version < {}.{}. Using Python of the "
+ "version < {0}.{1}. Using Python of the "
"master process to execute "
- "script.".format(*MIN_PY_VERSION))
+ "script. Note that this cannot be avoided, "
+ "because the script uses data structures from "
+ "Snakemake which are Python >={0}.{1} "
+ "only.".format(*MIN_PY_VERSION))
if singularity_img is not None:
# use python from image
py_exec = "python"
diff --git a/snakemake/version.py b/snakemake/version.py
index 85f6349..2cfe0c2 100644
--- a/snakemake/version.py
+++ b/snakemake/version.py
@@ -1,3 +1,3 @@
-__version__ = "4.3.0"
+__version__ = "4.3.1"
MIN_PY_VERSION = (3, 5)
diff --git a/snakemake/workflow.py b/snakemake/workflow.py
index 0610942..605d8c0 100644
--- a/snakemake/workflow.py
+++ b/snakemake/workflow.py
@@ -256,6 +256,7 @@ class Workflow:
list_code_changes=False,
list_input_changes=False,
list_params_changes=False,
+ list_conda_envs=False,
summary=False,
archive=None,
detailed_summary=False,
@@ -500,8 +501,16 @@ class Workflow:
if items:
print(*items, sep="\n")
return True
+ elif list_conda_envs:
+ from snakemake.utils import simplify_path
+ dag.create_conda_envs(init_only=True, forceall=True)
+ print("environment", "location", sep="\t")
+ for env in set(job.conda_env for job in dag.jobs):
+ if env:
+ print(simplify_path(env.file), simplify_path(env.path), sep="\t")
+ return True
- if not keep_shadow:
+ if not keep_shadow and not dryrun:
self.persistence.cleanup_shadow()
if self.use_conda:
diff --git a/snakemake/wrapper.py b/snakemake/wrapper.py
index 8eff4d8..f838a50 100644
--- a/snakemake/wrapper.py
+++ b/snakemake/wrapper.py
@@ -7,8 +7,11 @@ __license__ = "MIT"
import os
import posixpath
-from snakemake.script import script
+from urllib.error import URLError
+from urllib.request import urlopen
+from snakemake.exceptions import WorkflowError
+from snakemake.script import script
def is_script(path):
return path.endswith("wrapper.py") or path.endswith("wrapper.R")
@@ -22,11 +25,35 @@ def get_path(path, prefix=None):
return path
+def is_local(path):
+ return path.startswith("file:")
+
+
+def find_extension(path, extensions=[".py", ".R", ".Rmd"]):
+ for ext in extensions:
+ if path.endswith("wrapper{}".format(ext)):
+ return path
+ for ext in extensions:
+ script = "/wrapper{}".format(ext)
+ if is_local(path):
+ if path.startswith("file://"):
+ p = path[7:]
+ elif path.startswith("file:"):
+ p = path[5:]
+ if os.path.exists(p + script):
+ return path + script
+ else:
+ try:
+ urlopen(path + script)
+ return path + script
+ except URLError:
+ continue
+ return path + "/wrapper.py" # default case
+
+
def get_script(path, prefix=None):
path = get_path(path, prefix=prefix)
- if not is_script(path):
- path += "/wrapper.py"
- return path
+ return find_extension(path)
def get_conda_env(path, prefix=None):
@@ -54,7 +81,7 @@ def wrapper(path,
prefix):
"""
Load a wrapper from https://bitbucket.org/snakemake/snakemake-wrappers under
- the given path + wrapper.py and execute it.
+ the given path + wrapper.(py|R|Rmd) and execute it.
"""
path = get_script(path, prefix=prefix)
script(path, "", input, output, params, wildcards, threads, resources,
diff --git a/test-environment.yml b/test-environment.yml
index 5f5f7d1..1cd3233 100644
--- a/test-environment.yml
+++ b/test-environment.yml
@@ -19,9 +19,9 @@ dependencies:
- appdirs
- pytools
- docutils
- - pandoc
+ - pandoc <2.0 # pandoc has changed the CLI API so that it is no longer compatible with the version of r-markdown below
- r-base >=3.4.1
- - r-rmarkdown
+ - r-rmarkdown =1.6
- xorg-libxrender
- xorg-libxext
- xorg-libxau
diff --git a/tests/test_r_wrapper/Snakefile b/tests/test_r_wrapper/Snakefile
new file mode 100644
index 0000000..1d9318d
--- /dev/null
+++ b/tests/test_r_wrapper/Snakefile
@@ -0,0 +1,5 @@
+rule r_wrapper:
+ output:
+ "deleteme"
+ wrapper:
+ "file://tests/test_r_wrapper"
diff --git a/tests/test_r_wrapper/wrapper.R b/tests/test_r_wrapper/wrapper.R
new file mode 100644
index 0000000..86c3425
--- /dev/null
+++ b/tests/test_r_wrapper/wrapper.R
@@ -0,0 +1,3 @@
+outfile = snakemake at output[[1]]
+x <- data.frame()
+write.table(x, file=outfile, col.names=FALSE)
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/snakemake.git
More information about the debian-med-commit
mailing list