[med-svn] [Git][med-team/snakemake][master] 4 commits: New upstream version 5.5.4
Michael R. Crusoe
gitlab at salsa.debian.org
Wed Jul 31 15:27:39 BST 2019
Michael R. Crusoe pushed to branch master at Debian Med / snakemake
Commits:
2ae04d6a by Michael R. Crusoe at 2019-07-31T09:22:33Z
New upstream version 5.5.4
- - - - -
83fa8e5e by Michael R. Crusoe at 2019-07-31T09:23:17Z
Update upstream source from tag 'upstream/5.5.4'
Update to upstream version '5.5.4'
with Debian dir 56097cffc5945c72dfe93da70a2c54ed76a3d502
- - - - -
776fde08 by Michael R. Crusoe at 2019-07-31T09:27:16Z
5.5.4-1
- - - - -
dbc52a27 by Michael R. Crusoe at 2019-07-31T14:26:06Z
run all of the tests
- - - - -
27 changed files:
- CHANGELOG.rst
- debian/README.source
- debian/changelog
- debian/control
- + debian/docs
- debian/patches/0012-reproducible-build.patch
- + debian/patches/0013-remove-duplicate-keyword-argument.patch
- + debian/patches/boto3_is_just_boto
- debian/patches/series
- debian/rules
- debian/source/lintian-overrides
- docs/getting_started/installation.rst
- docs/index.rst
- docs/tutorial/advanced.rst
- docs/tutorial/basics.rst
- environment.yml
- misc/vim/syntax/snakemake.vim
- setup.py
- snakemake/_version.py
- snakemake/logging.py
- snakemake/report/__init__.py
- snakemake/report/report.html
- snakemake/rules.py
- snakemake/script.py
- snakemake/workflow.py
- test-environment.yml
- tests/test_report/expected-results/report.html
Changes:
=====================================
CHANGELOG.rst
=====================================
@@ -1,3 +1,9 @@
+[5.5.4] - 2019-07-21
+====================
+Changed
+-------
+- Reports now automatically include workflow code and configuration for improved transparency.
+
[5.5.3] - 2019-07-11
====================
Changed
=====================================
debian/README.source
=====================================
@@ -13,3 +13,17 @@ with in 0011-fix-privacy-breach.patch. Here are the remaining issues:
should be packaged to solve the two remaining issues.
-- Andreas Tille <tille at debian.org> Tue, 12 Dec 2017 11:32:28 +0100
+
+Missing (optional) dependencies:
+
+- conda
+- gfal-copy and other gfal-* commands for gfal/gridftp (missing from gfal2?)
+
+Missing (optional) Python dependencies:
+
+- moto
+- google.cloud (google-cloud-sdk)
+- ftputil
+- pysftp
+- XRootD
+- aioeasywebdav
=====================================
debian/changelog
=====================================
@@ -1,9 +1,19 @@
-snakemake (5.5.3-2) UNRELEASED; urgency=medium
+snakemake (5.5.4-1) UNRELEASED; urgency=medium
+ [ Dylan Aïssi ]
* Apply patch from Chris Lamb to make the build reproducible
(Closes: #932116).
- -- Dylan Aïssi <daissi at debian.org> Tue, 16 Jul 2019 08:13:48 +0200
+ [ Michael R. Crusoe ]
+ * New upstream release
+ * debian/control: Add missing build-deps (as shown in errors during the
+ documenation building)
+ * debian/README.source: Document missing (optional?) dependencies
+ * debian/rules: Fix test invocation, as almost all tests were missed;
+ Document what is known about the skipped tests
+ * Apply patch from upstream to fix duplicated argument (Closes: #933527)
+
+ -- Michael R. Crusoe <michael.crusoe at gmail.com> Wed, 31 Jul 2019 11:24:24 +0200
snakemake (5.5.3-1) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -6,10 +6,15 @@ Priority: optional
Build-Depends: debhelper (>= 11~),
dh-python,
ca-certificates,
+ cwltool,
+ imagemagick,
python3,
python3-boto,
+ python3-botocore,
python3-configargparse,
python3-datrie,
+ python3-dropbox,
+ python3-flask,
python3-git,
python3-jsonschema,
python3-networkx,
@@ -29,6 +34,7 @@ Build-Depends: debhelper (>= 11~),
python3-wrapt,
python3-yaml,
r-cran-rmarkdown
+# python3-irodsclient, # when that enters testing
Standards-Version: 4.3.0
Vcs-Browser: https://salsa.debian.org/med-team/snakemake
Vcs-Git: https://salsa.debian.org/med-team/snakemake.git
@@ -54,8 +60,14 @@ Depends: ${misc:Depends},
python3-setuptools,
python3-wrapt,
python3-yaml
-Recommends: python3-boto,
- r-cran-rmarkdown
+Recommends: cwltool,
+ python3-boto,
+ python3-botocore,
+ python3-dropbox,
+ python3-flask,
+ r-cran-rmarkdown,
+ imagemagick
+# python3-irodsclient, when it enters testing
Description: pythonic workflow management system
Build systems like GNU Make are frequently used to create complicated
workflows, e.g. in bioinformatics. This project aims to reduce the
=====================================
debian/docs
=====================================
@@ -0,0 +1 @@
+tests/
=====================================
debian/patches/0012-reproducible-build.patch
=====================================
@@ -2,9 +2,9 @@ Description: Make the build reproducible
Author: Chris Lamb <lamby at debian.org>
Last-Update: 2019-07-15
---- snakemake-5.5.3.orig/snakemake/report/__init__.py
-+++ snakemake-5.5.3/snakemake/report/__init__.py
-@@ -98,10 +98,12 @@ def data_uri_from_file(file, defaultenc=
+--- snakemake.orig/snakemake/report/__init__.py
++++ snakemake/snakemake/report/__init__.py
+@@ -100,10 +100,12 @@
def report(text, path,
@@ -18,9 +18,9 @@ Last-Update: 2019-07-15
outmime, _ = mimetypes.guess_type(path)
if outmime != "text/html":
raise ValueError("Path to report output has to be an HTML file.")
---- snakemake-5.5.3.orig/snakemake/utils.py
-+++ snakemake-5.5.3/snakemake/utils.py
-@@ -189,7 +189,7 @@ def makedirs(dirnames):
+--- snakemake.orig/snakemake/utils.py
++++ snakemake/snakemake/utils.py
+@@ -189,7 +189,7 @@
def report(text, path,
@@ -29,7 +29,7 @@ Last-Update: 2019-07-15
defaultenc="utf8",
template=None,
metadata=None, **files):
-@@ -233,6 +233,8 @@ def report(text, path,
+@@ -233,6 +233,8 @@
metadata (str): E.g. an optional author name or email address.
"""
=====================================
debian/patches/0013-remove-duplicate-keyword-argument.patch
=====================================
@@ -0,0 +1,23 @@
+From 6d013348a3501b6c183438cfb44bf78704128925 Mon Sep 17 00:00:00 2001
+From: Alistair Miles <alimanfoo at googlemail.com>
+Date: Mon, 29 Jul 2019 14:53:04 +0000
+Subject: [PATCH] Merged in
+ alimanfoo/snakemake/Alistair-Miles/remove-duplicate-keyword-argument-1563308166092
+ (pull request #397)
+
+remove duplicate keyword argument
+---
+ snakemake/remote/gfal.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+--- snakemake.orig/snakemake/remote/gfal.py
++++ snakemake/snakemake/remote/gfal.py
+@@ -26,7 +26,7 @@
+ supports_default = True
+ allows_directories = True
+
+- def __init__(self, *args, keep_local=False, stay_on_remote=False, is_default=False, stay_on_remote=False, retry=5, **kwargs):
++ def __init__(self, *args, keep_local=False, stay_on_remote=False, is_default=False, retry=5, **kwargs):
+ super(RemoteProvider, self).__init__(*args, keep_local=keep_local, stay_on_remote=stay_on_remote, is_default=is_default, **kwargs)
+ self.retry = retry
+
=====================================
debian/patches/boto3_is_just_boto
=====================================
@@ -0,0 +1,35 @@
+Author: Michael R. Crusoe <michael.crusoe at gmail.com>
+Description: In Debian, boto3 is just boto
+--- snakemake.orig/snakemake/remote/S3.py
++++ snakemake/snakemake/remote/S3.py
+@@ -16,7 +16,7 @@
+
+ try:
+ # third-party modules
+- import boto3
++ import boto as boto3
+ import botocore
+ except ImportError as e:
+ raise WorkflowError("The Python 3 package 'boto3' "
+--- snakemake.orig/snakemake/remote/S3Mocked.py
++++ snakemake/snakemake/remote/S3Mocked.py
+@@ -19,7 +19,7 @@
+
+ try:
+ # third-party
+- import boto3
++ import boto as boto3
+ from moto import mock_s3
+ except ImportError as e:
+ raise WorkflowError("The Python 3 packages 'moto' and boto3' " +
+--- snakemake.orig/tests/test_static_remote/S3MockedForStaticTest.py
++++ snakemake/tests/test_static_remote/S3MockedForStaticTest.py
+@@ -20,7 +20,7 @@
+
+ try:
+ # third-party
+- import boto3
++ import boto as boto3
+ from moto import mock_s3
+ import filechunkio
+ except ImportError as e:
=====================================
debian/patches/series
=====================================
@@ -10,3 +10,5 @@
# 0010-skip-test-without-rmarkdown.patch
0011-fix-privacy-breach.patch
0012-reproducible-build.patch
+0013-remove-duplicate-keyword-argument.patch
+boto3_is_just_boto
=====================================
debian/rules
=====================================
@@ -6,6 +6,17 @@
export PYBUILD_NAME=snakemake
export PYBUILD_DESTDIR_python3=debian/snakemake
export PYBUILD_BEFORE_TEST_python3=chmod +x {dir}/bin/snakemake; cp -r {dir}/bin {dir}/tests {build_dir}
+export PYBUILD_TEST_ARGS=python{version} -m pytest tests/test*.py -v -k 'not report and not ancient and not test_script and not default_remote and not issue635 and not convert_to_cwl and not issue1083 and not issue1092 and not issue1093'
+
+# test_report
+# test_ancient
+# test_script requires conda; manually disabling conda show the need for the binary 'julia'
+# test_default_remote requires network access
+# test_issue634 requires conda, but passes when conda is turned off manually
+# test_convert_to_cwl tries to build a singularity format software image from docker://quay.io/snakemake/snakemake:v5.5.4
+# test_issue1083 tries to build a singularity format software image from docker://bash
+# test_issue1093 fails due to conda usage; commenting that out and installing bwa produces a different ordering than desired
+
export PYBUILD_AFTER_TEST_python3=rm -fr {build_dir}/bin {build_dir}/tests
export PATH:=$(shell pybuild --print build_dir --interpreter python3 --name $(PYBUILD_NAME))/bin:$(PATH)
@@ -21,3 +32,6 @@ override_dh_auto_build:
PATH=$(shell pybuild --print build_dir --interpreter python3)/bin:$(PATH) \
PYTHONPATH=$(shell pybuild --print build_dir --interpreter python3) \
python3 setup.py build_sphinx
+
+override_dh_auto_test:
+ PYBUILD_SYSTEM=custom dh_auto_test
=====================================
debian/source/lintian-overrides
=====================================
@@ -1,3 +1,2 @@
# This is just pure test data we need to compare with
-snakemake source: source-is-missing tests/test_report/report.html line length is *
-snakemake source: source-is-missing tests/test_report/expected-results/report.html line length is *
\ No newline at end of file
+snakemake source: source-is-missing tests/test_report/expected-results/report.html line length is *
=====================================
docs/getting_started/installation.rst
=====================================
@@ -1,4 +1,4 @@
-.. getting_started-installation:
+.. _getting_started-installation:
============
Installation
@@ -7,6 +7,8 @@ Installation
Snakemake is available on PyPi as well as through Bioconda and also from source code.
You can use one of the following ways for installing Snakemake.
+.. _conda-install:
+
Installation via Conda
======================
=====================================
docs/index.rst
=====================================
@@ -179,6 +179,7 @@ Please consider to add your own.
getting_started/installation
getting_started/examples
tutorial/tutorial
+ tutorial/short
.. toctree::
=====================================
docs/tutorial/advanced.rst
=====================================
@@ -203,7 +203,7 @@ We modify the rule ``bwa_map`` accordingly:
initialization to the DAG phase. In contrast to input functions, these can
optionally take additional arguments ``input``, ``output``, ``threads``, and ``resources``.
-Similar to input and output files, ``params`` can be accessed from the shell command or the Python based ``run`` block (see :ref:`tutorial-report`).
+Similar to input and output files, ``params`` can be accessed from the shell command the Python based ``run`` block, or the script directive (see :ref:`tutorial-script`).
Exercise
........
=====================================
docs/tutorial/basics.rst
=====================================
@@ -365,8 +365,10 @@ Exercise
:align: center
+.. _tutorial-script:
+
Step 6: Using custom scripts
-::::::::::::::::::::
+::::::::::::::::::::::::::::
Usually, a workflow not only consists of invoking various tools, but also contains custom code to e.g. calculate summary statistics or create plots.
While Snakemake also allows you to directly :ref:`write Python code inside a rule <.. _snakefiles-rules>`_, it is usually reasonable to move such logic into separate scripts.
=====================================
environment.yml
=====================================
@@ -31,3 +31,4 @@ dependencies:
- xorg-libxrender
- xorg-libxpm
- gitpython
+ - pygments
=====================================
misc/vim/syntax/snakemake.vim
=====================================
@@ -26,25 +26,29 @@ source $VIMRUNTIME/syntax/python.vim
" XXX N.B. several of the new defs are missing from this table i.e.
" subworkflow, touch etc
"
-" rule = "rule" (identifier | "") ":" ruleparams
-" include = "include:" stringliteral
-" workdir = "workdir:" stringliteral
-" ni = NEWLINE INDENT
-" ruleparams = [ni input] [ni output] [ni params] [ni message] [ni threads] [ni (run | shell)] NEWLINE snakemake
-" input = "input" ":" parameter_list
-" output = "output" ":" parameter_list
-" params = "params" ":" parameter_list
-" message = "message" ":" stringliteral
-" threads = "threads" ":" integer
-" resources = "resources" ":" parameter_list
-" version = "version" ":" statement
-" run = "run" ":" ni statement
-" shell = "shell" ":" stringliteral
+" rule = "rule" (identifier | "") ":" ruleparams
+" include = "include:" stringliteral
+" workdir = "workdir:" stringliteral
+" ni = NEWLINE INDENT
+" ruleparams = [ni input] [ni output] [ni params] [ni message] [ni threads] [ni (run | shell)] NEWLINE snakemake
+" input = "input" ":" parameter_list
+" output = "output" ":" parameter_list
+" params = "params" ":" parameter_list
+" message = "message" ":" stringliteral
+" threads = "threads" ":" integer
+" resources = "resources" ":" parameter_list
+" version = "version" ":" statement
+" run = "run" ":" ni statement
+" shell = "shell" ":" stringliteral
+" singularity = "singularity" ":" stringliteral
+" conda = "conda" ":" stringliteral
+" shadow = "shadow" ":" stringliteral
+
syn keyword pythonStatement include workdir onsuccess onerror
syn keyword pythonStatement ruleorder localrules configfile
-syn keyword pythonStatement touch protected temp wrapper
-syn keyword pythonStatement input output params message threads resources
+syn keyword pythonStatement touch protected temp wrapper conda shadow
+syn keyword pythonStatement input output params message threads resources singularity
syn keyword pythonStatement version run shell benchmark snakefile log script
syn keyword pythonStatement rule subworkflow nextgroup=pythonFunction skipwhite
=====================================
setup.py
=====================================
@@ -49,7 +49,7 @@ setup(
install_requires=['wrapt', 'requests', 'ratelimiter', 'pyyaml',
'configargparse', 'appdirs', 'datrie', 'jsonschema',
'docutils', 'gitpython'],
- extras_require={"reports": ['jinja2', 'networkx']},
+ extras_require={"reports": ['jinja2', 'networkx', 'pygments']},
classifiers=
["Development Status :: 5 - Production/Stable", "Environment :: Console",
"Intended Audience :: Science/Research",
=====================================
snakemake/_version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (HEAD -> master, tag: v5.5.3)"
- git_full = "66c61c139e1f439e3f78c09df97927ad54363cc3"
- git_date = "2019-07-11 18:27:18 +0200"
+ git_refnames = " (tag: v5.5.4)"
+ git_full = "a98c6317d6269c6b339bdca70521bbeb155ba90f"
+ git_date = "2019-07-21 09:16:33 +0200"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
=====================================
snakemake/logging.py
=====================================
@@ -344,7 +344,7 @@ class Logger:
def format_dict(dict, omit_keys=[], omit_values=[]):
- return ", ".join("{}={}".format(name, value)
+ return ", ".join("{}={}".format(name, str(value))
for name, value in dict.items()
if name not in omit_keys and value not in omit_values)
=====================================
snakemake/report/__init__.py
=====================================
@@ -25,6 +25,7 @@ from docutils.parsers.rst.directives.images import Image, Figure
from docutils.parsers.rst import directives
from docutils.core import publish_file, publish_parts
+from snakemake import script, wrapper
from snakemake.utils import format
from snakemake.logging import logger
from snakemake.io import is_flagged, get_flag_value
@@ -32,6 +33,7 @@ from snakemake.exceptions import WorkflowError
from snakemake.script import Snakemake
from snakemake import __version__
from snakemake.common import num_if_possible
+from snakemake import logging
class EmbeddedMixin(object):
@@ -183,9 +185,11 @@ class Category:
class RuleRecord:
+
def __init__(self, job, job_rec):
import yaml
self.name = job_rec.rule
+ self._rule = job.rule
self.singularity_img_url = job_rec.singularity_img_url
self.conda_env = None
self._conda_env_raw = None
@@ -196,6 +200,32 @@ class RuleRecord:
self.output = list(job_rec.output)
self.id = uuid.uuid4()
+ def code(self):
+ try:
+ from pygments.lexers import get_lexer_by_name
+ from pygments.formatters import HtmlFormatter
+ from pygments import highlight
+ except ImportError:
+ raise WorkflowError("Python package pygments must be installed to create reports.")
+ source, language = None, None
+ if self._rule.shellcmd is not None:
+ source = self._rule.shellcmd
+ language = "bash"
+ elif self._rule.script is not None:
+ logger.info("Loading script code for rule {}".format(self.name))
+ _, source, language = script.get_source(self._rule.script, self._rule.basedir)
+ source = source.decode()
+ elif self._rule.wrapper is not None:
+ logger.info("Loading wrapper code for rule {}".format(self.name))
+ _, source, language = script.get_source(wrapper.get_script(self._rule.wrapper, prefix=self._rule.workflow.wrapper_prefix))
+ source = source.decode()
+
+ try:
+ lexer = get_lexer_by_name(language)
+ return highlight(source, lexer, HtmlFormatter(linenos=True, cssclass="source", wrapcode=True))
+ except pygments.utils.ClassNotFound:
+ return "<pre><code>source</code></pre>"
+
def add(self, job_rec):
self.n_jobs += 1
self.output.extend(job_rec.output)
@@ -206,8 +236,27 @@ class RuleRecord:
self.singularity_img_url == other.singularity_img_url)
+class ConfigfileRecord:
+ def __init__(self, configfile):
+ self.name = configfile
+
+ def code(self):
+ try:
+ from pygments.lexers import get_lexer_by_name
+ from pygments.formatters import HtmlFormatter
+ from pygments import highlight
+ except ImportError:
+ raise WorkflowError("Python package pygments must be installed to create reports.")
+
+ language = "yaml" if self.name.endswith(".yaml") or self.name.endswith(".yml") else "json"
+ lexer = get_lexer_by_name(language)
+ with open(self.name) as f:
+ return highlight(f.read(), lexer, HtmlFormatter(linenos=True, cssclass="source", wrapcode=True))
+
+
class JobRecord:
def __init__(self):
+ self.job = None
self.rule = None
self.starttime = sys.maxsize
self.endtime = 0
@@ -226,6 +275,8 @@ class FileRecord:
self.mime, _ = mime_from_file(self.path)
self.id = uuid.uuid4()
self.job = job
+ self.wildcards = logging.format_wildcards(job.wildcards)
+ self.params = logging.format_dict(job.params)
self.png_uri = None
self.category = category
if self.is_img:
@@ -419,6 +470,7 @@ def auto_report(dag, path):
rule = meta["rule"]
rec = records[(job_hash, rule)]
rec.rule = rule
+ rec.job = job
rec.starttime = min(rec.starttime, meta["starttime"])
rec.endtime = max(rec.endtime, meta["endtime"])
rec.conda_env_file = None
@@ -446,7 +498,7 @@ def auto_report(dag, path):
# prepare per-rule information
rules = defaultdict(list)
for rec in records.values():
- rule = RuleRecord(job, rec)
+ rule = RuleRecord(rec.job, rec)
if rec.rule not in rules:
rules[rec.rule].append(rule)
else:
@@ -462,6 +514,8 @@ def auto_report(dag, path):
# rulegraph
rulegraph, xmax, ymax = rulegraph_d3_spec(dag)
+ # configfiles
+ configfiles = [ConfigfileRecord(f) for f in dag.workflow.configfiles]
seen = set()
files = [seen.add(res.target) or res for cat in results.values() for res in cat if res.target not in seen]
@@ -499,11 +553,17 @@ def auto_report(dag, path):
now = "{} {}".format(datetime.datetime.now().ctime(), time.tzname[0])
results_size = sum(res.size for cat in results.values() for res in cat)
+ try:
+ from pygments.formatters import HtmlFormatter
+ except ImportError:
+ raise WorkflowError("Python package pygments must be installed to create reports.")
+
# render HTML
template = env.get_template("report.html")
with open(path, "w", encoding="utf-8") as out:
out.write(template.render(results=results,
results_size=results_size,
+ configfiles=configfiles,
text=text,
rulegraph_nodes=rulegraph["nodes"],
rulegraph_links=rulegraph["links"],
@@ -513,5 +573,6 @@ def auto_report(dag, path):
timeline=timeline,
rules=[rec for recs in rules.values() for rec in recs],
version=__version__,
- now=now))
+ now=now,
+ pygments_css=HtmlFormatter(style="trac").get_style_defs('.source')))
logger.info("Report created.")
=====================================
snakemake/report/report.html
=====================================
@@ -11,7 +11,8 @@
<!-- Bootstrap CSS -->
<style>{{ "https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"|get_resource_as_string }}</style>
<style>{{ "https://cdnjs.cloudflare.com/ajax/libs/ekko-lightbox/5.3.0/ekko-lightbox.css"|get_resource_as_string }}</style>
- <style>{{ "https://cdn.datatables.net/v/bs4/dt-1.10.18/r-2.2.2/datatables.min.css"|get_resource_as_string }}</style>
+ <style>{{ "https://cdn.datatables.net/v/bs4/dt-1.10.18/r-2.2.2/sl-1.3.0/datatables.min.css"|get_resource_as_string }}</style>
+ <style>{{ pygments_css|safe }}</style>
<!-- Custom styles for this template -->
<style>
@@ -203,6 +204,37 @@
.navbar {
opacity: 0.8;
}
+
+ .source {
+ background: none !important;
+ }
+
+ table.dataTable tbody tr.selected,
+ table.dataTable tbody th.selected,
+ table.dataTable tbody td.selected {
+ color: black;
+ font-weight: bold;
+ }
+
+ table.dataTable tbody tr.selected td:first-of-type {
+ font-weight: bold;
+ }
+
+ table.dataTable tbody tr.selected td {
+ border-top: 1px solid #007bff;
+ border-bottom: 1px solid #007bff;
+ }
+
+ table.dataTable tbody > tr.selected,
+ table.dataTable tbody > tr > .selected {
+ background-color: transparent;
+ }
+
+ table.dataTable tbody tr.selected a,
+ table.dataTable tbody th.selected a,
+ table.dataTable tbody td.selected a {
+ color: #007bff;
+ }
</style>
</head>
@@ -213,7 +245,7 @@
<p id="info">Loading {{ results_size|filesizeformat }}. For large reports, this can take a while.</p>
</div>
- <nav class="navbar navbar-dark fixed-top bg-dark flex-md-nowrap p-0 shadow">
+ <nav class="navbar fixed-top navbar-dark bg-dark flex-md-nowrap p-0 shadow">
<a class="navbar-brand col-sm-3 col-md-2 mr-0" href="#">Snakemake Report</a>
<ul class="nav navbar-nav navbar-right" style="padding-right: 10px;">
<li class="text-white">{{ now }}</li>
@@ -229,7 +261,7 @@
<ul class="nav flex-column">
<li class="nav-item">
<a class="nav-link active" href="#">
- <span data-feather="home"></span>
+ <span data-feather="git-pull-request"></span>
Workflow <span class="sr-only">(current)</span>
</a>
</li>
@@ -239,6 +271,14 @@
Statistics
</a>
</li>
+ {% if configfiles %}
+ <li class="nav-item">
+ <a class="nav-link" href="#configuration">
+ <span data-feather="settings"></span>
+ Configuration
+ </a>
+ </li>
+ {% endif %}
<li class="nav-item">
<a class="nav-link" href="#rules">
<span data-feather="list"></span>
@@ -273,7 +313,11 @@
<p>Detailed software versions can be found under <a href="#rules">Rules</a>.</p>
- <div id="rulegraph"></div>
+ <div class="row justify-content-center">
+ <div class="col-xs-6">
+ <div id="rulegraph"></div>
+ </div>
+ </div>
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
<h2 id="results">Results</h2>
@@ -298,6 +342,7 @@
<th>File</th>
<th>Size</th>
<th>Description</th>
+ <th>Job properties</th>
<th></th>
</tr>
</thead>
@@ -309,6 +354,25 @@
</th>
<td style="white-space:nowrap;">{{ res.size|filesizeformat }}</td>
<td>{{ res.caption }}</td>
+ <td>
+ <table class="table table-sm table-borderless">
+ <tbody>
+ <tr>
+ <th>Rule</th><td><a class="rule" href="javascript:void(0)">{{ res.job.rule }}</a></td>
+ </tr>
+ {% if res.wildcards %}
+ <tr>
+ <th>Wildcards</th><td>{{ res.wildcards }}</td>
+ </tr>
+ {% endif %}
+ {% if res.params %}
+ <tr>
+ <th>Params</th><td>{{ res.params }}</td>
+ </tr>
+ {% endif %}
+ </tbody>
+ </table>
+ </td>
<td class="preview" id="{{ res.id }}-preview">
</td>
</tr>
@@ -325,8 +389,42 @@
<h2 id="stats">Statistics</h2>
</div>
If the workflow has been executed in cluster/cloud, runtimes include the waiting time in the queue.
- <div id="runtimes" class="plot"></div>
- <div id="timeline" class="plot"></div>
+ <div class="row justify-content-center">
+ <div class="col-xs-6">
+ <div id="runtimes" class="plot"></div>
+ </div>
+ <div class="col-xs-6">
+ <div id="timeline" class="plot"></div>
+ </div>
+ </div>
+
+ {% if configfiles %}
+ <div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
+ <h2 id="configuration">Configuration</h2>
+ </div>
+ <table class="table">
+ <thead>
+ <tr>
+ <th>File</th>
+ <th>Code</th>
+ </tr>
+ </thead>
+ <tbody>
+ {% for configfile in configfiles %}
+ <tr>
+ <td>{{ configfile.name }}</td>
+ <td>
+ <a data-toggle="collapse" role="button" href="#configfile-{{ configfile.id }}" aria-expanded="false" aria-controls="collapse-env"><span data-feather="plus-circle"></span></a>
+ <div class="collapse" id="configfile-{{ configfile.id }}">
+ {{ configfile.code()|safe }}
+ </div>
+ </td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+ {% endif %}
+
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
<h2 id="rules">Rules</h2>
</div>
@@ -338,6 +436,7 @@
<th>Output</th>
<th>Singularity</th>
<th>Conda environment</th>
+ <th>Code</th>
</tr>
</thead>
<tbody>
@@ -368,6 +467,14 @@
</div>
{% endif %}
</td>
+ <td>
+ {% if rule.code is not none %}
+ <a data-toggle="collapse" role="button" href="#code-{{ rule.id }}" aria-expanded="false" aria-controls="collapse-env"><span data-feather="plus-circle"></span></a>
+ <div class="collapse" id="code-{{ rule.id }}">
+ {{ rule.code()|safe }}
+ </div>
+ {% endif %}
+ </td>
</tr>
{% endfor %}
</tbody>
@@ -378,17 +485,18 @@
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
- <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.slim.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"|get_resource_as_string }}</script>
<script>{{ "https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js"|get_resource_as_string }}</script>
<script>{{ "https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.1.1/js/bootstrap.min.js"|get_resource_as_string }}</script>
<script>{{ "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js"|get_resource_as_string }}</script>
- <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega/3.3.1/vega.min.js"|get_resource_as_string }}</script>
- <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega-lite/2.4.3/vega-lite.min.js"|get_resource_as_string }}</script>
- <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega-embed/3.13.2/vega-embed.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega/5.4.0/vega.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega-lite/3.3.0/vega-lite.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/vega-embed/4.2.1/vega-embed.min.js"|get_resource_as_string }}</script>
<script>{{ "https://cdnjs.cloudflare.com/ajax/libs/feather-icons/4.7.3/feather.min.js"|get_resource_as_string }}</script>
<script>{{ "https://cdnjs.cloudflare.com/ajax/libs/ekko-lightbox/5.3.0/ekko-lightbox.min.js"|get_resource_as_string }}</script>
<script>{{ "https://raw.githubusercontent.com/eligrey/FileSaver.js/2.0.0/src/FileSaver.js"|get_resource_as_string }}</script>
- <script>{{ "https://cdn.datatables.net/v/bs4/dt-1.10.18/r-2.2.2/datatables.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdn.datatables.net/v/bs4/dt-1.10.18/r-2.2.2/sl-1.3.0/datatables.min.js"|get_resource_as_string }}</script>
+ <script>{{ "https://cdnjs.cloudflare.com/ajax/libs/jquery-scrollTo/2.1.2/jquery.scrollTo.min.js"|get_resource_as_string }}</script>
<!-- Icons -->
<script>
@@ -520,11 +628,7 @@
}
]
};
- var view = new vega.View(vega.parse(rulegraph_spec))
- .renderer("canvas")
- .initialize("#rulegraph")
- .hover()
- .run();
+ vegaEmbed("#rulegraph", rulegraph_spec);
var runtimes_spec = {
"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
@@ -605,7 +709,7 @@
});
$('.results-table').DataTable();
- $('#rules-table').DataTable({
+ var rule_table = $('#rules-table').DataTable({
paging: false
});
@@ -622,6 +726,25 @@
{% endfor %}
{% endfor %}
+ // jump to row upon clicking rule
+ $("a.rule").click(function() {
+ var rule = $(this).html();
+ var row = rule_table.row(function ( idx, data, node ) {
+ return data[0] == rule;
+ });
+ if(row.length > 0) {
+ rule_table.rows().deselect();
+ var node = row.select().draw().node();
+ $("body").scrollTo(node);
+ }
+ });
+
+ $.extend($.scrollTo.defaults, {
+ axis: 'y',
+ duration: 800,
+ offset: -90
+ });
+
$(document).ready(function() {
// Hide loading screen when document is ready.
setTimeout(function(){
=====================================
snakemake/rules.py
=====================================
@@ -68,6 +68,7 @@ class Rule:
self.is_branched = False
self.is_checkpoint = False
self.restart_times = 0
+ self.basedir = None
elif len(args) == 1:
other = args[0]
self.name = other.name
@@ -108,6 +109,7 @@ class Rule:
self.is_branched = True
self.is_checkpoint = other.is_checkpoint
self.restart_times = other.restart_times
+ self.basedir = other.basedir
def dynamic_branch(self, wildcards, input=True):
def get_io(rule):
=====================================
snakemake/script.py
=====================================
@@ -227,13 +227,8 @@ class Snakemake:
return lookup[(stdout, stderr, append)].format(self.log)
-def script(path, basedir, input, output, params, wildcards, threads, resources,
- log, config, rulename, conda_env, singularity_img, singularity_args,
- bench_record, jobid, bench_iteration, shadow_dir):
- """
- Load a script from the given basedir + path and execute it.
- Supports Python 3 and R.
- """
+def get_source(path, basedir=None):
+ source = None
if not path.startswith("http") and not path.startswith("git+file"):
if path.startswith("file://"):
path = path[7:]
@@ -244,225 +239,245 @@ def script(path, basedir, input, output, params, wildcards, threads, resources,
path = "file://" + path
path = format(path, stepout=1)
if path.startswith("file://"):
- sourceurl = "file:"+pathname2url(path[7:])
+ sourceurl = "file:" + pathname2url(path[7:])
elif path.startswith("git+file"):
+ source = git_content(path)
(root_path, file_path, version) = split_git_path(path)
- dir = ".snakemake/wrappers"
- os.makedirs(dir, exist_ok=True)
- new_path = os.path.join(dir, version + "-"+ "-".join(file_path.split("/")))
- with open(new_path,'w') as wrapper:
- wrapper.write(git_content(path))
- sourceurl = "file:" + new_path
- path = path.rstrip("@" + version)
+ path = path.rstrip("@" + version)
else:
sourceurl = path
+
+ language = None
+ if path.endswith(".py"):
+ language = "python"
+ elif path.endswith(".R"):
+ language = "r"
+ elif path.endswith(".Rmd"):
+ language = "rmarkdown"
+ elif path.endswith(".jl"):
+ language = "julia"
+
+ if source is None:
+ with urlopen(sourceurl) as source:
+ return path, source.read(), language
+ else:
+ return path, source, language
+
+
+def script(path, basedir, input, output, params, wildcards, threads, resources,
+ log, config, rulename, conda_env, singularity_img, singularity_args,
+ bench_record, jobid, bench_iteration, shadow_dir):
+ """
+ Load a script from the given basedir + path and execute it.
+ Supports Python 3 and R.
+ """
f = None
try:
- with urlopen(sourceurl) as source:
- if path.endswith(".py"):
- wrapper_path = path[7:] if path.startswith("file://") else path
- snakemake = Snakemake(input, output, params, wildcards,
- threads, resources, log, config, rulename,
- bench_iteration,
- os.path.dirname(wrapper_path))
- snakemake = pickle.dumps(snakemake)
- # Obtain search path for current snakemake module.
- # The module is needed for unpickling in the script.
- # We append it at the end (as a fallback).
- searchpath = SNAKEMAKE_SEARCHPATH
- if singularity_img is not None:
- searchpath = singularity.SNAKEMAKE_MOUNTPOINT
- searchpath = '"{}"'.format(searchpath)
- # For local scripts, add their location to the path in case they use path-based imports
- if path.startswith("file://"):
- searchpath += ', "{}"'.format(os.path.dirname(path[7:]))
- preamble = textwrap.dedent("""
- ######## Snakemake header ########
- import sys; sys.path.extend([{searchpath}]); import pickle; snakemake = pickle.loads({snakemake}); from snakemake.logging import logger; logger.printshellcmds = {printshellcmds}; __real_file__ = __file__; __file__ = {file_override};
- ######## Original script #########
- """).format(
- searchpath=escape_backslash(searchpath),
- snakemake=snakemake,
- printshellcmds=logger.printshellcmds,
- file_override=repr(os.path.realpath(wrapper_path)))
- elif path.endswith(".R") or path.endswith(".Rmd"):
- preamble = textwrap.dedent("""
- ######## Snakemake header ########
- library(methods)
- Snakemake <- setClass(
- "Snakemake",
- slots = c(
- input = "list",
- output = "list",
- params = "list",
- wildcards = "list",
- threads = "numeric",
- log = "list",
- resources = "list",
- config = "list",
- rule = "character",
- bench_iteration = "numeric",
- scriptdir = "character",
- source = "function"
- )
+ path, source, language = get_source(path, basedir)
+ if language == "python":
+ wrapper_path = path[7:] if path.startswith("file://") else path
+ snakemake = Snakemake(input, output, params, wildcards,
+ threads, resources, log, config, rulename,
+ bench_iteration,
+ os.path.dirname(wrapper_path))
+ snakemake = pickle.dumps(snakemake)
+ # Obtain search path for current snakemake module.
+ # The module is needed for unpickling in the script.
+ # We append it at the end (as a fallback).
+ searchpath = SNAKEMAKE_SEARCHPATH
+ if singularity_img is not None:
+ searchpath = singularity.SNAKEMAKE_MOUNTPOINT
+ searchpath = '"{}"'.format(searchpath)
+ # For local scripts, add their location to the path in case they use path-based imports
+ if path.startswith("file://"):
+ searchpath += ', "{}"'.format(os.path.dirname(path[7:]))
+ preamble = textwrap.dedent("""
+ ######## Snakemake header ########
+ import sys; sys.path.extend([{searchpath}]); import pickle; snakemake = pickle.loads({snakemake}); from snakemake.logging import logger; logger.printshellcmds = {printshellcmds}; __real_file__ = __file__; __file__ = {file_override};
+ ######## Original script #########
+ """).format(
+ searchpath=escape_backslash(searchpath),
+ snakemake=snakemake,
+ printshellcmds=logger.printshellcmds,
+ file_override=repr(os.path.realpath(wrapper_path)))
+ elif language == "r" or language == "rmarkdown":
+ preamble = textwrap.dedent("""
+ ######## Snakemake header ########
+ library(methods)
+ Snakemake <- setClass(
+ "Snakemake",
+ slots = c(
+ input = "list",
+ output = "list",
+ params = "list",
+ wildcards = "list",
+ threads = "numeric",
+ log = "list",
+ resources = "list",
+ config = "list",
+ rule = "character",
+ bench_iteration = "numeric",
+ scriptdir = "character",
+ source = "function"
)
- snakemake <- Snakemake(
- input = {},
- output = {},
- params = {},
- wildcards = {},
- threads = {},
- log = {},
- resources = {},
- config = {},
- rule = {},
- bench_iteration = {},
- scriptdir = {},
- source = function(...){{
- wd <- getwd()
- setwd(snakemake at scriptdir)
- source(...)
- setwd(wd)
- }}
+ )
+ snakemake <- Snakemake(
+ input = {},
+ output = {},
+ params = {},
+ wildcards = {},
+ threads = {},
+ log = {},
+ resources = {},
+ config = {},
+ rule = {},
+ bench_iteration = {},
+ scriptdir = {},
+ source = function(...){{
+ wd <- getwd()
+ setwd(snakemake at scriptdir)
+ source(...)
+ setwd(wd)
+ }}
+ )
+
+ ######## Original script #########
+ """).format(REncoder.encode_namedlist(input),
+ REncoder.encode_namedlist(output),
+ REncoder.encode_namedlist(params),
+ REncoder.encode_namedlist(wildcards),
+ threads,
+ REncoder.encode_namedlist(log),
+ REncoder.encode_namedlist({
+ name: value
+ for name, value in resources.items()
+ if name != "_cores" and name != "_nodes"
+ }), REncoder.encode_dict(config), REncoder.encode_value(rulename),
+ REncoder.encode_numeric(bench_iteration),
+ REncoder.encode_value(os.path.dirname(path[7:]) if path.startswith("file://") else os.path.dirname(path)))
+ elif language == "julia":
+ preamble = textwrap.dedent("""
+ ######## Snakemake header ########
+ struct Snakemake
+ input::Dict
+ output::Dict
+ params::Dict
+ wildcards::Dict
+ threads::Int64
+ log::Dict
+ resources::Dict
+ config::Dict
+ rule::String
+ bench_iteration
+ scriptdir::String
+ #source::Any
+ end
+ snakemake = Snakemake(
+ {}, #input::Dict
+ {}, #output::Dict
+ {}, #params::Dict
+ {}, #wildcards::Dict
+ {}, #threads::Int64
+ {}, #log::Dict
+ {}, #resources::Dict
+ {}, #config::Dict
+ {}, #rule::String
+ {}, #bench_iteration::Int64
+ {}, #scriptdir::String
+ #, #source::Any
+ )
+ ######## Original script #########
+ """.format(
+ JuliaEncoder.encode_namedlist(input),
+ JuliaEncoder.encode_namedlist(output),
+ JuliaEncoder.encode_namedlist(params),
+ JuliaEncoder.encode_namedlist(wildcards),
+ JuliaEncoder.encode_value(threads),
+ JuliaEncoder.encode_namedlist(log),
+ JuliaEncoder.encode_namedlist({
+ name: value
+ for name, value in resources.items()
+ if name != "_cores" and name != "_nodes"
+ }),
+ JuliaEncoder.encode_dict(config),
+ JuliaEncoder.encode_value(rulename),
+ JuliaEncoder.encode_value(bench_iteration),
+ JuliaEncoder.encode_value(os.path.dirname(path[7:]) if path.startswith("file://") else os.path.dirname(path))
+ ).replace("\'","\"")
)
+ else:
+ raise ValueError(
+ "Unsupported script: Expecting either Python (.py), R (.R), RMarkdown (.Rmd) or Julia (.jl) script.")
- ######## Original script #########
- """).format(REncoder.encode_namedlist(input),
- REncoder.encode_namedlist(output),
- REncoder.encode_namedlist(params),
- REncoder.encode_namedlist(wildcards),
- threads,
- REncoder.encode_namedlist(log),
- REncoder.encode_namedlist({
- name: value
- for name, value in resources.items()
- if name != "_cores" and name != "_nodes"
- }), REncoder.encode_dict(config), REncoder.encode_value(rulename),
- REncoder.encode_numeric(bench_iteration),
- REncoder.encode_value(os.path.dirname(path[7:]) if path.startswith("file://") else os.path.dirname(path)))
- elif path.endswith(".jl"):
- preamble = textwrap.dedent("""
- ######## Snakemake header ########
- struct Snakemake
- input::Dict
- output::Dict
- params::Dict
- wildcards::Dict
- threads::Int64
- log::Dict
- resources::Dict
- config::Dict
- rule::String
- bench_iteration
- scriptdir::String
- #source::Any
- end
- snakemake = Snakemake(
- {}, #input::Dict
- {}, #output::Dict
- {}, #params::Dict
- {}, #wildcards::Dict
- {}, #threads::Int64
- {}, #log::Dict
- {}, #resources::Dict
- {}, #config::Dict
- {}, #rule::String
- {}, #bench_iteration::Int64
- {}, #scriptdir::String
- #, #source::Any
- )
- ######## Original script #########
- """.format(
- JuliaEncoder.encode_namedlist(input),
- JuliaEncoder.encode_namedlist(output),
- JuliaEncoder.encode_namedlist(params),
- JuliaEncoder.encode_namedlist(wildcards),
- JuliaEncoder.encode_value(threads),
- JuliaEncoder.encode_namedlist(log),
- JuliaEncoder.encode_namedlist({
- name: value
- for name, value in resources.items()
- if name != "_cores" and name != "_nodes"
- }),
- JuliaEncoder.encode_dict(config),
- JuliaEncoder.encode_value(rulename),
- JuliaEncoder.encode_value(bench_iteration),
- JuliaEncoder.encode_value(os.path.dirname(path[7:]) if path.startswith("file://") else os.path.dirname(path))
- ).replace("\'","\"")
- )
+ dir = ".snakemake/scripts"
+ os.makedirs(dir, exist_ok=True)
+
+ with tempfile.NamedTemporaryFile(
+ suffix="." + os.path.basename(path),
+ dir=dir,
+ delete=False) as f:
+ if not language == "rmarkdown":
+ f.write(preamble.encode())
+ f.write(source)
else:
- raise ValueError(
- "Unsupported script: Expecting either Python (.py), R (.R), RMarkdown (.Rmd) or Julia (.jl) script.")
-
- dir = ".snakemake/scripts"
- os.makedirs(dir, exist_ok=True)
-
- with tempfile.NamedTemporaryFile(
- suffix="." + os.path.basename(path),
- dir=dir,
- delete=False) as f:
- if not path.endswith(".Rmd"):
- f.write(preamble.encode())
- f.write(source.read())
- else:
- # Insert Snakemake object after the RMarkdown header
- code = source.read().decode()
- pos = next(islice(re.finditer(r"---\n", code), 1, 2)).start() + 3
- f.write(str.encode(code[:pos]))
- preamble = textwrap.dedent("""
- ```{r, echo=FALSE, message=FALSE, warning=FALSE}
- %s
- ```
- """ % preamble)
- f.write(preamble.encode())
- f.write(str.encode(code[pos:]))
-
- if path.endswith(".py"):
- py_exec = sys.executable
- if conda_env is not None:
- py = os.path.join(conda_env, "bin", "python")
- if os.path.exists(py):
- out = subprocess.check_output([py, "--version"],
- stderr=subprocess.STDOUT,
- universal_newlines=True)
- ver = tuple(map(int, PY_VER_RE.match(out).group("ver_min").split(".")))
- if ver >= MIN_PY_VERSION:
- # Python version is new enough, make use of environment
- # to execute script
- py_exec = "python"
- else:
- logger.warning("Conda environment defines Python "
- "version < {0}.{1}. Using Python of the "
- "master process to execute "
- "script. Note that this cannot be avoided, "
- "because the script uses data structures from "
- "Snakemake which are Python >={0}.{1} "
- "only.".format(*MIN_PY_VERSION))
- if singularity_img is not None:
- # use python from image
- py_exec = "python"
- # use the same Python as the running process or the one from the environment
- shell("{py_exec} {f.name:q}", bench_record=bench_record)
- elif path.endswith(".R"):
- if conda_env is not None and "R_LIBS" in os.environ:
- logger.warning("R script job uses conda environment but "
- "R_LIBS environment variable is set. This "
- "is likely not intended, as R_LIBS can "
- "interfere with R packages deployed via "
- "conda. Consider running `unset R_LIBS` or "
- "remove it entirely before executing "
- "Snakemake.")
- shell("Rscript --vanilla {f.name:q}", bench_record=bench_record)
- elif path.endswith(".Rmd"):
- if len(output) != 1:
- raise WorkflowError("RMarkdown scripts (.Rmd) may only have a single output file.")
- out = os.path.abspath(output[0])
- shell("Rscript --vanilla -e 'rmarkdown::render(\"{f.name}\", output_file=\"{out}\", quiet=TRUE, knit_root_dir = \"{workdir}\", params = list(rmd=\"{f.name}\"))'",
- bench_record=bench_record,
- workdir=os.getcwd())
- elif path.endswith(".jl"):
- shell("julia {f.name:q}", bench_record=bench_record)
+ # Insert Snakemake object after the RMarkdown header
+ code = source.decode()
+ pos = next(islice(re.finditer(r"---\n", code), 1, 2)).start() + 3
+ f.write(str.encode(code[:pos]))
+ preamble = textwrap.dedent("""
+ ```{r, echo=FALSE, message=FALSE, warning=FALSE}
+ %s
+ ```
+ """ % preamble)
+ f.write(preamble.encode())
+ f.write(str.encode(code[pos:]))
+
+ if language == "python":
+ py_exec = sys.executable
+ if conda_env is not None:
+ py = os.path.join(conda_env, "bin", "python")
+ if os.path.exists(py):
+ out = subprocess.check_output([py, "--version"],
+ stderr=subprocess.STDOUT,
+ universal_newlines=True)
+ ver = tuple(map(int, PY_VER_RE.match(out).group("ver_min").split(".")))
+ if ver >= MIN_PY_VERSION:
+ # Python version is new enough, make use of environment
+ # to execute script
+ py_exec = "python"
+ else:
+ logger.warning("Conda environment defines Python "
+ "version < {0}.{1}. Using Python of the "
+ "master process to execute "
+ "script. Note that this cannot be avoided, "
+ "because the script uses data structures from "
+ "Snakemake which are Python >={0}.{1} "
+ "only.".format(*MIN_PY_VERSION))
+ if singularity_img is not None:
+ # use python from image
+ py_exec = "python"
+ # use the same Python as the running process or the one from the environment
+ shell("{py_exec} {f.name:q}", bench_record=bench_record)
+ elif language == "r":
+ if conda_env is not None and "R_LIBS" in os.environ:
+ logger.warning("R script job uses conda environment but "
+ "R_LIBS environment variable is set. This "
+ "is likely not intended, as R_LIBS can "
+ "interfere with R packages deployed via "
+ "conda. Consider running `unset R_LIBS` or "
+ "remove it entirely before executing "
+ "Snakemake.")
+ shell("Rscript --vanilla {f.name:q}", bench_record=bench_record)
+ elif language == "rmarkdown":
+ if len(output) != 1:
+ raise WorkflowError("RMarkdown scripts (.Rmd) may only have a single output file.")
+ out = os.path.abspath(output[0])
+ shell("Rscript --vanilla -e 'rmarkdown::render(\"{f.name}\", output_file=\"{out}\", quiet=TRUE, knit_root_dir = \"{workdir}\", params = list(rmd=\"{f.name}\"))'",
+ bench_record=bench_record,
+ workdir=os.getcwd())
+ elif language == "julia":
+ shell("julia {f.name:q}", bench_record=bench_record)
except URLError as e:
raise WorkflowError(e)
=====================================
snakemake/workflow.py
=====================================
@@ -896,6 +896,7 @@ class Workflow:
rule.wrapper = ruleinfo.wrapper
rule.cwl = ruleinfo.cwl
rule.restart_times=self.restart_times
+ rule.basedir = self.current_basedir
ruleinfo.func.__name__ = "__{}".format(rule.name)
self.globals[ruleinfo.func.__name__] = ruleinfo.func
=====================================
test-environment.yml
=====================================
@@ -19,6 +19,7 @@ dependencies:
- appdirs
- pytools
- docutils
+ - pygments
- pandoc <2.0 # pandoc has changed the CLI API so that it is no longer compatible with the version of r-markdown below
- xorg-libxrender
- xorg-libxext
=====================================
tests/test_report/expected-results/report.html
=====================================
The diff for this file was not included because it is too large.
View it on GitLab: https://salsa.debian.org/med-team/snakemake/compare/703c50a3f500e83b1253f79237bd607ee426127c...dbc52a27594fb0ae7f1cea2c7c37b62e2c2f54ce
--
View it on GitLab: https://salsa.debian.org/med-team/snakemake/compare/703c50a3f500e83b1253f79237bd607ee426127c...dbc52a27594fb0ae7f1cea2c7c37b62e2c2f54ce
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20190731/673a1226/attachment-0001.html>
More information about the debian-med-commit
mailing list