[med-svn] [Git][med-team/snakemake][upstream] New upstream version 7.9.0
Andreas Tille (@tille)
gitlab at salsa.debian.org
Tue Jul 26 12:18:44 BST 2022
Andreas Tille pushed to branch upstream at Debian Med / snakemake
Commits:
da8fea00 by Andreas Tille at 2022-07-25T21:22:29+02:00
New upstream version 7.9.0
- - - - -
29 changed files:
- + .github/CODEOWNERS
- + .github/ISSUE_TEMPLATE/bug_report.md
- + .github/ISSUE_TEMPLATE/feature_request.md
- + .github/dependabot.yml
- + .github/pull_request_template.md
- + .github/workflows/conventional-prs.yml
- + .github/workflows/docker-publish.yml
- + .github/workflows/main.yml
- + .github/workflows/release-please.yml
- CHANGELOG.md
- docs/snakefiles/modularization.rst
- snakemake/__init__.py
- snakemake/_version.py
- snakemake/modules.py
- snakemake/parser.py
- snakemake/persistence.py
- snakemake/script.py
- snakemake/workflow.py
- + tests/test_conda_python_script/Snakefile
- + tests/test_conda_python_script/expected-results/version.txt
- + tests/test_conda_python_script/test_python_env.yaml
- + tests/test_conda_python_script/test_script.py
- + tests/test_modules_all_exclude/Snakefile
- + tests/test_modules_all_exclude/Snakefile_exclude
- + tests/test_modules_all_exclude/config/config.yaml
- + tests/test_modules_all_exclude/expected-results/results/testmodule/test.out
- + tests/test_modules_all_exclude/expected-results/results/testmodule/test2.out
- + tests/test_modules_all_exclude/module-test/Snakefile
- tests/tests.py
Changes:
=====================================
.github/CODEOWNERS
=====================================
@@ -0,0 +1 @@
+* @johanneskoester
=====================================
.github/ISSUE_TEMPLATE/bug_report.md
=====================================
@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+<!-- Please do not post usage questions here. Ask them on Stack Overflow: https://stackoverflow.com/questions/tagged/snakemake -->
+
+**Snakemake version**
+<!--Note the Snakemake version for which you experience the bug.
+Please only report bugs of the **latest stable release of Snakemake**.
+If possible please check whether the bug has been already fixed in the main branch.-->
+
+**Describe the bug**
+<!--A clear and concise description of what the bug is.-->
+
+**Logs**
+<!--If applicable, any terminal output to help explain your problem.-->
+
+**Minimal example**
+<!--Add a minimal example for reproducing the bug.-->
+
+**Additional context**
+<!--Add any other context about the problem here.-->
=====================================
.github/ISSUE_TEMPLATE/feature_request.md
=====================================
@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
=====================================
.github/dependabot.yml
=====================================
@@ -0,0 +1,6 @@
+version: 2
+updates:
+ - package-ecosystem: "github-actions"
+ directory: "/"
+ schedule:
+ interval: "weekly"
=====================================
.github/pull_request_template.md
=====================================
@@ -0,0 +1,9 @@
+### Description
+
+<!--Add a description of your PR here-->
+
+### QC
+<!-- Make sure that you can tick the boxes below. -->
+
+* [ ] The PR contains a test case for the changes or the changes are already covered by an existing test case.
+* [ ] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).
=====================================
.github/workflows/conventional-prs.yml
=====================================
@@ -0,0 +1,24 @@
+name: PR
+on:
+ pull_request_target:
+ types:
+ - opened
+ - reopened
+ - edited
+ - synchronize
+
+permissions:
+ contents: read
+
+jobs:
+ title-format:
+ permissions:
+ pull-requests: read # for amannn/action-semantic-pull-request to analyze PRs
+ statuses: write # for amannn/action-semantic-pull-request to mark status of analyzed PR
+ runs-on: ubuntu-latest
+ steps:
+ - uses: amannn/action-semantic-pull-request at v4.5.0
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ with:
+ validateSingleCommit: true
\ No newline at end of file
=====================================
.github/workflows/docker-publish.yml
=====================================
@@ -0,0 +1,18 @@
+name: Publish to Docker Hub
+
+on:
+ push:
+ branches:
+ - main
+
+jobs:
+ update:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout at master
+ - name: Publish to Registry
+ uses: elgohr/Publish-Docker-Github-Action at v4
+ with:
+ name: snakemake/snakemake
+ username: ${{ secrets.DOCKER_USERNAME }}
+ password: ${{ secrets.DOCKER_TOKEN }}
=====================================
.github/workflows/main.yml
=====================================
@@ -0,0 +1,242 @@
+name: CI
+
+on:
+ push:
+ branches:
+ - main
+ pull_request:
+ branches_ignore: []
+
+jobs:
+ cancel-previous:
+ runs-on: ubuntu-latest
+ if: github.ref != 'refs/heads/main'
+ steps:
+ - uses: khan/pull-request-workflow-cancel at 1.0.0
+ with:
+ workflows: "main.yml"
+ env:
+ GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}'
+
+ formatting:
+ permissions:
+ contents: read # for actions/checkout to fetch code
+ pull-requests: write # for marocchino/sticky-pull-request-comment to create or update PR comment
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout at v1
+
+ - name: Setup black environment
+ run: |
+ conda create -c conda-forge -y -q --name black black
+
+ - name: Check formatting
+ run: |
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate black
+ black --check --diff snakemake tests/tests.py tests/test_tes.py tests/test_io.py tests/common.py tests/test_google_lifesciences.py
+
+ - name: Comment PR
+ if: github.event_name == 'pull_request' && failure()
+ uses: marocchino/sticky-pull-request-comment at v2.2.0
+ with:
+ message: 'Please format your code with [black](https://black.readthedocs.io): `black snakemake tests/*.py`.'
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+ testing:
+ runs-on: ubuntu-latest
+ needs: formatting
+ env:
+ AWS_AVAILABLE: ${{ secrets.AWS_ACCESS_KEY_ID }}
+ GCP_AVAILABLE: ${{ secrets.GCP_SA_KEY }}
+ steps:
+ - uses: actions/checkout at v2
+ with:
+ fetch-depth: 0 # we need tags for versioneer to work
+
+ - name: Setup Snakemake environment
+ run: |
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ conda config --set channel_priority strict
+ conda install -c conda-forge -q mamba
+ # ensure that mamba is happy to write into the cache
+ sudo chown -R runner:docker /usr/share/miniconda/pkgs/cache
+ mamba env create -q --name snakemake --file test-environment.yml
+ # additionally add singularity
+ # TODO remove version constraint: needed because 3.8.7 fails with missing libz:
+ # bin/unsquashfs: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
+ mamba install -c conda-forge -n snakemake "singularity<=3.8.6"
+
+ - name: Setup apt dependencies
+ run: |
+ sudo apt install -y stress git wget
+
+ - name: Setup iRODS
+ run: |
+ docker build -t irods-server tests/test_remote_irods
+ docker run -d -p 1247:1247 --name provider irods-server -i run_irods
+ sleep 10
+ docker exec -u irods provider iput /incoming/infile
+ cp -r tests/test_remote_irods/setup-data ~/.irods
+
+ - name: Setup Gcloud
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud at v0.2.1
+ if: env.GCP_AVAILABLE
+ with:
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ export_default_credentials: true
+
+ - name: Setup AWS
+ uses: aws-actions/configure-aws-credentials at v1
+ if: env.AWS_AVAILABLE
+ with:
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+ aws-region: us-east-1
+
+ ###### slurm setup #####
+
+ - name: Download slurm ansible roles
+ run: |
+ ansible-galaxy install galaxyproject.slurm
+
+ - name: Define slurm playbook
+ uses: 1arp/create-a-file-action at 0.2
+ with:
+ file: slurm-playbook.yml
+ content: |
+ - name: Slurm all in One
+ hosts: localhost
+ vars:
+ slurm_roles: ['controller', 'exec']
+ roles:
+ - role: galaxyproject.slurm
+ become: true
+
+ - name: Setup slurm
+ run: |
+ ansible-playbook slurm-playbook.yml || (journalctl -xe && exit 1)
+
+ - name: Test slurm submission
+ run: |
+ srun echo "hello world"
+
+ ###### finalized slurm setup #####
+
+ - name: Run tests
+ env:
+ CI: true
+ ZENODO_SANDBOX_PAT: ${{ secrets.ZENODO_SANDBOX_PAT }}
+ run: |
+ # activate conda env
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate snakemake
+
+ pytest -v -x tests/test_expand.py tests/test_io.py tests/test_schema.py tests/test_linting.py tests/tests.py
+
+ - name: Build and publish docker image
+ if: "contains(github.event.pull_request.labels.*.name, 'update-container-image')"
+ uses: elgohr/Publish-Docker-Github-Action at v4
+ with:
+ name: snakemake/snakemake
+ username: ${{ secrets.DOCKER_USERNAME }}
+ password: ${{ secrets.DOCKER_TOKEN }}
+ tags: ${{ env.GITHUB_SHA }}
+
+ - name: Set container image
+ if: "contains(github.event.pull_request.labels.*.name, 'update-container-image')"
+ run: |
+ echo CONTAINER_IMAGE=snakemake/snakemake:$GITHUB_SHA >> $GITHUB_ENV
+
+ # TODO reactivate in April (we have no free resources left this month)
+ - name: Test Google Life Sciences Executor
+ if: env.GCP_AVAILABLE
+ run: |
+ # activate conda env
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate snakemake
+ pytest -s -v -x tests/test_google_lifesciences.py
+
+ # TODO reactivate in April (we have no free resources left this month)
+ - name: Test Kubernetes execution
+ if: env.GCP_AVAILABLE
+ env:
+ CI: true
+ run: |
+ # activate conda env
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate snakemake
+
+ pytest -s -v -x tests/test_kubernetes.py
+
+ # TODO temporarily deactivate and fix in separate PR.
+ - name: Test Tibanna (AWS) execution
+ if: env.AWS_AVAILABLE
+ env:
+ CI: true
+ run: |
+ # activate conda env
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate snakemake
+
+ # pytest -v -x -s tests/test_tibanna.py
+
+ - name: Test GA4GH TES executor
+ run: |
+ # activate conda env
+ export PATH="/usr/share/miniconda/bin:$PATH"
+ source activate snakemake
+ pytest -s -v -x tests/test_tes.py
+
+ - name: Delete container image
+ if: "contains(github.event.pull_request.labels.*.name, 'update-container-image') && always()"
+ run: |
+ docker run --rm lumir/remove-dockerhub-tag \
+ --user ${{ secrets.DOCKER_USERNAME }} \
+ --password ${{ secrets.DOCKER_TOKEN }} \
+ snakemake/snakemake:${{ env.GITHUB_SHA }}
+
+ - name: Build container image
+ if: "!contains(github.event.pull_request.labels.*.name, 'update-container-image')"
+ run: docker build .
+
+
+ testing-windows:
+ runs-on: windows-latest
+ needs: formatting
+
+ steps:
+ - uses: actions/checkout at v2
+ with:
+ fetch-depth: 0
+
+ - name: Remove unix-only dependencies
+ shell: python
+ run: |
+ import fileinput
+ excluded_on_win = ["environment-modules", "cwltool"]
+ for line in fileinput.input("test-environment.yml", inplace=True):
+ if all(pkg not in line for pkg in excluded_on_win):
+ print(line)
+
+ - name: Setup miniconda
+ uses: conda-incubator/setup-miniconda at v2
+ with:
+ activate-environment: snakemake
+ python-version: 3.9
+ channels: conda-forge, bioconda
+
+ - name: Setup Snakemake environment
+ run: |
+ conda config --set channel_priority strict
+ conda install -q --name base mamba
+ mamba env update -q --file test-environment.yml
+
+ - name: Run tests
+ env:
+ CI: true
+ ZENODO_SANDBOX_PAT: ${{ secrets.ZENODO_SANDBOX_PAT }}
+ run: |
+ python -m pytest -v -x tests/tests.py
=====================================
.github/workflows/release-please.yml
=====================================
@@ -0,0 +1,51 @@
+on:
+ push:
+ branches:
+ - main
+
+name: release-please
+
+jobs:
+ release-please:
+ runs-on: ubuntu-latest
+ steps:
+
+ - uses: GoogleCloudPlatform/release-please-action at v3
+ id: release
+ with:
+ release-type: python
+ package-name: snakemake
+
+ - uses: actions/checkout at v2
+ if: ${{ steps.release.outputs.release_created }}
+ with:
+ fetch-depth: 0
+
+ - name: Set up Python
+ if: ${{ steps.release.outputs.release_created }}
+ uses: actions/setup-python at v4
+ with:
+ python-version: '3.x'
+
+ - name: Build package
+ if: ${{ steps.release.outputs.release_created }}
+ run: |
+ python -m pip install --upgrade pip
+ pip install build
+ python -m build
+
+ - name: Publish to PyPI
+ if: ${{ steps.release.outputs.release_created }}
+ uses: pypa/gh-action-pypi-publish at master
+ with:
+ user: __token__
+ password: ${{ secrets.PYPI_TOKEN }}
+
+ - name: Publish to Registry
+ if: ${{ steps.release.outputs.release_created }}
+ uses: elgohr/Publish-Docker-Github-Action at v4
+ with:
+ name: snakemake/snakemake
+ username: ${{ secrets.DOCKER_USERNAME }}
+ password: ${{ secrets.DOCKER_TOKEN }}
+ tags: "v${{ steps.release.outputs.major }}.${{ steps.release.outputs.minor }}.${{ steps.release.outputs.patch }},stable"
=====================================
CHANGELOG.md
=====================================
@@ -1,5 +1,20 @@
# Changelog
+## [7.9.0](https://github.com/snakemake/snakemake/compare/v7.8.5...v7.9.0) (2022-07-19)
+
+
+### Features
+
+* make it possible to exclude rules that will be imported when using 'use rule' statement ([#1717](https://github.com/snakemake/snakemake/issues/1717)) ([d9e0611](https://github.com/snakemake/snakemake/commit/d9e061178bd22307cc710bea28a5994e866260d9))
+
+
+### Bug Fixes
+
+* add lock free mechanism for avoiding race conditions when writing persistence information; consider corrupt metadata records as non-existent ([#1745](https://github.com/snakemake/snakemake/issues/1745)) ([71fe952](https://github.com/snakemake/snakemake/commit/71fe9527bb7011ba01d25fdd21c102c135412c04))
+* conda python interpreter path on Windows ([#1711](https://github.com/snakemake/snakemake/issues/1711)) ([155c9d6](https://github.com/snakemake/snakemake/commit/155c9d6688a99db4b49c5b25d5a8a65a3aca532d))
+* ensures that REncoder also checks for numpy.bool_ in encode_value ([#1749](https://github.com/snakemake/snakemake/issues/1749)) ([10a6e1d](https://github.com/snakemake/snakemake/commit/10a6e1de50ea7957e8685ba55b2cc115101cc23f))
+* Move quiet default after profile parsing ([#1764](https://github.com/snakemake/snakemake/issues/1764)) ([6ade76d](https://github.com/snakemake/snakemake/commit/6ade76d1287c8f62056853491a4e67b08a4739a6))
+
## [7.8.5](https://github.com/snakemake/snakemake/compare/v7.8.4...v7.8.5) (2022-06-30)
=====================================
docs/snakefiles/modularization.rst
=====================================
@@ -130,11 +130,11 @@ With Snakemake 6.0 and later, it is possible to define external workflows as mod
# here, plain paths, URLs and the special markers for code hosting providers (see below) are possible.
"other_workflow/Snakefile"
- use rule * from other_workflow as other_*
+ use rule * from other_workflow exclude ruleC as other_*
The ``module other_workflow:`` statement registers the external workflow as a module, by defining the path to the main snakefile of ``other_workflow``.
Here, plain paths, HTTP/HTTPS URLs and special markers for code hosting providers like Github or Gitlab are possible (see :ref:`snakefile-code-hosting-providers`).
-The second statement, ``use rule * from other_workflow as other_*``, declares all rules of that module to be used in the current one.
+The second statement, ``use rule * from other_workflow exclude ruleC as other_*``, declares all rules of that module to be used in the current one, except for ruleC.
Thereby, the ``as other_*`` at the end renames all those rules with a common prefix.
This can be handy to avoid rule name conflicts (note that rules from modules can otherwise overwrite rules from your current workflow or other modules).
=====================================
snakemake/__init__.py
=====================================
@@ -2444,10 +2444,6 @@ def main(argv=None):
parser = get_argument_parser()
args = parser.parse_args(argv)
- if args.quiet is not None and len(args.quiet) == 0:
- # default case, set quiet to progress and rule
- args.quiet = ["progress", "rules"]
-
if args.profile:
# reparse args while inferring config file from profile
parser = get_argument_parser(args.profile)
@@ -2478,6 +2474,10 @@ def main(argv=None):
if args.report_stylesheet:
args.report_stylesheet = adjust_path(args.report_stylesheet)
+ if args.quiet is not None and len(args.quiet) == 0:
+ # default case, set quiet to progress and rule
+ args.quiet = ["progress", "rules"]
+
if args.bash_completion:
cmd = b"complete -o bashdefault -C snakemake-bash-completion snakemake"
sys.stdout.buffer.write(cmd)
=====================================
snakemake/_version.py
=====================================
@@ -22,9 +22,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (HEAD -> main, tag: v7.8.5)"
- git_full = "531ef4a6b962b06ef865ba34ea6e952a7d4b3a2b"
- git_date = "2022-06-30 15:19:34 +0200"
+ git_refnames = " (tag: v7.9.0)"
+ git_full = "9da571f29c3e4b2c77f8465edcb7e21c91f5feb7"
+ git_date = "2022-07-19 15:03:44 +0200"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
=====================================
snakemake/modules.py
=====================================
@@ -73,6 +73,7 @@ class ModuleInfo:
self,
rules=None,
name_modifier=None,
+ exclude_rules=None,
ruleinfo=None,
skip_global_report_caption=False,
):
@@ -84,6 +85,7 @@ class ModuleInfo:
skip_configfile=self.config is not None,
skip_validation=self.skip_validation,
skip_global_report_caption=skip_global_report_caption,
+ rule_exclude_list=exclude_rules,
rule_whitelist=self.get_rule_whitelist(rules),
rulename_modifier=get_name_modifier_func(rules, name_modifier),
ruleinfo_overwrite=ruleinfo,
@@ -140,6 +142,7 @@ class WorkflowModifier:
skip_global_report_caption=False,
rulename_modifier=None,
rule_whitelist=None,
+ rule_exclude_list=None,
ruleinfo_overwrite=None,
allow_rule_overwrite=False,
replace_prefix=None,
@@ -156,6 +159,7 @@ class WorkflowModifier:
self.skip_validation = parent_modifier.skip_validation
self.skip_global_report_caption = parent_modifier.skip_global_report_caption
self.rule_whitelist = parent_modifier.rule_whitelist
+ self.rule_exclude_list = parent_modifier.rule_exclude_list
self.ruleinfo_overwrite = parent_modifier.ruleinfo_overwrite
self.allow_rule_overwrite = parent_modifier.allow_rule_overwrite
self.path_modifier = parent_modifier.path_modifier
@@ -178,6 +182,7 @@ class WorkflowModifier:
self.skip_validation = skip_validation
self.skip_global_report_caption = skip_global_report_caption
self.rule_whitelist = rule_whitelist
+ self.rule_exclude_list = rule_exclude_list
self.ruleinfo_overwrite = ruleinfo_overwrite
self.allow_rule_overwrite = allow_rule_overwrite
self.path_modifier = PathModifier(replace_prefix, prefix, workflow)
@@ -185,7 +190,9 @@ class WorkflowModifier:
self.namespace = namespace
def skip_rule(self, rulename):
- return self.rule_whitelist is not None and rulename not in self.rule_whitelist
+ return (
+ self.rule_whitelist is not None and rulename not in self.rule_whitelist
+ ) or (self.rule_exclude_list is not None and rulename in self.rule_exclude_list)
def modify_rulename(self, rulename):
if self.rulename_modifier is not None:
=====================================
snakemake/parser.py
=====================================
@@ -947,6 +947,7 @@ class UseRule(GlobalKeywordState):
super().__init__(snakefile, base_indent=base_indent, dedent=dedent, root=root)
self.state = self.state_keyword_rule
self.rules = []
+ self.exclude_rules = []
self.has_with = False
self.name_modifier = []
self.from_module = None
@@ -955,8 +956,8 @@ class UseRule(GlobalKeywordState):
def end(self):
name_modifier = "".join(self.name_modifier) if self.name_modifier else None
- yield "@workflow.userule(rules={!r}, from_module={!r}, name_modifier={!r}, lineno={})".format(
- self.rules, self.from_module, name_modifier, self.lineno
+ yield "@workflow.userule(rules={!r}, from_module={!r}, exclude_rules={!r}, name_modifier={!r}, lineno={})".format(
+ self.rules, self.from_module, self.exclude_rules, name_modifier, self.lineno
)
yield "\n"
@@ -1060,6 +1061,9 @@ class UseRule(GlobalKeywordState):
if token.string == "as" and not self.name_modifier:
self.state = self.state_as
yield from ()
+ elif token.string == "exclude":
+ self.state = self.state_exclude
+ yield from ()
elif token.string == "with":
yield from self.handle_with(token)
else:
@@ -1111,9 +1115,43 @@ class UseRule(GlobalKeywordState):
yield from ()
else:
self.error(
- "Expecting colon after 'with' keyword in 'use rule' statement.", token
+ "Expecting colon after 'with' keyword in 'use rule' statement.",
+ token,
+ )
+
+ def state_exclude(self, token):
+ if is_name(token):
+ self.exclude_rules.append(token.string)
+ self.state = self.state_exclude_comma_or_end
+ yield from ()
+ else:
+ self.error(
+ "Expecting rule name(s) after 'exclude' keyword in 'use rule' statement.",
+ token,
)
+ def state_exclude_comma_or_end(self, token):
+ if is_name(token):
+ if token.string == "from" or token.string == "as":
+ if not self.exclude_rules:
+ self.error(
+ "Expecting rule names after 'exclude' statement.",
+ token,
+ )
+ if token.string == "from":
+ self.state = self.state_from
+ else:
+ self.state = self.state_as
+ yield from ()
+ else:
+ yield from ()
+ elif is_comma(token):
+ self.state = self.state_exclude
+ yield from ()
+ else:
+ self.state = self.state_modifier
+ yield from ()
+
def block_content(self, token):
if is_comment(token):
yield "\n", token
=====================================
snakemake/persistence.py
=====================================
@@ -9,6 +9,7 @@ import signal
import marshal
import pickle
import json
+import tempfile
import time
from base64 import urlsafe_b64encode, b64encode
from functools import lru_cache, partial
@@ -434,9 +435,20 @@ class Persistence:
def _record(self, subject, json_value, id):
recpath = self._record_path(subject, id)
- os.makedirs(os.path.dirname(recpath), exist_ok=True)
- with open(recpath, "w") as f:
- json.dump(json_value, f)
+ recdir = os.path.dirname(recpath)
+ os.makedirs(recdir, exist_ok=True)
+ # Write content to temporary file and rename it to the final file.
+ # This avoids race-conditions while writing (e.g. on NFS when the main job
+ # and the cluster node job propagate their content and the system has some
+ # latency including non-atomic propagation processes).
+ with tempfile.NamedTemporaryFile(
+ mode="w",
+ dir=recdir,
+ delete=False,
+ suffix=os.path.basename(recpath),
+ ) as tmpfile:
+ json.dump(json_value, tmpfile)
+ os.rename(tmpfile.name, recpath)
def _delete_record(self, subject, id):
try:
@@ -462,7 +474,14 @@ class Persistence:
if not self._exists_record(subject, id):
return dict()
with open(self._record_path(subject, id), "r") as f:
- return json.load(f)
+ try:
+ return json.load(f)
+ except json.JSONDecodeError as e:
+ pass
+ # case: file is corrupted, delete it
+ logger.warning(f"Deleting corrupted metadata record.")
+ self._delete_record(subject, id)
+ return dict()
def _exists_record(self, subject, id):
return os.path.exists(self._record_path(subject, id))
=====================================
snakemake/script.py
=====================================
@@ -180,6 +180,7 @@ class REncoder:
@classmethod
def encode_value(cls, value):
+
if value is None:
return "NULL"
elif isinstance(value, str):
@@ -202,6 +203,9 @@ class REncoder:
if isinstance(value, np.number):
return str(value)
+ elif isinstance(value, np.bool_):
+ return "TRUE" if value else "FALSE"
+
except ImportError:
pass
raise ValueError("Unsupported value for conversion into R: {}".format(value))
@@ -532,13 +536,18 @@ class PythonScript(ScriptBase):
fd.write(self.source.encode())
def _is_python_env(self):
- if self.conda_env is not None:
+ if self.conda_env is not None and ON_WINDOWS:
+ prefix = self.conda_env
+ elif self.conda_env is not None:
prefix = os.path.join(self.conda_env, "bin")
elif self.env_modules is not None:
prefix = self._execute_cmd("echo $PATH", read=True).split(":")[0]
else:
raise NotImplementedError()
- return os.path.exists(os.path.join(prefix, "python"))
+ if not ON_WINDOWS:
+ return os.path.exists(os.path.join(prefix, "python"))
+ else:
+ return os.path.exists(os.path.join(prefix, "python.exe"))
def _get_python_version(self):
out = self._execute_cmd(
=====================================
snakemake/workflow.py
=====================================
@@ -1941,7 +1941,14 @@ class Workflow:
prefix=prefix,
)
- def userule(self, rules=None, from_module=None, name_modifier=None, lineno=None):
+ def userule(
+ self,
+ rules=None,
+ from_module=None,
+ exclude_rules=None,
+ name_modifier=None,
+ lineno=None,
+ ):
def decorate(maybe_ruleinfo):
if from_module is not None:
try:
@@ -1955,6 +1962,7 @@ class Workflow:
module.use_rules(
rules,
name_modifier,
+ exclude_rules=exclude_rules,
ruleinfo=None if callable(maybe_ruleinfo) else maybe_ruleinfo,
skip_global_report_caption=self.report_text
is not None, # do not overwrite existing report text via module
=====================================
tests/test_conda_python_script/Snakefile
=====================================
@@ -0,0 +1,7 @@
+rule random_python_conda_script:
+ output:
+ "version.txt"
+ conda:
+ "test_python_env.yaml"
+ script:
+ "test_script.py"
=====================================
tests/test_conda_python_script/expected-results/version.txt
=====================================
@@ -0,0 +1 @@
+1.21.4
\ No newline at end of file
=====================================
tests/test_conda_python_script/test_python_env.yaml
=====================================
@@ -0,0 +1,6 @@
+channels:
+ - conda-forge
+ - defaults
+dependencies:
+ - numpy ==1.21.4
+ - python <3.10
=====================================
tests/test_conda_python_script/test_script.py
=====================================
@@ -0,0 +1,4 @@
+import numpy
+
+with open('version.txt', 'w') as f:
+ f.write(numpy.__version__)
=====================================
tests/test_modules_all_exclude/Snakefile
=====================================
@@ -0,0 +1,17 @@
+shell.executable("bash")
+
+configfile: "config/config.yaml"
+
+
+module test:
+ snakefile:
+ "module-test/Snakefile"
+ config:
+ config
+ replace_prefix:
+ {"results/": "results/testmodule/"}
+
+
+use rule * from test as test_*
+
+assert test.some_func() == 15
=====================================
tests/test_modules_all_exclude/Snakefile_exclude
=====================================
@@ -0,0 +1,17 @@
+shell.executable("bash")
+
+configfile: "config/config.yaml"
+
+
+module test:
+ snakefile:
+ "module-test/Snakefile"
+ config:
+ config
+ replace_prefix:
+ {"results/": "results/testmodule/"}
+
+
+use rule * from test exclude b, d
+
+assert test.some_func() == 15
=====================================
tests/test_modules_all_exclude/config/config.yaml
=====================================
@@ -0,0 +1,4 @@
+test: 1
+testb: 2
+testc: 3
+testd: 4
=====================================
tests/test_modules_all_exclude/expected-results/results/testmodule/test.out
=====================================
@@ -0,0 +1 @@
+1
=====================================
tests/test_modules_all_exclude/expected-results/results/testmodule/test2.out
=====================================
@@ -0,0 +1 @@
+3
=====================================
tests/test_modules_all_exclude/module-test/Snakefile
=====================================
@@ -0,0 +1,35 @@
+configfile: "config.yaml" # does not exist, but this statement should be ignored on module import
+
+
+def some_func():
+ return 15
+
+
+rule all:
+ input:
+ "results/test.out", "results/test2.out"
+
+
+rule a:
+ output:
+ "results/test.out"
+ shell:
+ "echo {config[test]} > {output}"
+
+rule b:
+ output:
+ "results/test2.out"
+ shell:
+ "echo {config[testb]} > {output}"
+
+rule c:
+ output:
+ "results/test2.out"
+ shell:
+ "echo {config[testc]} > {output}"
+
+rule d:
+ output:
+ "results/test2.out"
+ shell:
+ "echo {config[testc]} > {output}"
=====================================
tests/tests.py
=====================================
@@ -1378,6 +1378,20 @@ def test_modules_all():
run(dpath("test_modules_all"), targets=["a"])
+def test_modules_all_exclude_1():
+ # Fail due to conflicting rules
+ run(dpath("test_modules_all_exclude"), shouldfail=True)
+
+
+def test_modules_all_exclude_2():
+ # Successed since the conflicting rules have been excluded
+ run(
+ dpath("test_modules_all_exclude"),
+ snakefile="Snakefile_exclude",
+ shouldfail=False,
+ )
+
+
@skip_on_windows
def test_modules_prefix():
run(dpath("test_modules_prefix"), targets=["a"])
@@ -1659,3 +1673,7 @@ def test_conda_pin_file():
@skip_on_windows # sufficient to test this on linux
def test_github_issue1618():
run(dpath("test_github_issue1618"), cores=5)
+
+
+def test_conda_python_script():
+ run(dpath("test_conda_python_script"), use_conda=True)
View it on GitLab: https://salsa.debian.org/med-team/snakemake/-/commit/da8fea00be9f98e3473ce44f5d9fdbc21aa4a7b2
--
View it on GitLab: https://salsa.debian.org/med-team/snakemake/-/commit/da8fea00be9f98e3473ce44f5d9fdbc21aa4a7b2
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20220726/eab20af2/attachment-0001.htm>
More information about the debian-med-commit
mailing list