[med-svn] [cwltool] 01/05: New upstream version 1.0.20171221100033
Andreas Tille
tille at debian.org
Fri Dec 22 08:35:29 UTC 2017
This is an automated email from the git hooks/post-receive script.
tille pushed a commit to branch master
in repository cwltool.
commit fc0ce1f6a6d2d99b2fa479d8d3a51b2be7c25039
Author: Andreas Tille <tille at debian.org>
Date: Fri Dec 22 08:19:37 2017 +0100
New upstream version 1.0.20171221100033
---
MANIFEST.in | 1 +
Makefile | 6 +-
PKG-INFO | 93 +++++++++++++++++++++------
README.rst | 91 ++++++++++++++++++++------
cwltool.egg-info/PKG-INFO | 93 +++++++++++++++++++++------
cwltool.egg-info/SOURCES.txt | 16 +++++
cwltool/draft2tool.py | 33 ++++++----
cwltool/load_tool.py | 110 +++++++++++++++++++++++--------
cwltool/main.py | 139 ++++++++++++++++++++++++++++------------
cwltool/pack.py | 37 +++++++++--
cwltool/pathmapper.py | 4 +-
cwltool/process.py | 13 +++-
cwltool/workflow.py | 14 ++--
setup.cfg | 2 +-
tests/override/echo-job-ov.yml | 6 ++
tests/override/echo-job-ov2.yml | 7 ++
tests/override/echo-job.yml | 1 +
tests/override/echo-wf.cwl | 14 ++++
tests/override/echo.cwl | 19 ++++++
tests/override/ov.yml | 5 ++
tests/override/ov2.yml | 5 ++
tests/override/ov3.yml | 5 ++
tests/test_ext.py | 1 -
tests/test_fetch.py | 8 ++-
tests/test_override.py | 66 +++++++++++++++++++
tests/test_pack.py | 101 ++++++++++++++++++++++++++---
tests/wf/count-lines1-wf.cwl | 25 ++++++++
tests/wf/formattest-job.json | 7 ++
tests/wf/formattest.cwl | 20 ++++++
tests/wf/parseInt-tool.cwl | 16 +++++
tests/wf/wc-job.json | 6 ++
tests/wf/wc-tool.cwl | 17 +++++
tests/wf/whale.txt | 16 +++++
33 files changed, 829 insertions(+), 168 deletions(-)
diff --git a/MANIFEST.in b/MANIFEST.in
index 47e3080..effba2c 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -2,6 +2,7 @@ include gittaggers.py Makefile cwltool.py
include tests/*
include tests/tmp1/tmp2/tmp3/.gitkeep
include tests/wf/*
+include tests/override/*
include cwltool/schemas/v1.0/*.yml
include cwltool/schemas/draft-2/*.yml
include cwltool/schemas/draft-3/*.yml
diff --git a/Makefile b/Makefile
index 2b6a380..bd4a87d 100644
--- a/Makefile
+++ b/Makefile
@@ -89,7 +89,7 @@ pydocstyle_report.txt: $(PYSOURCES)
pydocstyle setup.py $^ > pydocstyle_report.txt 2>&1 || true
diff_pydocstyle_report: pydocstyle_report.txt
- diff-quality --violations=pep8 $^
+ diff-quality --violations=pycodestyle $^
## autopep8 : fix most Python code indentation and formatting
autopep8: $(PYSOURCES)
@@ -160,7 +160,7 @@ mypy2: ${PYSOURCES}
rm -Rf typeshed/2and3/schema_salad
ln -s $(shell python -c 'from __future__ import print_function; import schema_salad; import os.path; print(os.path.dirname(schema_salad.__file__))') \
typeshed/2and3/schema_salad
- MYPYPATH=$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \
+ MYPYPATH=$$MYPYPATH:typeshed/2.7:typeshed/2and3 mypy --py2 --disallow-untyped-calls \
--warn-redundant-casts \
cwltool
@@ -171,7 +171,7 @@ mypy3: ${PYSOURCES}
rm -Rf typeshed/2and3/schema_salad
ln -s $(shell python3 -c 'from __future__ import print_function; import schema_salad; import os.path; print(os.path.dirname(schema_salad.__file__))') \
typeshed/2and3/schema_salad
- MYPYPATH=$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \
+ MYPYPATH=$$MYPYPATH:typeshed/3:typeshed/2and3 mypy --disallow-untyped-calls \
--warn-redundant-casts \
cwltool
diff --git a/PKG-INFO b/PKG-INFO
index d3ce41e..694e059 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: cwltool
-Version: 1.0.20171107133715
+Version: 1.0.20171221100033
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Author: Common workflow language working group
@@ -41,7 +41,7 @@ Description: ==================================================================
virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
source venv/bin/activate # Activate environment before installing `cwltool`
- 1. Installing the official package from PyPi (will install "cwltool" package as
+ Installing the official package from PyPi (will install "cwltool" package as
well)
.. code:: bash
@@ -54,7 +54,7 @@ Description: ==================================================================
pip install cwltool
- 2. To install from source
+ Or you can install from source:
.. code:: bash
@@ -72,9 +72,16 @@ Description: ==================================================================
- Running basic tests ``(/tests)``:
- We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
- to run various tests in all supported Python environments.
- You can run the test suite by simply running the following in the terminal:
+ To run the basis tests after installing `cwltool` execute the following:
+
+ .. code:: bash
+
+ pip install pytest mock
+ py.test --ignore cwltool/schemas/ --pyarg cwltool
+
+ To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments
+ first downloading the complete code repository (see the ``git clone`` instructions above) and then run
+ the following in the terminal:
``pip install tox; tox``
List of all environment can be seen using:
@@ -116,6 +123,21 @@ Description: ==================================================================
.. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
:target: https://ci.commonwl.org/job/cwltool-conformance/
+ Running user-space implementations of Docker
+ --------------------------------------------
+
+ Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+ Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+ For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+ Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+ .. code:: bash
+
+ cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+
Tool or workflow loading from remote or local locations
-------------------------------------------------------
@@ -379,6 +401,50 @@ Description: ==================================================================
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
+ Overriding workflow requirements at load time
+ ---------------------------------------------
+
+ Sometimes a workflow needs additional requirements to run in a particular
+ environment or with a particular dataset. To avoid the need to modify the
+ underlying workflow, cwltool supports requirement "overrides".
+
+ The format of the "overrides" object is a mapping of item identifier (workflow,
+ workflow step, or command line tool) followed by a list of ProcessRequirements
+ that should be applied.
+
+ .. code:: yaml
+
+ cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+
+ Overrides can be specified either on the command line, or as part of the job
+ input document. Workflow steps are identified using the name of the workflow
+ file followed by the step name as a document fragment identifier "#id".
+ Override identifiers are relative to the toplevel workflow document.
+
+ .. code:: bash
+
+ cwltool --overrides overrides.yml my-tool.cwl my-job.yml
+
+ .. code:: yaml
+
+ input_parameter1: value1
+ input_parameter2: value2
+ cwltool:overrides:
+ workflow.cwl#step1:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+ .. code:: bash
+
+ cwltool my-tool.cwl my-job-with-overrides.yml
+
+
CWL Tool Control Flow
---------------------
@@ -511,21 +577,6 @@ Description: ==================================================================
Handler object for logging.
- Running user-space implementations of Docker
- --------------------------------------------
-
- Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
-
- Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
-
- For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
-
- Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
-
- ```
- cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
- ```
-
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
diff --git a/README.rst b/README.rst
index e44935e..f817cf0 100644
--- a/README.rst
+++ b/README.rst
@@ -31,7 +31,7 @@ It is highly recommended to setup virtual environment before installing `cwltool
virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
source venv/bin/activate # Activate environment before installing `cwltool`
-1. Installing the official package from PyPi (will install "cwltool" package as
+Installing the official package from PyPi (will install "cwltool" package as
well)
.. code:: bash
@@ -44,7 +44,7 @@ If installing alongside another CWL implementation then
pip install cwltool
-2. To install from source
+Or you can install from source:
.. code:: bash
@@ -62,9 +62,16 @@ Running tests locally
- Running basic tests ``(/tests)``:
-We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
-to run various tests in all supported Python environments.
-You can run the test suite by simply running the following in the terminal:
+To run the basis tests after installing `cwltool` execute the following:
+
+.. code:: bash
+
+ pip install pytest mock
+ py.test --ignore cwltool/schemas/ --pyarg cwltool
+
+To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments
+first downloading the complete code repository (see the ``git clone`` instructions above) and then run
+the following in the terminal:
``pip install tox; tox``
List of all environment can be seen using:
@@ -106,6 +113,21 @@ and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
.. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
:target: https://ci.commonwl.org/job/cwltool-conformance/
+Running user-space implementations of Docker
+--------------------------------------------
+
+Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+.. code:: bash
+
+ cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+
Tool or workflow loading from remote or local locations
-------------------------------------------------------
@@ -369,6 +391,50 @@ at the following links:
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
+Overriding workflow requirements at load time
+---------------------------------------------
+
+Sometimes a workflow needs additional requirements to run in a particular
+environment or with a particular dataset. To avoid the need to modify the
+underlying workflow, cwltool supports requirement "overrides".
+
+The format of the "overrides" object is a mapping of item identifier (workflow,
+workflow step, or command line tool) followed by a list of ProcessRequirements
+that should be applied.
+
+.. code:: yaml
+
+ cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+
+Overrides can be specified either on the command line, or as part of the job
+input document. Workflow steps are identified using the name of the workflow
+file followed by the step name as a document fragment identifier "#id".
+Override identifiers are relative to the toplevel workflow document.
+
+.. code:: bash
+
+ cwltool --overrides overrides.yml my-tool.cwl my-job.yml
+
+.. code:: yaml
+
+ input_parameter1: value1
+ input_parameter2: value2
+ cwltool:overrides:
+ workflow.cwl#step1:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+.. code:: bash
+
+ cwltool my-tool.cwl my-job-with-overrides.yml
+
+
CWL Tool Control Flow
---------------------
@@ -500,18 +566,3 @@ logger_handler
logging.Handler
Handler object for logging.
-
-Running user-space implementations of Docker
---------------------------------------------
-
-Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
-
-Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
-
-For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
-
-Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
-
-```
-cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
-```
diff --git a/cwltool.egg-info/PKG-INFO b/cwltool.egg-info/PKG-INFO
index d3ce41e..694e059 100644
--- a/cwltool.egg-info/PKG-INFO
+++ b/cwltool.egg-info/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: cwltool
-Version: 1.0.20171107133715
+Version: 1.0.20171221100033
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Author: Common workflow language working group
@@ -41,7 +41,7 @@ Description: ==================================================================
virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
source venv/bin/activate # Activate environment before installing `cwltool`
- 1. Installing the official package from PyPi (will install "cwltool" package as
+ Installing the official package from PyPi (will install "cwltool" package as
well)
.. code:: bash
@@ -54,7 +54,7 @@ Description: ==================================================================
pip install cwltool
- 2. To install from source
+ Or you can install from source:
.. code:: bash
@@ -72,9 +72,16 @@ Description: ==================================================================
- Running basic tests ``(/tests)``:
- We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
- to run various tests in all supported Python environments.
- You can run the test suite by simply running the following in the terminal:
+ To run the basis tests after installing `cwltool` execute the following:
+
+ .. code:: bash
+
+ pip install pytest mock
+ py.test --ignore cwltool/schemas/ --pyarg cwltool
+
+ To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments
+ first downloading the complete code repository (see the ``git clone`` instructions above) and then run
+ the following in the terminal:
``pip install tox; tox``
List of all environment can be seen using:
@@ -116,6 +123,21 @@ Description: ==================================================================
.. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
:target: https://ci.commonwl.org/job/cwltool-conformance/
+ Running user-space implementations of Docker
+ --------------------------------------------
+
+ Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+ Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+ For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+ Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+ .. code:: bash
+
+ cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+
Tool or workflow loading from remote or local locations
-------------------------------------------------------
@@ -379,6 +401,50 @@ Description: ==================================================================
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
+ Overriding workflow requirements at load time
+ ---------------------------------------------
+
+ Sometimes a workflow needs additional requirements to run in a particular
+ environment or with a particular dataset. To avoid the need to modify the
+ underlying workflow, cwltool supports requirement "overrides".
+
+ The format of the "overrides" object is a mapping of item identifier (workflow,
+ workflow step, or command line tool) followed by a list of ProcessRequirements
+ that should be applied.
+
+ .. code:: yaml
+
+ cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+
+ Overrides can be specified either on the command line, or as part of the job
+ input document. Workflow steps are identified using the name of the workflow
+ file followed by the step name as a document fragment identifier "#id".
+ Override identifiers are relative to the toplevel workflow document.
+
+ .. code:: bash
+
+ cwltool --overrides overrides.yml my-tool.cwl my-job.yml
+
+ .. code:: yaml
+
+ input_parameter1: value1
+ input_parameter2: value2
+ cwltool:overrides:
+ workflow.cwl#step1:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: override_value
+
+ .. code:: bash
+
+ cwltool my-tool.cwl my-job-with-overrides.yml
+
+
CWL Tool Control Flow
---------------------
@@ -511,21 +577,6 @@ Description: ==================================================================
Handler object for logging.
- Running user-space implementations of Docker
- --------------------------------------------
-
- Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
-
- Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
-
- For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
-
- Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
-
- ```
- cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
- ```
-
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
diff --git a/cwltool.egg-info/SOURCES.txt b/cwltool.egg-info/SOURCES.txt
index f4bebd2..149d8d9 100644
--- a/cwltool.egg-info/SOURCES.txt
+++ b/cwltool.egg-info/SOURCES.txt
@@ -173,21 +173,33 @@ tests/test_ext.py
tests/test_fetch.py
tests/test_http_input.py
tests/test_js_sandbox.py
+tests/test_override.py
tests/test_pack.py
tests/test_pathmapper.py
tests/test_rdfprint.py
tests/test_relax_path_checks.py
tests/test_toolargparse.py
tests/util.py
+tests/override/echo-job-ov.yml
+tests/override/echo-job-ov2.yml
+tests/override/echo-job.yml
+tests/override/echo-wf.cwl
+tests/override/echo.cwl
+tests/override/ov.yml
+tests/override/ov2.yml
+tests/override/ov3.yml
tests/tmp1/tmp2/tmp3/.gitkeep
tests/wf/badout1.cwl
tests/wf/badout2.cwl
tests/wf/badout3.cwl
tests/wf/cat.cwl
+tests/wf/count-lines1-wf.cwl
tests/wf/default_path.cwl
tests/wf/echo.cwl
tests/wf/empty.ttl
tests/wf/expect_packed.cwl
+tests/wf/formattest-job.json
+tests/wf/formattest.cwl
tests/wf/hello-workflow.cwl
tests/wf/hello.txt
tests/wf/hello_single_tool.cwl
@@ -201,6 +213,7 @@ tests/wf/missing_cwlVersion.cwl
tests/wf/mut.cwl
tests/wf/mut2.cwl
tests/wf/mut3.cwl
+tests/wf/parseInt-tool.cwl
tests/wf/revsort-job.json
tests/wf/revsort.cwl
tests/wf/revtool.cwl
@@ -211,5 +224,8 @@ tests/wf/updatedir_inplace.cwl
tests/wf/updateval.cwl
tests/wf/updateval.py
tests/wf/updateval_inplace.cwl
+tests/wf/wc-job.json
+tests/wf/wc-tool.cwl
tests/wf/wffail.cwl
+tests/wf/whale.txt
tests/wf/wrong_cwlVersion.cwl
\ No newline at end of file
diff --git a/cwltool/draft2tool.py b/cwltool/draft2tool.py
index bb56692..5d572bb 100644
--- a/cwltool/draft2tool.py
+++ b/cwltool/draft2tool.py
@@ -1,14 +1,16 @@
from __future__ import absolute_import
import copy
import hashlib
+import locale
import json
import logging
import os
import re
import shutil
import tempfile
-from functools import partial
-from typing import Any, Callable, Dict, Generator, List, Optional, Set, Text, Union, cast
+from functools import partial, cmp_to_key
+from typing import (Any, Callable, Dict, Generator, List, Optional, Set, Text,
+ Union, cast)
from six import string_types, u
@@ -208,7 +210,7 @@ class CommandLineTool(Process):
})
dockerReq = self.requirements[0]
if default_container == windows_default_container_id and use_container and onWindows():
- _logger.warning(DEFAULT_CONTAINER_MSG%(windows_default_container_id, windows_default_container_id))
+ _logger.warning(DEFAULT_CONTAINER_MSG % (windows_default_container_id, windows_default_container_id))
if dockerReq and use_container:
return DockerCommandLineJob()
@@ -523,8 +525,8 @@ class CommandLineTool(Process):
for i, port in enumerate(ports):
def makeWorkflowException(msg):
return WorkflowException(
- u"Error collecting output for parameter '%s':\n%s"
- % (shortname(port["id"]), msg))
+ u"Error collecting output for parameter '%s':\n%s"
+ % (shortname(port["id"]), msg))
with SourceLine(ports, i, makeWorkflowException, debug):
fragment = shortname(port["id"])
ret[fragment] = self.collect_output(port, builder, outdir, fs_access,
@@ -575,16 +577,25 @@ class CommandLineTool(Process):
elif gb == ".":
gb = outdir
elif gb.startswith("/"):
- raise WorkflowException("glob patterns must not start with '/'")
+ raise WorkflowException(
+ "glob patterns must not start with '/'")
try:
prefix = fs_access.glob(outdir)
r.extend([{"location": g,
- "path": fs_access.join(builder.outdir, g[len(prefix[0])+1:]),
+ "path": fs_access.join(builder.outdir,
+ g[len(prefix[0])+1:]),
"basename": os.path.basename(g),
- "nameroot": os.path.splitext(os.path.basename(g))[0],
- "nameext": os.path.splitext(os.path.basename(g))[1],
- "class": "File" if fs_access.isfile(g) else "Directory"}
- for g in fs_access.glob(fs_access.join(outdir, gb))])
+ "nameroot": os.path.splitext(
+ os.path.basename(g))[0],
+ "nameext": os.path.splitext(
+ os.path.basename(g))[1],
+ "class": "File" if fs_access.isfile(g)
+ else "Directory"}
+ for g in sorted(fs_access.glob(
+ fs_access.join(outdir, gb)),
+ key=cmp_to_key(cast(
+ Callable[[Text, Text],
+ int], locale.strcoll)))])
except (OSError, IOError) as e:
_logger.warning(Text(e))
except:
diff --git a/cwltool/load_tool.py b/cwltool/load_tool.py
index 8f30b90..59fa62e 100644
--- a/cwltool/load_tool.py
+++ b/cwltool/load_tool.py
@@ -8,7 +8,8 @@ import re
import uuid
import hashlib
import json
-from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast
+import copy
+from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast, Iterable
import requests.sessions
from six import itervalues, string_types
@@ -23,19 +24,65 @@ from six.moves import urllib
from . import process, update
from .errors import WorkflowException
-from .process import Process, shortname
+from .process import Process, shortname, get_schema
from .update import ALLUPDATES
_logger = logging.getLogger("cwltool")
jobloaderctx = {
u"cwl": "https://w3id.org/cwl/cwl#",
+ u"cwltool": "http://commonwl.org/cwltool#",
u"path": {u"@type": u"@id"},
u"location": {u"@type": u"@id"},
u"format": {u"@type": u"@id"},
u"id": u"@id"
}
+
+overrides_ctx = {
+ u"overrideTarget": {u"@type": u"@id"},
+ u"cwltool": "http://commonwl.org/cwltool#",
+ u"overrides": {
+ "@id": "cwltool:overrides",
+ "mapSubject": "overrideTarget",
+ "mapPredicate": "override"
+ },
+ u"override": {
+ "@id": "cwltool:override",
+ "mapSubject": "class"
+ }
+} # type: Dict[Text, Union[Dict[Any, Any], Text, Iterable[Text]]]
+
+def resolve_tool_uri(argsworkflow, # type: Text
+ resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text]
+ fetcher_constructor=None,
+ # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]
+ document_loader=None # type: Loader
+):
+ # type: (...) -> Tuple[Text, Text]
+
+ uri = None # type: Text
+ split = urllib.parse.urlsplit(argsworkflow)
+ # In case of Windows path, urlsplit misjudge Drive letters as scheme, here we are skipping that
+ if split.scheme and split.scheme in [u'http',u'https',u'file']:
+ uri = argsworkflow
+ elif os.path.exists(os.path.abspath(argsworkflow)):
+ uri = file_uri(str(os.path.abspath(argsworkflow)))
+ elif resolver:
+ if document_loader is None:
+ document_loader = Loader(jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore
+ uri = resolver(document_loader, argsworkflow)
+
+ if uri is None:
+ raise ValidationException("Not found: '%s'" % argsworkflow)
+
+ if argsworkflow != uri:
+ _logger.info("Resolved '%s' to '%s'", argsworkflow, uri)
+
+ fileuri = urllib.parse.urldefrag(uri)[0]
+ return uri, fileuri
+
+
def fetch_document(argsworkflow, # type: Union[Text, Dict[Text, Any]]
resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text]
fetcher_constructor=None
@@ -49,22 +96,7 @@ def fetch_document(argsworkflow, # type: Union[Text, Dict[Text, Any]]
uri = None # type: Text
workflowobj = None # type: CommentedMap
if isinstance(argsworkflow, string_types):
- split = urllib.parse.urlsplit(argsworkflow)
- # In case of Windows path, urlsplit misjudge Drive letters as scheme, here we are skipping that
- if split.scheme and split.scheme in [u'http',u'https',u'file']:
- uri = argsworkflow
- elif os.path.exists(os.path.abspath(argsworkflow)):
- uri = file_uri(str(os.path.abspath(argsworkflow)))
- elif resolver:
- uri = resolver(document_loader, argsworkflow)
-
- if uri is None:
- raise ValidationException("Not found: '%s'" % argsworkflow)
-
- if argsworkflow != uri:
- _logger.info("Resolved '%s' to '%s'", argsworkflow, uri)
-
- fileuri = urllib.parse.urldefrag(uri)[0]
+ uri, fileuri = resolve_tool_uri(argsworkflow, resolver=resolver, document_loader=document_loader)
workflowobj = document_loader.fetch(fileuri)
elif isinstance(argsworkflow, dict):
uri = "#" + Text(id(argsworkflow))
@@ -139,8 +171,9 @@ def validate_document(document_loader, # type: Loader
strict=True, # type: bool
preprocess_only=False, # type: bool
fetcher_constructor=None,
- skip_schemas=None
+ skip_schemas=None,
# type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]
+ overrides=None # type: List[Dict]
):
# type: (...) -> Tuple[Loader, Names, Union[Dict[Text, Any], List[Dict[Text, Any]]], Dict[Text, Any], Text]
"""Validate a CWL document."""
@@ -155,9 +188,15 @@ def validate_document(document_loader, # type: Loader
jobobj = None
if "cwl:tool" in workflowobj:
- jobobj, _ = document_loader.resolve_all(workflowobj, uri)
+ job_loader = Loader(jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore
+ jobobj, _ = job_loader.resolve_all(workflowobj, uri)
uri = urllib.parse.urljoin(uri, workflowobj["https://w3id.org/cwl/cwl#tool"])
del cast(dict, jobobj)["https://w3id.org/cwl/cwl#tool"]
+
+ if "http://commonwl.org/cwltool#overrides" in jobobj:
+ overrides.extend(resolve_overrides(jobobj, uri, uri))
+ del jobobj["http://commonwl.org/cwltool#overrides"]
+
workflowobj = fetch_document(uri, fetcher_constructor=fetcher_constructor)[1]
fileuri = urllib.parse.urldefrag(uri)[0]
@@ -225,6 +264,9 @@ def validate_document(document_loader, # type: Loader
if jobobj:
metadata[u"cwl:defaults"] = jobobj
+ if overrides:
+ metadata[u"cwltool:overrides"] = overrides
+
return document_loader, avsc_names, processobj, metadata, uri
@@ -239,10 +281,13 @@ def make_tool(document_loader, # type: Loader
"""Make a Python CWL object."""
resolveduri = document_loader.resolve_ref(uri)[0]
+ processobj = None
if isinstance(resolveduri, list):
- if len(resolveduri) == 1:
- processobj = resolveduri[0]
- else:
+ for obj in resolveduri:
+ if obj['id'].endswith('#main'):
+ processobj = obj
+ break
+ if not processobj:
raise WorkflowException(
u"Tool file contains graph of multiple objects, must specify "
"one of #%s" % ", #".join(
@@ -277,7 +322,8 @@ def load_tool(argsworkflow, # type: Union[Text, Dict[Text, Any]]
enable_dev=False, # type: bool
strict=True, # type: bool
resolver=None, # type: Callable[[Loader, Union[Text, Dict[Text, Any]]], Text]
- fetcher_constructor=None # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]
+ fetcher_constructor=None, # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]
+ overrides=None
):
# type: (...) -> Process
@@ -285,6 +331,20 @@ def load_tool(argsworkflow, # type: Union[Text, Dict[Text, Any]]
fetcher_constructor=fetcher_constructor)
document_loader, avsc_names, processobj, metadata, uri = validate_document(
document_loader, workflowobj, uri, enable_dev=enable_dev,
- strict=strict, fetcher_constructor=fetcher_constructor)
+ strict=strict, fetcher_constructor=fetcher_constructor,
+ overrides=overrides)
return make_tool(document_loader, avsc_names, metadata, uri,
makeTool, kwargs if kwargs else {})
+
+def resolve_overrides(ov, ov_uri, baseurl): # type: (CommentedMap, Text, Text) -> List[Dict[Text, Any]]
+ ovloader = Loader(overrides_ctx)
+ ret, _ = ovloader.resolve_all(ov, baseurl)
+ if not isinstance(ret, CommentedMap):
+ raise Exception("Expected CommentedMap, got %s" % type(ret))
+ cwl_docloader = get_schema("v1.0")[0]
+ cwl_docloader.resolve_all(ret, ov_uri)
+ return ret["overrides"]
+
+def load_overrides(ov, base_url): # type: (Text, Text) -> List[Dict[Text, Any]]
+ ovloader = Loader(overrides_ctx)
+ return resolve_overrides(ovloader.fetch(ov), ov, base_url)
diff --git a/cwltool/main.py b/cwltool/main.py
index f57cafb..8f58cb1 100755
--- a/cwltool/main.py
+++ b/cwltool/main.py
@@ -11,7 +11,7 @@ import os
import sys
import tempfile
from typing import (IO, Any, AnyStr, Callable, Dict, List, Sequence, Text, Tuple,
- Union, cast)
+ Union, cast, Mapping, MutableMapping, Iterable)
import pkg_resources # part of setuptools
import requests
@@ -27,7 +27,8 @@ from . import draft2tool, workflow
from .builder import Builder
from .cwlrdf import printdot, printrdf
from .errors import UnsupportedRequirement, WorkflowException
-from .load_tool import fetch_document, make_tool, validate_document, jobloaderctx
+from .load_tool import (resolve_tool_uri, fetch_document, make_tool, validate_document,
+ jobloaderctx, resolve_overrides, load_overrides)
from .mutation import MutationManager
from .pack import pack
from .pathmapper import (adjustDirObjs, adjustFileObjs, get_listing,
@@ -36,7 +37,9 @@ from .process import (Process, cleanIntermediate, normalizeFilesDirs,
relocateOutputs, scandeps, shortname, use_custom_schema,
use_standard_schema)
from .resolver import ga4gh_tool_registries, tool_resolver
-from .software_requirements import DependenciesConfiguration, get_container_from_software_requirements, SOFTWARE_REQUIREMENTS_ENABLED
+from .software_requirements import (DependenciesConfiguration,
+ get_container_from_software_requirements,
+ SOFTWARE_REQUIREMENTS_ENABLED)
from .stdfsaccess import StdFsAccess
from .update import ALLUPDATES, UPDATES
from .utils import onWindows, windows_default_container_id
@@ -238,6 +241,10 @@ def arg_parser(): # type: () -> argparse.ArgumentParser
parser.add_argument("--no-read-only", action="store_true",
default=False, help="Do not set root directoy in the"
" container as read-only", dest="no_read_only")
+
+ parser.add_argument("--overrides", type=str,
+ default=None, help="Read process requirement overrides from file.")
+
parser.add_argument("workflow", type=Text, nargs="?", default=None)
parser.add_argument("job_order", nargs=argparse.REMAINDER)
@@ -509,14 +516,17 @@ def generate_input_template(tool):
-def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
- stdout=sys.stdout, make_fs_access=None, fetcher_constructor=None):
- # type: (argparse.Namespace, Process, IO[Any], bool, bool, IO[Any], Callable[[Text], StdFsAccess], Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]) -> Union[int, Tuple[Dict[Text, Any], Text]]
+def load_job_order(args, # type: argparse.Namespace
+ stdin, # type: IO[Any]
+ fetcher_constructor, # Fetcher
+ overrides, # type: List[Dict[Text, Any]]
+ tool_file_uri # type: Text
+):
+ # type: (...) -> Tuple[Dict[Text, Any], Text, Loader]
job_order_object = None
_jobloaderctx = jobloaderctx.copy()
- _jobloaderctx.update(t.metadata.get("$namespaces", {}))
loader = Loader(_jobloaderctx, fetcher_constructor=fetcher_constructor) # type: ignore
if len(args.job_order) == 1 and args.job_order[0][0] != "-":
@@ -531,14 +541,31 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
input_basedir = args.basedir if args.basedir else os.getcwd()
elif job_order_file:
input_basedir = args.basedir if args.basedir else os.path.abspath(os.path.dirname(job_order_file))
- try:
- job_order_object, _ = loader.resolve_ref(job_order_file, checklinks=False)
- except Exception as e:
- _logger.error(Text(e), exc_info=args.debug)
- return 1
- toolparser = None
- else:
+ job_order_object, _ = loader.resolve_ref(job_order_file, checklinks=False)
+
+ if job_order_object and "http://commonwl.org/cwltool#overrides" in job_order_object:
+ overrides.extend(resolve_overrides(job_order_object, file_uri(job_order_file), tool_file_uri))
+ del job_order_object["http://commonwl.org/cwltool#overrides"]
+
+ if not job_order_object:
input_basedir = args.basedir if args.basedir else os.getcwd()
+
+ return (job_order_object, input_basedir, loader)
+
+
+def init_job_order(job_order_object, # type: MutableMapping[Text, Any]
+ args, # type: argparse.Namespace
+ t, # type: Process
+ print_input_deps=False, # type: bool
+ relative_deps=False, # type: bool
+ stdout=sys.stdout, # type: IO[Any]
+ make_fs_access=None, # type: Callable[[Text], StdFsAccess]
+ loader=None, # type: Loader
+ input_basedir="" # type: Text
+):
+ # (...) -> Tuple[Dict[Text, Any], Text]
+
+ if not job_order_object:
namemap = {} # type: Dict[Text, Text]
records = [] # type: List[Text]
toolparser = generate_parser(
@@ -546,7 +573,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
if toolparser:
if args.tool_help:
toolparser.print_help()
- return 0
+ exit(0)
cmd_line = vars(toolparser.parse_args(args.job_order))
for record_name in records:
record = {}
@@ -560,9 +587,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
if cmd_line["job_order"]:
try:
- input_basedir = args.basedir if args.basedir else os.path.abspath(
- os.path.dirname(cmd_line["job_order"]))
- job_order_object = loader.resolve_ref(cmd_line["job_order"])
+ job_order_object = cast(MutableMapping, loader.resolve_ref(cmd_line["job_order"])[0])
except Exception as e:
_logger.error(Text(e), exc_info=args.debug)
return 1
@@ -590,12 +615,12 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
toolparser.print_help()
_logger.error("")
_logger.error("Input object required, use --help for details")
- return 1
+ exit(1)
if print_input_deps:
printdeps(job_order_object, loader, stdout, relative_deps, "",
- basedir=file_uri(input_basedir + "/"))
- return 0
+ basedir=file_uri(str(input_basedir) + "/"))
+ exit(0)
def pathToLoc(p):
if "location" not in p and "path" in p:
@@ -613,8 +638,16 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
else:
return # best effort
+ ns = {} # type: Dict[Text, Union[Dict[Any, Any], Text, Iterable[Text]]]
+ ns.update(t.metadata.get("$namespaces", {}))
+ ld = Loader(ns)
+ def expand_formats(p):
+ if "format" in p:
+ p["format"] = ld.expand_url(p["format"], "")
+
visit_class(job_order_object, ("File", "Directory"), pathToLoc)
- visit_class(job_order_object, ("File"), addSizes)
+ visit_class(job_order_object, ("File",), addSizes)
+ visit_class(job_order_object, ("File",), expand_formats)
adjustDirObjs(job_order_object, trim_listing)
normalizeFilesDirs(job_order_object)
@@ -623,7 +656,7 @@ def load_job_order(args, t, stdin, print_input_deps=False, relative_deps=False,
if "id" in job_order_object:
del job_order_object["id"]
- return (job_order_object, input_basedir)
+ return job_order_object
def makeRelative(base, ob):
@@ -637,7 +670,7 @@ def makeRelative(base, ob):
def printdeps(obj, document_loader, stdout, relative_deps, uri, basedir=None):
- # type: (Dict[Text, Any], Loader, IO[Any], bool, Text, Text) -> None
+ # type: (Mapping[Text, Any], Loader, IO[Any], bool, Text, Text) -> None
deps = {"class": "File",
"location": uri} # type: Dict[Text, Any]
@@ -699,7 +732,7 @@ def main(argsl=None, # type: List[str]
stdout=sys.stdout, # type: IO[Any]
stderr=sys.stderr, # type: IO[Any]
versionfunc=versionstring, # type: Callable[[], Text]
- job_order_object=None, # type: Union[Tuple[Dict[Text, Any], Text], int]
+ job_order_object=None, # type: MutableMapping[Text, Any]
make_fs_access=StdFsAccess, # type: Callable[[Text], StdFsAccess]
fetcher_constructor=None, # type: Callable[[Dict[Text, Text], requests.sessions.Session], Fetcher]
resolver=tool_resolver,
@@ -757,7 +790,8 @@ def main(argsl=None, # type: List[str]
'enable_ga4gh_tool_registry': False,
'ga4gh_tool_registries': [],
'find_default_container': None,
- 'make_template': False
+ 'make_template': False,
+ 'overrides': None
}):
if not hasattr(args, k):
setattr(args, k, v)
@@ -802,8 +836,26 @@ def main(argsl=None, # type: List[str]
else:
use_standard_schema("v1.0")
+ uri, tool_file_uri = resolve_tool_uri(args.workflow,
+ resolver=resolver,
+ fetcher_constructor=fetcher_constructor)
+
+ overrides = [] # type: List[Dict[Text, Any]]
+
+ try:
+ job_order_object, input_basedir, jobloader = load_job_order(args,
+ stdin,
+ fetcher_constructor,
+ overrides,
+ tool_file_uri)
+ except Exception as e:
+ _logger.error(Text(e), exc_info=args.debug)
+
+ if args.overrides:
+ overrides.extend(load_overrides(file_uri(os.path.abspath(args.overrides)), tool_file_uri))
+
try:
- document_loader, workflowobj, uri = fetch_document(args.workflow, resolver=resolver,
+ document_loader, workflowobj, uri = fetch_document(uri, resolver=resolver,
fetcher_constructor=fetcher_constructor)
if args.print_deps:
@@ -815,16 +867,15 @@ def main(argsl=None, # type: List[str]
enable_dev=args.enable_dev, strict=args.strict,
preprocess_only=args.print_pre or args.pack,
fetcher_constructor=fetcher_constructor,
- skip_schemas=args.skip_schemas)
-
- if args.pack:
- stdout.write(print_pack(document_loader, processobj, uri, metadata))
- return 0
+ skip_schemas=args.skip_schemas,
+ overrides=overrides)
if args.print_pre:
stdout.write(json.dumps(processobj, indent=4))
return 0
+ overrides.extend(metadata.get("cwltool:overrides", []))
+
conf_file = getattr(args, "beta_dependency_resolvers_configuration", None) # Text
use_conda_dependencies = getattr(args, "beta_conda_dependencies", None) # Text
@@ -836,6 +887,7 @@ def main(argsl=None, # type: List[str]
make_tool_kwds["job_script_provider"] = dependencies_configuration
make_tool_kwds["find_default_container"] = functools.partial(find_default_container, args)
+ make_tool_kwds["overrides"] = overrides
tool = make_tool(document_loader, avsc_names, metadata, uri,
makeTool, make_tool_kwds)
@@ -846,6 +898,11 @@ def main(argsl=None, # type: List[str]
return 0
if args.validate:
+ _logger.info("Tool definition is valid")
+ return 0
+
+ if args.pack:
+ stdout.write(print_pack(document_loader, processobj, uri, metadata))
return 0
if args.print_rdf:
@@ -893,13 +950,13 @@ def main(argsl=None, # type: List[str]
setattr(args, "tmp_outdir_prefix", args.cachedir)
try:
- if job_order_object is None:
- job_order_object = load_job_order(args, tool, stdin,
- print_input_deps=args.print_input_deps,
- relative_deps=args.relative_deps,
- stdout=stdout,
- make_fs_access=make_fs_access,
- fetcher_constructor=fetcher_constructor)
+ job_order_object = init_job_order(job_order_object, args, tool,
+ print_input_deps=args.print_input_deps,
+ relative_deps=args.relative_deps,
+ stdout=stdout,
+ make_fs_access=make_fs_access,
+ loader=jobloader,
+ input_basedir=input_basedir)
except SystemExit as e:
return e.code
@@ -907,10 +964,10 @@ def main(argsl=None, # type: List[str]
return job_order_object
try:
- setattr(args, 'basedir', job_order_object[1])
+ setattr(args, 'basedir', input_basedir)
del args.workflow
del args.job_order
- (out, status) = executor(tool, job_order_object[0],
+ (out, status) = executor(tool, job_order_object,
makeTool=makeTool,
select_resources=selectResources,
make_fs_access=make_fs_access,
diff --git a/cwltool/pack.py b/cwltool/pack.py
index 898f548..dc2c414 100644
--- a/cwltool/pack.py
+++ b/cwltool/pack.py
@@ -1,9 +1,11 @@
from __future__ import absolute_import
import copy
+import re
from typing import Any, Callable, Dict, List, Set, Text, Union, cast
-from schema_salad.ref_resolver import Loader
+from schema_salad.ref_resolver import Loader, SubLoader
from six.moves import urllib
+from ruamel.yaml.comments import CommentedSeq, CommentedMap
from .process import shortname, uniquename
import six
@@ -64,7 +66,12 @@ def replace_refs(d, rewrite, stem, newstem):
if v in rewrite:
d[s] = rewrite[v]
elif v.startswith(stem):
- d[s] = newstem + v[len(stem):]
+ id_ = v[len(stem):]
+ # prevent appending newstems if tool is already packed
+ if id_.startswith(newstem.strip("#")):
+ d[s] = "#" + id_
+ else:
+ d[s] = newstem + id_
replace_refs(v, rewrite, stem, newstem)
def import_embed(d, seen):
@@ -84,12 +91,25 @@ def import_embed(d, seen):
seen.add(this)
break
- for v in d.values():
- import_embed(v, seen)
+ for k in sorted(d.keys()):
+ import_embed(d[k], seen)
def pack(document_loader, processobj, uri, metadata):
# type: (Loader, Union[Dict[Text, Any], List[Dict[Text, Any]]], Text, Dict[Text, Text]) -> Dict[Text, Any]
+
+ document_loader = SubLoader(document_loader)
+ document_loader.idx = {}
+ if isinstance(processobj, dict):
+ document_loader.idx[processobj["id"]] = CommentedMap(six.iteritems(processobj))
+ elif isinstance(processobj, list):
+ path, frag = urllib.parse.urldefrag(uri)
+ for po in processobj:
+ if not frag:
+ if po["id"].endswith("#main"):
+ uri = po["id"]
+ document_loader.idx[po["id"]] = CommentedMap(six.iteritems(po))
+
def loadref(b, u):
# type: (Text, Text) -> Union[Dict, List, Text]
return document_loader.resolve_ref(u, base_url=b)[0]
@@ -111,7 +131,8 @@ def pack(document_loader, processobj, uri, metadata):
if r == mainuri:
rewrite[r] = "#main"
elif r.startswith(mainuri) and r[len(mainuri)] in ("#", "/"):
- pass
+ path, frag = urllib.parse.urldefrag(r)
+ rewrite[r] = "#"+frag
else:
path, frag = urllib.parse.urldefrag(r)
if path == mainpath:
@@ -128,10 +149,14 @@ def pack(document_loader, processobj, uri, metadata):
packed = {"$graph": [], "cwlVersion": metadata["cwlVersion"]
} # type: Dict[Text, Any]
+ namespaces = metadata.get('$namespaces', None)
schemas = set() # type: Set[Text]
for r in sorted(runs):
dcr, metadata = document_loader.resolve_ref(r)
+ if isinstance(dcr, CommentedSeq):
+ dcr = dcr[0]
+ dcr = cast(CommentedMap, dcr)
if not isinstance(dcr, dict):
continue
for doc in (dcr, metadata):
@@ -161,5 +186,7 @@ def pack(document_loader, processobj, uri, metadata):
# duplicate 'cwlVersion' inside $graph when there is a single item
# because we're printing contents inside '$graph' rather than whole dict
packed["$graph"][0]["cwlVersion"] = packed["cwlVersion"]
+ if namespaces:
+ packed["$graph"][0]["$namespaces"] = dict(cast(Dict, namespaces))
return packed
diff --git a/cwltool/pathmapper.py b/cwltool/pathmapper.py
index 7127d23..c3aed1e 100644
--- a/cwltool/pathmapper.py
+++ b/cwltool/pathmapper.py
@@ -10,7 +10,7 @@ from tempfile import NamedTemporaryFile
import requests
from cachecontrol import CacheControl
from cachecontrol.caches import FileCache
-from typing import Any, Callable, Dict, Iterable, List, Set, Text, Tuple, Union
+from typing import Any, Callable, Dict, Iterable, List, Set, Text, Tuple, Union, MutableMapping
import schema_salad.validate as validate
from schema_salad.ref_resolver import uri_file_path
@@ -60,7 +60,7 @@ def adjustDirObjs(rec, op):
visit_class(rec, ("Directory",), op)
def normalizeFilesDirs(job):
- # type: (Union[List[Dict[Text, Any]], Dict[Text, Any]]) -> None
+ # type: (Union[List[Dict[Text, Any]], MutableMapping[Text, Any]]) -> None
def addLocation(d):
if "location" not in d:
if d["class"] == "File" and ("contents" not in d):
diff --git a/cwltool/process.py b/cwltool/process.py
index bdbee58..e9356f0 100644
--- a/cwltool/process.py
+++ b/cwltool/process.py
@@ -422,6 +422,15 @@ def avroize_type(field_type, name_prefix=""):
avroize_type(field_type["items"], name_prefix)
return field_type
+def get_overrides(overrides, toolid): # type: (List[Dict[Text, Any]], Text) -> List[Dict[Text, Any]]
+ req = [] # type: List[Dict[Text, Any]]
+ if not isinstance(overrides, list):
+ raise validate.ValidationException("Expected overrides to be a list, but was %s" % type(overrides))
+ for ov in overrides:
+ if ov["overrideTarget"] == toolid:
+ req.extend(ov["override"])
+ return req
+
class Process(six.with_metaclass(abc.ABCMeta, object)):
def __init__(self, toolpath_object, **kwargs):
# type: (Dict[Text, Any], **Any) -> None
@@ -456,7 +465,9 @@ class Process(six.with_metaclass(abc.ABCMeta, object)):
else:
self.names = names
self.tool = toolpath_object
- self.requirements = kwargs.get("requirements", []) + self.tool.get("requirements", [])
+ self.requirements = (kwargs.get("requirements", []) +
+ self.tool.get("requirements", []) +
+ get_overrides(kwargs.get("overrides", []), self.tool["id"]))
self.hints = kwargs.get("hints", []) + self.tool.get("hints", [])
self.formatgraph = None # type: Graph
if "loader" in kwargs:
diff --git a/cwltool/workflow.py b/cwltool/workflow.py
index 2081430..ea9def2 100644
--- a/cwltool/workflow.py
+++ b/cwltool/workflow.py
@@ -15,7 +15,7 @@ from schema_salad.sourceline import SourceLine, cmap
from . import draft2tool, expression
from .errors import WorkflowException
from .load_tool import load_tool
-from .process import Process, shortname, uniquename
+from .process import Process, shortname, uniquename, get_overrides
from .utils import aslist
import six
from six.moves import range
@@ -521,6 +521,8 @@ class Workflow(Process):
try:
self.steps.append(WorkflowStep(step, n, **kwargs))
except validate.ValidationException as v:
+ if _logger.isEnabledFor(logging.DEBUG):
+ _logger.exception("Validation failed at")
validation_errors.append(v)
if validation_errors:
@@ -623,8 +625,7 @@ def static_checker(workflow_inputs, workflow_outputs, step_inputs, step_outputs)
all_exception_msg = "\n".join(exception_msgs)
if warnings:
- _logger.warning("Workflow checker warning:")
- _logger.warning(all_warning_msg)
+ _logger.warning("Workflow checker warning:\n%s" % all_warning_msg)
if exceptions:
raise validate.ValidationException(all_exception_msg)
@@ -666,7 +667,9 @@ class WorkflowStep(Process):
else:
self.id = "#step" + Text(pos)
- kwargs["requirements"] = kwargs.get("requirements", []) + toolpath_object.get("requirements", [])
+ kwargs["requirements"] = (kwargs.get("requirements", []) +
+ toolpath_object.get("requirements", []) +
+ get_overrides(kwargs.get("overrides", []), self.id))
kwargs["hints"] = kwargs.get("hints", []) + toolpath_object.get("hints", [])
try:
@@ -678,7 +681,8 @@ class WorkflowStep(Process):
enable_dev=kwargs.get("enable_dev"),
strict=kwargs.get("strict"),
fetcher_constructor=kwargs.get("fetcher_constructor"),
- resolver=kwargs.get("resolver"))
+ resolver=kwargs.get("resolver"),
+ overrides=kwargs.get("overrides"))
except validate.ValidationException as v:
raise WorkflowException(
u"Tool definition %s failed validation:\n%s" %
diff --git a/setup.cfg b/setup.cfg
index 8336837..773cc2f 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -13,6 +13,6 @@ addopts = --ignore cwltool/schemas
testpaths = tests
[egg_info]
-tag_build = .20171107133715
+tag_build = .20171221100033
tag_date = 0
diff --git a/tests/override/echo-job-ov.yml b/tests/override/echo-job-ov.yml
new file mode 100644
index 0000000..6170fd2
--- /dev/null
+++ b/tests/override/echo-job-ov.yml
@@ -0,0 +1,6 @@
+m1: zing
+cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: hello3
diff --git a/tests/override/echo-job-ov2.yml b/tests/override/echo-job-ov2.yml
new file mode 100644
index 0000000..3e8da10
--- /dev/null
+++ b/tests/override/echo-job-ov2.yml
@@ -0,0 +1,7 @@
+m1: zing
+cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: hello4
+cwl:tool: echo.cwl
diff --git a/tests/override/echo-job.yml b/tests/override/echo-job.yml
new file mode 100644
index 0000000..b1ac4c4
--- /dev/null
+++ b/tests/override/echo-job.yml
@@ -0,0 +1 @@
+m1: zing
\ No newline at end of file
diff --git a/tests/override/echo-wf.cwl b/tests/override/echo-wf.cwl
new file mode 100644
index 0000000..a9f86af
--- /dev/null
+++ b/tests/override/echo-wf.cwl
@@ -0,0 +1,14 @@
+cwlVersion: v1.0
+class: Workflow
+inputs:
+ m1: string
+outputs:
+ out:
+ type: string
+ outputSource: step1/out
+steps:
+ step1:
+ in:
+ m1: m1
+ out: [out]
+ run: echo.cwl
diff --git a/tests/override/echo.cwl b/tests/override/echo.cwl
new file mode 100644
index 0000000..1c6c19a
--- /dev/null
+++ b/tests/override/echo.cwl
@@ -0,0 +1,19 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+ ShellCommandRequirement: {}
+hints:
+ EnvVarRequirement:
+ envDef:
+ MESSAGE: hello1
+inputs:
+ m1: string
+outputs:
+ - id: out
+ type: string
+ outputBinding:
+ glob: out.txt
+ loadContents: true
+ outputEval: $(self[0].contents)
+arguments: ["echo", "-n", $(inputs.m1), {shellQuote: false, valueFrom: "$MESSAGE"}]
+stdout: out.txt
diff --git a/tests/override/ov.yml b/tests/override/ov.yml
new file mode 100644
index 0000000..6a2d903
--- /dev/null
+++ b/tests/override/ov.yml
@@ -0,0 +1,5 @@
+cwltool:overrides:
+ echo.cwl:
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: hello2
diff --git a/tests/override/ov2.yml b/tests/override/ov2.yml
new file mode 100644
index 0000000..05c90ba
--- /dev/null
+++ b/tests/override/ov2.yml
@@ -0,0 +1,5 @@
+cwltool:overrides:
+ "echo-wf.cwl#step1":
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: hello5
diff --git a/tests/override/ov3.yml b/tests/override/ov3.yml
new file mode 100644
index 0000000..5ee06b1
--- /dev/null
+++ b/tests/override/ov3.yml
@@ -0,0 +1,5 @@
+cwltool:overrides:
+ "echo-wf.cwl":
+ - class: EnvVarRequirement
+ envDef:
+ MESSAGE: hello6
diff --git a/tests/test_ext.py b/tests/test_ext.py
index 6f690d3..76c8e08 100644
--- a/tests/test_ext.py
+++ b/tests/test_ext.py
@@ -6,7 +6,6 @@ import unittest
import pytest
import cwltool.expression as expr
-import cwltool.factory
import cwltool.pathmapper
import cwltool.process
import cwltool.workflow
diff --git a/tests/test_fetch.py b/tests/test_fetch.py
index e30fb47..cebcaf0 100644
--- a/tests/test_fetch.py
+++ b/tests/test_fetch.py
@@ -27,7 +27,7 @@ inputs: []
outputs: []
"""
else:
- raise RuntimeError("Not foo.cwl")
+ raise RuntimeError("Not foo.cwl, was %s" % url)
def check_exists(self, url): # type: (unicode) -> bool
if url == "baz:bar/foo.cwl":
@@ -46,7 +46,11 @@ outputs: []
return urllib.parse.urljoin(base, url)
def test_resolver(d, a):
- return "baz:bar/" + a
+ if a.startswith("baz:bar/"):
+ return a
+ else:
+ return "baz:bar/" + a
+
load_tool("foo.cwl", defaultMakeTool, resolver=test_resolver, fetcher_constructor=TestFetcher)
diff --git a/tests/test_override.py b/tests/test_override.py
new file mode 100644
index 0000000..d49410c
--- /dev/null
+++ b/tests/test_override.py
@@ -0,0 +1,66 @@
+from __future__ import absolute_import
+import unittest
+
+import cwltool.expression as expr
+import cwltool.pathmapper
+import cwltool.process
+import cwltool.workflow
+import pytest
+import json
+from cwltool.main import main
+from cwltool.utils import onWindows
+from six import StringIO
+
+from .util import get_data
+
+
+class TestOverride(unittest.TestCase):
+ @pytest.mark.skipif(onWindows(),
+ reason="Instance of Cwltool is used, On windows that invoke a default docker Container")
+ def test_overrides(self):
+ sio = StringIO()
+
+ self.assertEquals(main([get_data('tests/override/echo.cwl'),
+ get_data('tests/override/echo-job.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello1"}, json.loads(sio.getvalue()))
+
+ sio = StringIO()
+ self.assertEquals(main(["--overrides", get_data('tests/override/ov.yml'),
+ get_data('tests/override/echo.cwl'),
+ get_data('tests/override/echo-job.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello2"}, json.loads(sio.getvalue()))
+
+ sio = StringIO()
+ self.assertEquals(main([get_data('tests/override/echo.cwl'),
+ get_data('tests/override/echo-job-ov.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello3"}, json.loads(sio.getvalue()))
+
+ sio = StringIO()
+ self.assertEquals(main([get_data('tests/override/echo-job-ov2.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello4"}, json.loads(sio.getvalue()))
+
+
+ sio = StringIO()
+ self.assertEquals(main(["--overrides", get_data('tests/override/ov.yml'),
+ get_data('tests/override/echo-wf.cwl'),
+ get_data('tests/override/echo-job.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello2"}, json.loads(sio.getvalue()))
+
+ sio = StringIO()
+ self.assertEquals(main(["--overrides", get_data('tests/override/ov2.yml'),
+ get_data('tests/override/echo-wf.cwl'),
+ get_data('tests/override/echo-job.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello5"}, json.loads(sio.getvalue()))
+
+ sio = StringIO()
+ self.assertEquals(main(["--overrides", get_data('tests/override/ov3.yml'),
+ get_data('tests/override/echo-wf.cwl'),
+ get_data('tests/override/echo-job.yml')],
+ stdout=sio), 0)
+ self.assertEquals({"out": "zing hello6"}, json.loads(sio.getvalue()))
diff --git a/tests/test_pack.py b/tests/test_pack.py
index 34a14e6..728a1e0 100644
--- a/tests/test_pack.py
+++ b/tests/test_pack.py
@@ -1,22 +1,28 @@
from __future__ import absolute_import
+
import json
-import os
import unittest
+
+import os
from functools import partial
+import tempfile
+
+import pytest
+from six import StringIO
import cwltool.pack
-from cwltool.main import print_pack as print_pack
import cwltool.workflow
from cwltool.load_tool import fetch_document, validate_document
-from cwltool.main import makeRelative
+from cwltool.main import makeRelative, main, print_pack
from cwltool.pathmapper import adjustDirObjs, adjustFileObjs
-
+from cwltool.utils import onWindows
from .util import get_data
class TestPack(unittest.TestCase):
+ maxDiff = None
+
def test_pack(self):
- self.maxDiff = None
document_loader, workflowobj, uri = fetch_document(
get_data("tests/wf/revsort.cwl"))
@@ -38,13 +44,11 @@ class TestPack(unittest.TestCase):
def test_pack_missing_cwlVersion(self):
"""Test to ensure the generated pack output is not missing
the `cwlVersion` in case of single tool workflow and single step workflow"""
- # Since diff is longer than 3174 characters
- self.maxDiff = None
# Testing single tool workflow
document_loader, workflowobj, uri = fetch_document(
get_data("tests/wf/hello_single_tool.cwl"))
- document_loader, avsc_names, processobj, metadata, uri = validate_document(
+ document_loader, _, processobj, metadata, uri = validate_document(
document_loader, workflowobj, uri)
# generate pack output dict
packed = json.loads(print_pack(document_loader, processobj, uri, metadata))
@@ -54,9 +58,88 @@ class TestPack(unittest.TestCase):
# Testing single step workflow
document_loader, workflowobj, uri = fetch_document(
get_data("tests/wf/hello-workflow.cwl"))
- document_loader, avsc_names, processobj, metadata, uri = validate_document(
+ document_loader, _, processobj, metadata, uri = validate_document(
document_loader, workflowobj, uri)
# generate pack output dict
packed = json.loads(print_pack(document_loader, processobj, uri, metadata))
self.assertEqual('v1.0', packed["cwlVersion"])
+
+ def test_pack_idempotence_tool(self):
+ """Test to ensure that pack produces exactly the same document for
+ an already packed document"""
+
+ # Testing single tool
+ self._pack_idempotently("tests/wf/hello_single_tool.cwl")
+
+ def test_pack_idempotence_workflow(self):
+ """Test to ensure that pack produces exactly the same document for
+ an already packed document"""
+
+ # Testing workflow
+ self._pack_idempotently("tests/wf/count-lines1-wf.cwl")
+
+ def _pack_idempotently(self, document):
+ document_loader, workflowobj, uri = fetch_document(
+ get_data(document))
+ document_loader, avsc_names, processobj, metadata, uri = validate_document(
+ document_loader, workflowobj, uri)
+ # generate pack output dict
+ packed = json.loads(print_pack(document_loader, processobj, uri, metadata))
+
+ document_loader, workflowobj, uri2 = fetch_document(packed)
+ document_loader, avsc_names, processobj, metadata, uri2 = validate_document(
+ document_loader, workflowobj, uri)
+ double_packed = json.loads(print_pack(document_loader, processobj, uri2, metadata))
+ self.assertEqual(packed, double_packed)
+
+ @pytest.mark.skipif(onWindows(),
+ reason="Instance of cwltool is used, on Windows it invokes a default docker container"
+ "which is not supported on AppVeyor")
+ def test_packed_workflow_execution(self):
+ test_wf = "tests/wf/count-lines1-wf.cwl"
+ test_wf_job = "tests/wf/wc-job.json"
+ document_loader, workflowobj, uri = fetch_document(
+ get_data(test_wf))
+ document_loader, avsc_names, processobj, metadata, uri = validate_document(
+ document_loader, workflowobj, uri)
+ packed = json.loads(print_pack(document_loader, processobj, uri, metadata))
+ temp_packed_path = tempfile.mkstemp()[1]
+ with open(temp_packed_path, 'w') as f:
+ json.dump(packed, f)
+ normal_output = StringIO()
+ packed_output = StringIO()
+ self.assertEquals(main(['--debug', get_data(temp_packed_path),
+ get_data(test_wf_job)],
+ stdout=packed_output), 0)
+ self.assertEquals(main([get_data(test_wf),
+ get_data(test_wf_job)],
+ stdout=normal_output), 0)
+ self.assertEquals(json.loads(packed_output.getvalue()), json.loads(normal_output.getvalue()))
+ os.remove(temp_packed_path)
+
+ @pytest.mark.skipif(onWindows(),
+ reason="Instance of cwltool is used, on Windows it invokes a default docker container"
+ "which is not supported on AppVeyor")
+ def test_preserving_namespaces(self):
+ test_wf = "tests/wf/formattest.cwl"
+ test_wf_job = "tests/wf/formattest-job.json"
+ document_loader, workflowobj, uri = fetch_document(
+ get_data(test_wf))
+ document_loader, avsc_names, processobj, metadata, uri = validate_document(
+ document_loader, workflowobj, uri)
+ packed = json.loads(print_pack(document_loader, processobj, uri, metadata))
+ assert "$namespaces" in packed
+ temp_packed_path = tempfile.mkstemp()[1]
+ with open(temp_packed_path, 'w') as f:
+ json.dump(packed, f)
+ normal_output = StringIO()
+ packed_output = StringIO()
+ self.assertEquals(main(['--debug', get_data(temp_packed_path),
+ get_data(test_wf_job)],
+ stdout=packed_output), 0)
+ self.assertEquals(main([get_data(test_wf),
+ get_data(test_wf_job)],
+ stdout=normal_output), 0)
+ self.assertEquals(json.loads(packed_output.getvalue()), json.loads(normal_output.getvalue()))
+ os.remove(temp_packed_path)
diff --git a/tests/wf/count-lines1-wf.cwl b/tests/wf/count-lines1-wf.cwl
new file mode 100644
index 0000000..77cbf3a
--- /dev/null
+++ b/tests/wf/count-lines1-wf.cwl
@@ -0,0 +1,25 @@
+#!/usr/bin/env cwl-runner
+class: Workflow
+cwlVersion: v1.0
+
+inputs:
+ file1:
+ type: File
+
+outputs:
+ count_output:
+ type: int
+ outputSource: step2/output
+
+steps:
+ step1:
+ run: wc-tool.cwl
+ in:
+ file1: file1
+ out: [output]
+
+ step2:
+ run: parseInt-tool.cwl
+ in:
+ file1: step1/output
+ out: [output]
diff --git a/tests/wf/formattest-job.json b/tests/wf/formattest-job.json
new file mode 100644
index 0000000..0ff0240
--- /dev/null
+++ b/tests/wf/formattest-job.json
@@ -0,0 +1,7 @@
+{
+ "input": {
+ "class": "File",
+ "location": "whale.txt",
+ "format": "edam:format_2330"
+ }
+}
diff --git a/tests/wf/formattest.cwl b/tests/wf/formattest.cwl
new file mode 100644
index 0000000..19168e8
--- /dev/null
+++ b/tests/wf/formattest.cwl
@@ -0,0 +1,20 @@
+$namespaces:
+ edam: "http://edamontology.org/"
+cwlVersion: v1.0
+class: CommandLineTool
+doc: "Reverse each line using the `rev` command"
+inputs:
+ input:
+ type: File
+ inputBinding: {}
+ format: edam:format_2330
+
+outputs:
+ output:
+ type: File
+ outputBinding:
+ glob: output.txt
+ format: edam:format_2330
+
+baseCommand: rev
+stdout: output.txt
\ No newline at end of file
diff --git a/tests/wf/parseInt-tool.cwl b/tests/wf/parseInt-tool.cwl
new file mode 100644
index 0000000..42f166b
--- /dev/null
+++ b/tests/wf/parseInt-tool.cwl
@@ -0,0 +1,16 @@
+#!/usr/bin/env cwl-runner
+
+class: ExpressionTool
+requirements:
+ - class: InlineJavascriptRequirement
+cwlVersion: v1.0
+
+inputs:
+ file1:
+ type: File
+ inputBinding: { loadContents: true }
+
+outputs:
+ output: int
+
+expression: "$({'output': parseInt(inputs.file1.contents)})"
diff --git a/tests/wf/wc-job.json b/tests/wf/wc-job.json
new file mode 100644
index 0000000..598568d
--- /dev/null
+++ b/tests/wf/wc-job.json
@@ -0,0 +1,6 @@
+{
+ "file1": {
+ "class": "File",
+ "location": "whale.txt"
+ }
+}
diff --git a/tests/wf/wc-tool.cwl b/tests/wf/wc-tool.cwl
new file mode 100644
index 0000000..1655854
--- /dev/null
+++ b/tests/wf/wc-tool.cwl
@@ -0,0 +1,17 @@
+#!/usr/bin/env cwl-runner
+
+class: CommandLineTool
+cwlVersion: v1.0
+
+inputs:
+ file1: File
+
+outputs:
+ output:
+ type: File
+ outputBinding: { glob: output }
+
+baseCommand: [wc, -l]
+
+stdin: $(inputs.file1.path)
+stdout: output
diff --git a/tests/wf/whale.txt b/tests/wf/whale.txt
new file mode 100644
index 0000000..425d1ed
--- /dev/null
+++ b/tests/wf/whale.txt
@@ -0,0 +1,16 @@
+Call me Ishmael. Some years ago--never mind how long precisely--having
+little or no money in my purse, and nothing particular to interest me on
+shore, I thought I would sail about a little and see the watery part of
+the world. It is a way I have of driving off the spleen and regulating
+the circulation. Whenever I find myself growing grim about the mouth;
+whenever it is a damp, drizzly November in my soul; whenever I find
+myself involuntarily pausing before coffin warehouses, and bringing up
+the rear of every funeral I meet; and especially whenever my hypos get
+such an upper hand of me, that it requires a strong moral principle to
+prevent me from deliberately stepping into the street, and methodically
+knocking people's hats off--then, I account it high time to get to sea
+as soon as I can. This is my substitute for pistol and ball. With a
+philosophical flourish Cato throws himself upon his sword; I quietly
+take to the ship. There is nothing surprising in this. If they but knew
+it, almost all men in their degree, some time or other, cherish very
+nearly the same feelings towards the ocean with me.
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/cwltool.git
More information about the debian-med-commit
mailing list