[med-svn] [cwltool] 01/01: New upstream version 1.0.20171107133715
Andreas Tille
tille at debian.org
Wed Dec 6 15:59:24 UTC 2017
This is an automated email from the git hooks/post-receive script.
tille pushed a commit to annotated tag upstream/1.0.20171107133715
in repository cwltool.
commit d827d3c931525d0456949ca3b2ba709aeb04e78c
Author: Andreas Tille <tille at debian.org>
Date: Wed Dec 6 16:28:57 2017 +0100
New upstream version 1.0.20171107133715
---
MANIFEST.in | 1 +
PKG-INFO | 135 +++++----
README.rst | 132 ++++++---
cwltool.egg-info/PKG-INFO | 135 +++++----
cwltool.egg-info/SOURCES.txt | 29 +-
cwltool.egg-info/pbr.json | 1 -
cwltool.egg-info/requires.txt | 16 +-
cwltool/builder.py | 12 +-
cwltool/cwlNodeEngine.js | 14 +-
cwltool/cwlNodeEngineJSConsole.js | 32 +++
cwltool/cwlrdf.py | 6 +-
cwltool/draft2tool.py | 93 +++---
cwltool/expression.py | 24 +-
cwltool/job.py | 191 ++++++++-----
cwltool/load_tool.py | 5 +-
cwltool/main.py | 18 +-
cwltool/mutation.py | 4 +
cwltool/pathmapper.py | 21 +-
cwltool/process.py | 20 +-
cwltool/sandboxjs.py | 71 +++--
cwltool/schemas/draft-3/index.md | 0
.../salad/schema_salad/metaschema/metaschema2.yml | 317 ---------------------
cwltool/workflow.py | 8 +-
setup.cfg | 3 +-
setup.py | 5 +-
tests/cat.cwl | 8 -
tests/cat2.cwl | 9 -
tests/env.cwl | 5 -
tests/item1.yml | 3 -
tests/output.txt | 16 --
tests/test_examples.py | 61 ++++
tests/test_rdfprint.py | 12 +
tests/value | 1 -
tests/wf/4c637b77-8130-4158-807c-ccc78ea7b563 | 1 -
tests/wf/c5b8320e-6e31-4bbb-8453-03b054b3254e | 1 -
tests/wf/conditional.cwl | 22 --
tests/wf/emptyscatter.cwl | 18 --
tests/wf/emptyscatter.yml | 1 -
tests/wf/foo1.txt | 1 -
tests/wf/foo3.txt | 1 -
tests/wf/fooa.txt | 1 -
tests/wf/foob.txt | 1 -
tests/wf/foofoo5.txt.txt | 1 -
tests/wf/js_output.cwl | 9 +
tests/wf/js_output_workflow.cwl | 15 +
tests/wf/loop.cwl | 36 ---
tests/wf/output.txt | 16 --
tests/wf/revsort-ovr2-job.json | 15 -
tests/wf/value | 1 -
tests/wf/whale.txt | 16 --
tests/writeable.cwl | 11 -
51 files changed, 707 insertions(+), 868 deletions(-)
diff --git a/MANIFEST.in b/MANIFEST.in
index 2808c92..47e3080 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -17,6 +17,7 @@ include cwltool/schemas/v1.1.0-dev1/*.md
include cwltool/schemas/v1.1.0-dev1/salad/schema_salad/metaschema/*.yml
include cwltool/schemas/v1.1.0-dev1/salad/schema_salad/metaschema/*.md
include cwltool/cwlNodeEngine.js
+include cwltool/cwlNodeEngineJSConsole.js
include cwltool/extensions.yml
global-exclude *~
global-exclude *.pyc
diff --git a/PKG-INFO b/PKG-INFO
index 059a944..d3ce41e 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,54 +1,67 @@
Metadata-Version: 1.1
Name: cwltool
-Version: 1.0.20170810192106
+Version: 1.0.20171107133715
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Author: Common workflow language working group
Author-email: common-workflow-language at googlegroups.com
License: UNKNOWN
Download-URL: https://github.com/common-workflow-language/cwltool
+Description-Content-Type: UNKNOWN
Description: ==================================================================
- Common workflow language tool description reference implementation
+ Common Workflow Language tool description reference implementation
==================================================================
- CWL Conformance test: |Build Status|
-
- Travis: |Unix Build Status|
+ CWL conformance tests: |Build Status| Travis CI: |Unix Build Status|
.. |Unix Build Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/cwltool
This is the reference implementation of the Common Workflow Language. It is
- intended to be feature complete and provide comprehensive validation of CWL
+ intended to feature complete and provide comprehensive validation of CWL
files as well as provide other tools related to working with CWL.
- This is written and tested for Python 2.7.
+ This is written and tested for Python ``2.7 and 3.x {x = 3, 4, 5, 6}``
- The reference implementation consists of two packages. The "cwltool" package
+ The reference implementation consists of two packages. The ``cwltool`` package
is the primary Python module containing the reference implementation in the
- "cwltool" module and console executable by the same name.
+ ``cwltool`` module and console executable by the same name.
- The "cwlref-runner" package is optional and provides an additional entry point
- under the alias "cwl-runner", which is the implementation-agnostic name for the
+ The ``cwlref-runner`` package is optional and provides an additional entry point
+ under the alias ``cwl-runner``, which is the implementation-agnostic name for the
default CWL interpreter installed on a host.
Install
-------
- Installing the official package from PyPi (will install "cwltool" package as
- well)::
+ It is highly recommended to setup virtual environment before installing `cwltool`:
+
+ .. code:: bash
+
+ virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
+ source venv/bin/activate # Activate environment before installing `cwltool`
+
+ 1. Installing the official package from PyPi (will install "cwltool" package as
+ well)
+
+ .. code:: bash
pip install cwlref-runner
- If installing alongside another CWL implementation then::
+ If installing alongside another CWL implementation then
+
+ .. code:: bash
pip install cwltool
- To install from source::
+ 2. To install from source
+
+ .. code:: bash
- git clone https://github.com/common-workflow-language/cwltool.git
- cd cwltool && python setup.py install
- cd cwlref-runner && python setup.py install # co-installing? skip this
+ git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
+ cd cwltool # Switch to source directory
+ pip install . # Install `cwltool` from source
+ cwltool --version # Check if the installation works correctly
Remember, if co-installing multiple CWL implementations then you need to
maintain which implementation ``cwl-runner`` points to via a symbolic file
@@ -59,9 +72,15 @@ Description: ==================================================================
- Running basic tests ``(/tests)``:
- .. code:: bash
+ We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
+ to run various tests in all supported Python environments.
+ You can run the test suite by simply running the following in the terminal:
+ ``pip install tox; tox``
- python setup.py test
+ List of all environment can be seen using:
+ ``tox --listenvs``
+ and running a specfic test env using:
+ ``tox -e <env name>``
- Running the entire suite of CWL conformance tests:
@@ -132,13 +151,17 @@ Description: ==================================================================
Import as a module
------------------
- Add::
+ Add
+
+ .. code:: python
import cwltool
to your script.
- The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory::
+ The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
+
+ .. code:: python
import cwltool.factory
fac = cwltool.factory.Factory()
@@ -151,7 +174,7 @@ Description: ==================================================================
Leveraging SoftwareRequirements (Beta)
--------------------------------------
- CWL tools may be decoarated with ``SoftwareRequirement`` hints that cwltool
+ CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
may in turn use to resolve to packages in various package managers or
dependency management systems such as `Environment Modules
<http://modules.sourceforge.net/>`__.
@@ -180,8 +203,8 @@ Description: ==================================================================
- r93
Now imagine deploying cwltool on a cluster with Software Modules installed
- and that a ``seqtk`` module is avaialble at version ``r93``. This means cluster
- users likely won't have the ``seqtk`` the binary on their ``PATH`` by default but after
+ and that a ``seqtk`` module is available at version ``r93``. This means cluster
+ users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
available on the ``PATH``. A simple dependency resolvers configuration file, called
``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
@@ -197,10 +220,11 @@ Description: ==================================================================
is required for all plugins. The available plugins and the parameters
available for each are documented (incompletely) `here
<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
- Unfortunately, this documentation is in the context of Galaxy tool ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
+ Unfortunately, this documentation is in the context of Galaxy tool
+ ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
cwltool is distributed with an example of such seqtk tool and sample corresponding
- job. It could executed from the cwltool root using a dependency resolvers
+ job. It could executed from the cwltool root using a dependency resolvers
configuration file such as the above one using the command::
cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
@@ -217,8 +241,8 @@ Description: ==================================================================
"Galaxy packages" are a lighter weight alternative to Environment Modules that are
really just defined by a way to lay out directories into packages and versions
to find little scripts that are sourced to modify the environment. They have
- been used for years in Galaxy community to adapt Galaxy tools to cluster
- environments but require neither knowledge of Galaxy nor any special tools to
+ been used for years in Galaxy community to adapt Galaxy tools to cluster
+ environments but require neither knowledge of Galaxy nor any special tools to
setup. These should work just fine for CWL tools.
The cwltool source code repository's test directory is setup with a very simple
@@ -247,7 +271,7 @@ Description: ==================================================================
the corresponding tool. That ``env.sh`` script is only responsible for modifying
the job's ``PATH`` to add the required binaries.
- This is a full example that works since resolving "Galaxy packages" has no
+ This is a full example that works since resolving "Galaxy packages" has no
external requirements. Try it out by executing the following command from cwltool's
root directory::
@@ -311,7 +335,7 @@ Description: ==================================================================
on both Linux and Mac OS X.
The Conda plugin can be endlessly configured, but a sensible set of defaults
- that has proven a powerful stack for dependency management within the Galaxy tool
+ that has proven a powerful stack for dependency management within the Galaxy tool
development ecosystem can be enabled by simply passing cwltool the
``--beta-conda-dependencies`` flag.
@@ -355,41 +379,41 @@ Description: ==================================================================
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
- Cwltool control flow
- --------------------
+ CWL Tool Control Flow
+ ---------------------
Technical outline of how cwltool works internally, for maintainers.
- #. Use CWL `load_tool()` to load document.
+ #. Use CWL ``load_tool()`` to load document.
#. Fetches the document from file or URL
#. Applies preprocessing (syntax/identifier expansion and normalization)
#. Validates the document based on cwlVersion
#. If necessary, updates the document to latest spec
- #. Constructs a Process object using `make_tool()` callback. This yields a
+ #. Constructs a Process object using ``make_tool()``` callback. This yields a
CommandLineTool, Workflow, or ExpressionTool. For workflows, this
recursively constructs each workflow step.
#. To construct custom types for CommandLineTool, Workflow, or
- ExpressionTool, provide a custom `make_tool()`
+ ExpressionTool, provide a custom ``make_tool()``
- #. Iterate on the `job()` method of the Process object to get back runnable jobs.
+ #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
- #. `job()` is a generator method (uses the Python iterator protocol)
- #. Each time the `job()` method is invoked in an iteration, it returns one
- of: a runnable item (an object with a `run()` method), `None` (indicating
+ #. ``job()`` is a generator method (uses the Python iterator protocol)
+ #. Each time the ``job()`` method is invoked in an iteration, it returns one
+ of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
there is currently no work ready to run) or end of iteration (indicating
the process is complete.)
- #. Invoke the runnable item by calling `run()`. This runs the tool and gets output.
+ #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
#. Output of a process is reported by an output callback.
- #. `job()` may be iterated over multiple times. It will yield all the work
+ #. ``job()`` may be iterated over multiple times. It will yield all the work
that is currently ready to run and then yield None.
- #. "Workflow" objects create a corresponding "WorkflowJob" and "WorkflowJobStep" objects to hold the workflow state for the duration of the job invocation.
+ #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
#. The WorkflowJob iterates over each WorkflowJobStep and determines if the
inputs the step are ready.
#. When a step is ready, it constructs an input object for that step and
- iterates on the `job()` method of the workflow job step.
+ iterates on the ``job()`` method of the workflow job step.
#. Each runnable item is yielded back up to top level run loop
#. When a step job completes and receives an output callback, the
job outputs are assigned to the output of the workflow step.
@@ -397,10 +421,10 @@ Description: ==================================================================
workflow output, intermediate directories are deleted, and the output
callback for the workflow is called.
- #. "CommandLineTool" job() objects yield a single runnable object.
+ #. ``CommandLineTool`` job() objects yield a single runnable object.
- #. The CommandLineTool `job()` method calls `makeJobRunner()` to create a
- `CommandLineJob` object
+ #. The CommandLineTool ``job()`` method calls ``makeJobRunner()`` to create a
+ ``CommandLineJob`` object
#. The job method configures the CommandLineJob object by setting public
attributes
#. The job method iterates over file and directories inputs to the
@@ -411,7 +435,7 @@ Description: ==================================================================
#. Files are staged to targets paths using either Docker volume binds (when
using containers) or symlinks (if not). This staging step enables files
to be logically rearranged or renamed independent of their source layout.
- #. The run() method of CommandLineJob executes the command line tool or
+ #. The ``run()`` method of CommandLineJob executes the command line tool or
Docker container, waits for it to complete, collects output, and makes
the output callback.
@@ -487,6 +511,21 @@ Description: ==================================================================
Handler object for logging.
+ Running user-space implementations of Docker
+ --------------------------------------------
+
+ Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+ Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+ For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+ Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+ ```
+ cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+ ```
+
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
diff --git a/README.rst b/README.rst
index 369504e..e44935e 100644
--- a/README.rst
+++ b/README.rst
@@ -1,45 +1,57 @@
==================================================================
-Common workflow language tool description reference implementation
+Common Workflow Language tool description reference implementation
==================================================================
-CWL Conformance test: |Build Status|
-
-Travis: |Unix Build Status|
+CWL conformance tests: |Build Status| Travis CI: |Unix Build Status|
.. |Unix Build Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/cwltool
This is the reference implementation of the Common Workflow Language. It is
-intended to be feature complete and provide comprehensive validation of CWL
+intended to feature complete and provide comprehensive validation of CWL
files as well as provide other tools related to working with CWL.
-This is written and tested for Python 2.7.
+This is written and tested for Python ``2.7 and 3.x {x = 3, 4, 5, 6}``
-The reference implementation consists of two packages. The "cwltool" package
+The reference implementation consists of two packages. The ``cwltool`` package
is the primary Python module containing the reference implementation in the
-"cwltool" module and console executable by the same name.
+``cwltool`` module and console executable by the same name.
-The "cwlref-runner" package is optional and provides an additional entry point
-under the alias "cwl-runner", which is the implementation-agnostic name for the
+The ``cwlref-runner`` package is optional and provides an additional entry point
+under the alias ``cwl-runner``, which is the implementation-agnostic name for the
default CWL interpreter installed on a host.
Install
-------
-Installing the official package from PyPi (will install "cwltool" package as
-well)::
+It is highly recommended to setup virtual environment before installing `cwltool`:
+
+.. code:: bash
+
+ virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
+ source venv/bin/activate # Activate environment before installing `cwltool`
+
+1. Installing the official package from PyPi (will install "cwltool" package as
+well)
+
+.. code:: bash
pip install cwlref-runner
-If installing alongside another CWL implementation then::
+If installing alongside another CWL implementation then
+
+.. code:: bash
pip install cwltool
-To install from source::
+2. To install from source
+
+.. code:: bash
- git clone https://github.com/common-workflow-language/cwltool.git
- cd cwltool && python setup.py install
- cd cwlref-runner && python setup.py install # co-installing? skip this
+ git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
+ cd cwltool # Switch to source directory
+ pip install . # Install `cwltool` from source
+ cwltool --version # Check if the installation works correctly
Remember, if co-installing multiple CWL implementations then you need to
maintain which implementation ``cwl-runner`` points to via a symbolic file
@@ -50,9 +62,15 @@ Running tests locally
- Running basic tests ``(/tests)``:
-.. code:: bash
+We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
+to run various tests in all supported Python environments.
+You can run the test suite by simply running the following in the terminal:
+``pip install tox; tox``
- python setup.py test
+List of all environment can be seen using:
+``tox --listenvs``
+and running a specfic test env using:
+``tox -e <env name>``
- Running the entire suite of CWL conformance tests:
@@ -123,13 +141,17 @@ For this example, grab the test.json (and input file) from https://github.com/Ca
Import as a module
------------------
-Add::
+Add
+
+.. code:: python
import cwltool
to your script.
-The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory::
+The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
+
+.. code:: python
import cwltool.factory
fac = cwltool.factory.Factory()
@@ -142,7 +164,7 @@ The easiest way to use cwltool to run a tool or workflow from Python is to use a
Leveraging SoftwareRequirements (Beta)
--------------------------------------
-CWL tools may be decoarated with ``SoftwareRequirement`` hints that cwltool
+CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
may in turn use to resolve to packages in various package managers or
dependency management systems such as `Environment Modules
<http://modules.sourceforge.net/>`__.
@@ -171,8 +193,8 @@ following ``hint`` definition for an example CWL tool.
- r93
Now imagine deploying cwltool on a cluster with Software Modules installed
-and that a ``seqtk`` module is avaialble at version ``r93``. This means cluster
-users likely won't have the ``seqtk`` the binary on their ``PATH`` by default but after
+and that a ``seqtk`` module is available at version ``r93``. This means cluster
+users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
available on the ``PATH``. A simple dependency resolvers configuration file, called
``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
@@ -188,10 +210,11 @@ for the plugin above, this is ``type`` and defines the plugin type. This paramet
is required for all plugins. The available plugins and the parameters
available for each are documented (incompletely) `here
<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
-Unfortunately, this documentation is in the context of Galaxy tool ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
+Unfortunately, this documentation is in the context of Galaxy tool
+``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
cwltool is distributed with an example of such seqtk tool and sample corresponding
-job. It could executed from the cwltool root using a dependency resolvers
+job. It could executed from the cwltool root using a dependency resolvers
configuration file such as the above one using the command::
cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
@@ -208,8 +231,8 @@ the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
"Galaxy packages" are a lighter weight alternative to Environment Modules that are
really just defined by a way to lay out directories into packages and versions
to find little scripts that are sourced to modify the environment. They have
-been used for years in Galaxy community to adapt Galaxy tools to cluster
-environments but require neither knowledge of Galaxy nor any special tools to
+been used for years in Galaxy community to adapt Galaxy tools to cluster
+environments but require neither knowledge of Galaxy nor any special tools to
setup. These should work just fine for CWL tools.
The cwltool source code repository's test directory is setup with a very simple
@@ -238,7 +261,7 @@ Then cwltool will simply find that ``env.sh`` file and source it before executin
the corresponding tool. That ``env.sh`` script is only responsible for modifying
the job's ``PATH`` to add the required binaries.
-This is a full example that works since resolving "Galaxy packages" has no
+This is a full example that works since resolving "Galaxy packages" has no
external requirements. Try it out by executing the following command from cwltool's
root directory::
@@ -302,7 +325,7 @@ user, install its own Conda environment and manage multiple versions of Conda pa
on both Linux and Mac OS X.
The Conda plugin can be endlessly configured, but a sensible set of defaults
-that has proven a powerful stack for dependency management within the Galaxy tool
+that has proven a powerful stack for dependency management within the Galaxy tool
development ecosystem can be enabled by simply passing cwltool the
``--beta-conda-dependencies`` flag.
@@ -346,41 +369,41 @@ at the following links:
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
-Cwltool control flow
---------------------
+CWL Tool Control Flow
+---------------------
Technical outline of how cwltool works internally, for maintainers.
-#. Use CWL `load_tool()` to load document.
+#. Use CWL ``load_tool()`` to load document.
#. Fetches the document from file or URL
#. Applies preprocessing (syntax/identifier expansion and normalization)
#. Validates the document based on cwlVersion
#. If necessary, updates the document to latest spec
- #. Constructs a Process object using `make_tool()` callback. This yields a
+ #. Constructs a Process object using ``make_tool()``` callback. This yields a
CommandLineTool, Workflow, or ExpressionTool. For workflows, this
recursively constructs each workflow step.
#. To construct custom types for CommandLineTool, Workflow, or
- ExpressionTool, provide a custom `make_tool()`
+ ExpressionTool, provide a custom ``make_tool()``
-#. Iterate on the `job()` method of the Process object to get back runnable jobs.
+#. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
- #. `job()` is a generator method (uses the Python iterator protocol)
- #. Each time the `job()` method is invoked in an iteration, it returns one
- of: a runnable item (an object with a `run()` method), `None` (indicating
+ #. ``job()`` is a generator method (uses the Python iterator protocol)
+ #. Each time the ``job()`` method is invoked in an iteration, it returns one
+ of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
there is currently no work ready to run) or end of iteration (indicating
the process is complete.)
- #. Invoke the runnable item by calling `run()`. This runs the tool and gets output.
+ #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
#. Output of a process is reported by an output callback.
- #. `job()` may be iterated over multiple times. It will yield all the work
+ #. ``job()`` may be iterated over multiple times. It will yield all the work
that is currently ready to run and then yield None.
-#. "Workflow" objects create a corresponding "WorkflowJob" and "WorkflowJobStep" objects to hold the workflow state for the duration of the job invocation.
+#. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
#. The WorkflowJob iterates over each WorkflowJobStep and determines if the
inputs the step are ready.
#. When a step is ready, it constructs an input object for that step and
- iterates on the `job()` method of the workflow job step.
+ iterates on the ``job()`` method of the workflow job step.
#. Each runnable item is yielded back up to top level run loop
#. When a step job completes and receives an output callback, the
job outputs are assigned to the output of the workflow step.
@@ -388,10 +411,10 @@ Technical outline of how cwltool works internally, for maintainers.
workflow output, intermediate directories are deleted, and the output
callback for the workflow is called.
-#. "CommandLineTool" job() objects yield a single runnable object.
+#. ``CommandLineTool`` job() objects yield a single runnable object.
- #. The CommandLineTool `job()` method calls `makeJobRunner()` to create a
- `CommandLineJob` object
+ #. The CommandLineTool ``job()`` method calls ``makeJobRunner()`` to create a
+ ``CommandLineJob`` object
#. The job method configures the CommandLineJob object by setting public
attributes
#. The job method iterates over file and directories inputs to the
@@ -402,7 +425,7 @@ Technical outline of how cwltool works internally, for maintainers.
#. Files are staged to targets paths using either Docker volume binds (when
using containers) or symlinks (if not). This staging step enables files
to be logically rearranged or renamed independent of their source layout.
- #. The run() method of CommandLineJob executes the command line tool or
+ #. The ``run()`` method of CommandLineJob executes the command line tool or
Docker container, waits for it to complete, collects output, and makes
the output callback.
@@ -477,3 +500,18 @@ logger_handler
logging.Handler
Handler object for logging.
+
+Running user-space implementations of Docker
+--------------------------------------------
+
+Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+```
+cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+```
diff --git a/cwltool.egg-info/PKG-INFO b/cwltool.egg-info/PKG-INFO
index 059a944..d3ce41e 100644
--- a/cwltool.egg-info/PKG-INFO
+++ b/cwltool.egg-info/PKG-INFO
@@ -1,54 +1,67 @@
Metadata-Version: 1.1
Name: cwltool
-Version: 1.0.20170810192106
+Version: 1.0.20171107133715
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Author: Common workflow language working group
Author-email: common-workflow-language at googlegroups.com
License: UNKNOWN
Download-URL: https://github.com/common-workflow-language/cwltool
+Description-Content-Type: UNKNOWN
Description: ==================================================================
- Common workflow language tool description reference implementation
+ Common Workflow Language tool description reference implementation
==================================================================
- CWL Conformance test: |Build Status|
-
- Travis: |Unix Build Status|
+ CWL conformance tests: |Build Status| Travis CI: |Unix Build Status|
.. |Unix Build Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/cwltool
This is the reference implementation of the Common Workflow Language. It is
- intended to be feature complete and provide comprehensive validation of CWL
+ intended to feature complete and provide comprehensive validation of CWL
files as well as provide other tools related to working with CWL.
- This is written and tested for Python 2.7.
+ This is written and tested for Python ``2.7 and 3.x {x = 3, 4, 5, 6}``
- The reference implementation consists of two packages. The "cwltool" package
+ The reference implementation consists of two packages. The ``cwltool`` package
is the primary Python module containing the reference implementation in the
- "cwltool" module and console executable by the same name.
+ ``cwltool`` module and console executable by the same name.
- The "cwlref-runner" package is optional and provides an additional entry point
- under the alias "cwl-runner", which is the implementation-agnostic name for the
+ The ``cwlref-runner`` package is optional and provides an additional entry point
+ under the alias ``cwl-runner``, which is the implementation-agnostic name for the
default CWL interpreter installed on a host.
Install
-------
- Installing the official package from PyPi (will install "cwltool" package as
- well)::
+ It is highly recommended to setup virtual environment before installing `cwltool`:
+
+ .. code:: bash
+
+ virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
+ source venv/bin/activate # Activate environment before installing `cwltool`
+
+ 1. Installing the official package from PyPi (will install "cwltool" package as
+ well)
+
+ .. code:: bash
pip install cwlref-runner
- If installing alongside another CWL implementation then::
+ If installing alongside another CWL implementation then
+
+ .. code:: bash
pip install cwltool
- To install from source::
+ 2. To install from source
+
+ .. code:: bash
- git clone https://github.com/common-workflow-language/cwltool.git
- cd cwltool && python setup.py install
- cd cwlref-runner && python setup.py install # co-installing? skip this
+ git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
+ cd cwltool # Switch to source directory
+ pip install . # Install `cwltool` from source
+ cwltool --version # Check if the installation works correctly
Remember, if co-installing multiple CWL implementations then you need to
maintain which implementation ``cwl-runner`` points to via a symbolic file
@@ -59,9 +72,15 @@ Description: ==================================================================
- Running basic tests ``(/tests)``:
- .. code:: bash
+ We use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_
+ to run various tests in all supported Python environments.
+ You can run the test suite by simply running the following in the terminal:
+ ``pip install tox; tox``
- python setup.py test
+ List of all environment can be seen using:
+ ``tox --listenvs``
+ and running a specfic test env using:
+ ``tox -e <env name>``
- Running the entire suite of CWL conformance tests:
@@ -132,13 +151,17 @@ Description: ==================================================================
Import as a module
------------------
- Add::
+ Add
+
+ .. code:: python
import cwltool
to your script.
- The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory::
+ The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
+
+ .. code:: python
import cwltool.factory
fac = cwltool.factory.Factory()
@@ -151,7 +174,7 @@ Description: ==================================================================
Leveraging SoftwareRequirements (Beta)
--------------------------------------
- CWL tools may be decoarated with ``SoftwareRequirement`` hints that cwltool
+ CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
may in turn use to resolve to packages in various package managers or
dependency management systems such as `Environment Modules
<http://modules.sourceforge.net/>`__.
@@ -180,8 +203,8 @@ Description: ==================================================================
- r93
Now imagine deploying cwltool on a cluster with Software Modules installed
- and that a ``seqtk`` module is avaialble at version ``r93``. This means cluster
- users likely won't have the ``seqtk`` the binary on their ``PATH`` by default but after
+ and that a ``seqtk`` module is available at version ``r93``. This means cluster
+ users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
available on the ``PATH``. A simple dependency resolvers configuration file, called
``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
@@ -197,10 +220,11 @@ Description: ==================================================================
is required for all plugins. The available plugins and the parameters
available for each are documented (incompletely) `here
<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
- Unfortunately, this documentation is in the context of Galaxy tool ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
+ Unfortunately, this documentation is in the context of Galaxy tool
+ ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
cwltool is distributed with an example of such seqtk tool and sample corresponding
- job. It could executed from the cwltool root using a dependency resolvers
+ job. It could executed from the cwltool root using a dependency resolvers
configuration file such as the above one using the command::
cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
@@ -217,8 +241,8 @@ Description: ==================================================================
"Galaxy packages" are a lighter weight alternative to Environment Modules that are
really just defined by a way to lay out directories into packages and versions
to find little scripts that are sourced to modify the environment. They have
- been used for years in Galaxy community to adapt Galaxy tools to cluster
- environments but require neither knowledge of Galaxy nor any special tools to
+ been used for years in Galaxy community to adapt Galaxy tools to cluster
+ environments but require neither knowledge of Galaxy nor any special tools to
setup. These should work just fine for CWL tools.
The cwltool source code repository's test directory is setup with a very simple
@@ -247,7 +271,7 @@ Description: ==================================================================
the corresponding tool. That ``env.sh`` script is only responsible for modifying
the job's ``PATH`` to add the required binaries.
- This is a full example that works since resolving "Galaxy packages" has no
+ This is a full example that works since resolving "Galaxy packages" has no
external requirements. Try it out by executing the following command from cwltool's
root directory::
@@ -311,7 +335,7 @@ Description: ==================================================================
on both Linux and Mac OS X.
The Conda plugin can be endlessly configured, but a sensible set of defaults
- that has proven a powerful stack for dependency management within the Galaxy tool
+ that has proven a powerful stack for dependency management within the Galaxy tool
development ecosystem can be enabled by simply passing cwltool the
``--beta-conda-dependencies`` flag.
@@ -355,41 +379,41 @@ Description: ==================================================================
- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
- Cwltool control flow
- --------------------
+ CWL Tool Control Flow
+ ---------------------
Technical outline of how cwltool works internally, for maintainers.
- #. Use CWL `load_tool()` to load document.
+ #. Use CWL ``load_tool()`` to load document.
#. Fetches the document from file or URL
#. Applies preprocessing (syntax/identifier expansion and normalization)
#. Validates the document based on cwlVersion
#. If necessary, updates the document to latest spec
- #. Constructs a Process object using `make_tool()` callback. This yields a
+ #. Constructs a Process object using ``make_tool()``` callback. This yields a
CommandLineTool, Workflow, or ExpressionTool. For workflows, this
recursively constructs each workflow step.
#. To construct custom types for CommandLineTool, Workflow, or
- ExpressionTool, provide a custom `make_tool()`
+ ExpressionTool, provide a custom ``make_tool()``
- #. Iterate on the `job()` method of the Process object to get back runnable jobs.
+ #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
- #. `job()` is a generator method (uses the Python iterator protocol)
- #. Each time the `job()` method is invoked in an iteration, it returns one
- of: a runnable item (an object with a `run()` method), `None` (indicating
+ #. ``job()`` is a generator method (uses the Python iterator protocol)
+ #. Each time the ``job()`` method is invoked in an iteration, it returns one
+ of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
there is currently no work ready to run) or end of iteration (indicating
the process is complete.)
- #. Invoke the runnable item by calling `run()`. This runs the tool and gets output.
+ #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
#. Output of a process is reported by an output callback.
- #. `job()` may be iterated over multiple times. It will yield all the work
+ #. ``job()`` may be iterated over multiple times. It will yield all the work
that is currently ready to run and then yield None.
- #. "Workflow" objects create a corresponding "WorkflowJob" and "WorkflowJobStep" objects to hold the workflow state for the duration of the job invocation.
+ #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
#. The WorkflowJob iterates over each WorkflowJobStep and determines if the
inputs the step are ready.
#. When a step is ready, it constructs an input object for that step and
- iterates on the `job()` method of the workflow job step.
+ iterates on the ``job()`` method of the workflow job step.
#. Each runnable item is yielded back up to top level run loop
#. When a step job completes and receives an output callback, the
job outputs are assigned to the output of the workflow step.
@@ -397,10 +421,10 @@ Description: ==================================================================
workflow output, intermediate directories are deleted, and the output
callback for the workflow is called.
- #. "CommandLineTool" job() objects yield a single runnable object.
+ #. ``CommandLineTool`` job() objects yield a single runnable object.
- #. The CommandLineTool `job()` method calls `makeJobRunner()` to create a
- `CommandLineJob` object
+ #. The CommandLineTool ``job()`` method calls ``makeJobRunner()`` to create a
+ ``CommandLineJob`` object
#. The job method configures the CommandLineJob object by setting public
attributes
#. The job method iterates over file and directories inputs to the
@@ -411,7 +435,7 @@ Description: ==================================================================
#. Files are staged to targets paths using either Docker volume binds (when
using containers) or symlinks (if not). This staging step enables files
to be logically rearranged or renamed independent of their source layout.
- #. The run() method of CommandLineJob executes the command line tool or
+ #. The ``run()`` method of CommandLineJob executes the command line tool or
Docker container, waits for it to complete, collects output, and makes
the output callback.
@@ -487,6 +511,21 @@ Description: ==================================================================
Handler object for logging.
+ Running user-space implementations of Docker
+ --------------------------------------------
+
+ Some compute environments disallow user-space installation of Docker due to incompatiblities in libraries or to meet security requirements. The CWL reference supports using a user space implementation with the `--user-space-docker-cmd` option.
+
+ Example using `dx-docker` (https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images):
+
+ For use on Linux, install the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
+
+ Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
+
+ ```
+ cwltool --user-space-docker-cmd=dx-docker --outdir=/tmp/tmpidytmp v1.0/test-cwl-out2.cwl v1.0/empty.json
+ ```
+
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
diff --git a/cwltool.egg-info/SOURCES.txt b/cwltool.egg-info/SOURCES.txt
index 51d419e..f4bebd2 100644
--- a/cwltool.egg-info/SOURCES.txt
+++ b/cwltool.egg-info/SOURCES.txt
@@ -9,6 +9,7 @@ cwltool/__init__.py
cwltool/__main__.py
cwltool/builder.py
cwltool/cwlNodeEngine.js
+cwltool/cwlNodeEngineJSConsole.js
cwltool/cwlrdf.py
cwltool/docker.py
cwltool/docker_id.py
@@ -36,7 +37,6 @@ cwltool.egg-info/PKG-INFO
cwltool.egg-info/SOURCES.txt
cwltool.egg-info/dependency_links.txt
cwltool.egg-info/entry_points.txt
-cwltool.egg-info/pbr.json
cwltool.egg-info/requires.txt
cwltool.egg-info/top_level.txt
cwltool.egg-info/zip-safe
@@ -51,7 +51,6 @@ cwltool/schemas/draft-3/UserGuide.yml
cwltool/schemas/draft-3/Workflow.yml
cwltool/schemas/draft-3/concepts.md
cwltool/schemas/draft-3/contrib.md
-cwltool/schemas/draft-3/index.md
cwltool/schemas/draft-3/index.yml
cwltool/schemas/draft-3/intro.md
cwltool/schemas/draft-3/invocation.md
@@ -105,7 +104,6 @@ cwltool/schemas/v1.0/salad/schema_salad/metaschema/map_res_proc.yml
cwltool/schemas/v1.0/salad/schema_salad/metaschema/map_res_schema.yml
cwltool/schemas/v1.0/salad/schema_salad/metaschema/map_res_src.yml
cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema.yml
-cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema2.yml
cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema_base.yml
cwltool/schemas/v1.0/salad/schema_salad/metaschema/salad.md
cwltool/schemas/v1.0/salad/schema_salad/metaschema/typedsl_res.yml
@@ -151,15 +149,10 @@ cwltool/schemas/v1.1.0-dev1/salad/schema_salad/metaschema/vocab_res_src.yml
tests/2.fasta
tests/2.fastq
tests/__init__.py
-tests/cat.cwl
-tests/cat2.cwl
tests/echo-cwlrun-job.yaml
tests/echo-job.yaml
tests/echo.cwl
-tests/env.cwl
-tests/item1.yml
tests/listing-job.yml
-tests/output.txt
tests/random_lines.cwl
tests/random_lines_job.json
tests/random_lines_mapping.cwl
@@ -182,45 +175,33 @@ tests/test_http_input.py
tests/test_js_sandbox.py
tests/test_pack.py
tests/test_pathmapper.py
+tests/test_rdfprint.py
tests/test_relax_path_checks.py
tests/test_toolargparse.py
tests/util.py
-tests/value
-tests/writeable.cwl
tests/tmp1/tmp2/tmp3/.gitkeep
-tests/wf/4c637b77-8130-4158-807c-ccc78ea7b563
tests/wf/badout1.cwl
tests/wf/badout2.cwl
tests/wf/badout3.cwl
-tests/wf/c5b8320e-6e31-4bbb-8453-03b054b3254e
tests/wf/cat.cwl
-tests/wf/conditional.cwl
tests/wf/default_path.cwl
tests/wf/echo.cwl
tests/wf/empty.ttl
-tests/wf/emptyscatter.cwl
-tests/wf/emptyscatter.yml
tests/wf/expect_packed.cwl
-tests/wf/foo1.txt
-tests/wf/foo3.txt
-tests/wf/fooa.txt
-tests/wf/foob.txt
-tests/wf/foofoo5.txt.txt
tests/wf/hello-workflow.cwl
tests/wf/hello.txt
tests/wf/hello_single_tool.cwl
+tests/wf/js_output.cwl
+tests/wf/js_output_workflow.cwl
tests/wf/listing_deep.cwl
tests/wf/listing_none.cwl
tests/wf/listing_shallow.cwl
tests/wf/listing_v1_0.cwl
-tests/wf/loop.cwl
tests/wf/missing_cwlVersion.cwl
tests/wf/mut.cwl
tests/wf/mut2.cwl
tests/wf/mut3.cwl
-tests/wf/output.txt
tests/wf/revsort-job.json
-tests/wf/revsort-ovr2-job.json
tests/wf/revsort.cwl
tests/wf/revtool.cwl
tests/wf/scatterfail.cwl
@@ -230,7 +211,5 @@ tests/wf/updatedir_inplace.cwl
tests/wf/updateval.cwl
tests/wf/updateval.py
tests/wf/updateval_inplace.cwl
-tests/wf/value
tests/wf/wffail.cwl
-tests/wf/whale.txt
tests/wf/wrong_cwlVersion.cwl
\ No newline at end of file
diff --git a/cwltool.egg-info/pbr.json b/cwltool.egg-info/pbr.json
deleted file mode 100644
index 75a451d..0000000
--- a/cwltool.egg-info/pbr.json
+++ /dev/null
@@ -1 +0,0 @@
-{"is_release": false, "git_version": "0bb2f210"}
\ No newline at end of file
diff --git a/cwltool.egg-info/requires.txt b/cwltool.egg-info/requires.txt
index df3e6f8..4ec6827 100644
--- a/cwltool.egg-info/requires.txt
+++ b/cwltool.egg-info/requires.txt
@@ -1,11 +1,11 @@
setuptools
-requests >= 2.12.4
-ruamel.yaml >= 0.12.4, < 0.15
-rdflib >= 4.2.2, < 4.3.0
-shellescape >= 3.4.1, < 3.5
-schema-salad >= 2.6, < 3
-typing >= 3.5.3
-six >= 1.8.0
+requests>=2.4.3
+ruamel.yaml<0.15,>=0.12.4
+rdflib<4.3.0,>=4.2.2
+shellescape<3.5,>=3.4.1
+schema-salad<3,>=2.6.20170927145003
+typing>=3.5.3
+six>=1.8.0
[deps]
-galaxy-lib >= 17.09.3
+galaxy-lib>=17.09.3
diff --git a/cwltool/builder.py b/cwltool/builder.py
index 20f67b2..b3a3d1d 100644
--- a/cwltool/builder.py
+++ b/cwltool/builder.py
@@ -51,6 +51,7 @@ class Builder(object):
self.stagedir = None # type: Text
self.make_fs_access = None # type: Type[StdFsAccess]
self.debug = False # type: bool
+ self.js_console = False # type: bool
self.mutation_manager = None # type: MutationManager
self.force_docker_pull = False # type: bool
@@ -62,8 +63,8 @@ class Builder(object):
self.job_script_provider = None # type: Any
def build_job_script(self, commands):
- # type: (List[bytes]) -> Text
- build_job_script_method = getattr(self.job_script_provider, "build_job_script", None) # type: Callable[[Builder, List[bytes]], Text]
+ # type: (List[Text]) -> Text
+ build_job_script_method = getattr(self.job_script_provider, "build_job_script", None) # type: Callable[[Builder, Union[List[str],List[Text]]], Text]
if build_job_script_method:
return build_job_script_method(self, commands)
else:
@@ -206,7 +207,7 @@ class Builder(object):
def generate_arg(self, binding): # type: (Dict[Text,Any]) -> List[Text]
value = binding.get("datum")
if "valueFrom" in binding:
- with SourceLine(binding, "valueFrom", WorkflowException):
+ with SourceLine(binding, "valueFrom", WorkflowException, _logger.isEnabledFor(logging.DEBUG)):
value = self.do_eval(binding["valueFrom"], context=value)
prefix = binding.get("prefix")
@@ -256,5 +257,6 @@ class Builder(object):
self.resources,
context=context, pull_image=pull_image,
timeout=self.timeout,
- force_docker_pull=self.force_docker_pull,
- debug=self.debug)
+ debug=self.debug,
+ js_console=self.js_console,
+ force_docker_pull=self.force_docker_pull)
diff --git a/cwltool/cwlNodeEngine.js b/cwltool/cwlNodeEngine.js
index ce43516..b3b6ff8 100755
--- a/cwltool/cwlNodeEngine.js
+++ b/cwltool/cwlNodeEngine.js
@@ -5,9 +5,17 @@ process.stdin.on("data", function(chunk) {
incoming += chunk;
var i = incoming.indexOf("\n");
if (i > -1) {
- var fn = JSON.parse(incoming.substr(0, i));
- incoming = incoming.substr(i+1);
- process.stdout.write(JSON.stringify(require("vm").runInNewContext(fn, {})) + "\n");
+ try{
+ var fn = JSON.parse(incoming.substr(0, i));
+ incoming = incoming.substr(i+1);
+ process.stdout.write(JSON.stringify(require("vm").runInNewContext(fn, {})) + "\n");
+ }
+ catch(e){
+ console.error(e)
+ }
+ /*strings to indicate the process has finished*/
+ console.log("r1cepzbhUTxtykz5XTC4");
+ console.error("r1cepzbhUTxtykz5XTC4");
}
});
process.stdin.on("end", process.exit);
diff --git a/cwltool/cwlNodeEngineJSConsole.js b/cwltool/cwlNodeEngineJSConsole.js
new file mode 100644
index 0000000..2ebe8bd
--- /dev/null
+++ b/cwltool/cwlNodeEngineJSConsole.js
@@ -0,0 +1,32 @@
+"use strict";
+function js_console_log(){
+ console.error("[log] "+require("util").format.apply(this, arguments).split("\n").join("\n[log] "));
+}
+function js_console_err(){
+ console.error("[err] "+require("util").format.apply(this, arguments).split("\n").join("\n[err] "));
+}
+process.stdin.setEncoding("utf8");
+var incoming = "";
+process.stdin.on("data", function(chunk) {
+ incoming += chunk;
+ var i = incoming.indexOf("\n");
+ if (i > -1) {
+ try{
+ var fn = JSON.parse(incoming.substr(0, i));
+ incoming = incoming.substr(i+1);
+ process.stdout.write(JSON.stringify(require("vm").runInNewContext(fn, {
+ console: {
+ log: js_console_log,
+ error: js_console_err
+ }
+ })) + "\n");
+ }
+ catch(e){
+ console.error(e)
+ }
+ /*strings to indicate the process has finished*/
+ console.log("r1cepzbhUTxtykz5XTC4");
+ console.error("r1cepzbhUTxtykz5XTC4");
+ }
+});
+process.stdin.on("end", process.exit);
diff --git a/cwltool/cwlrdf.py b/cwltool/cwlrdf.py
index fc662db..b957530 100644
--- a/cwltool/cwlrdf.py
+++ b/cwltool/cwlrdf.py
@@ -20,9 +20,9 @@ def gather(tool, ctx): # type: (Process, ContextType) -> Graph
return g
-def printrdf(wf, ctx, sr, stdout):
- # type: (Process, ContextType, Text, IO[Any]) -> None
- stdout.write(gather(wf, ctx).serialize(format=sr))
+def printrdf(wf, ctx, sr):
+ # type: (Process, ContextType, Text) -> Text
+ return gather(wf, ctx).serialize(format=sr).decode('utf-8')
def lastpart(uri): # type: (Any) -> Text
diff --git a/cwltool/draft2tool.py b/cwltool/draft2tool.py
index 682012c..bb56692 100644
--- a/cwltool/draft2tool.py
+++ b/cwltool/draft2tool.py
@@ -119,38 +119,35 @@ def revmap_file(builder, outdir, f):
# builder.outdir is the inner (container/compute node) output directory
# outdir is the outer (host/storage system) output directory
- if "location" in f:
+ if "location" in f and "path" not in f:
if f["location"].startswith("file://"):
- path = convert_pathsep_to_unix(uri_file_path(f["location"]))
- revmap_f = builder.pathmapper.reversemap(path)
-
- if revmap_f and not builder.pathmapper.mapper(revmap_f[0]).type.startswith("Writable"):
- f["basename"] = os.path.basename(path)
- f["location"] = revmap_f[1]
- elif path == builder.outdir:
- f["location"] = outdir
- elif path.startswith(builder.outdir):
- f["location"] = builder.fs_access.join(outdir, path[len(builder.outdir) + 1:])
- elif f["location"].startswith(outdir):
- revmap_f = builder.pathmapper.reversemap(builder.fs_access.join(builder.outdir, f["location"][len(outdir) + 1:]))
- if revmap_f and not builder.pathmapper.mapper(revmap_f[0]).type.startswith("Writable"):
- f["basename"] = os.path.basename(path)
- f["location"] = revmap_f[1]
- return f
+ f["path"] = convert_pathsep_to_unix(uri_file_path(f["location"]))
+ else:
+ return f
if "path" in f:
path = f["path"]
+ uripath = file_uri(path)
del f["path"]
+
+ if "basename" not in f:
+ f["basename"] = os.path.basename(path)
+
revmap_f = builder.pathmapper.reversemap(path)
- if revmap_f:
+
+ if revmap_f and not builder.pathmapper.mapper(revmap_f[0]).type.startswith("Writable"):
f["location"] = revmap_f[1]
- return f
- elif path.startswith(builder.outdir):
+ elif uripath == outdir or uripath.startswith(outdir+os.sep):
+ f["location"] = file_uri(path)
+ elif path == builder.outdir or path.startswith(builder.outdir+os.sep):
f["location"] = builder.fs_access.join(outdir, path[len(builder.outdir) + 1:])
- return f
+ elif not os.path.isabs(path):
+ f["location"] = builder.fs_access.join(outdir, path)
else:
raise WorkflowException(u"Output file path %s must be within designated output directory (%s) or an input "
u"file pass through." % (path, builder.outdir))
+ return f
+
raise WorkflowException(u"Output File object is missing both `location` and `path` fields: %s" % f)
@@ -247,7 +244,6 @@ class CommandLineTool(Process):
# type: (...) -> Generator[Union[JobBase, CallbackJob], None, None]
jobname = uniquename(kwargs.get("name", shortname(self.tool.get("id", "job"))))
-
if kwargs.get("cachedir"):
cacheargs = kwargs.copy()
cacheargs["outdir"] = "/out"
@@ -268,11 +264,15 @@ class CommandLineTool(Process):
dockerimg = docker_req.get("dockerImageId") or docker_req.get("dockerPull")
elif kwargs.get("default_container", None) is not None and kwargs.get("use_container"):
dockerimg = kwargs.get("default_container")
+ else:
+ dockerimg = None
if dockerimg:
cmdline = ["docker", "run", dockerimg] + cmdline
keydict = {u"cmdline": cmdline}
+ if "stdout" in self.tool:
+ keydict["stdout"] = self.tool["stdout"]
for location, f in cachebuilder.pathmapper.items():
if f.type == "File":
checksum = next((e['checksum'] for e in cachebuilder.files
@@ -349,7 +349,9 @@ class CommandLineTool(Process):
j.hints = self.hints
j.name = jobname
- if _logger.isEnabledFor(logging.DEBUG):
+ debug = _logger.isEnabledFor(logging.DEBUG)
+
+ if debug:
_logger.debug(u"[job %s] initializing from %s%s",
j.name,
self.tool.get("id", ""),
@@ -408,28 +410,28 @@ class CommandLineTool(Process):
self.updatePathmap(builder.outdir, builder.pathmapper, l)
visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
- if _logger.isEnabledFor(logging.DEBUG):
+ if debug:
_logger.debug(u"[job %s] path mappings is %s", j.name,
json.dumps({p: builder.pathmapper.mapper(p) for p in builder.pathmapper.files()}, indent=4))
if self.tool.get("stdin"):
- with SourceLine(self.tool, "stdin", validate.ValidationException):
+ with SourceLine(self.tool, "stdin", validate.ValidationException, debug):
j.stdin = builder.do_eval(self.tool["stdin"])
reffiles.append({"class": "File", "path": j.stdin})
if self.tool.get("stderr"):
- with SourceLine(self.tool, "stderr", validate.ValidationException):
+ with SourceLine(self.tool, "stderr", validate.ValidationException, debug):
j.stderr = builder.do_eval(self.tool["stderr"])
if os.path.isabs(j.stderr) or ".." in j.stderr:
raise validate.ValidationException("stderr must be a relative path, got '%s'" % j.stderr)
if self.tool.get("stdout"):
- with SourceLine(self.tool, "stdout", validate.ValidationException):
+ with SourceLine(self.tool, "stdout", validate.ValidationException, debug):
j.stdout = builder.do_eval(self.tool["stdout"])
if os.path.isabs(j.stdout) or ".." in j.stdout or not j.stdout:
raise validate.ValidationException("stdout must be a relative path, got '%s'" % j.stdout)
- if _logger.isEnabledFor(logging.DEBUG):
+ if debug:
_logger.debug(u"[job %s] command line bindings is %s", j.name, json.dumps(builder.bindings, indent=4))
dockerReq = self.get_requirement("DockerRequirement")[0]
@@ -508,37 +510,31 @@ class CommandLineTool(Process):
def collect_output_ports(self, ports, builder, outdir, compute_checksum=True, jobname="", readers=None):
# type: (Set[Dict[Text, Any]], Builder, Text, bool, Text, Dict[Text, Any]) -> Dict[Text, Union[Text, List[Any], Dict[Text, Any]]]
ret = {} # type: Dict[Text, Union[Text, List[Any], Dict[Text, Any]]]
+ debug = _logger.isEnabledFor(logging.DEBUG)
try:
fs_access = builder.make_fs_access(outdir)
custom_output = fs_access.join(outdir, "cwl.output.json")
if fs_access.exists(custom_output):
with fs_access.open(custom_output, "r") as f:
ret = json.load(f)
- if _logger.isEnabledFor(logging.DEBUG):
+ if debug:
_logger.debug(u"Raw output from %s: %s", custom_output, json.dumps(ret, indent=4))
else:
for i, port in enumerate(ports):
- with SourceLine(ports, i, WorkflowException):
- fragment = shortname(port["id"])
- try:
- ret[fragment] = self.collect_output(port, builder, outdir, fs_access,
- compute_checksum=compute_checksum)
- except Exception as e:
- _logger.debug(
- u"Error collecting output for parameter '%s'"
- % shortname(port["id"]), exc_info=True)
- raise WorkflowException(
+ def makeWorkflowException(msg):
+ return WorkflowException(
u"Error collecting output for parameter '%s':\n%s"
- % (shortname(port["id"]), indent(u(str(e)))))
-
+ % (shortname(port["id"]), msg))
+ with SourceLine(ports, i, makeWorkflowException, debug):
+ fragment = shortname(port["id"])
+ ret[fragment] = self.collect_output(port, builder, outdir, fs_access,
+ compute_checksum=compute_checksum)
if ret:
revmap = partial(revmap_file, builder, outdir)
adjustDirObjs(ret, trim_listing)
visit_class(ret, ("File", "Directory"), cast(Callable[[Any], Any], revmap))
visit_class(ret, ("File", "Directory"), remove_path)
normalizeFilesDirs(ret)
- if builder.mutation_manager:
- adjustFileObjs(ret, builder.mutation_manager.set_generation)
visit_class(ret, ("File", "Directory"), partial(check_valid_locations, fs_access))
if compute_checksum:
@@ -546,6 +542,8 @@ class CommandLineTool(Process):
validate.validate_ex(self.names.get_name("outputs_record_schema", ""), ret,
strict=False, logger=_logger_validation_warnings)
+ if ret is not None and builder.mutation_manager is not None:
+ adjustFileObjs(ret, builder.mutation_manager.set_generation)
return ret if ret is not None else {}
except validate.ValidationException as e:
raise WorkflowException("Error validating output record. " + Text(e) + "\n in " + json.dumps(ret, indent=4))
@@ -557,6 +555,7 @@ class CommandLineTool(Process):
def collect_output(self, schema, builder, outdir, fs_access, compute_checksum=True):
# type: (Dict[Text, Any], Builder, Text, StdFsAccess, bool) -> Union[Dict[Text, Any], List[Union[Dict[Text, Any], Text]]]
r = [] # type: List[Any]
+ debug = _logger.isEnabledFor(logging.DEBUG)
if "outputBinding" in schema:
binding = schema["outputBinding"]
globpatterns = [] # type: List[Text]
@@ -564,7 +563,7 @@ class CommandLineTool(Process):
revmap = partial(revmap_file, builder, outdir)
if "glob" in binding:
- with SourceLine(binding, "glob", WorkflowException):
+ with SourceLine(binding, "glob", WorkflowException, debug):
for gb in aslist(binding["glob"]):
gb = builder.do_eval(gb)
if gb:
@@ -629,12 +628,12 @@ class CommandLineTool(Process):
single = True
if "outputEval" in binding:
- with SourceLine(binding, "outputEval", WorkflowException):
+ with SourceLine(binding, "outputEval", WorkflowException, debug):
r = builder.do_eval(binding["outputEval"], context=r)
if single:
if not r and not optional:
- with SourceLine(binding, "glob", WorkflowException):
+ with SourceLine(binding, "glob", WorkflowException, debug):
raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
elif not r and optional:
pass
@@ -645,7 +644,7 @@ class CommandLineTool(Process):
r = r[0]
if "secondaryFiles" in schema:
- with SourceLine(schema, "secondaryFiles", WorkflowException):
+ with SourceLine(schema, "secondaryFiles", WorkflowException, debug):
for primary in aslist(r):
if isinstance(primary, dict):
primary.setdefault("secondaryFiles", [])
diff --git a/cwltool/expression.py b/cwltool/expression.py
index 4db5aa1..19606ba 100644
--- a/cwltool/expression.py
+++ b/cwltool/expression.py
@@ -153,8 +153,8 @@ def next_seg(remain, obj): # type: (Text, Any) -> Any
return obj
-def evaluator(ex, jslib, obj, fullJS=False, timeout=None, force_docker_pull=False, debug=False):
- # type: (Text, Text, Dict[Text, Any], bool, int, bool, bool) -> JSON
+def evaluator(ex, jslib, obj, fullJS=False, timeout=None, force_docker_pull=False, debug=False, js_console=False):
+ # type: (Text, Text, Dict[Text, Any], bool, int, bool, bool, bool) -> JSON
m = param_re.match(ex)
if m:
if m.end(1)+1 == len(ex) and m.group(1) == "null":
@@ -164,7 +164,7 @@ def evaluator(ex, jslib, obj, fullJS=False, timeout=None, force_docker_pull=Fals
except Exception as w:
raise WorkflowException("%s%s" % (m.group(1), w))
elif fullJS:
- return sandboxjs.execjs(ex, jslib, timeout=timeout, force_docker_pull=force_docker_pull, debug=debug)
+ return sandboxjs.execjs(ex, jslib, timeout=timeout, force_docker_pull=force_docker_pull, debug=debug, js_console=js_console)
else:
raise sandboxjs.JavascriptException(
"Syntax error in parameter reference '%s' or used Javascript code without specifying InlineJavascriptRequirement.",
@@ -172,9 +172,9 @@ def evaluator(ex, jslib, obj, fullJS=False, timeout=None, force_docker_pull=Fals
def interpolate(scan, rootvars,
- timeout=None, fullJS=None, jslib="",force_docker_pull=False,
- debug=False):
- # type: (Text, Dict[Text, Any], int, bool, Union[str, Text], bool, bool) -> JSON
+ timeout=None, fullJS=None, jslib="", force_docker_pull=False,
+ debug=False, js_console=False):
+ # type: (Text, Dict[Text, Any], int, bool, Union[str, Text], bool, bool, bool) -> JSON
scan = scan.strip()
parts = []
w = scanner(scan)
@@ -184,7 +184,7 @@ def interpolate(scan, rootvars,
if scan[w[0]] == '$':
e = evaluator(scan[w[0] + 1:w[1]], jslib, rootvars, fullJS=fullJS,
timeout=timeout, force_docker_pull=force_docker_pull,
- debug=debug)
+ debug=debug, js_console=js_console)
if w[0] == 0 and w[1] == len(scan):
return e
leaf = json.dumps(e, sort_keys=True)
@@ -202,8 +202,8 @@ def interpolate(scan, rootvars,
def do_eval(ex, jobinput, requirements, outdir, tmpdir, resources,
- context=None, pull_image=True, timeout=None, force_docker_pull=False, debug=False):
- # type: (Union[dict, AnyStr], Dict[Text, Union[Dict, List, Text]], List[Dict[Text, Any]], Text, Text, Dict[Text, Union[int, Text]], Any, bool, int, bool, bool) -> Any
+ context=None, pull_image=True, timeout=None, force_docker_pull=False, debug=False, js_console=False):
+ # type: (Union[dict, AnyStr], Dict[Text, Union[Dict, List, Text]], List[Dict[Text, Any]], Text, Text, Dict[Text, Union[int, Text]], Any, bool, int, bool, bool, bool) -> Any
runtime = copy.copy(resources)
runtime["tmpdir"] = docker_windows_path_adjust(tmpdir)
@@ -214,7 +214,7 @@ def do_eval(ex, jobinput, requirements, outdir, tmpdir, resources,
u"self": context,
u"runtime": runtime}
- if isinstance(ex, (str, Text)):
+ if isinstance(ex, (str, Text)) and ("$(" in ex or "${" in ex):
fullJS = False
jslib = u""
for r in reversed(requirements):
@@ -230,7 +230,9 @@ def do_eval(ex, jobinput, requirements, outdir, tmpdir, resources,
fullJS=fullJS,
jslib=jslib,
force_docker_pull=force_docker_pull,
- debug=debug)
+ debug=debug,
+ js_console=js_console)
+
except Exception as e:
raise WorkflowException("Expression evaluation error:\n%s" % e)
else:
diff --git a/cwltool/job.py b/cwltool/job.py
index 4e82b7e..6e6e4a5 100644
--- a/cwltool/job.py
+++ b/cwltool/job.py
@@ -21,7 +21,7 @@ from . import docker
from .builder import Builder
from .docker_id import docker_vm_id
from .errors import WorkflowException
-from .pathmapper import PathMapper
+from .pathmapper import PathMapper, ensure_writable
from .process import (UnsupportedRequirement, empty_subtree, get_feature,
stageFiles)
from .utils import bytes2str_in_dicts
@@ -98,24 +98,26 @@ def deref_links(outputs): # type: (Any) -> None
for v in outputs:
deref_links(v)
-def relink_initialworkdir(pathmapper, inplace_update=False):
- # type: (PathMapper, bool) -> None
+def relink_initialworkdir(pathmapper, host_outdir, container_outdir, inplace_update=False):
+ # type: (PathMapper, Text, Text, bool) -> None
for src, vol in pathmapper.items():
if not vol.staged:
continue
+
if vol.type in ("File", "Directory") or (inplace_update and
vol.type in ("WritableFile", "WritableDirectory")):
- if os.path.islink(vol.target) or os.path.isfile(vol.target):
- os.remove(vol.target)
- elif os.path.isdir(vol.target):
- shutil.rmtree(vol.target)
+ host_outdir_tgt = os.path.join(host_outdir, vol.target[len(container_outdir)+1:])
+ if os.path.islink(host_outdir_tgt) or os.path.isfile(host_outdir_tgt):
+ os.remove(host_outdir_tgt)
+ elif os.path.isdir(host_outdir_tgt):
+ shutil.rmtree(host_outdir_tgt)
if onWindows():
if vol.type in ("File", "WritableFile"):
- shutil.copy(vol.resolved,vol.target)
+ shutil.copy(vol.resolved, host_outdir_tgt)
elif vol.type in ("Directory", "WritableDirectory"):
- copytree_with_merge(vol.resolved, vol.target)
+ copytree_with_merge(vol.resolved, host_outdir_tgt)
else:
- os.symlink(vol.resolved, vol.target)
+ os.symlink(vol.resolved, host_outdir_tgt)
class JobBase(object):
def __init__(self): # type: () -> None
@@ -160,7 +162,7 @@ class JobBase(object):
make_path_mapper_kwargs = make_path_mapper_kwargs.copy()
del make_path_mapper_kwargs["basedir"]
self.generatemapper = self.make_pathmapper(cast(List[Any], self.generatefiles["listing"]),
- self.outdir, basedir=self.outdir, separateDirs=False, **make_path_mapper_kwargs)
+ self.builder.outdir, basedir=self.outdir, separateDirs=False, **make_path_mapper_kwargs)
_logger.debug(u"[job %s] initial work dir %s", self.name,
json.dumps({p: self.generatemapper.mapper(p) for p in self.generatemapper.files()}, indent=4))
@@ -207,7 +209,7 @@ class JobBase(object):
os.makedirs(dn)
stdout_path = absout
- commands = [Text(x).encode('utf-8') for x in runtime + self.command_line]
+ commands = [Text(x) for x in (runtime + self.command_line)]
job_script_contents = None # type: Text
builder = getattr(self, "builder", None) # type: Builder
if builder is not None:
@@ -234,7 +236,7 @@ class JobBase(object):
processStatus = "permanentFail"
if self.generatefiles["listing"]:
- relink_initialworkdir(self.generatemapper, inplace_update=self.inplace_update)
+ relink_initialworkdir(self.generatemapper, self.outdir, self.builder.outdir, inplace_update=self.inplace_update)
outputs = self.collect_outputs(self.outdir)
outputs = bytes2str_in_dicts(outputs) # type: ignore
@@ -297,51 +299,67 @@ class CommandLineJob(JobBase):
env["TMPDIR"] = str(self.tmpdir) if onWindows() else self.tmpdir
if "PATH" not in env:
env["PATH"] = str(os.environ["PATH"]) if onWindows() else os.environ["PATH"]
+ if "SYSTEMROOT" not in env and "SYSTEMROOT" in os.environ:
+ env["SYSTEMROOT"] = str(os.environ["SYSTEMROOT"]) if onWindows() else os.environ["SYSTEMROOT"]
stageFiles(self.pathmapper, ignoreWritable=True, symLink=True)
if self.generatemapper:
stageFiles(self.generatemapper, ignoreWritable=self.inplace_update, symLink=True)
- relink_initialworkdir(self.generatemapper, inplace_update=self.inplace_update)
+ relink_initialworkdir(self.generatemapper, self.outdir, self.builder.outdir, inplace_update=self.inplace_update)
self._execute([], env, rm_tmpdir=rm_tmpdir, move_outputs=move_outputs)
class DockerCommandLineJob(JobBase):
- def add_volumes(self, pathmapper, runtime, stage_output):
- # type: (PathMapper, List[Text], bool) -> None
+ def add_volumes(self, pathmapper, runtime):
+ # type: (PathMapper, List[Text]) -> None
host_outdir = self.outdir
container_outdir = self.builder.outdir
for src, vol in pathmapper.items():
if not vol.staged:
continue
- if stage_output:
- containertgt = container_outdir + vol.target[len(host_outdir):]
+ if vol.target.startswith(container_outdir+"/"):
+ host_outdir_tgt = os.path.join(
+ host_outdir, vol.target[len(container_outdir)+1:])
else:
- containertgt = vol.target
+ host_outdir_tgt = None
if vol.type in ("File", "Directory"):
if not vol.resolved.startswith("_:"):
- runtime.append(u"--volume=%s:%s:ro" % (docker_windows_path_adjust(vol.resolved), docker_windows_path_adjust(containertgt)))
+ runtime.append(u"--volume=%s:%s:ro" % (
+ docker_windows_path_adjust(vol.resolved),
+ docker_windows_path_adjust(vol.target)))
elif vol.type == "WritableFile":
if self.inplace_update:
- runtime.append(u"--volume=%s:%s:rw" % (docker_windows_path_adjust(vol.resolved), docker_windows_path_adjust(containertgt)))
+ runtime.append(u"--volume=%s:%s:rw" % (
+ docker_windows_path_adjust(vol.resolved),
+ docker_windows_path_adjust(vol.target)))
else:
- shutil.copy(vol.resolved, vol.target)
+ shutil.copy(vol.resolved, host_outdir_tgt)
+ ensure_writable(host_outdir_tgt)
elif vol.type == "WritableDirectory":
if vol.resolved.startswith("_:"):
os.makedirs(vol.target, 0o0755)
else:
if self.inplace_update:
- runtime.append(u"--volume=%s:%s:rw" % (docker_windows_path_adjust(vol.resolved), docker_windows_path_adjust(containertgt)))
+ runtime.append(u"--volume=%s:%s:rw" % (
+ docker_windows_path_adjust(vol.resolved),
+ docker_windows_path_adjust(vol.target)))
else:
- shutil.copytree(vol.resolved, vol.target)
+ shutil.copytree(vol.resolved, host_outdir_tgt)
+ ensure_writable(host_outdir_tgt)
elif vol.type == "CreateFile":
- createtmp = os.path.join(host_outdir, os.path.basename(vol.target))
- with open(createtmp, "wb") as f:
- f.write(vol.resolved.encode("utf-8"))
- runtime.append(u"--volume=%s:%s:ro" % (docker_windows_path_adjust(createtmp), docker_windows_path_adjust(vol.target)))
-
+ if host_outdir_tgt:
+ with open(host_outdir_tgt, "wb") as f:
+ f.write(vol.resolved.encode("utf-8"))
+ else:
+ fd, createtmp = tempfile.mkstemp(dir=self.tmpdir)
+ with os.fdopen(fd, "wb") as f:
+ f.write(vol.resolved.encode("utf-8"))
+ runtime.append(u"--volume=%s:%s:rw" % (
+ docker_windows_path_adjust(createtmp),
+ docker_windows_path_adjust(vol.target)))
def run(self, pull_image=True, rm_container=True,
rm_tmpdir=True, move_outputs="move", **kwargs):
@@ -351,57 +369,89 @@ class DockerCommandLineJob(JobBase):
img_id = None
env = None # type: MutableMapping[Text, Text]
- try:
- env = cast(MutableMapping[Text, Text], os.environ)
- if docker_req and kwargs.get("use_container"):
- img_id = docker.get_from_requirements(docker_req, True, pull_image)
- if img_id is None:
- if self.builder.find_default_container:
- default_container = self.builder.find_default_container()
- if default_container:
- img_id = default_container
- env = cast(MutableMapping[Text, Text], os.environ)
-
- if docker_req and img_id is None and kwargs.get("use_container"):
- raise Exception("Docker image not available")
- except Exception as e:
- _logger.debug("Docker error", exc_info=True)
- if docker_is_req:
- raise UnsupportedRequirement(
- "Docker is required to run this tool: %s" % e)
+ user_space_docker_cmd = kwargs.get("user_space_docker_cmd")
+ if docker_req and user_space_docker_cmd:
+ # For user-space docker implementations, a local image name or ID
+ # takes precedence over a network pull
+ if 'dockerImageId' in docker_req:
+ img_id = str(docker_req["dockerImageId"])
+ elif 'dockerPull' in docker_req:
+ img_id = str(docker_req["dockerPull"])
else:
- raise WorkflowException(
- "Docker is not available for this tool, try --no-container"
- " to disable Docker: %s" % e)
+ raise Exception("Docker image must be specified as "
+ "'dockerImageId' or 'dockerPull' when using user "
+ "space implementations of Docker")
+ else:
+ try:
+ env = cast(MutableMapping[Text, Text], os.environ)
+ if docker_req and kwargs.get("use_container"):
+ img_id = str(docker.get_from_requirements(
+ docker_req, True, pull_image))
+ if img_id is None:
+ if self.builder.find_default_container:
+ default_container = self.builder.find_default_container()
+ if default_container:
+ img_id = str(default_container)
+ env = cast(MutableMapping[Text, Text], os.environ)
+
+ if docker_req and img_id is None and kwargs.get("use_container"):
+ raise Exception("Docker image not available")
+ except Exception as e:
+ _logger.debug("Docker error", exc_info=True)
+ if docker_is_req:
+ raise UnsupportedRequirement(
+ "Docker is required to run this tool: %s" % e)
+ else:
+ raise WorkflowException(
+ "Docker is not available for this tool, try "
+ "--no-container to disable Docker, or install "
+ "a user space Docker replacement like uDocker with "
+ "--user-space-docker-cmd.: %s" % e)
self._setup(kwargs)
- runtime = [u"docker", u"run", u"-i"]
+ if user_space_docker_cmd:
+ runtime = [user_space_docker_cmd, u"run"]
+ else:
+ runtime = [u"docker", u"run", u"-i"]
- runtime.append(u"--volume=%s:%s:rw" % (docker_windows_path_adjust(os.path.realpath(self.outdir)), self.builder.outdir))
- runtime.append(u"--volume=%s:%s:rw" % (docker_windows_path_adjust(os.path.realpath(self.tmpdir)), "/tmp"))
+ runtime.append(u"--volume=%s:%s:rw" % (
+ docker_windows_path_adjust(os.path.realpath(self.outdir)),
+ self.builder.outdir))
+ runtime.append(u"--volume=%s:%s:rw" % (
+ docker_windows_path_adjust(os.path.realpath(self.tmpdir)), "/tmp"))
- self.add_volumes(self.pathmapper, runtime, False)
+ self.add_volumes(self.pathmapper, runtime)
if self.generatemapper:
- self.add_volumes(self.generatemapper, runtime, True)
+ self.add_volumes(self.generatemapper, runtime)
- runtime.append(u"--workdir=%s" % (docker_windows_path_adjust(self.builder.outdir)))
- runtime.append(u"--read-only=true")
+ if user_space_docker_cmd:
+ runtime = [x.replace(":ro", "") for x in runtime]
+ runtime = [x.replace(":rw", "") for x in runtime]
- if kwargs.get("custom_net", None) is not None:
- runtime.append(u"--net={0}".format(kwargs.get("custom_net")))
- elif kwargs.get("disable_net", None):
- runtime.append(u"--net=none")
+ runtime.append(u"--workdir=%s" % (
+ docker_windows_path_adjust(self.builder.outdir)))
+ if not user_space_docker_cmd:
- if self.stdout:
- runtime.append("--log-driver=none")
+ if not kwargs.get("no_read_only"):
+ runtime.append(u"--read-only=true")
+
+ if kwargs.get("custom_net", None) is not None:
+ runtime.append(u"--net={0}".format(kwargs.get("custom_net")))
+ elif kwargs.get("disable_net", None):
+ runtime.append(u"--net=none")
+
+ if self.stdout:
+ runtime.append("--log-driver=none")
- euid, egid = docker_vm_id()
- if not onWindows(): # MS Windows does not have getuid() or geteuid() functions
- euid, egid = euid or os.geteuid(), egid or os.getgid()
+ euid, egid = docker_vm_id()
+ if not onWindows():
+ # MS Windows does not have getuid() or geteuid() functions
+ euid, egid = euid or os.geteuid(), egid or os.getgid()
- if kwargs.get("no_match_user", None) is False and (euid, egid) != (None, None):
- runtime.append(u"--user=%d:%d" % (euid, egid))
+ if kwargs.get("no_match_user", None) is False \
+ and (euid, egid) != (None, None):
+ runtime.append(u"--user=%d:%d" % (euid, egid))
if rm_container:
runtime.append(u"--rm")
@@ -418,11 +468,12 @@ class DockerCommandLineJob(JobBase):
runtime.append(img_id)
- self._execute(runtime, env, rm_tmpdir=rm_tmpdir, move_outputs=move_outputs)
+ self._execute(
+ runtime, env, rm_tmpdir=rm_tmpdir, move_outputs=move_outputs)
def _job_popen(
- commands, # type: List[bytes]
+ commands, # type: List[Text]
stdin_path, # type: Text
stdout_path, # type: Text
stderr_path, # type: Text
diff --git a/cwltool/load_tool.py b/cwltool/load_tool.py
index 491fc3a..8f30b90 100644
--- a/cwltool/load_tool.py
+++ b/cwltool/load_tool.py
@@ -6,6 +6,8 @@ import logging
import os
import re
import uuid
+import hashlib
+import json
from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast
import requests.sessions
@@ -88,7 +90,8 @@ def _convert_stdstreams_to_files(workflowobj):
if streamtype in workflowobj:
filename = workflowobj[streamtype]
else:
- filename = Text(uuid.uuid4())
+ filename = Text(hashlib.sha1(json.dumps(workflowobj,
+ sort_keys=True).encode('utf-8')).hexdigest())
workflowobj[streamtype] = filename
out['type'] = 'File'
out['outputBinding'] = {'glob': filename}
diff --git a/cwltool/main.py b/cwltool/main.py
index 526c6c2..f57cafb 100755
--- a/cwltool/main.py
+++ b/cwltool/main.py
@@ -158,6 +158,12 @@ def arg_parser(): # type: () -> argparse.ArgumentParser
exgroup.add_argument("--quiet", action="store_true", help="Only print warnings and errors.")
exgroup.add_argument("--debug", action="store_true", help="Print even more logging")
+ parser.add_argument("--js-console", action="store_true", help="Enable javascript console output")
+ parser.add_argument("--user-space-docker-cmd",
+ help="(Linux/OS X only) Specify a user space docker "
+ "command (like udocker or dx-docker) that will be "
+ "used to call 'pull' and 'run'")
+
dependency_resolvers_configuration_help = argparse.SUPPRESS
dependencies_directory_help = argparse.SUPPRESS
use_biocontainers_help = argparse.SUPPRESS
@@ -229,6 +235,9 @@ def arg_parser(): # type: () -> argparse.ArgumentParser
parser.add_argument("--force-docker-pull", action="store_true",
default=False, help="Pull latest docker image even if"
" it is locally present", dest="force_docker_pull")
+ parser.add_argument("--no-read-only", action="store_true",
+ default=False, help="Do not set root directoy in the"
+ " container as read-only", dest="no_read_only")
parser.add_argument("workflow", type=Text, nargs="?", default=None)
parser.add_argument("job_order", nargs=argparse.REMAINDER)
@@ -730,6 +739,7 @@ def main(argsl=None, # type: List[str]
'cachedir': None,
'quiet': False,
'debug': False,
+ 'js_console': False,
'version': False,
'enable_dev': False,
'enable_ext': False,
@@ -839,7 +849,7 @@ def main(argsl=None, # type: List[str]
return 0
if args.print_rdf:
- printrdf(tool, document_loader.ctx, args.rdf_serializer, stdout)
+ stdout.write(printrdf(tool, document_loader.ctx, args.rdf_serializer))
return 0
if args.print_dot:
@@ -910,11 +920,17 @@ def main(argsl=None, # type: List[str]
if out is not None:
def locToPath(p):
+ for field in ("path", "nameext", "nameroot", "dirname"):
+ if field in p:
+ del p[field]
if p["location"].startswith("file://"):
p["path"] = uri_file_path(p["location"])
visit_class(out, ("File", "Directory"), locToPath)
+ # Unsetting the Generation fron final output object
+ visit_class(out,("File",), MutationManager().unset_generation)
+
if isinstance(out, six.string_types):
stdout.write(out)
else:
diff --git a/cwltool/mutation.py b/cwltool/mutation.py
index d7f3dbe..c8373cc 100644
--- a/cwltool/mutation.py
+++ b/cwltool/mutation.py
@@ -67,3 +67,7 @@ class MutationManager(object):
loc = obj["location"]
current = self.generations.get(loc, MutationState(0,[], ""))
obj[_generation] = current.generation
+
+ def unset_generation(self, obj):
+ # type: (Dict) -> None
+ obj.pop(_generation, None)
diff --git a/cwltool/pathmapper.py b/cwltool/pathmapper.py
index dd7c09a..7127d23 100644
--- a/cwltool/pathmapper.py
+++ b/cwltool/pathmapper.py
@@ -168,6 +168,25 @@ def downloadHttpFile(httpurl):
r.close()
return f.name
+def ensure_writable(path):
+ # type: (Text) -> None
+ if os.path.isdir(path):
+ for root, dirs, files in os.walk(path):
+ for name in files:
+ j = os.path.join(root, name)
+ st = os.stat(j)
+ mode = stat.S_IMODE(st.st_mode)
+ os.chmod(j, mode|stat.S_IWUSR)
+ for name in dirs:
+ j = os.path.join(root, name)
+ st = os.stat(j)
+ mode = stat.S_IMODE(st.st_mode)
+ os.chmod(j, mode|stat.S_IWUSR)
+ else:
+ st = os.stat(path)
+ mode = stat.S_IMODE(st.st_mode)
+ os.chmod(path, mode|stat.S_IWUSR)
+
class PathMapper(object):
"""Mapping of files from relative path provided in the file to a tuple of
(absolute local path, absolute container path)
@@ -235,7 +254,7 @@ class PathMapper(object):
if "contents" in obj and obj["location"].startswith("_:"):
self._pathmap[obj["location"]] = MapperEnt(obj["contents"], tgt, "CreateFile", staged)
else:
- with SourceLine(obj, "location", validate.ValidationException):
+ with SourceLine(obj, "location", validate.ValidationException, _logger.isEnabledFor(logging.DEBUG)):
deref = ab
if urllib.parse.urlsplit(deref).scheme in ['http','https']:
deref = downloadHttpFile(path)
diff --git a/cwltool/process.py b/cwltool/process.py
index 0e76607..bdbee58 100644
--- a/cwltool/process.py
+++ b/cwltool/process.py
@@ -34,7 +34,8 @@ from .utils import cmp_like_py2
from .builder import Builder
from .errors import UnsupportedRequirement, WorkflowException
from .pathmapper import (PathMapper, adjustDirObjs, get_listing,
- normalizeFilesDirs, visit_class, trim_listing)
+ normalizeFilesDirs, visit_class, trim_listing,
+ ensure_writable)
from .stdfsaccess import StdFsAccess
from .utils import aslist, get_feature, copytree_with_merge, onWindows
@@ -230,15 +231,17 @@ def stageFiles(pm, stageFunc=None, ignoreWritable=False, symLink=True):
os.makedirs(p.target, 0o0755)
elif p.type == "WritableFile" and not ignoreWritable:
shutil.copy(p.resolved, p.target)
+ ensure_writable(p.target)
elif p.type == "WritableDirectory" and not ignoreWritable:
if p.resolved.startswith("_:"):
os.makedirs(p.target, 0o0755)
else:
shutil.copytree(p.resolved, p.target)
- elif p.type == "CreateFile" and not ignoreWritable:
+ ensure_writable(p.target)
+ elif p.type == "CreateFile":
with open(p.target, "wb") as n:
n.write(p.resolved.encode("utf-8"))
-
+ ensure_writable(p.target)
def collectFilesAndDirs(obj, out):
# type: (Union[Dict[Text, Any], List[Dict[Text, Any]]], List[Dict[Text, Any]]) -> None
@@ -276,9 +279,13 @@ def relocateOutputs(outputObj, outdir, output_dirs, action, fs_access):
if src != dst:
_logger.debug("Copying %s to %s", src, dst)
if os.path.isdir(src):
+ if os.path.isdir(dst):
+ shutil.rmtree(dst)
+ elif os.path.isfile(dst):
+ os.unlink(dst)
shutil.copytree(src, dst)
else:
- shutil.copy(src, dst)
+ shutil.copy2(src, dst)
outfiles = [] # type: List[Dict[Text, Any]]
collectFilesAndDirs(outputObj, outfiles)
@@ -385,7 +392,7 @@ def checkFormat(actualFile, inputFormats, ontology):
def fillInDefaults(inputs, job):
# type: (List[Dict[Text, Text]], Dict[Text, Union[Dict[Text, Any], List, Text]]) -> None
for e, inp in enumerate(inputs):
- with SourceLine(inputs, e, WorkflowException):
+ with SourceLine(inputs, e, WorkflowException, _logger.isEnabledFor(logging.DEBUG)):
fieldname = shortname(inp[u"id"])
if job.get(fieldname) is not None:
pass
@@ -532,6 +539,8 @@ class Process(six.with_metaclass(abc.ABCMeta, object)):
tmpdir: tmpdir on host for this job
stagedir: stagedir on host for this job
select_resources: callback to select compute resources
+ debug: enable debugging output
+ js_console: enable javascript console output
"""
builder = Builder()
@@ -556,6 +565,7 @@ class Process(six.with_metaclass(abc.ABCMeta, object)):
builder.resources = {}
builder.timeout = kwargs.get("eval_timeout")
builder.debug = kwargs.get("debug")
+ builder.js_console = kwargs.get("js_console")
builder.mutation_manager = kwargs.get("mutation_manager")
builder.make_fs_access = kwargs.get("make_fs_access") or StdFsAccess
diff --git a/cwltool/sandboxjs.py b/cwltool/sandboxjs.py
index fa08519..e2a513f 100644
--- a/cwltool/sandboxjs.py
+++ b/cwltool/sandboxjs.py
@@ -3,6 +3,7 @@ import errno
import json
import logging
import os
+import re
import select
import subprocess
import threading
@@ -53,11 +54,17 @@ def check_js_threshold_version(working_alias):
return False
-def new_js_proc(force_docker_pull=False):
- # type: (bool) -> subprocess.Popen
+def new_js_proc(force_docker_pull=False, js_console=False):
+ # type: (bool, bool) -> subprocess.Popen
+
+ cwl_node_engine_js = 'cwlNodeEngine.js'
+ if js_console:
+ cwl_node_engine_js = 'cwlNodeEngineJSConsole.js'
+ _logger.warn("Running with support for javascript console in expressions (DO NOT USE IN PRODUCTION)")
+
+ res = resource_stream(__name__, cwl_node_engine_js)
+ nodecode = res.read().decode('utf-8')
- res = resource_stream(__name__, 'cwlNodeEngine.js')
- nodecode = res.read()
required_node_version, docker = (False,)*2
nodejs = None
trynodes = ("nodejs", "node")
@@ -124,10 +131,10 @@ def new_js_proc(force_docker_pull=False):
return nodejs
-def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False): # type: (Union[Mapping, Text], Any, int, bool, bool) -> JSON
+def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False, js_console=False): # type: (Union[Mapping, Text], Any, int, bool, bool, bool) -> JSON
if not hasattr(localdata, "proc") or localdata.proc.poll() is not None or onWindows():
- localdata.proc = new_js_proc(force_docker_pull=force_docker_pull)
+ localdata.proc = new_js_proc(force_docker_pull=force_docker_pull, js_console=js_console)
nodejs = localdata.proc
@@ -157,6 +164,12 @@ def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False): # ty
rselect = [nodejs.stdout, nodejs.stderr] # type: List[BytesIO]
wselect = [nodejs.stdin] # type: List[BytesIO]
+ PROCESS_FINISHED_STR = "r1cepzbhUTxtykz5XTC4\n"
+
+ def process_finished(): # type: () -> bool
+ return stdout_buf.getvalue().decode().endswith(PROCESS_FINISHED_STR) and \
+ stderr_buf.getvalue().decode().endswith(PROCESS_FINISHED_STR)
+
# On windows system standard input/output are not handled properly by select module
# (modules like pywin32, msvcrt, gevent don't work either)
if sys.platform=='win32':
@@ -207,11 +220,9 @@ def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False): # ty
error_thread.daemon=True
error_thread.start()
- # mark if output/error is ready
- output_ready=False
- error_ready=False
+ finished = False
- while (len(wselect) + len(rselect)) > 0:
+ while not finished and tm.is_alive():
try:
if nodejs.stdin in wselect:
if not input_queue.empty():
@@ -220,58 +231,39 @@ def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False): # ty
wselect = []
if nodejs.stdout in rselect:
if not output_queue.empty():
- output_ready = True
stdout_buf.write(output_queue.get())
- elif output_ready:
- rselect = []
- no_more_output.release()
- no_more_error.release()
- output_thread.join()
if nodejs.stderr in rselect:
if not error_queue.empty():
- error_ready = True
- stderr_buf.write(error_queue.get())
- elif error_ready:
- rselect = []
- no_more_output.release()
- no_more_error.release()
- output_thread.join()
- error_thread.join()
- if stdout_buf.getvalue().endswith("\n"):
- rselect = []
+ stderr_buf.write(error_queue.get())
+
+ if process_finished() and error_queue.empty() and output_queue.empty():
+ finished = True
no_more_output.release()
no_more_error.release()
- output_thread.join()
except OSError as e:
break
else:
- while (len(wselect) + len(rselect)) > 0:
+ while not process_finished() and tm.is_alive():
rready, wready, _ = select.select(rselect, wselect, [])
try:
if nodejs.stdin in wready:
b = stdin_buf.read(select.PIPE_BUF)
if b:
os.write(nodejs.stdin.fileno(), b)
- else:
- wselect = []
for pipes in ((nodejs.stdout, stdout_buf), (nodejs.stderr, stderr_buf)):
if pipes[0] in rready:
b = os.read(pipes[0].fileno(), select.PIPE_BUF)
if b:
pipes[1].write(b)
- else:
- rselect.remove(pipes[0])
- if stdout_buf.getvalue().endswith("\n".encode()):
- rselect = []
except OSError as e:
break
tm.cancel()
stdin_buf.close()
- stdoutdata = stdout_buf.getvalue()
- stderrdata = stderr_buf.getvalue()
+ stdoutdata = stdout_buf.getvalue()[:-len(PROCESS_FINISHED_STR) - 1]
+ stderrdata = stderr_buf.getvalue()[:-len(PROCESS_FINISHED_STR) - 1]
def fn_linenum(): # type: () -> Text
lines = fn.splitlines()
@@ -289,6 +281,13 @@ def execjs(js, jslib, timeout=None, force_docker_pull=False, debug=False): # ty
nodejs.poll()
+ if js_console:
+ if len(stderrdata) > 0:
+ _logger.info("Javascript console output:")
+ _logger.info("----------------------------------------")
+ _logger.info('\n'.join(re.findall(r'^[[](?:log|err)[]].*$', stderrdata.decode('utf-8'), flags=re.MULTILINE)))
+ _logger.info("----------------------------------------")
+
if debug:
info = u"returncode was: %s\nscript was:\n%s\nstdout was: %s\nstderr was: %s\n" %\
(nodejs.returncode, fn_linenum(), stdfmt(stdoutdata.decode('utf-8')), stdfmt(stderrdata.decode('utf-8')))
diff --git a/cwltool/schemas/draft-3/index.md b/cwltool/schemas/draft-3/index.md
deleted file mode 100644
index e69de29..0000000
diff --git a/cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema2.yml b/cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema2.yml
deleted file mode 100644
index c928928..0000000
--- a/cwltool/schemas/v1.0/salad/schema_salad/metaschema/metaschema2.yml
+++ /dev/null
@@ -1,317 +0,0 @@
-$base: "https://w3id.org/cwl/salad#"
-
-$namespaces:
- sld: "https://w3id.org/cwl/salad#"
- dct: "http://purl.org/dc/terms/"
- rdf: "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
- rdfs: "http://www.w3.org/2000/01/rdf-schema#"
- xsd: "http://www.w3.org/2001/XMLSchema#"
-
-$graph:
-
-- name: "Semantic_Annotations_for_Linked_Avro_Data"
- type: documentation
- doc:
- - $include: salad.md
- - $import: field_name.yml
- - $import: ident_res.yml
- - $import: link_res.yml
- - $import: vocab_res.yml
- - $include: import_include.md
-
-- name: "Link_Validation"
- type: documentation
- doc: |
- # Link validation
-
- Once a document has been preprocessed, an implementation may validate
- links. The link validation traversal may visit fields which the schema
- designates as link fields and check that each URI references an existing
- object in the current document, an imported document, file system, or
- network resource. Failure to validate links may be a fatal error. Link
- validation behavior for individual fields may be modified by `identity` and
- `noLinkCheck` in the `jsonldPredicate` section of the field schema.
-
-
-- name: "Schema_validation"
- type: documentation
- doc: ""
-
-
-# - name: "JSON_LD_Context"
-# type: documentation
-# doc: |
-# # Generating JSON-LD Context
-
-# How to generate the json-ld context...
-
-
-- $import: metaschema_base.yml
-
-- name: JsonldPredicate
- type: record
- doc: |
- Attached to a record field to define how the parent record field is handled for
- URI resolution and JSON-LD context generation.
- fields:
- - name: _id
- type: string?
- jsonldPredicate:
- _id: sld:_id
- _type: "@id"
- identity: true
- doc: |
- The predicate URI that this field corresponds to.
- Corresponds to JSON-LD `@id` directive.
- - name: _type
- type: string?
- doc: |
- The context type hint, corresponds to JSON-LD `@type` directive.
-
- * If the value of this field is `@id` and `identity` is false or
- unspecified, the parent field must be resolved using the link
- resolution rules. If `identity` is true, the parent field must be
- resolved using the identifier expansion rules.
-
- * If the value of this field is `@vocab`, the parent field must be
- resolved using the vocabulary resolution rules.
-
- - name: _container
- type: string?
- doc: |
- Structure hint, corresponds to JSON-LD `@container` directive.
- - name: identity
- type: boolean?
- doc: |
- If true and `_type` is `@id` this indicates that the parent field must
- be resolved according to identity resolution rules instead of link
- resolution rules. In addition, the field value is considered an
- assertion that the linked value exists; absence of an object in the loaded document
- with the URI is not an error.
- - name: noLinkCheck
- type: boolean?
- doc: |
- If true, this indicates that link validation traversal must stop at
- this field. This field (it is is a URI) or any fields under it (if it
- is an object or array) are not subject to link checking.
- - name: mapSubject
- type: string?
- doc: |
- If the value of the field is a JSON object, it must be transformed
- into an array of JSON objects, where each key-value pair from the
- source JSON object is a list item, the list items must be JSON objects,
- and the key is assigned to the field specified by `mapSubject`.
- - name: mapPredicate
- type: string?
- doc: |
- Only applies if `mapSubject` is also provided. If the value of the
- field is a JSON object, it is transformed as described in `mapSubject`,
- with the addition that when the value of a map item is not an object,
- the item is transformed to a JSON object with the key assigned to the
- field specified by `mapSubject` and the value assigned to the field
- specified by `mapPredicate`.
- - name: refScope
- type: int?
- doc: |
- If the field contains a relative reference, it must be resolved by
- searching for valid document references in each successive parent scope
- in the document fragment. For example, a reference of `foo` in the
- context `#foo/bar/baz` will first check for the existence of
- `#foo/bar/baz/foo`, followed by `#foo/bar/foo`, then `#foo/foo` and
- then finally `#foo`. The first valid URI in the search order shall be
- used as the fully resolved value of the identifier. The value of the
- refScope field is the specified number of levels from the containing
- identifer scope before starting the search, so if `refScope: 2` then
- "baz" and "bar" must be stripped to get the base `#foo` and search
- `#foo/foo` and the `#foo`. The last scope searched must be the top
- level scope before determining if the identifier cannot be resolved.
- - name: typeDSL
- type: boolean?
- doc: |
- Field must be expanded based on the the Schema Salad type DSL.
-
-
-- name: SpecializeDef
- type: record
- fields:
- - name: specializeFrom
- type: string
- doc: "The data type to be replaced"
- jsonldPredicate:
- _id: "sld:specializeFrom"
- _type: "@id"
- refScope: 1
-
- - name: specializeTo
- type: string
- doc: "The new data type to replace with"
- jsonldPredicate:
- _id: "sld:specializeTo"
- _type: "@id"
- refScope: 1
-
-
-- name: NamedType
- type: record
- abstract: true
- fields:
- - name: name
- type: string
- jsonldPredicate: "@id"
- doc: "The identifier for this type"
-
-
-- name: DocType
- type: record
- abstract: true
- fields:
- - name: doc
- type:
- - string?
- - string[]?
- doc: "A documentation string for this type, or an array of strings which should be concatenated."
- jsonldPredicate: "rdfs:comment"
-
- - name: docParent
- type: string?
- doc: |
- Hint to indicate that during documentation generation, documentation
- for this type should appear in a subsection under `docParent`.
- jsonldPredicate:
- _id: "sld:docParent"
- _type: "@id"
-
- - name: docChild
- type:
- - string?
- - string[]?
- doc: |
- Hint to indicate that during documentation generation, documentation
- for `docChild` should appear in a subsection under this type.
- jsonldPredicate:
- _id: "sld:docChild"
- _type: "@id"
-
- - name: docAfter
- type: string?
- doc: |
- Hint to indicate that during documentation generation, documentation
- for this type should appear after the `docAfter` section at the same
- level.
- jsonldPredicate:
- _id: "sld:docAfter"
- _type: "@id"
-
-
-- name: SchemaDefinedType
- type: record
- extends: DocType
- doc: |
- Abstract base for schema-defined types.
- abstract: true
- fields:
- - name: jsonldPredicate
- type:
- - string?
- - JsonldPredicate?
- doc: |
- Annotate this type with linked data context.
- jsonldPredicate: sld:jsonldPredicate
-
- - name: documentRoot
- type: boolean?
- doc: |
- If true, indicates that the type is a valid at the document root. At
- least one type in a schema must be tagged with `documentRoot: true`.
-
-
-- name: SaladRecordField
- type: record
- extends: RecordField
- doc: "A field of a record."
- fields:
- - name: jsonldPredicate
- type:
- - string?
- - JsonldPredicate?
- doc: |
- Annotate this type with linked data context.
- jsonldPredicate: "sld:jsonldPredicate"
-
-
-- name: SaladRecordSchema
- type: record
- extends: [NamedType, RecordSchema, SchemaDefinedType]
- documentRoot: true
- specialize:
- RecordField: SaladRecordField
- fields:
- - name: abstract
- type: boolean?
- doc: |
- If true, this record is abstract and may be used as a base for other
- records, but is not valid on its own.
-
- - name: extends
- type:
- - string?
- - string[]?
- jsonldPredicate:
- _id: "sld:extends"
- _type: "@id"
- refScope: 1
- doc: |
- Indicates that this record inherits fields from one or more base records.
-
- - name: specialize
- type:
- - SpecializeDef[]?
- doc: |
- Only applies if `extends` is declared. Apply type specialization using the
- base record as a template. For each field inherited from the base
- record, replace any instance of the type `specializeFrom` with
- `specializeTo`.
- jsonldPredicate:
- _id: "sld:specialize"
- mapSubject: specializeFrom
- mapPredicate: specializeTo
-
-- name: SaladEnumSchema
- type: record
- extends: [EnumSchema, SchemaDefinedType]
- documentRoot: true
- doc: |
- Define an enumerated type.
- fields:
- - name: extends
- type:
- - string?
- - string[]?
- jsonldPredicate:
- _id: "sld:extends"
- _type: "@id"
- refScope: 1
- doc: |
- Indicates that this enum inherits symbols from a base enum.
-
-
-- name: Documentation
- type: record
- extends: [NamedType, DocType]
- documentRoot: true
- doc: |
- A documentation section. This type exists to facilitate self-documenting
- schemas but has no role in formal validation.
- fields:
- type:
- doc: {foo: "Must be `documentation`"}
- type:
- name: Documentation_symbol
- type: enum
- symbols:
- - "sld:documentation"
- jsonldPredicate:
- _id: "sld:type"
- _type: "@vocab"
- typeDSL: true
- refScope: 2
diff --git a/cwltool/workflow.py b/cwltool/workflow.py
index 4bd926b..2081430 100644
--- a/cwltool/workflow.py
+++ b/cwltool/workflow.py
@@ -337,6 +337,10 @@ class WorkflowJob(object):
def try_make_job(self, step, final_output_callback, **kwargs):
# type: (WorkflowJobStep, Callable[[Any, Any], Any], **Any) -> Generator
+
+ js_console = kwargs.get("js_console", False)
+ debug = kwargs.get("debug", False)
+
inputparms = step.tool["inputs"]
outputparms = step.tool["outputs"]
@@ -375,7 +379,7 @@ class WorkflowJob(object):
if k in valueFrom:
return expression.do_eval(
valueFrom[k], shortio, self.workflow.requirements,
- None, None, {}, context=v)
+ None, None, {}, context=v, debug=debug, js_console=js_console)
else:
return v
@@ -445,7 +449,7 @@ class WorkflowJob(object):
del kwargs["outdir"]
for e, i in enumerate(self.tool["inputs"]):
- with SourceLine(self.tool["inputs"], e, WorkflowException):
+ with SourceLine(self.tool["inputs"], e, WorkflowException, _logger.isEnabledFor(logging.DEBUG)):
iid = shortname(i["id"])
if iid in joborder:
self.state[i["id"]] = WorkflowStateItem(i, copy.deepcopy(joborder[iid]), "success")
diff --git a/setup.cfg b/setup.cfg
index b2580d2..8336837 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -13,7 +13,6 @@ addopts = --ignore cwltool/schemas
testpaths = tests
[egg_info]
-tag_build = .20170810192106
+tag_build = .20171107133715
tag_date = 0
-tag_svn_revision = 0
diff --git a/setup.py b/setup.py
index 989ccb2..8f8fd65 100755
--- a/setup.py
+++ b/setup.py
@@ -44,15 +44,16 @@ setup(name='cwltool',
'schemas/v1.1.0-dev1/salad/schema_salad/metaschema/*.yml',
'schemas/v1.1.0-dev1/salad/schema_salad/metaschema/*.md',
'cwlNodeEngine.js',
+ 'cwlNodeEngineJSConsole.js',
'extensions.yml']},
include_package_data=True,
install_requires=[
'setuptools',
- 'requests >= 2.12.4',
+ 'requests >= 2.4.3',
'ruamel.yaml >= 0.12.4, < 0.15',
'rdflib >= 4.2.2, < 4.3.0',
'shellescape >= 3.4.1, < 3.5',
- 'schema-salad >= 2.6, < 3',
+ 'schema-salad >= 2.6.20170927145003, < 3',
'typing >= 3.5.3',
'six >= 1.8.0',
],
diff --git a/tests/cat.cwl b/tests/cat.cwl
deleted file mode 100644
index 93af517..0000000
--- a/tests/cat.cwl
+++ /dev/null
@@ -1,8 +0,0 @@
-cwlVersion: v1.0
-class: CommandLineTool
-inputs:
- - id: inp
- type: File
- inputBinding: {}
-outputs: []
-baseCommand: cat
diff --git a/tests/cat2.cwl b/tests/cat2.cwl
deleted file mode 100644
index 29d42a8..0000000
--- a/tests/cat2.cwl
+++ /dev/null
@@ -1,9 +0,0 @@
-cwlVersion: v1.0
-class: CommandLineTool
-inputs:
- - id: inp
- type: Directory
- inputBinding:
- valueFrom: $(self.listing[0].path)
-outputs: []
-baseCommand: cat
diff --git a/tests/env.cwl b/tests/env.cwl
deleted file mode 100644
index 97d2b31..0000000
--- a/tests/env.cwl
+++ /dev/null
@@ -1,5 +0,0 @@
-class: CommandLineTool
-cwlVersion: v1.0
-inputs: []
-outputs: []
-baseCommand: env
\ No newline at end of file
diff --git a/tests/item1.yml b/tests/item1.yml
deleted file mode 100644
index 6f30df9..0000000
--- a/tests/item1.yml
+++ /dev/null
@@ -1,3 +0,0 @@
-inp:
- class: File
- location: "keep:6bf7c0d0f95d3ca89ce7a1d64ea4ff0a+56"
\ No newline at end of file
diff --git a/tests/output.txt b/tests/output.txt
deleted file mode 100644
index 97b096c..0000000
--- a/tests/output.txt
+++ /dev/null
@@ -1,16 +0,0 @@
-yrev hsirehc ,rehto ro emit emos ,eerged rieht ni nem lla tsomla ,ti
-ylteiuq I ;drows sih nopu flesmih sworht otaC hsiruolf lacihposolihp
-yllacidohtem dna ,teerts eht otni gnippets yletarebiled morf em tneverp
-wenk tub yeht fI .siht ni gnisirprus gnihton si erehT .pihs eht ot ekat
-teg sopyh ym revenehw yllaicepse dna ;teem I larenuf yreve fo raer eht
-pu gnignirb dna ,sesuoheraw niffoc erofeb gnisuap yliratnulovni flesym
-ot elpicnirp larom gnorts a seriuqer ti taht ,em fo dnah reppu na hcus
-no em tseretni ot ralucitrap gnihton dna ,esrup ym ni yenom on ro elttil
-gnivah--ylesicerp gnol woh dnim reven--oga sraey emoS .leamhsI em llaC
-gnitaluger dna neelps eht ffo gnivird fo evah I yaw a si tI .dlrow eht
-fo trap yretaw eht ees dna elttil a tuoba lias dluow I thguoht I ,erohs
-dnif I revenehw ;luos ym ni rebmevoN ylzzird ,pmad a si ti revenehw
-aes ot teg ot emit hgih ti tnuocca I ,neht--ffo stah s'elpoep gnikconk
-a htiW .llab dna lotsip rof etutitsbus ym si sihT .nac I sa noos sa
-;htuom eht tuoba mirg gniworg flesym dnif I revenehW .noitalucric eht
-.em htiw naeco eht sdrawot sgnileef emas eht ylraen
diff --git a/tests/test_examples.py b/tests/test_examples.py
index 641cb5b..0b0c609 100644
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -1,5 +1,21 @@
from __future__ import absolute_import
import unittest
+import pytest
+import subprocess
+from os import path
+import sys
+
+from io import StringIO
+
+from cwltool.utils import onWindows
+
+try:
+ reload
+except:
+ try:
+ from imp import reload
+ except:
+ from importlib import reload
import cwltool.expression as expr
import cwltool.factory
@@ -516,5 +532,50 @@ class TestPrintDot(unittest.TestCase):
self.assertEquals(main(["--print-dot", get_data('tests/wf/revsort.cwl')]), 0)
+class TestCmdLine(unittest.TestCase):
+ def get_main_stderr(self, new_args):
+ process = subprocess.Popen([
+ sys.executable,
+ "-m",
+ "cwltool"
+ ] + new_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ stdout, stderr = process.communicate()
+ return process.returncode, stderr.decode()
+
+
+class TestJsConsole(TestCmdLine):
+
+ def test_js_console_cmd_line_tool(self):
+ for test_file in ("js_output.cwl", "js_output_workflow.cwl"):
+ error_code, output = self.get_main_stderr(["--js-console", "--no-container",
+ get_data("tests/wf/" + test_file)])
+
+ self.assertIn("[log] Log message", output)
+ self.assertIn("[err] Error message", output)
+
+ self.assertEquals(error_code, 0, output)
+
+ def test_no_js_console(self):
+ for test_file in ("js_output.cwl", "js_output_workflow.cwl"):
+ error_code, output = self.get_main_stderr(["--no-container",
+ get_data("tests/wf/" + test_file)])
+
+ self.assertNotIn("[log] Log message", output)
+ self.assertNotIn("[err] Error message", output)
+
+
+ at pytest.mark.skipif(onWindows(),
+ reason="Instance of cwltool is used, on Windows it invokes a default docker container"
+ "which is not supported on AppVeyor")
+class TestCache(TestCmdLine):
+ def test_wf_without_container(self):
+ test_file = "hello-workflow.cwl"
+ error_code, output = self.get_main_stderr(["--cachedir", "cache",
+ get_data("tests/wf/" + test_file), "--usermessage", "hello"])
+ self.assertIn("completed success", output)
+ self.assertEquals(error_code, 0)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/tests/test_rdfprint.py b/tests/test_rdfprint.py
new file mode 100644
index 0000000..7b37099
--- /dev/null
+++ b/tests/test_rdfprint.py
@@ -0,0 +1,12 @@
+from __future__ import absolute_import
+import unittest
+
+from six import StringIO
+from cwltool.main import main
+
+from .util import get_data
+
+class RDF_Print(unittest.TestCase):
+
+ def test_rdf_print(self):
+ self.assertEquals(main(['--print-rdf', get_data('tests/wf/hello_single_tool.cwl')]), 0)
diff --git a/tests/value b/tests/value
deleted file mode 100644
index dc7b54a..0000000
--- a/tests/value
+++ /dev/null
@@ -1 +0,0 @@
-33
\ No newline at end of file
diff --git a/tests/wf/4c637b77-8130-4158-807c-ccc78ea7b563 b/tests/wf/4c637b77-8130-4158-807c-ccc78ea7b563
deleted file mode 100644
index 0cfbf08..0000000
--- a/tests/wf/4c637b77-8130-4158-807c-ccc78ea7b563
+++ /dev/null
@@ -1 +0,0 @@
-2
diff --git a/tests/wf/c5b8320e-6e31-4bbb-8453-03b054b3254e b/tests/wf/c5b8320e-6e31-4bbb-8453-03b054b3254e
deleted file mode 100644
index d00491f..0000000
--- a/tests/wf/c5b8320e-6e31-4bbb-8453-03b054b3254e
+++ /dev/null
@@ -1 +0,0 @@
-1
diff --git a/tests/wf/conditional.cwl b/tests/wf/conditional.cwl
deleted file mode 100644
index 57f3654..0000000
--- a/tests/wf/conditional.cwl
+++ /dev/null
@@ -1,22 +0,0 @@
-cwlVersion: v1.0
-class: Workflow
-inputs:
- a: int
-outputs: []
-steps:
- step1:
- in:
- a: a
- cond:
- - case: $(a > 1)
- run: greater.cwl
- - case: default
- run: default.cwl
- out: [out]
- step2:
- in:
- a: a
- cond:
- "$(inputs.a > 1)": greater.cwl
- default: default.cwl
- out: [out]
diff --git a/tests/wf/emptyscatter.cwl b/tests/wf/emptyscatter.cwl
deleted file mode 100644
index 10a17c1..0000000
--- a/tests/wf/emptyscatter.cwl
+++ /dev/null
@@ -1,18 +0,0 @@
-class: Workflow
-cwlVersion: v1.0
-requirements:
- ScatterFeatureRequirement: {}
-inputs:
- inp: string[]
-outputs:
- f:
- type: File[]
- outputSource: boo/out
-
-steps:
- boo:
- in:
- r: inp
- out: [out]
- scatter: r
- run: echo.cwl
diff --git a/tests/wf/emptyscatter.yml b/tests/wf/emptyscatter.yml
deleted file mode 100644
index dd3e18c..0000000
--- a/tests/wf/emptyscatter.yml
+++ /dev/null
@@ -1 +0,0 @@
-inp: []
\ No newline at end of file
diff --git a/tests/wf/foo1.txt b/tests/wf/foo1.txt
deleted file mode 100644
index d00491f..0000000
--- a/tests/wf/foo1.txt
+++ /dev/null
@@ -1 +0,0 @@
-1
diff --git a/tests/wf/foo3.txt b/tests/wf/foo3.txt
deleted file mode 100644
index 00750ed..0000000
--- a/tests/wf/foo3.txt
+++ /dev/null
@@ -1 +0,0 @@
-3
diff --git a/tests/wf/fooa.txt b/tests/wf/fooa.txt
deleted file mode 100644
index 7898192..0000000
--- a/tests/wf/fooa.txt
+++ /dev/null
@@ -1 +0,0 @@
-a
diff --git a/tests/wf/foob.txt b/tests/wf/foob.txt
deleted file mode 100644
index 6178079..0000000
--- a/tests/wf/foob.txt
+++ /dev/null
@@ -1 +0,0 @@
-b
diff --git a/tests/wf/foofoo5.txt.txt b/tests/wf/foofoo5.txt.txt
deleted file mode 100644
index 2182af9..0000000
--- a/tests/wf/foofoo5.txt.txt
+++ /dev/null
@@ -1 +0,0 @@
-foo5.txt
diff --git a/tests/wf/js_output.cwl b/tests/wf/js_output.cwl
new file mode 100644
index 0000000..68ea3af
--- /dev/null
+++ b/tests/wf/js_output.cwl
@@ -0,0 +1,9 @@
+class: CommandLineTool
+cwlVersion: v1.0
+requirements:
+ - class: InlineJavascriptRequirement
+inputs: []
+outputs: []
+arguments:
+ - valueFrom: ${console.log("Log message");console.error("Error message");return ["python", "-c", "True"]}
+ shellQuote: false
\ No newline at end of file
diff --git a/tests/wf/js_output_workflow.cwl b/tests/wf/js_output_workflow.cwl
new file mode 100644
index 0000000..fe52c9d
--- /dev/null
+++ b/tests/wf/js_output_workflow.cwl
@@ -0,0 +1,15 @@
+class: Workflow
+cwlVersion: v1.0
+requirements:
+ - class: InlineJavascriptRequirement
+inputs: []
+outputs: []
+steps:
+ - id: js_log
+ in: []
+ out: []
+ run:
+ class: ExpressionTool
+ inputs: []
+ outputs: []
+ expression: ${console.log("Log message");console.error("Error message");return ["python", "-c", "True"];}
\ No newline at end of file
diff --git a/tests/wf/loop.cwl b/tests/wf/loop.cwl
deleted file mode 100644
index b56ac8b..0000000
--- a/tests/wf/loop.cwl
+++ /dev/null
@@ -1,36 +0,0 @@
-cwlVersion: v1.0
-class: Workflow
-inputs:
- a: int
-outputs: []
-steps:
- step1:
- in:
- a: a
- out: [out]
- while: $(inputs.a > 1)
- run: blah.cwl
- update:
- - source: out
- dest: a
- step2:
- in:
- a: a
- out: [out]
- while: $(inputs.a > 1)
- run: blah.cwl
- update:
- a: out
- step3:
- in:
- a: a
- out:
- out:
- default: "foo"
- scatter: a
- while: $(inputs.a < 3)
- cond:
- "$(inputs.a < 1)": inc.cwl
- else: default.cwl
- update:
- a: out
diff --git a/tests/wf/output.txt b/tests/wf/output.txt
deleted file mode 100644
index 97b096c..0000000
--- a/tests/wf/output.txt
+++ /dev/null
@@ -1,16 +0,0 @@
-yrev hsirehc ,rehto ro emit emos ,eerged rieht ni nem lla tsomla ,ti
-ylteiuq I ;drows sih nopu flesmih sworht otaC hsiruolf lacihposolihp
-yllacidohtem dna ,teerts eht otni gnippets yletarebiled morf em tneverp
-wenk tub yeht fI .siht ni gnisirprus gnihton si erehT .pihs eht ot ekat
-teg sopyh ym revenehw yllaicepse dna ;teem I larenuf yreve fo raer eht
-pu gnignirb dna ,sesuoheraw niffoc erofeb gnisuap yliratnulovni flesym
-ot elpicnirp larom gnorts a seriuqer ti taht ,em fo dnah reppu na hcus
-no em tseretni ot ralucitrap gnihton dna ,esrup ym ni yenom on ro elttil
-gnivah--ylesicerp gnol woh dnim reven--oga sraey emoS .leamhsI em llaC
-gnitaluger dna neelps eht ffo gnivird fo evah I yaw a si tI .dlrow eht
-fo trap yretaw eht ees dna elttil a tuoba lias dluow I thguoht I ,erohs
-dnif I revenehw ;luos ym ni rebmevoN ylzzird ,pmad a si ti revenehw
-aes ot teg ot emit hgih ti tnuocca I ,neht--ffo stah s'elpoep gnikconk
-a htiW .llab dna lotsip rof etutitsbus ym si sihT .nac I sa noos sa
-;htuom eht tuoba mirg gniworg flesym dnif I revenehW .noitalucric eht
-.em htiw naeco eht sdrawot sgnileef emas eht ylraen
diff --git a/tests/wf/revsort-ovr2-job.json b/tests/wf/revsort-ovr2-job.json
deleted file mode 100644
index 4928b9d..0000000
--- a/tests/wf/revsort-ovr2-job.json
+++ /dev/null
@@ -1,15 +0,0 @@
-{
- "input": {
- "class": "File",
- "location": "whale.txt"
- },
- "cwltool:overrides": [
- {
- "overrideTarget": "sorttool.cwl",
- "override": [{
- "class": "DockerRequirement",
- "dockerPull": "ubuntu:14.04"
- }]
- }
- ]
-}
diff --git a/tests/wf/value b/tests/wf/value
deleted file mode 100644
index 43c451e..0000000
--- a/tests/wf/value
+++ /dev/null
@@ -1 +0,0 @@
-54
\ No newline at end of file
diff --git a/tests/wf/whale.txt b/tests/wf/whale.txt
deleted file mode 100644
index 425d1ed..0000000
--- a/tests/wf/whale.txt
+++ /dev/null
@@ -1,16 +0,0 @@
-Call me Ishmael. Some years ago--never mind how long precisely--having
-little or no money in my purse, and nothing particular to interest me on
-shore, I thought I would sail about a little and see the watery part of
-the world. It is a way I have of driving off the spleen and regulating
-the circulation. Whenever I find myself growing grim about the mouth;
-whenever it is a damp, drizzly November in my soul; whenever I find
-myself involuntarily pausing before coffin warehouses, and bringing up
-the rear of every funeral I meet; and especially whenever my hypos get
-such an upper hand of me, that it requires a strong moral principle to
-prevent me from deliberately stepping into the street, and methodically
-knocking people's hats off--then, I account it high time to get to sea
-as soon as I can. This is my substitute for pistol and ball. With a
-philosophical flourish Cato throws himself upon his sword; I quietly
-take to the ship. There is nothing surprising in this. If they but knew
-it, almost all men in their degree, some time or other, cherish very
-nearly the same feelings towards the ocean with me.
diff --git a/tests/writeable.cwl b/tests/writeable.cwl
deleted file mode 100644
index a635e79..0000000
--- a/tests/writeable.cwl
+++ /dev/null
@@ -1,11 +0,0 @@
-class: CommandLineTool
-cwlVersion: v1.0
-baseCommand: [ls, -l]
-requirements:
- InitialWorkDirRequirement:
- listing:
- - entry: $(inputs.blah)
- writable: true
-inputs:
- blah: Directory
-outputs: []
\ No newline at end of file
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/cwltool.git
More information about the debian-med-commit
mailing list