[med-svn] [Git][med-team/cwltool][upstream] New upstream version 3.1.20241112140730
Michael R. Crusoe (@crusoe)
gitlab at salsa.debian.org
Sun Dec 15 23:12:25 GMT 2024
Michael R. Crusoe pushed to branch upstream at Debian Med / cwltool
Commits:
c7d4649c by Michael R. Crusoe at 2024-12-16T00:00:40+01:00
New upstream version 3.1.20241112140730
- - - - -
12 changed files:
- .circleci/config.yml
- .github/workflows/wheels.yml
- PKG-INFO
- README.rst
- conformance-test.sh
- cwltool.egg-info/PKG-INFO
- cwltool/_version.py
- cwltool/executors.py
- cwltool/singularity.py
- lint-requirements.txt
- pyproject.toml
- tests/test_environment.py
Changes:
=====================================
.circleci/config.yml
=====================================
@@ -15,8 +15,8 @@ jobs:
type: string
machine:
- image: ubuntu-2204:current
- resource_class: arm.medium # two vCPUs
+ image: ubuntu-2404:current
+ resource_class: arm.medium # 2 vCPUs
environment:
CIBW_ARCHS: "aarch64"
@@ -84,20 +84,17 @@ workflows:
- arm-wheels:
name: arm-wheels-manylinux_2_28
filters:
+ branches:
+ only: main
tags:
only: /.*/
build: "*manylinux*"
image: quay.io/pypa/manylinux_2_28_aarch64
- - arm-wheels:
- name: arm-wheels-musllinux_1_1
- filters:
- tags:
- only: /.*/
- build: "*musllinux*"
- image: quay.io/pypa/musllinux_1_1_aarch64
- arm-wheels:
name: arm-wheels-musllinux_1_2
filters:
+ branches:
+ only: main
tags:
only: /.*/
build: "*musllinux*"
=====================================
.github/workflows/wheels.yml
=====================================
@@ -5,7 +5,6 @@ on:
types: [published]
workflow_dispatch: {}
repository_dispatch: {}
- pull_request:
push:
branches:
- main
@@ -23,8 +22,6 @@ jobs:
include:
- image: manylinux_2_28_x86_64
build: "*manylinux*"
- - image: musllinux_1_1_x86_64
- build: "*musllinux*"
- image: musllinux_1_2_x86_64
build: "*musllinux*"
=====================================
PKG-INFO
=====================================
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: cwltool
-Version: 3.1.20241024121129
+Version: 3.1.20241112140730
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Download-URL: https://github.com/common-workflow-language/cwltool
@@ -241,6 +241,22 @@ and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
$ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
+
+Docker default platform on macOS with Apple Silicon
+===================================================
+
+If macOS users want to run CWL tools/workflows using ``cwltool`` with Docker and their software containers only have support for amd64 (64-bit x86) CPUs, but they have an Apple Silicon (aarch64/arm64) CPU,
+they run into the error:
+
+ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.
+
+To fix this, export the following environment variable before executing `cwltool`:
+
+``export DOCKER_DEFAULT_PLATFORM=linux/amd64``
+
+To automatically have this variable set in the future, add the same command to ones respective shell profile (e.g. ``~/.zshrc``, ``~/.bash_profile``).
+
+
Using uDocker
=============
=====================================
README.rst
=====================================
@@ -189,6 +189,22 @@ and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
$ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
+
+Docker default platform on macOS with Apple Silicon
+===================================================
+
+If macOS users want to run CWL tools/workflows using ``cwltool`` with Docker and their software containers only have support for amd64 (64-bit x86) CPUs, but they have an Apple Silicon (aarch64/arm64) CPU,
+they run into the error:
+
+ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.
+
+To fix this, export the following environment variable before executing `cwltool`:
+
+``export DOCKER_DEFAULT_PLATFORM=linux/amd64``
+
+To automatically have this variable set in the future, add the same command to ones respective shell profile (e.g. ``~/.zshrc``, ``~/.bash_profile``).
+
+
Using uDocker
=============
=====================================
conformance-test.sh
=====================================
@@ -97,19 +97,7 @@ if [[ "$VERSION" = *dev* ]]
then
CWLTOOL_OPTIONS+=" --enable-dev"
fi
-if [[ "$CONTAINER" = "singularity" ]]; then
- CWLTOOL_OPTIONS+=" --singularity"
- # This test fails because Singularity and Docker have
- # different views on how to deal with this.
- exclusions+=(docker_entrypoint)
- if [[ "${VERSION}" = "v1.1" ]]; then
- # This fails because of a difference (in Singularity vs Docker) in
- # the way filehandles are passed to processes in the container and
- # wc can tell somehow.
- # See issue #1440
- exclusions+=(stdin_shorcut)
- fi
-elif [[ "$CONTAINER" = "podman" ]]; then
+if [[ "$CONTAINER" = "podman" ]]; then
CWLTOOL_OPTIONS+=" --podman"
fi
=====================================
cwltool.egg-info/PKG-INFO
=====================================
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: cwltool
-Version: 3.1.20241024121129
+Version: 3.1.20241112140730
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Download-URL: https://github.com/common-workflow-language/cwltool
@@ -241,6 +241,22 @@ and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
$ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
+
+Docker default platform on macOS with Apple Silicon
+===================================================
+
+If macOS users want to run CWL tools/workflows using ``cwltool`` with Docker and their software containers only have support for amd64 (64-bit x86) CPUs, but they have an Apple Silicon (aarch64/arm64) CPU,
+they run into the error:
+
+ WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.
+
+To fix this, export the following environment variable before executing `cwltool`:
+
+``export DOCKER_DEFAULT_PLATFORM=linux/amd64``
+
+To automatically have this variable set in the future, add the same command to ones respective shell profile (e.g. ``~/.zshrc``, ``~/.bash_profile``).
+
+
Using uDocker
=============
=====================================
cwltool/_version.py
=====================================
@@ -12,5 +12,5 @@ __version__: str
__version_tuple__: VERSION_TUPLE
version_tuple: VERSION_TUPLE
-__version__ = version = '3.1.20241024121129'
-__version_tuple__ = version_tuple = (3, 1, 20241024121129)
+__version__ = version = '3.1.20241112140730'
+__version_tuple__ = version_tuple = (3, 1, 20241112140730)
=====================================
cwltool/executors.py
=====================================
@@ -273,7 +273,7 @@ class MultithreadedJobExecutor(JobExecutor):
self.pending_jobs_lock = threading.Lock()
self.max_ram = int(psutil.virtual_memory().available / 2**20)
- self.max_cores = float(psutil.cpu_count())
+ self.max_cores = float(psutil.cpu_count() or 1)
self.max_cuda = cuda_version_and_device_count()[1]
self.allocated_ram = float(0)
self.allocated_cores = float(0)
@@ -429,7 +429,7 @@ class MultithreadedJobExecutor(JobExecutor):
logger: logging.Logger,
runtime_context: RuntimeContext,
) -> None:
- self.taskqueue: TaskQueue = TaskQueue(threading.Lock(), psutil.cpu_count())
+ self.taskqueue: TaskQueue = TaskQueue(threading.Lock(), int(math.ceil(self.max_cores)))
try:
jobiter = process.job(job_order_object, self.output_callback, runtime_context)
=====================================
cwltool/singularity.py
=====================================
@@ -75,6 +75,14 @@ def is_apptainer_1_or_newer() -> bool:
return v[0][0] >= 1
+def is_apptainer_1_1_or_newer() -> bool:
+ """Check if apptainer singularity distribution is version 1.1 or higher."""
+ v = get_version()
+ if v[1] != "apptainer":
+ return False
+ return v[0][0] >= 2 or (v[0][0] >= 1 and v[0][1] >= 1)
+
+
def is_version_2_6() -> bool:
"""
Check if this singularity version is exactly version 2.6.
@@ -119,6 +127,12 @@ def is_version_3_9_or_newer() -> bool:
return v[0][0] >= 4 or (v[0][0] == 3 and v[0][1] >= 9)
+def is_version_3_10_or_newer() -> bool:
+ """Detect if Singularity v3.10+ is available."""
+ v = get_version()
+ return v[0][0] >= 4 or (v[0][0] == 3 and v[0][1] >= 10)
+
+
def _normalize_image_id(string: str) -> str:
return string.replace("/", "_") + ".img"
@@ -464,14 +478,18 @@ class SingularityCommandLineJob(ContainerCommandLineJob):
) -> tuple[list[str], Optional[str]]:
"""Return the Singularity runtime list of commands and options."""
any_path_okay = self.builder.get_requirement("DockerRequirement")[1] or False
+
runtime = [
"singularity",
"--quiet",
- "exec",
+ "run" if is_apptainer_1_1_or_newer() or is_version_3_10_or_newer() else "exec",
"--contain",
"--ipc",
"--cleanenv",
]
+ if is_apptainer_1_1_or_newer() or is_version_3_10_or_newer():
+ runtime.append("--no-eval")
+
if singularity_supports_userns():
runtime.append("--userns")
else:
=====================================
lint-requirements.txt
=====================================
@@ -1,3 +1,3 @@
-flake8-bugbear<24.9
+flake8-bugbear<24.11
black==24.*
codespell
=====================================
pyproject.toml
=====================================
@@ -6,7 +6,7 @@ requires = [
"types-requests",
"types-psutil",
"importlib_resources>=1.4;python_version<'3.9'",
- "ruamel.yaml>=0.16.0,<0.18",
+ "ruamel.yaml>=0.16.0,<0.19",
"schema-salad>=8.7,<9",
"cwl-utils>=0.32",
"toml",
@@ -18,7 +18,7 @@ build-backend = "setuptools.build_meta"
write_to = "cwltool/_version.py"
[tool.cibuildwheel]
-test-command = "python -m pytest -n 2 --junitxml={project}/test-results/junit_$(python -V | awk '{print $2}')_${AUDITWHEEL_PLAT}.xml -k 'not (test_bioconda or test_env_filtering or test_udocker)' --pyargs cwltool"
+test-command = "python -m pytest --ignore cwltool/schemas -n logical --dist worksteal --junitxml={project}/test-results/junit_$(python -V | awk '{print $2}')_${AUDITWHEEL_PLAT}.xml -k 'not (test_bioconda or test_env_filtering or test_udocker)' --pyargs cwltool"
test-requires = "-r test-requirements.txt"
test-extras = "deps"
skip = "pp*"
=====================================
tests/test_environment.py
=====================================
@@ -159,6 +159,9 @@ class Singularity(CheckHolder):
return v.startswith(tmp_prefix) and v.endswith(":/tmp")
sing_vars["SINGULARITY_BIND"] = BIND
+ if vminor >= 10:
+ sing_vars["SINGULARITY_COMMAND"] = "run"
+ sing_vars["SINGULARITY_NO_EVAL"] = None
result.update(sing_vars)
View it on GitLab: https://salsa.debian.org/med-team/cwltool/-/commit/c7d4649c10bc15ba7ca56f3fce3c295c33c130fb
--
View it on GitLab: https://salsa.debian.org/med-team/cwltool/-/commit/c7d4649c10bc15ba7ca56f3fce3c295c33c130fb
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20241215/c2a33ce7/attachment-0001.htm>
More information about the debian-med-commit
mailing list