[med-svn] [Git][med-team/toil][master] 4 commits: TODO: make local-test
Michael R. Crusoe
gitlab at salsa.debian.org
Sun Feb 7 12:27:16 GMT 2021
Michael R. Crusoe pushed to branch master at Debian Med / toil
Commits:
d47fa861 by Michael R. Crusoe at 2021-01-26T18:01:34+01:00
TODO: make local-test
- - - - -
dda26a49 by Michael R. Crusoe at 2021-02-04T12:17:00+01:00
Enable toil-wdl-runner now that python3-wdlparse is in Debian.
- - - - -
2539e2d9 by Michael R. Crusoe at 2021-02-07T13:25:07+01:00
Runs the tests during the build
and as autopkgtests!
- - - - -
96eb64c9 by Michael R. Crusoe at 2021-02-07T13:26:16+01:00
routine-update: Ready to upload to unstable
- - - - -
13 changed files:
- debian/changelog
- debian/clean
- debian/control
- + debian/patches/proxy
- + debian/patches/python3
- debian/patches/series
- + debian/patches/ship_tests
- + debian/patches/soften-pydocker-dep
- − debian/patches/wdlparse_not_available
- debian/rules
- debian/tests/control
- + debian/tests/run-unit-tests
- debian/tests/smoke-test
Changes:
=====================================
debian/changelog
=====================================
@@ -1,3 +1,10 @@
+toil (5.2.0-2) unstable; urgency=high
+
+ * Enable toil-wdl-runner now that python3-wdlparse is in Debian.
+ * Enabled running the the tests during the build!
+
+ -- Michael R. Crusoe <crusoe at debian.org> Sun, 07 Feb 2021 13:26:16 +0100
+
toil (5.2.0-1) unstable; urgency=medium
* New upstream version
=====================================
debian/clean
=====================================
@@ -1,4 +1,6 @@
+toilWorkflowRun
version_template.pyc
+.pytest_cache/
src/toil/version.pyc
src/toil.egg-info/
docs/vendor/sphinxcontrib/__pycache__/
=====================================
debian/control
=====================================
@@ -11,10 +11,16 @@ Build-Depends: debhelper-compat (= 13),
python3-docker,
python3-dill,
python3-future,
- python3-pytest-xdist,
cwltool (>= 3.0.20201203173111),
python3-distutils,
python3-enlighten,
+ python3-wdlparse,
+ python3-pytest <!nocheck>,
+ python3-mock <!nocheck>,
+ python3-boto3 <!nocheck>,
+ python3-boto <!nocheck>,
+ python3-stubserver <!nocheck>,
+ rsync <!nocheck>,
# documentation
python3-urllib3,
python3-sphinx
@@ -32,6 +38,8 @@ Depends: ${python3:Depends},
cwltool
Recommends: python3-boto,
python3-boto3,
+# for WDL support
+ python3-wdlparse,
# for google cloud support
python3-libcloud
# for mesos support, but needs python3-mesos first
=====================================
debian/patches/proxy
=====================================
@@ -0,0 +1,27 @@
+From: Michael R. Crusoe <crusoe at debian.org>
+Subject: cope with an invalid HTTP_PROXY
+Forwarded: https://github.com/DataBiosphere/toil/pull/3447
+
+Debian purposely sets an invalid HTTP_PROXY during some build situations
+--- toil.orig/src/toil/jobStores/abstractJobStore.py
++++ toil/src/toil/jobStores/abstractJobStore.py
+@@ -37,6 +37,10 @@
+
+ logger = logging.getLogger(__name__)
+
++try:
++ from botocore.exceptions import ProxyConnectionError
++except ImportError:
++ ProxyConnectionError = None
+
+ class InvalidImportExportUrlException(Exception):
+ def __init__(self, url):
+@@ -243,7 +247,7 @@
+ from importlib import import_module
+ try:
+ module = import_module(moduleName)
+- except ImportError:
++ except (ImportError, ProxyConnectionError):
+ logger.debug("Unable to import '%s' as is expected if the corresponding extra was "
+ "omitted at installation time.", moduleName)
+ else:
=====================================
debian/patches/python3
=====================================
@@ -0,0 +1,41 @@
+From: Michael R. Crusoe <crusoe at debian.org>
+Subject: Ensure that python3 and not python (2) is used
+Forwarded: https://github.com/DataBiosphere/toil/pull/3446
+--- toil.orig/src/toil/test/utils/toilDebugTest.py
++++ toil/src/toil/test/utils/toilDebugTest.py
+@@ -19,7 +19,7 @@
+
+ from toil.test import ToilTest, slow, travis_test
+ from toil.utils.toilDebugFile import recursiveGlob
+-
++from toil.version import python
+ logger = logging.getLogger(__name__)
+
+ class ToilDebugFileTest(ToilTest):
+@@ -27,7 +27,7 @@
+
+ def setUp(self):
+ """Initial set up of variables for the test."""
+- subprocess.check_call(['python', os.path.abspath('src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py')])
++ subprocess.check_call([python, os.path.abspath('src/toil/test/utils/ABCWorkflowDebug/debugWorkflow.py')])
+ self.jobStoreDir = os.path.abspath('toilWorkflowRun')
+ self.tempDir = self._createTempDir(purpose='tempDir')
+ self.outputDir = os.path.abspath('testoutput')
+@@ -54,7 +54,7 @@
+
+ contents = ['A.txt', 'B.txt', 'C.txt', 'ABC.txt', 'mkFile.py']
+
+- subprocess.check_call(['python', os.path.abspath('src/toil/utils/toilDebugFile.py'), self.jobStoreDir, '--listFilesInJobStore=True'])
++ subprocess.check_call([python, os.path.abspath('src/toil/utils/toilDebugFile.py'), self.jobStoreDir, '--listFilesInJobStore=True'])
+ jobstoreFileContents = os.path.abspath('jobstore_files.txt')
+ files = []
+ match = 0
+@@ -95,7 +95,7 @@
+ then delete them.
+ """
+ contents = ['A.txt', 'B.txt', 'C.txt', 'ABC.txt', 'mkFile.py']
+- cmd = ['python', os.path.abspath('src/toil/utils/toilDebugFile.py'),
++ cmd = [python, os.path.abspath('src/toil/utils/toilDebugFile.py'),
+ self.jobStoreDir,
+ '--fetch', '*A.txt', '*B.txt', '*C.txt', '*ABC.txt', '*mkFile.py',
+ '--localFilePath=' + self.outputDir,
=====================================
debian/patches/series
=====================================
@@ -3,4 +3,7 @@ setting_version.patch
no_galaxy_lib
debianize_docs
soften-mesos-deps
-wdlparse_not_available
+soften-pydocker-dep
+python3
+proxy
+ship_tests
=====================================
debian/patches/ship_tests
=====================================
@@ -0,0 +1,20 @@
+From: Michael R. Crusoe <crusoe at debian.org>
+Subject: ship the tests
+Forwarded: not-needed
+--- toil.orig/setup.py
++++ toil/setup.py
+@@ -132,12 +132,9 @@
+ 'wdl': wdl_reqs,
+ 'all': all_reqs},
+ package_dir={'': 'src'},
+- packages=find_packages(where='src',
+- # Note that we intentionally include the top-level `test` package for
+- # functionality like the @experimental and @integrative decorators:
+- exclude=['*.test.*']),
++ packages=find_packages(where='src'),
+ package_data={
+- '': ['*.yml', 'cloud-config'],
++ '': ['*.yml', 'cloud-config', '*.cwl'],
+ },
+ # Unfortunately, the names of the entry points are hard-coded elsewhere in the code base so
+ # you can't just change them here. Luckily, most of them are pretty unique strings, and thus
=====================================
debian/patches/soften-pydocker-dep
=====================================
@@ -0,0 +1,14 @@
+Author: Michael R. Crusoe <crusoe at debian.org>
+Description: Allow for use of python3-docker package
+Forwarded: not-needed
+--- toil.orig/setup.py
++++ toil/setup.py
+@@ -40,7 +40,7 @@
+ pytz = 'pytz>=2012'
+ dill = 'dill>=0.3.2, <0.4'
+ requests = 'requests>=2, <3'
+- docker = 'docker==4.3.1'
++ docker = 'docker'
+ dateutil = 'python-dateutil'
+ enlighten = 'enlighten>=1.5.2, <2'
+ wdlparse = 'wdlparse==0.1.0'
=====================================
debian/patches/wdlparse_not_available deleted
=====================================
@@ -1,49 +0,0 @@
-Author: Michael R. Crusoe <crusoe at debian.org>
-Description: wdlparse is not yet packaged for Debian
-Forwarded: not-needed
---- toil.orig/setup.py
-+++ toil/setup.py
-@@ -43,7 +43,6 @@
- docker = 'docker==4.3.1'
- dateutil = 'python-dateutil'
- enlighten = 'enlighten>=1.5.2, <2'
-- wdlparse = 'wdlparse==0.1.0'
-
- core_reqs = [
- dill,
-@@ -74,9 +73,6 @@
- mesos_reqs = [
- pymesos,
- psutil]
-- wdl_reqs = [
-- wdlparse
-- ]
-
- # htcondor is not supported by apple
- # this is tricky to conditionally support in 'all' due
-@@ -88,8 +84,7 @@
- encryption_reqs + \
- google_reqs + \
- kubernetes_reqs + \
-- mesos_reqs + \
-- wdl_reqs
-+ mesos_reqs
-
- setup(
- name='toil',
-@@ -129,7 +124,6 @@
- 'htcondor:sys_platform!="darwin"': htcondor_reqs,
- 'kubernetes': kubernetes_reqs,
- 'mesos': mesos_reqs,
-- 'wdl': wdl_reqs,
- 'all': all_reqs},
- package_dir={'': 'src'},
- packages=find_packages(where='src',
-@@ -148,7 +142,6 @@
- '_toil_worker = toil.worker:main',
- 'cwltoil = toil.cwl.cwltoil:cwltoil_was_removed [cwl]',
- 'toil-cwl-runner = toil.cwl.cwltoil:main [cwl]',
-- 'toil-wdl-runner = toil.wdl.toilwdl:main',
- '_toil_mesos_executor = toil.batchSystems.mesos.executor:main [mesos]',
- '_toil_kubernetes_executor = toil.batchSystems.kubernetes:executor [kubernetes]']})
-
=====================================
debian/rules
=====================================
@@ -5,9 +5,8 @@ export PYBUILD_NAME=toil
export PYBUILD_DESTDIR_python3=debian/toil/
-export PYBUILD_DISABLE=test
+#export PYBUILD_DISABLE=test
export PYBUILD_DISABLE_python2=1
-#CURDIR ?= $(realpath $(dir $(firstword $(MAKEFILE_LIST)))/..)
%:
dh $@ --with python3 --buildsystem=pybuild
@@ -21,15 +20,14 @@ override_dh_auto_install:
# remove the below once mesos is packaged and tested
find $(CURDIR)/debian -name _toil_mesos_executor -delete
-# more py3 test fixes need to be cherry-picked from upstream
-# override_dh_auto_test:
-# PYBUILD_SYSTEM=custom \
-# PYBUILD_TEST_ARGS='TOIL_SKIP_DOCKER=True {interpreter} -m pytest -vv \
-# -W ignore \
-# -k "not test_bioconda and not test_run_conformance and not testImportFtpFile" -n auto\
- {dir}/src/toil/test' dh_auto_test --buildsystem pybuild
+override_dh_auto_test:
+ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
+ PYBUILD_SYSTEM=custom \
+ PYBUILD_TEST_ARGS='HOME={home_dir} {interpreter} setup.py develop --user && PYTHONPATH={dir}/src:$$PYTHONPATH PATH={home_dir}/.local/bin/:$$PATH TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True {interpreter} -m pytest -vv -W ignore --ignore src/toil/test/provisioners/aws/awsProvisionerTest.py --ignore src/toil/test/wdl/toilwdlTest.py --ignore src/toil/test/cwl/cwlTest.py -k "not test_bioconda and not test_run_conformance and not testImportFtpFile and not ToilWdlIntegrationTest and not SortTest" {dir}/src/toil/test' \
+ dh_auto_test
+endif
- # If you need to rebuild the Sphinx documentation
+# If you need to rebuild the Sphinx documentation
# Add spinxdoc to the dh --with line
#override_dh_auto_build:
# dh_auto_build
=====================================
debian/tests/control
=====================================
@@ -1,3 +1,7 @@
Tests: smoke-test
-Depends: toil
+Depends: toil, python3-wdlparse
Restrictions: superficial
+
+Tests: run-unit-tests
+Depends: toil, python3-wdlparse, python3-pytest, python3-mock, python3-boto3, rsync, python3-boto
+Restrictions: allow-stderr
=====================================
debian/tests/run-unit-tests
=====================================
@@ -0,0 +1,13 @@
+#!/bin/sh -ex
+
+pkg=toil
+
+export LC_ALL=C.UTF-8
+# if [ "${AUTOPKGTEST_TMP}" = "" ] ; then
+# AUTOPKGTEST_TMP=$(mktemp -d /tmp/${pkg}-test.XXXXXX)
+# trap "rm -rf ${AUTOPKGTEST_TMP}" 0 INT QUIT ABRT PIPE TERM
+# fi
+#
+# cd "${AUTOPKGTEST_TMP}"
+
+TOIL_TEST_QUICK=True TOIL_SKIP_DOCKER=True python3 -m pytest -vv -W ignore -k "not test_bioconda and not testCwlexample and not CWLv10Test and not CWLv11Test and not CWLv12Test" --ignore-glob '*cwlTest*' --pyargs toil.test
=====================================
debian/tests/smoke-test
=====================================
@@ -1,3 +1,4 @@
#!/bin/sh -e
toil --help
toil-cwl-runner --help
+toil-wdl-runner --help
View it on GitLab: https://salsa.debian.org/med-team/toil/-/compare/f63eb972952f6be44b41d31fdad93363a28baa8b...96eb64c9f838600841ec3f54e2463662e0df7296
--
View it on GitLab: https://salsa.debian.org/med-team/toil/-/compare/f63eb972952f6be44b41d31fdad93363a28baa8b...96eb64c9f838600841ec3f54e2463662e0df7296
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20210207/a968edf9/attachment-0001.html>
More information about the debian-med-commit
mailing list