[med-svn] [Git][med-team/snakemake][debian-v7] 3 commits: Revert debugging changes.
Rebecca N. Palmer (@rnpalmer-guest)
gitlab at salsa.debian.org
Wed Aug 7 07:59:05 BST 2024
Rebecca N. Palmer pushed to branch debian-v7 at Debian Med / snakemake
Commits:
f2c932ea by Rebecca N. Palmer at 2024-08-07T07:31:50+01:00
Revert debugging changes.
- - - - -
2fee8b0b by Rebecca N. Palmer at 2024-08-07T07:33:30+01:00
test-depend on glpk-utils
- - - - -
ecd73d1f by Rebecca N. Palmer at 2024-08-07T07:58:32+01:00
Tests: allow extra time to avoid failing on riscv64.
- - - - -
8 changed files:
- debian/changelog
- debian/control
- − debian/patches/cancelscript_debug.patch
- debian/patches/series
- + debian/patches/test_extra_time.patch
- debian/rules
- debian/tests/control
- debian/tests/run-unit-test
Changes:
=====================================
debian/changelog
=====================================
@@ -1,21 +1,10 @@
-snakemake (7.32.4-4+ppa3) unstable; urgency=medium
+snakemake (7.32.4-5) unstable; urgency=medium
- * Tests: put the cache somewhere we're allowed to write to.
-
- -- Rebecca N. Palmer <rebecca_palmer at zoho.com> Mon, 05 Aug 2024 21:41:16 +0100
-
-snakemake (7.32.4-4+ppa2) unstable; urgency=medium
-
- * Tests: ignore nocheck to get a log on riscv64.
-
- -- Rebecca N. Palmer <rebecca_palmer at zoho.com> Mon, 05 Aug 2024 08:02:27 +0100
-
-snakemake (7.32.4-4+ppa1) unstable; urgency=medium
-
- * fail tests to get a log, extra debug output
+ * Tests: revert extra debug output.
* Tests: make test_*_solver actually call the solver.
+ * Tests: allow extra time to avoid failing on riscv64.
- -- Rebecca N. Palmer <rebecca_palmer at zoho.com> Sun, 04 Aug 2024 15:34:39 +0100
+ -- Rebecca N. Palmer <rebecca_palmer at zoho.com> Wed, 07 Aug 2024 07:57:41 +0100
snakemake (7.32.4-4) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -9,38 +9,38 @@ Build-Depends: ca-certificates,
dh-python,
pybuild-plugin-pyproject,
environment-modules,
- glpk-utils [amd64],
+ glpk-utils <!nocheck>,
imagemagick,
python3,
- python3-platformdirs,
- python3-biopython,
- python3-boto3,
- python3-botocore,
+ python3-platformdirs <!nocheck>,
+ python3-biopython <!nocheck>,
+ python3-boto3 <!nocheck>,
+ python3-botocore <!nocheck>,
python3-configargparse,
python3-connection-pool,
python3-datrie,
python3-docutils,
- python3-dropbox,
- python3-flask,
+ python3-dropbox <!nocheck>,
+ python3-flask <!nocheck>,
python3-git,
python3-jinja2 (>= 3.0~),
python3-jsonschema,
python3-nbformat,
python3-packaging,
- python3-pandas,
+ python3-pandas <!nocheck>,
python3-pkg-resources,
python3-psutil,
python3-pulp (>= 2.0~),
- python3-pygments,
- python3-pytest,
- python3-pytools,
+ python3-pygments <!nocheck>,
+ python3-pytest <!nocheck>,
+ python3-pytools <!nocheck>,
python3-ratelimiter,
python3-recommonmark <!nodoc>,
python3-requests,
- python3-requests-mock,
+ python3-requests-mock <!nocheck>,
python3-retry,
- python3-pytest-mock,
- python3-rpy2,
+ python3-pytest-mock <!nocheck>,
+ python3-rpy2 <!nocheck>,
python3-setuptools,
python3-smart-open (>= 3.0~),
python3-sphinx (>= 3.0~) <!nodoc>,
@@ -49,13 +49,13 @@ Build-Depends: ca-certificates,
python3-sphinx-rtd-theme <!nodoc>,
python3-stopit,
python3-tabulate,
- python3-throttler,
- python3-toposort,
+ python3-throttler <!nocheck>,
+ python3-toposort <!nocheck>,
python3-wrapt,
python3-yaml,
- python3-yte,
+ python3-yte <!nocheck>,
r-cran-rmarkdown,
- stress
+ stress <!nocheck>
Standards-Version: 4.7.0
Vcs-Browser: https://salsa.debian.org/med-team/snakemake
Vcs-Git: https://salsa.debian.org/med-team/snakemake.git
@@ -63,7 +63,7 @@ Homepage: https://snakemake.readthedocs.io/
Rules-Requires-Root: no
Package: snakemake
-Architecture: any
+Architecture: all
Depends: ${misc:Depends},
${python3:Depends},
ca-certificates,
=====================================
debian/patches/cancelscript_debug.patch deleted
=====================================
@@ -1,164 +0,0 @@
-Description: Extra debug output from test_cluster_cancel*
-
-(failing on riscv64)
-
-Author: Rebecca N. Palmer <rebecca_palmer at zoho.com>
-Forwarded: not-needed
-
---- a/snakemake/sourcecache.py
-+++ b/snakemake/sourcecache.py
-@@ -350,6 +350,7 @@ class SourceCache:
- self.cache = Path(
- os.path.join(get_appdirs().user_cache_dir, "snakemake/source-cache")
- )
-+ print("cache path", self.cache)
- os.makedirs(self.cache, exist_ok=True)
- if runtime_cache_path is None:
- runtime_cache_parent = self.cache / "runtime-cache"
---- a/tests/common.py
-+++ b/tests/common.py
-@@ -204,6 +204,7 @@ def run(
- if not shellcmd.startswith("snakemake"):
- raise ValueError("shellcmd does not start with snakemake")
- shellcmd = "{} -m {}".format(sys.executable, shellcmd)
-+ run_stdout = ""
- try:
- if sigint_after is None:
- subprocess.run(
-@@ -226,6 +227,7 @@ def run(
- process.send_signal(signal.SIGINT)
- time.sleep(2)
- success = process.returncode == 0
-+ run_stdout = process.stdout.read().decode()
- except subprocess.CalledProcessError as e:
- success = False
- print(e.stdout.decode(), file=sys.stderr)
-@@ -251,6 +253,10 @@ def run(
- )
-
- if shouldfail:
-+ if shellcmd and "cancel" in shellcmd:
-+ import io
-+ print("buffer size",io.DEFAULT_BUFFER_SIZE,"cancel stdout:",run_stdout,"\nWorkdir:")
-+ print_tree(tmpdir, exclude=".snakemake/conda")
- assert not success, "expected error on execution"
- else:
- if not success:
---- a/tests/tests.py
-+++ b/tests/tests.py
-@@ -133,11 +133,13 @@ def test_cluster_cancelscript():
- cleanup=False,
- sigint_after=4,
- )
-+ assert os.path.exists(outdir+"/scancel.txt"), str(glob.glob(outdir+'/**/*',recursive=True))
- scancel_txt = open("%s/scancel.txt" % outdir).read()
- scancel_lines = scancel_txt.splitlines()
- assert len(scancel_lines) == 1
- assert scancel_lines[0].startswith("cancel")
- assert len(scancel_lines[0].split(" ")) == 3
-+ assert False, "failing to show the logs, files="+ str(glob.glob(outdir+'/**/*',recursive=True))
-
-
- @skip_on_windows
-@@ -162,6 +164,7 @@ def test_cluster_cancelscript_nargs1():
- cleanup=False,
- sigint_after=4,
- )
-+ assert os.path.exists(outdir+"/scancel.txt"), str(glob.glob(outdir+'/**/*',recursive=True))
- scancel_txt = open("%s/scancel.txt" % outdir).read()
- scancel_lines = scancel_txt.splitlines()
- assert len(scancel_lines) == 2
-@@ -169,6 +172,7 @@ def test_cluster_cancelscript_nargs1():
- assert scancel_lines[1].startswith("cancel")
- assert len(scancel_lines[0].split(" ")) == 2
- assert len(scancel_lines[1].split(" ")) == 2
-+ assert False, "failing to show the logs, files="+ str(glob.glob(outdir+'/**/*',recursive=True))
-
-
- @skip_on_windows
-@@ -188,11 +192,13 @@ def test15():
-
- def test_glpk_solver():
- run(dpath("test_solver"), scheduler_ilp_solver="GLPK_CMD")
-+ assert False, "failing to show the logs"
-
-
- @skip_on_windows
- def test_coin_solver():
- run(dpath("test_solver"), scheduler_ilp_solver="COIN_CMD")
-+ assert False, "failing to show the logs"
-
-
- def test_directory():
---- a/snakemake/exceptions.py
-+++ b/snakemake/exceptions.py
-@@ -143,6 +143,7 @@ def print_exception(ex, linemaps):
- )
- elif isinstance(ex, KeyboardInterrupt):
- logger.info("Cancelling snakemake on user request.")
-+ traceback.print_exception(type(ex), ex, ex.__traceback__)
- else:
- traceback.print_exception(type(ex), ex, ex.__traceback__)
-
---- a/snakemake/executors/__init__.py
-+++ b/snakemake/executors/__init__.py
-@@ -704,6 +704,7 @@ class CPUExecutor(RealExecutor):
- self.pool.shutdown()
-
- def cancel(self):
-+ logger.info('cancelling CPUExecutor')
- self.pool.shutdown()
-
- def _callback(
-@@ -900,6 +901,7 @@ class ClusterExecutor(RealExecutor):
- shutil.rmtree(self.tmpdir)
-
- def cancel(self):
-+ logger.info('cancelling ClusterExecutor')
- self.shutdown()
-
- def _run(self, job: ExecutorJobInterface, callback=None, error_callback=None):
-@@ -1150,6 +1152,7 @@ class GenericClusterExecutor(ClusterExec
- thread_wait.start()
-
- def cancel(self):
-+ logger.info(f"cancelling with cancelcmd={self.cancelcmd}")
- def _chunks(lst, n):
- """Yield successive n-sized chunks from lst."""
- for i in range(0, len(lst), n):
-@@ -1176,7 +1179,7 @@ class GenericClusterExecutor(ClusterExec
- )
- except subprocess.SubprocessError:
- failures += 1
-- if failures:
-+ if True:#failures:
- logger.info(
- (
- "{} out of {} calls to --cluster-cancel failed. This is safe to "
---- a/snakemake/scheduler.py
-+++ b/snakemake/scheduler.py
-@@ -636,7 +636,7 @@ class JobScheduler(JobSchedulerExecutorI
- self.run(runjobs)
- except (KeyboardInterrupt, SystemExit):
- logger.info(
-- "Terminating processes on user request, this might take some time."
-+ f"Terminating processes on user request, this might take some time. type={type(self._executor)}"
- )
- self._executor.cancel()
- return False
-@@ -762,6 +762,7 @@ class JobScheduler(JobSchedulerExecutorI
- "Using greedy selector because only single job has to be scheduled."
- )
- return self.job_selector_greedy(jobs)
-+ logger.debug(f"Using ILP selector for {len(jobs)} jobs")
-
- with self._lock:
- if not self.resources["_cores"]:
-@@ -925,6 +926,7 @@ class JobScheduler(JobSchedulerExecutorI
- finally:
- os.environ["PATH"] = old_path
- solver.msg = self.workflow.verbose
-+ logger.debug(f"calling ILP {self.scheduler_ilp_solver} {solver}")
- prob.solve(solver)
-
- def required_by_job(self, temp_file, job):
=====================================
debian/patches/series
=====================================
@@ -18,5 +18,5 @@ skip-test_workflow_calling.patch
no_mix_tabs_spaces.patch
fstring.patch
1068010_appdirs_to_platformdirs.patch
-cancelscript_debug.patch
test_solver.patch
+test_extra_time.patch
=====================================
debian/patches/test_extra_time.patch
=====================================
@@ -0,0 +1,31 @@
+Description: Allow more time for test*_cancelscript*
+
+They were failing on riscv64 because
+the SIGINT was before they'd finished loading
+
+(the different times for the two tests are to get more information
+if this fails, I expect them to load in similar amounts of time)
+
+Author: Rebecca N. Palmer <rebecca_palmer at zoho.com>
+Forwarded: no
+
+--- a/tests/tests.py
++++ b/tests/tests.py
+@@ -131,7 +131,7 @@ def test_cluster_cancelscript():
+ ),
+ shouldfail=True,
+ cleanup=False,
+- sigint_after=4,
++ sigint_after=20,
+ )
+ scancel_txt = open("%s/scancel.txt" % outdir).read()
+ scancel_lines = scancel_txt.splitlines()
+@@ -160,7 +160,7 @@ def test_cluster_cancelscript_nargs1():
+ ),
+ shouldfail=True,
+ cleanup=False,
+- sigint_after=4,
++ sigint_after=40,
+ )
+ scancel_txt = open("%s/scancel.txt" % outdir).read()
+ scancel_lines = scancel_txt.splitlines()
=====================================
debian/rules
=====================================
@@ -47,13 +47,9 @@ export SPHINXBUILD=/usr/share/sphinx/scripts/python3/sphinx-build
dh $@ --with python3,sphinxdoc --buildsystem=pybuild
# LC_ALL is for reproducibility
-# tests here (bypassing nocheck) because this PPA exists to run them in riscv64
execute_after_dh_auto_build:
PYTHONPATH=$(shell pybuild --print build_dir --interpreter python3) \
cd docs && LC_ALL=C.UTF-8 LANGUAGE=C.UTF-8 make html
- mkdir $(shell pybuild --print build_dir --interpreter python3)/snakemaketmp000
- chmod +x $(shell pybuild --print build_dir --interpreter python3)/../../../bin/snakemake; cp -r $(shell pybuild --print build_dir --interpreter python3)/../../../bin $(shell pybuild --print build_dir --interpreter python3)/../../../tests $(shell pybuild --print build_dir --interpreter python3)
- . /etc/profile.d/modules.sh && HOME=$(shell pybuild --print build_dir --interpreter python3)/snakemaketmp000 python3.12 -m pytest -o faulthandler_timeout=60 -v $(shell pybuild --print build_dir --interpreter python3)/tests/test*.py -k 'not test_cwl and not test_cwl_singularity and not test_url_include and not test_wrapper and not test_issue1083 and not test_github_issue78 and not test_container and not test_singularity and not test_singularity_conda and not test_convert_to_cwl and not test_report and not test_report_zip and not test_archive and not test_jupyter_notebook and not test_conda and not test_upstream_conda and not test_conda_custom_prefix and not test_script and not test_issue635 and not test_issue1093 and not test_default_resources and not test_default_remote and not test_remote and not test_output_file_cache_remote and not test_tibanna and not test_module_complex and not test_module_complex2 and not test_module_with_script and not test_module_report and not test_modules_meta_wrapper and not test_tes and not test_deploy_script and not test_deploy_hashing and not test_peppy and not test_modules_peppy and not test_report_display_code and not test_wrapper_local_git_prefix and not test_github_issue1396 and not test_az_batch_executor and not test_modules_prefix and not test_ancient and not test_github_issue1062 and not test_symlink_time_handling and not test_env_modules and not test_github_issue1460 and not test_rule_inheritance_globals and not test_prebuilt_conda_script and not test_persistent_dict and not test_issue328' || true
execute_after_dh_installdocs:
find debian -name '__pycache__' -o name '.gitignore' | xargs rm -rf
@@ -61,7 +57,7 @@ execute_after_dh_installdocs:
# the modules.sh is a workaround for #928826; putting it in PYBUILD_BEFORE_TEST_python3 doesn't work
override_dh_auto_test:
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
- python3 -c "import io;print(io.DEFAULT_BUFFER_SIZE)"
+ . /etc/profile.d/modules.sh && PYBUILD_SYSTEM=custom dh_auto_test
endif
override_dh_auto_clean:
=====================================
debian/tests/control
=====================================
@@ -7,6 +7,7 @@ Restrictions: allow-stderr
Tests: run-unit-test
Depends: snakemake,
environment-modules,
+ glpk-utils,
graphviz,
r-base-core,
python3-connection-pool,
=====================================
debian/tests/run-unit-test
=====================================
@@ -21,8 +21,6 @@ export HOME="${AUTOPKGTEST_TMP}"
# workaround for #928826
. /etc/profile.d/modules.sh
-python3 -c "import io;print('buffer size',io.DEFAULT_BUFFER_SIZE)"
-
#See debian/rules for why these are excluded
python3 -m pytest -v tests/test*.py -k 'not test_cwl and not test_cwl_singularity and not test_url_include and not test_wrapper and not test_issue1083 and not test_github_issue78 and not test_container and not test_singularity and not test_singularity_conda and not test_convert_to_cwl and not test_report and not test_report_zip and not test_archive and not test_jupyter_notebook and not test_conda and not test_upstream_conda and not test_conda_custom_prefix and not test_script and not test_issue635 and not test_issue1093 and not test_default_resources and not test_default_remote and not test_remote and not test_output_file_cache_remote and not test_tibanna and not test_module_complex and not test_module_complex2 and not test_module_with_script and not test_module_report and not test_modules_meta_wrapper and not test_tes and not test_deploy_script and not test_deploy_hashing and not test_peppy and not test_modules_peppy and not test_report_display_code and not test_wrapper_local_git_prefix and not test_github_issue1396 and not test_az_batch_executor and not test_modules_prefix and not test_env_modules and not test_github_issue1062 and not test_pipes_fail and not test_ancient and not test_symlink_time_handling and not test_github_issue1460 and not test_rule_inheritance_globals and not test_prebuilt_conda_script'
View it on GitLab: https://salsa.debian.org/med-team/snakemake/-/compare/8c8b0eaa387a94736e7279ab05f1ab01cf81fdec...ecd73d1fb3b3d965ee527e91d4f186b227382466
--
View it on GitLab: https://salsa.debian.org/med-team/snakemake/-/compare/8c8b0eaa387a94736e7279ab05f1ab01cf81fdec...ecd73d1fb3b3d965ee527e91d4f186b227382466
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20240807/adf17fb8/attachment-0001.htm>
More information about the debian-med-commit
mailing list