[med-svn] [Git][med-team/paleomix][master] 6 commits: routine-update: New upstream version
Étienne Mollier (@emollier)
gitlab at salsa.debian.org
Sat Oct 9 20:13:35 BST 2021
Étienne Mollier pushed to branch master at Debian Med / paleomix
Commits:
0a45fd77 by Étienne Mollier at 2021-10-09T20:51:07+02:00
routine-update: New upstream version
- - - - -
74e9e914 by Étienne Mollier at 2021-10-09T20:51:08+02:00
New upstream version 1.3.4
- - - - -
6b6e4b3a by Étienne Mollier at 2021-10-09T20:51:11+02:00
Update upstream source from tag 'upstream/1.3.4'
Update to upstream version '1.3.4'
with Debian dir 42a739ee71ea56553d6d576683cf45b0136b4872
- - - - -
6d568376 by Étienne Mollier at 2021-10-09T20:57:56+02:00
d/control: update maintainer address
- - - - -
f038af08 by Étienne Mollier at 2021-10-09T21:07:22+02:00
update changelog
- - - - -
9de3110a by Étienne Mollier at 2021-10-09T21:10:11+02:00
routine-update: Ready to upload to unstable
- - - - -
20 changed files:
- CHANGES.md
- debian/changelog
- debian/control
- docs/conf.py
- docs/installation.rst
- paleomix/__init__.py
- paleomix/common/argparse.py
- paleomix/common/logging.py
- paleomix/nodegraph.py
- paleomix/nodes/picard.py
- paleomix/pipeline.py
- paleomix/pipelines/bam/config.py
- paleomix/pipelines/bam/makefile.py
- paleomix/pipelines/bam/parts/lane.py
- paleomix/pipelines/phylo/example.py
- paleomix/pipelines/phylo/pipeline.py
- paleomix/pipelines/zonkey/parts/report.py
- paleomix/pipelines/zonkey/parts/summary.py
- paleomix/resources/examples/bam_pipeline/makefile.yaml
- paleomix_environment.yaml
Changes:
=====================================
CHANGES.md
=====================================
@@ -1,5 +1,28 @@
# Changelog
+## [1.3.4] - 2021-09-23
+
+### Added
+ - Added support for the `--collapse-conservatively` AdapterRemoval option.
+
+### Changed
+ - Avoid creating log files on invalid commandline arguments.
+ - The directory for the log-file is created automatically if it does not exist.
+ - No longer prints stack-trace if the user terminates a pipeline wiht Ctrl + C.
+ - Log-level command-line options are now case insensitive.
+ - The default number of threads used by AdapterRemova, Bowtie2, and BWA are now scaled
+ based on the available number of cores instead of defaulting to 1 thread.
+ - Less exhaustive validation of .bai index files using picard ValidateSamFile. The
+ overhead of validating these files was excessive in light of the small benefit.
+
+### Fixed
+ - Fixed regression causing certain option to not be applied when mapping with BWA.
+ - Fixed --log-level not having an effect.
+ - Fixed possible infinite recursion when using lazily created log-files.
+ - Fixed BAM pipeline failing if mapDamage feature was not explicitly set.
+ - Fixed default values of 0 or 1 not being listed in commnad-line help text.
+
+
## [1.3.3] - 2021-04-06
### Fixed
@@ -729,7 +752,8 @@ the (partially) updated documentation now hosted on ReadTheDocs.
- Switching to more traditional version-number tracking.
-[Unreleased]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.3...HEAD
+[Unreleased]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.4...HEAD
+[1.3.3]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.3...v1.3.4
[1.3.3]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.2...v1.3.3
[1.3.2]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.1...v1.3.2
[1.3.1]: https://github.com/MikkelSchubert/paleomix/compare/v1.3.0...v1.3.1
=====================================
debian/changelog
=====================================
@@ -1,3 +1,10 @@
+paleomix (1.3.4-1) unstable; urgency=medium
+
+ * New upstream version
+ * d/control: update maintainer address
+
+ -- Étienne Mollier <emollier at debian.org> Sat, 09 Oct 2021 21:07:53 +0200
+
paleomix (1.3.3-1) unstable; urgency=medium
[ Étienne Mollier ]
=====================================
debian/control
=====================================
@@ -1,7 +1,7 @@
Source: paleomix
Maintainer: Debian Med Packaging Team <debian-med-packaging at lists.alioth.debian.org>
Uploaders: Andreas Tille <tille at debian.org>,
- Étienne Mollier <etienne.mollier at mailoo.org>
+ Étienne Mollier <emollier at debian.org>
Section: science
Priority: optional
Build-Depends: debhelper-compat (= 13),
=====================================
docs/conf.py
=====================================
@@ -26,7 +26,7 @@ author = "Mikkel Schubert"
# The short X.Y version
version = "1.3"
# The full version, including alpha/beta/rc tags
-release = "1.3.3"
+release = "1.3.4"
# -- General configuration ---------------------------------------------------
=====================================
docs/installation.rst
=====================================
@@ -15,19 +15,15 @@ In addition, some libraries used by PALEOMIX may require additional development
$ sudo apt-get install libz-dev libbz2-dev liblzma-dev python3-dev
-Once all requirements have been installed, PALEOMIX may be installed using `pip`:
+Once all requirements have been installed, PALEOMIX may be installed using `pip`::
-.. parsed-literal::
+ $ python3 -m pip install paleomix==1.3.4
- $ python3 -m pip install paleomix==\ |release|
-
-To verify that the installation was carried out correctly, run the command `paleomix`:
-
-.. parsed-literal::
+To verify that the installation was carried out correctly, run the command `paleomix`::
$ paleomix
PALEOMIX - pipelines and tools for NGS data analyses
- Version: \ |release|
+ Version: v1.3.4
...
@@ -46,12 +42,10 @@ This installation method requires the `venv` module. On Debian based systems, th
$ sudo apt-get install python3-venv
-Once `venv` is installed, creation of a virtual environment and installation of PALEOMIX may be carried out as shown here:
-
-.. parsed-literal::
+Once `venv` is installed, creation of a virtual environment and installation of PALEOMIX may be carried out as shown here::
$ python3 -m venv venv
- $ ./venv/bin/pip install paleomix==\ |release|
+ $ ./venv/bin/pip install paleomix==v1.3.4
Following successful completion of these commands, the `paleomix` executable will be accessible in the `./venv/bin/` folder. However, as this folder also contains a copy of Python itself, it is not recommended to add it to your `PATH`. Instead, simply link the `paleomix` executable to a folder in your `PATH`. This can be accomplished as follows::
@@ -78,20 +72,15 @@ To upgrade an installation a self-contained installation, simply call the `pip`
Conda installation
------------------
-To have a completely contained environment that includes all software dependencies, you can create a `conda`_ environment.
+`Conda`_ can be used to automatically setup a self-contained environment that includes the software required by PALEOMIX.
-To install `conda` and also set it up so it can use the `bioconda`_ bioinformatics tool repository, you can follow the instructions on the bioconda website `here`_.
+To install `conda` and also set it up so it can use the `bioconda`_ bioinformatics repository, follow the instructions on the bioconda website `here`_.
-Once set-up, you can create a conda environment named `paleomix` using the following commands:
+Next, run the following commands to download the conda environment template for this release of PALEOMIX and to create a new conda environment named `paleomix` using that template::
-.. parsed-literal::
-
- $ curl https://raw.githubusercontent.com/MikkelSchubert/paleomix/v\ |release|/paleomix_environment.yaml > paleomix_environment.yaml
+ $ curl -fL https://github.com/MikkelSchubert/paleomix/releases/download/v1.3.4/paleomix_environment.yaml > paleomix_environment.yaml
$ conda env create -n paleomix -f paleomix_environment.yaml
-.. note::
- The above only installs the dependencies for the BAM pipeline.
-
You can now activate the paleomix environment with::
$ conda activate paleomix
=====================================
paleomix/__init__.py
=====================================
@@ -21,5 +21,5 @@
# SOFTWARE.
#
-__version_info__ = (1, 3, 3)
+__version_info__ = (1, 3, 4)
__version__ = "%i.%i.%i" % __version_info__
=====================================
paleomix/common/argparse.py
=====================================
@@ -19,7 +19,7 @@ class ArgumentDefaultsHelpFormatter(configargparse.ArgumentDefaultsHelpFormatter
def _get_help_string(self, action):
# The following values look silly as part of a help string
- if action.default in [None, True, False, []]:
+ if isinstance(action.default, bool) or action.default in [None, [], ()]:
return action.help
# The subclass does not allow modification to the defaults string, so instead
=====================================
paleomix/common/logging.py
=====================================
@@ -57,6 +57,7 @@ def initialize(log_level="error", log_file=None, name="paleomix"):
log_level = _LOG_LEVELS[log_level.lower()]
root = logging.getLogger()
+ root.setLevel(logging.NOTSET)
if log_file:
handler = logging.FileHandler(log_file)
@@ -77,12 +78,15 @@ def add_argument_group(parser, default="error"):
object to have these options."""
group = parser.add_argument_group("Logging")
group.add_argument(
- "--log-file", default=None, help="Write log-messages to this file.",
+ "--log-file",
+ default=None,
+ help="Write log-messages to this file.",
)
group.add_argument(
"--log-level",
default=default,
choices=("info", "warning", "error", "debug"),
+ type=str.lower,
help="Log messages at the specified level. If a filename is not specified with "
"--log-file, PALEOMIX will automatically create a log file in the current "
"directory should events be logged at this level or above",
@@ -103,6 +107,11 @@ class LazyLogfile(logging.FileHandler):
self._template = self.baseFilename
self._log_level = log_level
+ def emit(self, record):
+ # Don't try to log self-emitted log events, to avoid recursive loops
+ if record.name != __name__:
+ super().emit(record)
+
def _open(self):
"""Try to open a new logfile, taking steps to ensure that
existing logfiles using the same template are not clobbered."""
@@ -110,6 +119,7 @@ class LazyLogfile(logging.FileHandler):
for start in itertools.count(start=1):
filename = self._template % (start,)
+ os.makedirs(os.path.dirname(filename), exist_ok=True)
try:
stream = os.fdopen(os.open(filename, flags), "w")
=====================================
paleomix/nodegraph.py
=====================================
@@ -128,14 +128,14 @@ class NodeGraph:
if not self._check_auxiliary_files(self._reverse_dependencies):
raise NodeGraphError(
"Please refer to the PALEOMIX installation instructions at "
- "https://paleomix.readthedocs.io/"
+ "https://paleomix.readthedocs.io/en/stable/"
)
self._logger.info("Checking required software")
if not self._check_version_requirements(self._reverse_dependencies):
raise NodeGraphError(
"Please refer to the PALEOMIX installation instructions at "
- "https://paleomix.readthedocs.io/"
+ "https://paleomix.readthedocs.io/en/stable/"
)
self._logger.info("Determining states")
=====================================
paleomix/nodes/picard.py
=====================================
@@ -68,6 +68,9 @@ class ValidateBAMNode(PicardNode):
for check in ignored_checks:
builder.add_option("IGNORE", check, sep="=")
+ # Exhaustive index validation is expensive and doesn't add a whole lot
+ builder.add_option("INDEX_VALIDATION_STRINGENCY", "LESS_EXHAUSTIVE", sep="=")
+
output_log = output_log or swap_ext(input_bam, ".validated")
builder.set_kwargs(
IN_BAM=input_bam, IN_INDEX=input_index, OUT_STDOUT=output_log
=====================================
paleomix/pipeline.py
=====================================
@@ -25,6 +25,7 @@ import logging
import multiprocessing
import os
import signal
+import sys
import traceback
from queue import Empty
@@ -311,8 +312,9 @@ class Pypeline:
"Press CTRL-C again to force termination."
)
else:
+ self._logger.warning("Terminating pipeline!")
self._pool.terminate()
- raise signal.default_int_handler(signum, frame)
+ sys.exit(-signum)
@classmethod
def _get_finished_node(cls, queue, running, blocking):
=====================================
paleomix/pipelines/bam/config.py
=====================================
@@ -99,19 +99,19 @@ def add_run_command(subparsers):
group.add_argument(
"--adapterremoval-max-threads",
type=int,
- default=1,
+ default=min(3, multiprocessing.cpu_count()),
help="Max number of threads to use per AdapterRemoval instance",
)
group.add_argument(
"--bowtie2-max-threads",
type=int,
- default=1,
+ default=max(1, min(8, multiprocessing.cpu_count() // 2)),
help="Max number of threads to use per Bowtie2 instance",
)
group.add_argument(
"--bwa-max-threads",
type=int,
- default=1,
+ default=max(1, min(8, multiprocessing.cpu_count() // 2)),
help="Max number of threads to use per BWA instance",
)
=====================================
paleomix/pipelines/bam/makefile.py
=====================================
@@ -124,7 +124,7 @@ _VALID_FEATURES_DICT = {
"Coverage": IsBoolean(default=True),
"Depths": IsBoolean(default=True),
"DuplicateHist": RemovedOption(),
- "mapDamage": StringIn(("rescale", "model", "plot", True, False)),
+ "mapDamage": StringIn(("rescale", "model", "plot", True, False), default="plot"),
"PCRDuplicates": StringIn((True, False, "mark", "filter"), default="filter"),
"RawBAM": RemovedOption(),
"RealignedBAM": RemovedOption(),
@@ -172,6 +172,7 @@ _VALIDATION_OPTIONS = {
"--trimwindows": Or(IsInt, IsFloat),
"--preserve5p": Or(IsNone, IsBoolean),
"--collapse-deterministic": Or(IsNone, IsBoolean),
+ "--collapse-conservatively": Or(IsNone, IsBoolean),
},
# Which aliger/mapper to use (BWA/Bowtie2)
"Aligners": {
@@ -305,7 +306,7 @@ def _mangle_options(makefile):
if invalid_features:
raise MakefileError(
"Some features (%s) may only be specified at root level, not at %r"
- % (", ".join(invalid_features), " :: ".join(path),)
+ % (", ".join(invalid_features), " :: ".join(path))
)
# Fill out missing values using those of prior levels
=====================================
paleomix/pipelines/bam/parts/lane.py
=====================================
@@ -149,7 +149,7 @@ class Lane:
threads=parameters["threads"],
prefix=parameters["prefix"],
reference=parameters["reference"],
- mapping_options=self.options["Aligners"]["BWA"],
+ mapping_options=options,
dependencies=parameters["dependencies"],
)
=====================================
paleomix/pipelines/phylo/example.py
=====================================
@@ -47,9 +47,7 @@ def build_parser():
def main(argv):
parser = build_parser()
config = parser.parse_args(argv)
- paleomix.common.logging.initialize(
- log_level=config.log_level, log_file=config.log_file, name="phylo_pipeline"
- )
+ paleomix.common.logging.initialize_console_logging()
if paleomix.resources.copy_example("phylo_pipeline", config):
return 1
=====================================
paleomix/pipelines/phylo/pipeline.py
=====================================
@@ -59,10 +59,6 @@ def main(argv):
elif "makefile" in config.commands or "mkfile" in config.commands:
return mkfile.main(config.files)
- paleomix.common.logging.initialize(
- log_level=config.log_level, log_file=config.log_file, name="phylo_pipeline"
- )
-
commands = []
for key in config.commands:
func = _COMMANDS.get(key)
@@ -72,6 +68,12 @@ def main(argv):
commands.append((key, func))
+ paleomix.common.logging.initialize(
+ log_level=config.log_level,
+ log_file=config.log_file,
+ name="phylo_pipeline",
+ )
+
if not os.path.exists(config.temp_root):
try:
os.makedirs(config.temp_root)
@@ -115,4 +117,5 @@ def main(argv):
if not pipeline.run(max_threads=config.max_threads, dry_run=config.dry_run):
return 1
+
return 0
=====================================
paleomix/pipelines/zonkey/parts/report.py
=====================================
@@ -506,11 +506,11 @@ _INTRODUCTION = """
<br/>
<div>
For more information, please refer to the
- <a href="http://paleomix.readthedocs.org/en/latest/zonkey_pipeline/index.html">
+ <a href="http://paleomix.readthedocs.org/en/stable/zonkey_pipeline/index.html">
the documentation for the Zonkey pipeline
</a>
or
- <a href="http://paleomix.readthedocs.org/en/latest/">
+ <a href="http://paleomix.readthedocs.org/en/stable/">
the documentation for the PALEOMIX pipeline,
</a>
on which the Zonkey pipeline is based.
=====================================
paleomix/pipelines/zonkey/parts/summary.py
=====================================
@@ -328,11 +328,11 @@ _HTML_HEADER = """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
<br/>
<div>
For more information, please refer to the
- <a href="http://paleomix.readthedocs.org/en/latest/zonkey_pipeline/index.html">
+ <a href="http://paleomix.readthedocs.org/en/stable/zonkey_pipeline/index.html">
the documentation for the Zonkey pipeline
</a>
or
- <a href="http://paleomix.readthedocs.org/en/latest/">
+ <a href="http://paleomix.readthedocs.org/en/stable/">
the documentation for the PALEOMIX pipeline,
</a>
on which the Zonkey pipeline is based.
=====================================
paleomix/resources/examples/bam_pipeline/makefile.yaml
=====================================
@@ -92,7 +92,7 @@ Options:
# 'mark', PCR duplicates are flagged with bit 0x400, and not removed from the
# output files; if set to 'no', the reads are assumed to not have been amplified.
PCRDuplicates: filter
- # Set to 'no' to disable mapDamage; set to 'plots' to build basic mapDamage plots;
+ # Set to 'no' to disable mapDamage; set to 'plot' to build basic mapDamage plots;
# set to 'model' to build plots and post-mortem damage models; and set to 'rescale'
# to build plots, models, and BAMs with rescaled quality scores. All analyses are
# carried out per library.
=====================================
paleomix_environment.yaml
=====================================
@@ -1,19 +1,20 @@
channels:
- - defaults
- - bioconda
- - conda-forge
+ - defaults
+ - bioconda
+ - conda-forge
dependencies:
+ - python>=3.7,<3.10
- pip
- - adapterremoval>=2.2.0
- - samtools>=1.3.0
- - picard>=1.137
- - bowtie2>=2.3.0,<2.4.0
- - bwa>=0.7.15
- - mapdamage2>=2.2.1
+ - adapterremoval>=2.3,<3
+ - samtools>=1.6,<2
+ - picard>=2.10,<3
+ - bowtie2>=2.3,<2.5
+ - bwa>=0.7.15,<0.8
+ - mapDamage2>=2.2.1,<2.3
- r-base
- r-rcpp
- r-rcppgsl
- r-gam
- r-inline
- pip:
- - paleomix==1.3.3
+ - paleomix==1.3.4
View it on GitLab: https://salsa.debian.org/med-team/paleomix/-/compare/ee8247d9190b6217892cba9a37476e2c92024913...9de3110a61e00ab3b6776fd729683edac7e5f4fa
--
View it on GitLab: https://salsa.debian.org/med-team/paleomix/-/compare/ee8247d9190b6217892cba9a37476e2c92024913...9de3110a61e00ab3b6776fd729683edac7e5f4fa
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20211009/ce751fb8/attachment-0001.htm>
More information about the debian-med-commit
mailing list