[med-svn] [Git][med-team/python-cutadapt][upstream] New upstream version 3.4
Nilesh Patra
gitlab at salsa.debian.org
Wed Apr 21 14:20:31 BST 2021
Nilesh Patra pushed to branch upstream at Debian Med / python-cutadapt
Commits:
a93b33f4 by Nilesh Patra at 2021-04-21T13:15:04+00:00
New upstream version 3.4
- - - - -
30 changed files:
- .github/workflows/ci.yml
- .github/workflows/pyinstaller.yml
- CHANGES.rst
- README.rst
- doc/guide.rst
- doc/installation.rst
- pyproject.toml
- setup.py
- src/cutadapt/__main__.py
- src/cutadapt/_align.pyi
- src/cutadapt/adapters.py
- src/cutadapt/align.py
- src/cutadapt/filters.py
- src/cutadapt/log.py
- src/cutadapt/modifiers.py
- src/cutadapt/parser.py
- src/cutadapt/pipeline.py
- src/cutadapt/report.py
- src/cutadapt/utils.py
- tests/conftest.py
- + tests/cut/info-rc.txt
- + tests/data/info-rc.fasta
- tests/test_command.py
- tests/test_commandline.py
- tests/test_main.py
- tests/test_modifiers.py
- tests/test_paired.py
- tests/test_parser.py
- tests/utils.py
- tox.ini
Changes:
=====================================
.github/workflows/ci.yml
=====================================
@@ -4,7 +4,7 @@ on: [push, pull_request]
jobs:
lint:
- timeout-minutes: 5
+ timeout-minutes: 10
runs-on: ubuntu-latest
strategy:
matrix:
@@ -22,7 +22,7 @@ jobs:
run: tox -e ${{ matrix.toxenv }}
test:
- timeout-minutes: 5
+ timeout-minutes: 10
runs-on: ${{ matrix.os }}
strategy:
matrix:
@@ -31,6 +31,8 @@ jobs:
include:
- python-version: 3.8
os: macos-latest
+ - python-version: 3.8
+ os: windows-latest
steps:
- uses: actions/checkout at v2
- name: Set up Python ${{ matrix.python-version }}
@@ -45,7 +47,7 @@ jobs:
uses: codecov/codecov-action at v1
deploy:
- timeout-minutes: 5
+ timeout-minutes: 10
runs-on: ubuntu-latest
needs: [lint, test]
if: startsWith(github.ref, 'refs/tags')
=====================================
.github/workflows/pyinstaller.yml
=====================================
@@ -10,7 +10,7 @@ jobs:
- name: Python
uses: actions/setup-python at v1
with:
- python-version: '3.7'
+ python-version: '3.9'
- name: Install
run: |
python -m venv venv
@@ -18,9 +18,9 @@ jobs:
venv/Scripts/pip install .
- name: Make exe
run: |
- echo "from cutadapt.__main__ import main" > script.py
- echo "sys.exit(main())" >> script.py
- venv/Scripts/pyinstaller -F -w -n cutadapt script.py
+ echo "from cutadapt.__main__ import main_cli" > script.py
+ echo "sys.exit(main_cli())" >> script.py
+ venv/Scripts/pyinstaller -F -n cutadapt script.py
- name: Run it
run: dist/cutadapt.exe --version
- uses: actions/upload-artifact at v2
=====================================
CHANGES.rst
=====================================
@@ -2,6 +2,30 @@
Changes
=======
+v3.4 (2021-03-30)
+-----------------
+
+* :issue:`481`: An experimental single-file Windows executable of Cutadapt
+ is `available for download on the GitHub "releases"
+ page <https://github.com/marcelm/cutadapt/releases>`_.
+* :issue:`517`: Report correct sequence in info file if read was reverse complemented
+* :issue:`517`: Added a column to the info file that shows whether the read was
+ reverse-complemented (if ``--revcomp`` was used)
+* :issue:`320`: Fix (again) "Too many open files" when demultiplexing
+
+v3.3 (2021-03-04)
+-----------------
+
+* :issue:`504`: Fix a crash on Windows.
+* :issue:`490`: When ``--rename`` is used with ``--revcomp``, disable adding the
+ ``rc`` suffix to reads that were reverse-complemented.
+* Also, there is now a ``{rc}` template variable for the ``--rename`` option, which
+ is replaced with "rc" if the read was reverse-complemented (and the empty string if not).
+* :issue:`512`: Fix issue :issue:`128` once more (the “Reads written” figure in the report
+ incorrectly included both trimmed and untrimmed reads if ``--untrimmed-output`` was used).
+* :issue:`515`: The report is now send to stderr if any output file is
+ written to stdout
+
v3.2 (2021-01-07)
-----------------
=====================================
README.rst
=====================================
@@ -1,4 +1,4 @@
-.. image:: https://travis-ci.org/marcelm/cutadapt.svg?branch=master
+.. image:: https://github.com/marcelm/cutadapt/workflows/CI/badge.svg
:target: https://travis-ci.org/marcelm/cutadapt
:alt:
=====================================
doc/guide.rst
=====================================
@@ -1099,6 +1099,8 @@ The following placeholders are currently available for single-end reads:
used with a positive length argument)
* ``{cut_suffix}`` -- the suffix removed by the ``--cut`` (or ``-u``) option (that is, when
used with a negative length argument)
+* ``{rc}`` -- this is replaced with the string ``rc`` if the read was reverse complemented.
+ This only applies when :ref:`reverse complementing <reverse-complement>` was requested.
For example, assume you have this input read in ``in.fasta``::
@@ -1115,6 +1117,10 @@ Will result in this modified read::
AAAA
+.. versionadded:: 3.3
+ The ``{rc}`` template variable.
+
+
``--rename`` also renames paired-end reads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -2021,7 +2027,8 @@ Cutadapt supports the following options to deal with ``N`` bases in your reads:
``--max-n COUNT``
Discard reads containing more than *COUNT* ``N`` bases. A fractional *COUNT*
between 0 and 1 can also be given and will be treated as the proportion of
- maximally allowed ``N`` bases in the read.
+ maximally allowed ``N`` bases in the read. For example, ``--max-n 0``
+ removes all reads that contain any ``N`` bases.
``--trim-n``
Remove flanking ``N`` bases from each read. That is, a read such as this::
@@ -2185,7 +2192,7 @@ from the final FASTA/FASTQ output due to filtering options
(such as ``--minimum-length``). Which fields are output in each row
depends on whether an adapter match was found in the read or not.
-The fields in a row that describes a match are:
+If an adapter match was found, these fields are output in each row:
1. Read name
2. Number of errors
@@ -2198,6 +2205,8 @@ The fields in a row that describes a match are:
9. Quality values corresponding to sequence left of the adapter match (can be empty)
10. Quality values corresponding to sequence matched to the adapter (can be empty)
11. Quality values corresponding to sequence to the right of the adapter match (can be empty)
+12. Flag indicating whether the read was reverse complemented: 1 if yes, 0 if not,
+ and empty if ``--revcomp`` was not used.
The concatenation of the fields 5-7 yields the full read sequence. Column 8 identifies
the found adapter. `The section about named adapters <named-adapters>` describes
@@ -2205,6 +2214,10 @@ how to give a name to an adapter. Adapters without a name are numbered starting
from 1. Fields 9-11 are empty if quality values are not available.
Concatenating them yields the full sequence of quality values.
+If the adapter match was found on the reverse complement of the read, fields 5 to 7
+show the reverse-complemented sequence, and fields 9-11 contain the qualities in
+reversed order.
+
If no adapter was found, the format is as follows:
1. Read name
@@ -2237,3 +2250,6 @@ a match of the 3' adapter, the string ``;2`` is added. If there are two rows, th
.. versionadded:: 2.8
Linked adapters in info files work.
+
+.. versionadded:: 3.4
+ Column 12 (revcomp flag) added
=====================================
doc/installation.rst
=====================================
@@ -129,6 +129,39 @@ system-installed packages::
sudo ln -s ../cutadapt/bin/cutadapt
+Installation on Windows
+-----------------------
+
+For some releases of Cutadapt, a single-file executable (``cutadapt.exe``)
+is made available on the
+`GitHub releases page <https://github.com/marcelm/cutadapt/releases>`_. Try that
+first, and if it does not work for you, please report the issue.
+
+To install Cutadapt manually, keep reading.
+
+There is no Bioconda package for Windows because Bioconda does not produce
+Windows packages. To install Cutadapt, you can use ``pip``, but because
+Cutadapt contains components that need to be compiled, you also need to install
+a compiler.
+
+1. Download a recent version (at least 3.6) of Python for Windows from
+ <https://www.python.org/> and install it.
+2. Download and install “Build Tools for Visual Studio 2019” from
+ <https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019>.
+ (There are many similarly named downloads on that page, ensure you get the
+ right one.)
+
+ During installation, when the dialog about which components to install pops
+ up, ensure that “C++ Build tools” is ticked. The download is quite big and
+ can take a long time.
+3. Open the command line (``cmd.exe``) and run ``py -m pip install cutadapt``.
+4. Test whether it worked by running ``py -m cutadapt --version``. You should
+ see the version number of Cutadapt.
+
+When running Cutadapt this way, you will need to remember to write
+``py -m cutadapt`` instead of just ``cutadapt``.
+
+
Uninstalling
------------
=====================================
pyproject.toml
=====================================
@@ -3,3 +3,8 @@ requires = ["setuptools", "wheel", "setuptools_scm", "cython"]
[black.tool]
line-length = 100
+
+[tool.pytest.ini_options]
+filterwarnings = [
+ "error",
+]
=====================================
setup.py
=====================================
@@ -101,7 +101,7 @@ setup(
entry_points={'console_scripts': ['cutadapt = cutadapt.__main__:main_cli']},
install_requires=[
'dnaio~=0.5.0',
- 'xopen~=1.0.0',
+ 'xopen~=1.1.0',
"dataclasses>=0.8; python_version<'3.7'",
],
extras_require={
=====================================
src/cutadapt/__main__.py
=====================================
@@ -23,7 +23,7 @@
"""
cutadapt version {version}
-Copyright (C) 2010-2020 Marcel Martin <marcel.martin at scilifelab.se>
+Copyright (C) 2010-2021 Marcel Martin <marcel.martin at scilifelab.se>
cutadapt removes adapter sequences from high-throughput sequencing reads..
@@ -688,17 +688,18 @@ def pipeline_from_parsed_args(args, paired, file_opener, adapters, adapters2) ->
args.action,
args.times,
args.reverse_complement,
+ not args.rename, # no "rc" suffix if --rename is used
args.index,
)
for modifier in modifiers_applying_to_both_ends_if_paired(args):
pipeline_add(modifier)
- if args.rename:
- if args.prefix or args.suffix:
- raise CommandLineError(
- "Option --rename cannot be combined with --prefix (-x) or --suffix (-y)"
- )
+ if args.rename and (args.prefix or args.suffix):
+ raise CommandLineError(
+ "Option --rename cannot be combined with --prefix (-x) or --suffix (-y)"
+ )
+ if args.rename and args.rename != "{header}":
try:
if paired:
pipeline.add_paired_modifier(PairedEndRenamer(args.rename))
@@ -781,6 +782,7 @@ def add_adapter_cutter(
action: Optional[str],
times: int,
reverse_complement: bool,
+ add_rc_suffix: bool,
allow_index: bool,
):
if pair_adapters:
@@ -808,7 +810,8 @@ def add_adapter_cutter(
elif adapter_cutter:
if reverse_complement:
modifier = ReverseComplementer(
- adapter_cutter
+ adapter_cutter,
+ rc_suffix=" rc" if add_rc_suffix else None,
) # type: Union[AdapterCutter,ReverseComplementer]
else:
modifier = adapter_cutter
@@ -860,9 +863,7 @@ def main(cmdlineargs, default_outfile=sys.stdout.buffer) -> Statistics:
# Setup logging only if there are not already any handlers (can happen when
# this function is being called externally such as from unit tests)
if not logging.root.handlers:
- # If results are to be sent to stdout, logging needs to go to stderr
- log_to_stdout = args.output is not None and args.output != "-" and args.paired_output != "-"
- setup_logging(logger, stdout=log_to_stdout,
+ setup_logging(logger, log_to_stderr=is_any_output_stdout(args),
quiet=args.quiet, minimal=args.report == 'minimal', debug=args.debug)
log_header(cmdlineargs)
profiler = setup_profiler_if_requested(args.profile)
@@ -879,8 +880,8 @@ def main(cmdlineargs, default_outfile=sys.stdout.buffer) -> Statistics:
cores = available_cpu_count() if args.cores == 0 else args.cores
file_opener = FileOpener(
- compression_level=args.compression_level, threads=0 if cores == 1 else None)
- if sys.stderr.isatty() and not args.quiet:
+ compression_level=args.compression_level, threads=estimate_compression_threads(cores))
+ if sys.stderr.isatty() and not args.quiet and not args.debug:
progress = Progress()
else:
progress = DummyProgress()
@@ -944,7 +945,7 @@ def setup_runner(
try:
if cores > 1:
return ParallelPipelineRunner(
- pipeline, inpaths, outfiles, progress, n_workers=cores, buffer_size=buffer_size)
+ pipeline, inpaths, outfiles, file_opener, progress, n_workers=cores, buffer_size=buffer_size)
else:
infiles = inpaths.open(file_opener)
return SerialPipelineRunner(pipeline, infiles, outfiles, progress)
@@ -972,5 +973,26 @@ def warn_if_en_dashes(args):
)
+def estimate_compression_threads(cores: int) -> Optional[int]:
+ return max(0, min(cores, 4))
+
+
+def is_any_output_stdout(args):
+ return any([
+ args.output is None,
+ args.output == "-",
+ args.paired_output == "-",
+ args.untrimmed_output == "-",
+ args.untrimmed_paired_output == "-",
+ args.too_short_output == "-",
+ args.too_short_paired_output == "-",
+ args.too_long_output == "-",
+ args.too_long_paired_output == "-",
+ args.rest_file == "-",
+ args.info_file == "-",
+ args.wildcard_file == "-",
+ ])
+
+
if __name__ == '__main__': # pragma: no cover
sys.exit(main_cli())
=====================================
src/cutadapt/_align.pyi
=====================================
@@ -4,7 +4,7 @@ class DPMatrix:
m: int
n: int
def __init__(self, reference: str, query: str): ...
- def set_entry(self, i: int, j: int, cost: int): ...
+ def set_entry(self, i: int, j: int, cost: int) -> None: ...
def __str__(self) -> str: ...
class Aligner:
=====================================
src/cutadapt/adapters.py
=====================================
@@ -49,6 +49,7 @@ class EndStatistics:
self.sequence = adapter.sequence # type: str
self.effective_length = adapter.effective_length # type: int
self.has_wildcards = adapter.adapter_wildcards # type: bool
+ self.allows_partial_matches: bool = adapter.allows_partial_matches
# self.errors[l][e] == n iff n times a sequence of length l matching at e errors was removed
self.errors = defaultdict(returns_defaultdict_int) # type: Dict[int, Dict[int, int]]
self.adjacent_bases = {'A': 0, 'C': 0, 'G': 0, 'T': 0, '': 0}
@@ -395,6 +396,8 @@ class SingleAdapter(Adapter, ABC):
unique number.
"""
+ allows_partial_matches: bool = True
+
def __init__(
self,
sequence: str,
@@ -617,6 +620,7 @@ class PrefixAdapter(NonInternalFrontAdapter):
"""An anchored 5' adapter"""
description = "anchored 5'"
+ allows_partial_matches = False
def _aligner(self):
if not self.indels: # TODO or if error rate allows 0 errors anyway
@@ -635,6 +639,7 @@ class SuffixAdapter(NonInternalBackAdapter):
"""An anchored 3' adapter"""
description = "anchored 3'"
+ allows_partial_matches = False
def _aligner(self):
if not self.indels: # TODO or if error rate allows 0 errors anyway
=====================================
src/cutadapt/align.py
=====================================
@@ -8,6 +8,8 @@ __all__ = [
'edit_distance',
]
+from typing import Iterator, Tuple
+
from cutadapt._align import Aligner, PrefixComparer, SuffixComparer
# flags for global alignment
@@ -28,7 +30,7 @@ STOP_WITHIN_SEQ2 = 8
SEMIGLOBAL = START_WITHIN_SEQ1 | START_WITHIN_SEQ2 | STOP_WITHIN_SEQ1 | STOP_WITHIN_SEQ2
-def edit_distance(s: str, t: str):
+def edit_distance(s: str, t: str) -> int:
"""
Return the edit distance between the strings s and t.
The edit distance is the sum of the numbers of insertions, deletions,
@@ -54,7 +56,7 @@ def edit_distance(s: str, t: str):
return costs[-1]
-def hamming_sphere(s, k):
+def hamming_sphere(s: str, k: int) -> Iterator[str]:
"""
Yield all strings t for which the hamming distance between s and t is exactly k,
assuming the alphabet is A, C, G, T.
@@ -79,7 +81,7 @@ def hamming_sphere(s, k):
yield y
-def hamming_environment(s, k):
+def hamming_environment(s: str, k: int) -> Iterator[Tuple[str, int, int]]:
"""
Find all strings t for which the hamming distance between s and t is at most k,
assuming the alphabet is A, C, G, T.
@@ -93,7 +95,7 @@ def hamming_environment(s, k):
yield t, e, n - e
-def naive_edit_environment(s: str, k: int):
+def naive_edit_environment(s: str, k: int) -> Iterator[str]:
"""
Apply all possible edits up to edit distance k to string s.
A string may be returned more than once.
@@ -114,7 +116,7 @@ def naive_edit_environment(s: str, k: int):
yield s[:i] + s[i+1:]
-def edit_environment(s: str, k: int):
+def edit_environment(s: str, k: int) -> Iterator[Tuple[str, int, int]]:
"""
Find all strings t for which the edit distance between s and t is at most k,
assuming the alphabet is A, C, G, T.
@@ -134,7 +136,7 @@ def edit_environment(s: str, k: int):
yield t, errors, matches
-def slow_edit_environment(s: str, k: int):
+def slow_edit_environment(s: str, k: int) -> Iterator[Tuple[str, int, int]]:
"""
Find all strings t for which the edit distance between s and t is at most k,
assuming the alphabet is A, C, G, T.
=====================================
src/cutadapt/filters.py
=====================================
@@ -23,6 +23,8 @@ from .modifiers import ModificationInfo
# Constants used when returning from a Filter’s __call__ method to improve
# readability (it is unintuitive that "return True" means "discard the read").
+from .utils import reverse_complemented_sequence
+
DISCARD = True
KEEP = False
@@ -118,7 +120,7 @@ class PairedNoFilter(PairedEndFilterWithStatistics):
return DISCARD
-class Redirector(SingleEndFilterWithStatistics):
+class Redirector(SingleEndFilter):
"""
Redirect discarded reads to the given writer. This is for single-end reads.
"""
@@ -137,12 +139,11 @@ class Redirector(SingleEndFilterWithStatistics):
self.filtered += 1
if self.writer is not None:
self.writer.write(read)
- self.update_statistics(read)
return DISCARD
return KEEP
-class PairedRedirector(PairedEndFilterWithStatistics):
+class PairedRedirector(PairedEndFilter):
"""
Redirect paired-end reads matching a filtering criterion to a writer..
Different filtering styles are supported, differing by which of the
@@ -195,7 +196,6 @@ class PairedRedirector(PairedEndFilterWithStatistics):
self.filtered += 1
if self.writer is not None:
self.writer.write(read1, read2)
- self.update_statistics(read1, read2)
return DISCARD
return KEEP
@@ -416,16 +416,21 @@ class WildcardFileWriter(SingleEndFilter):
class InfoFileWriter(SingleEndFilter):
+ RC_MAP = {None: "", True: "1", False: "0"}
+
def __init__(self, file):
self.file = file
def __call__(self, read, info: ModificationInfo):
current_read = info.original_read
+ if info.is_rc:
+ current_read = reverse_complemented_sequence(current_read)
if info.matches:
for match in info.matches:
for info_record in match.get_info_records(current_read):
# info_record[0] is the read name suffix
- print(read.name + info_record[0], *info_record[1:], sep='\t', file=self.file)
+ print(read.name + info_record[0], *info_record[1:], self.RC_MAP[info.is_rc],
+ sep='\t', file=self.file)
current_read = match.trimmed(current_read)
else:
seq = read.sequence
=====================================
src/cutadapt/log.py
=====================================
@@ -30,7 +30,7 @@ class NiceFormatter(logging.Formatter):
return super().format(record)
-def setup_logging(logger, stdout=False, minimal=False, quiet=False, debug=0):
+def setup_logging(logger, log_to_stderr=True, minimal=False, quiet=False, debug=0):
"""
Attach handler to the global logger object
"""
@@ -39,7 +39,7 @@ def setup_logging(logger, stdout=False, minimal=False, quiet=False, debug=0):
# INFO level (and the ERROR level would give us an 'ERROR:' prefix).
logging.addLevelName(REPORT, 'REPORT')
- stream_handler = CrashingHandler(sys.stdout if stdout else sys.stderr)
+ stream_handler = CrashingHandler(sys.stderr if log_to_stderr else sys.stdout)
stream_handler.setFormatter(NiceFormatter())
# debug overrides quiet overrides minimal
if debug > 0:
=====================================
src/cutadapt/modifiers.py
=====================================
@@ -437,6 +437,7 @@ class Renamer(SingleEndModifier):
- {cut_prefix} -- prefix removed by UnconditionalCutter (with positive length argument)
- {cut_suffix} -- suffix removed by UnconditionalCutter (with negative length argument)
- {adapter_name} -- name of the *last* adapter match or no_adapter if there was none
+ - {rc} -- the string 'rc' if the read was reverse complemented (with --revcomp) or '' otherwise
"""
variables = {
"header",
@@ -445,6 +446,7 @@ class Renamer(SingleEndModifier):
"cut_prefix",
"cut_suffix",
"adapter_name",
+ "rc",
}
def __init__(self, template: str):
@@ -455,6 +457,9 @@ class Renamer(SingleEndModifier):
self.raise_if_invalid_variable(self._tokens, self.variables)
self._template = template
+ def __repr__(self):
+ return f"Renamer('{self._template}')"
+
@staticmethod
def raise_if_invalid_variable(tokens: List[Token], allowed: Set[str]) -> None:
for token in tokens:
@@ -484,6 +489,7 @@ class Renamer(SingleEndModifier):
cut_prefix=info.cut_prefix if info.cut_prefix else "",
cut_suffix=info.cut_suffix if info.cut_suffix else "",
adapter_name=info.matches[-1].adapter.name if info.matches else "no_adapter",
+ rc="rc" if info.is_rc else "",
)
return read
@@ -512,9 +518,8 @@ class PairedEndRenamer(PairedEndModifier):
@staticmethod
def _get_allowed_variables() -> Set[str]:
- allowed = Renamer.variables.copy()
- allowed.add("rn")
- for v in Renamer.variables - {"id"}:
+ allowed = (Renamer.variables - {"rc"}) | {"rn"}
+ for v in Renamer.variables - {"id", "rc"}:
allowed.add("r1." + v)
allowed.add("r2." + v)
return allowed
=====================================
src/cutadapt/parser.py
=====================================
@@ -340,6 +340,8 @@ class AdapterParser:
adapter_class = aspec.adapter_class() # type: Type[Adapter]
if aspec.parameters.pop('anywhere', False) and adapter_class in (FrontAdapter, BackAdapter):
aspec.parameters['force_anywhere'] = True
+ if 'required' in aspec.parameters:
+ raise ValueError("'required' and 'optional' can only be used within linked adapters")
parameters = self.default_parameters.copy()
parameters.update(aspec.parameters)
return adapter_class(
=====================================
src/cutadapt/pipeline.py
=====================================
@@ -12,7 +12,6 @@ import multiprocessing.connection
from multiprocessing.connection import Connection
import traceback
-from xopen import xopen
import dnaio
from .utils import Progress, FileOpener
@@ -530,7 +529,7 @@ class ReaderProcess(Process):
and finally sends the stop token -1 ("poison pills") to all connections.
"""
- def __init__(self, path: str, path2: Optional[str], connections, queue, buffer_size, stdin_fd):
+ def __init__(self, path: str, path2: Optional[str], opener: FileOpener, connections, queue, buffer_size, stdin_fd):
"""
queue -- a Queue of worker indices. A worker writes its own index into this
queue to notify the reader that it is ready to receive more data.
@@ -543,15 +542,16 @@ class ReaderProcess(Process):
self.queue = queue
self.buffer_size = buffer_size
self.stdin_fd = stdin_fd
+ self._opener = opener
def run(self):
if self.stdin_fd != -1:
sys.stdin.close()
sys.stdin = os.fdopen(self.stdin_fd)
try:
- with xopen(self.path, 'rb') as f:
+ with self._opener.xopen(self.path, 'rb') as f:
if self.path2:
- with xopen(self.path2, 'rb') as f2:
+ with self._opener.xopen(self.path2, 'rb') as f2:
for chunk_index, (chunk1, chunk2) in enumerate(
dnaio.read_paired_chunks(f, f2, self.buffer_size)):
self.send_to_worker(chunk_index, chunk1, chunk2)
@@ -740,6 +740,7 @@ class ParallelPipelineRunner(PipelineRunner):
pipeline: Pipeline,
infiles: InputPaths,
outfiles: OutputFiles,
+ opener: FileOpener,
progress: Progress,
n_workers: int,
buffer_size: int = 4 * 1024**2,
@@ -748,8 +749,9 @@ class ParallelPipelineRunner(PipelineRunner):
self._n_workers = n_workers
self._need_work_queue = Queue() # type: Queue
self._buffer_size = buffer_size
- self._assign_input(infiles.path1, infiles.path2, infiles.interleaved)
self._outfiles = outfiles
+ self._opener = opener
+ self._assign_input(infiles.path1, infiles.path2, infiles.interleaved)
def _assign_input(
self,
@@ -768,7 +770,7 @@ class ParallelPipelineRunner(PipelineRunner):
# This happens during tests: pytest sets sys.stdin to an object
# that does not have a file descriptor.
fileno = -1
- self._reader_process = ReaderProcess(path1, path2, connw,
+ self._reader_process = ReaderProcess(path1, path2, self._opener, connw,
self._need_work_queue, self._buffer_size, fileno)
self._reader_process.daemon = True
self._reader_process.start()
@@ -798,8 +800,7 @@ class ParallelPipelineRunner(PipelineRunner):
n = 0 # A running total of the number of processed reads (for progress indicator)
while connections:
ready_connections = multiprocessing.connection.wait(connections)
- for connection in ready_connections:
- assert isinstance(connection, Connection)
+ for connection in ready_connections: # type: Any
chunk_index = connection.recv()
if chunk_index == -1:
# the worker is done
=====================================
src/cutadapt/report.py
=====================================
@@ -210,18 +210,21 @@ class Statistics:
def error_ranges(adapter_statistics: EndStatistics) -> str:
length = adapter_statistics.effective_length
error_rate = adapter_statistics.max_error_rate
- prev = 1
- s = ""
- for errors in range(1, int(error_rate * length) + 1):
- r = int(errors / error_rate)
- s += "{}-{} bp: {}; ".format(prev, r - 1, errors - 1)
- prev = r
- if prev == length:
- s += "{} bp: {}".format(length, int(error_rate * length))
+ if adapter_statistics.allows_partial_matches:
+ prev = 1
+ s = "\n"
+ for errors in range(1, int(error_rate * length) + 1):
+ r = int(errors / error_rate)
+ s += "{}-{} bp: {}; ".format(prev, r - 1, errors - 1)
+ prev = r
+ if prev == length:
+ s += "{} bp: {}".format(length, int(error_rate * length))
+ else:
+ s += "{}-{} bp: {}".format(prev, length, int(error_rate * length))
else:
- s += "{}-{} bp: {}".format(prev, length, int(error_rate * length))
+ s = f" {int(error_rate * length)}"
- return "No. of allowed errors:\n" + s + "\n"
+ return "No. of allowed errors:" + s + "\n"
def histogram(end_statistics: EndStatistics, n: int, gc_content: float) -> str:
@@ -303,7 +306,8 @@ def full_report(stats: Statistics, time: float, gc_content: float) -> str: # no
"""Print report to standard output."""
if stats.n == 0:
return "No reads processed!"
-
+ if time == 0:
+ time = 1E-6
sio = StringIO()
def print_s(*args, **kwargs):
=====================================
src/cutadapt/utils.py
=====================================
@@ -42,6 +42,23 @@ def available_cpu_count():
return multiprocessing.cpu_count()
+def open_raise_limit(func, *args, **kwargs):
+ """
+ Run 'func' (which should be some kind of open() function and return its result.
+ If "Too many open files" occurs, increase limit and try again.
+ """
+ try:
+ f = func(*args, **kwargs)
+ except OSError as e:
+ if e.errno == errno.EMFILE: # Too many open files
+ logger.debug("Too many open files, attempting to raise soft limit")
+ raise_open_files_limit(8)
+ f = func(*args, **kwargs)
+ else:
+ raise
+ return f
+
+
def raise_open_files_limit(n):
if resource is None:
return
@@ -91,6 +108,8 @@ class Progress:
delta = total - self._n
if delta < 1:
return
+ if time_delta == 0:
+ return
if not _final:
if time_delta < self._every:
return
@@ -163,8 +182,12 @@ class FileOpener:
self.threads = threads
def xopen(self, path, mode):
- logger.debug("Opening file '%s', mode '%s' with xopen", path, mode)
- return xopen(path, mode, compresslevel=self.compression_level, threads=self.threads)
+ threads = self.threads if "w" in mode else 0
+ f = open_raise_limit(
+ xopen, path, mode, compresslevel=self.compression_level, threads=threads
+ )
+ logger.debug("Opening '%s', mode '%s' with xopen resulted in %s", path, mode, f)
+ return f
def xopen_or_none(self, path, mode):
"""Return opened file or None if the path is None"""
@@ -181,22 +204,15 @@ class FileOpener:
return file1, file2
def dnaio_open(self, *args, **kwargs):
- logger.debug("Opening file '%s', mode '%s' with dnaio", args[0], kwargs['mode'])
kwargs["opener"] = self.xopen
- return dnaio.open(*args, **kwargs)
+ f = dnaio.open(*args, **kwargs)
+ logger.debug("Opening %r, mode '%s' with dnaio resulted in %s",
+ args[0], kwargs['mode'], f)
+ return f
def dnaio_open_raise_limit(self, *args, **kwargs):
"""
Open a FASTA/FASTQ file for writing. If it fails because the number of open files
would be exceeded, try to raise the soft limit and re-try.
"""
- try:
- f = self.dnaio_open(*args, **kwargs)
- except OSError as e:
- if e.errno == errno.EMFILE: # Too many open files
- logger.debug("Too many open files, attempting to raise soft limit")
- raise_open_files_limit(8)
- f = self.dnaio_open(*args, **kwargs)
- else:
- raise
- return f
+ return open_raise_limit(self.dnaio_open, *args, **kwargs)
=====================================
tests/conftest.py
=====================================
@@ -10,14 +10,14 @@ def cores(request):
@pytest.fixture
-def run(tmpdir):
+def run(tmp_path):
def _run(params, expected, inpath) -> Statistics:
if type(params) is str:
params = params.split()
- tmp_fastaq = str(tmpdir.join(expected))
+ tmp_fastaq = tmp_path / expected
params += ['-o', tmp_fastaq]
params += [datapath(inpath)]
- stats = main(params)
+ stats = main([str(p) for p in params])
# TODO redirect standard output
assert_files_equal(cutpath(expected), tmp_fastaq)
return stats
=====================================
tests/cut/info-rc.txt
=====================================
@@ -0,0 +1,2 @@
+s 0 20 26 AGGCGCTTGTAGCGTCGATT GAGTCG TGAC adapt 0
+r 0 20 26 AGGCGCTTGTAGCGTCGATT GAGTCG TGAC adapt 1
=====================================
tests/data/info-rc.fasta
=====================================
@@ -0,0 +1,4 @@
+>s
+AGGCGCTTGTAGCGTCGATTGAGTCGTGAC
+>r
+GTCACGACTCAATCGACGCTACAAGCGCCT
=====================================
tests/test_command.py
=====================================
@@ -2,6 +2,9 @@
import subprocess
import sys
+import os
+
+import pytest
from utils import datapath, assert_files_equal, cutpath
@@ -17,11 +20,12 @@ def test_run_as_module():
assert py.communicate()[0].decode().strip() == __version__
+ at pytest.mark.skipif(sys.platform == "win32", reason="Perhaps this can be fixed")
def test_standard_input_pipe(tmpdir, cores):
"""Read FASTQ from standard input"""
out_path = str(tmpdir.join("out.fastq"))
in_path = datapath("small.fastq")
- # Use 'cat' to simulate that no file name is available for stdin
+ # Simulate that no file name is available for stdin
with subprocess.Popen(["cat", in_path], stdout=subprocess.PIPE) as cat:
with subprocess.Popen([
sys.executable, "-m", "cutadapt", "--cores", str(cores),
@@ -72,8 +76,9 @@ def test_force_fasta_output(tmpdir, cores):
assert_files_equal(cutpath("small.fasta"), out_path)
+ at pytest.mark.skipif(sys.platform == "win32", reason="Maybe this can be made to work")
def test_non_utf8_locale():
subprocess.check_call(
- [sys.executable, "-m", "cutadapt", "-o", "/dev/null", datapath("small.fastq")],
+ [sys.executable, "-m", "cutadapt", "-o", os.devnull, datapath("small.fastq")],
env={"LC_CTYPE": "C"},
)
=====================================
tests/test_commandline.py
=====================================
@@ -1,6 +1,9 @@
import subprocess
import sys
+import os
from io import StringIO, BytesIO
+
+import dnaio
import pytest
from cutadapt.__main__ import main
@@ -81,15 +84,15 @@ def test_lowercase(run):
run('-a ttagacatatctccgtcg', 'lowercase.fastq', 'small.fastq')
-def test_rest(run, tmpdir, cores):
+def test_rest(run, tmp_path, cores):
"""-r/--rest-file"""
- rest = str(tmpdir.join("rest.tmp"))
+ rest = tmp_path / "rest.tmp"
run(['--cores', str(cores), '-b', 'ADAPTER', '-N', '-r', rest], "rest.fa", "rest.fa")
assert_files_equal(datapath('rest.txt'), rest)
-def test_restfront(run, tmpdir):
- path = str(tmpdir.join("rest.txt"))
+def test_restfront(run, tmp_path):
+ path = tmp_path / "rest.txt"
run(['-g', 'ADAPTER', '-N', '-r', path], "restfront.fa", "rest.fa")
assert_files_equal(datapath('restfront.txt'), path)
@@ -111,12 +114,13 @@ def test_extensiontxtgz(run):
def test_minimum_length(run):
"""-m/--minimum-length"""
- run("-m 5 -a TTAGACATATCTCCGTCG", "minlen.fa", "lengths.fa")
+ stats = run("-m 5 -a TTAGACATATCTCCGTCG", "minlen.fa", "lengths.fa")
+ assert stats.written_bp[0] == 45
+ assert stats.written == 6
-def test_too_short(run, tmpdir, cores):
- """--too-short-output"""
- too_short_path = str(tmpdir.join('tooshort.fa'))
+def test_too_short(run, tmp_path, cores):
+ too_short_path = tmp_path / 'tooshort.fa'
stats = run([
"--cores", str(cores),
"-m", "5",
@@ -127,14 +131,26 @@ def test_too_short(run, tmpdir, cores):
assert stats.too_short == 5
+ at pytest.mark.parametrize("redirect", (False, True))
+def test_too_short_statistics(redirect):
+ args = ["-a", "TTAGACATATCTCCGTCG", "-m", "24", "-o", os.devnull, datapath("small.fastq")]
+ if redirect:
+ args[:0] = ["--too-short-output", os.devnull]
+ stats = main(args)
+ assert stats.with_adapters[0] == 2
+ assert stats.written == 2
+ assert stats.written_bp[0] == 58
+ assert stats.too_short == 1
+
+
def test_maximum_length(run):
"""-M/--maximum-length"""
run("-M 5 -a TTAGACATATCTCCGTCG", "maxlen.fa", "lengths.fa")
-def test_too_long(run, tmpdir, cores):
+def test_too_long(run, tmp_path, cores):
"""--too-long-output"""
- too_long_path = str(tmpdir.join('toolong.fa'))
+ too_long_path = tmp_path / 'toolong.fa'
stats = run([
"--cores", str(cores),
"-M", "5",
@@ -153,19 +169,19 @@ def test_length_tag(run):
@pytest.mark.parametrize("length", list(range(3, 11)))
-def test_overlap_a(tmpdir, length):
+def test_overlap_a(tmp_path, length):
"""-O/--overlap with -a"""
adapter = "catatctccg"
record = ">read\nGAGACCATTCCAATG" + adapter[:length] + '\n'
- input = tmpdir.join("overlap.fasta")
- input.write(record)
+ input = tmp_path / "overlap.fasta"
+ input.write_text(record)
if length < 7:
expected = record
else:
expected = '>read\nGAGACCATTCCAATG\n'
- output = tmpdir.join("overlap-trimmed.fasta")
+ output = tmp_path / "overlap-trimmed.fasta"
main(["-O", "7", "-e", "0", "-a", adapter, "-o", str(output), str(input)])
- assert expected == output.read()
+ assert expected == output.read_text()
def test_overlap_b(run):
@@ -250,9 +266,9 @@ def test_read_wildcard(run):
("-a", "wildcard_adapter.fa"),
("-b", "wildcard_adapter_anywhere.fa"),
])
-def test_adapter_wildcard(adapter_type, expected, run, tmpdir, cores):
+def test_adapter_wildcard(adapter_type, expected, run, tmp_path, cores):
"""wildcards in adapter"""
- wildcard_path = str(tmpdir.join("wildcards.txt"))
+ wildcard_path = tmp_path / "wildcards.txt"
run([
"--cores", str(cores),
"--wildcard-file", wildcard_path,
@@ -324,39 +340,52 @@ def test_ellipsis_notation(run):
run('-a ...TTAGACATAT -g GAGATTGCCA --no-indels', 'no_indels.fasta', 'no_indels.fasta')
-def test_issue_46(run, tmpdir):
+def test_issue_46(run, tmp_path):
"""issue 46 - IndexError with --wildcard-file"""
run("--anywhere=AACGTN --wildcard-file={}".format(
- tmpdir.join("wildcards.txt")), "issue46.fasta", "issue46.fasta")
+ tmp_path / "wildcards.txt"), "issue46.fasta", "issue46.fasta")
def test_strip_suffix(run):
run("--strip-suffix _sequence -a XXXXXXX", "stripped.fasta", "simple.fasta")
-def test_info_file(run, tmpdir, cores):
+def test_info_file(run, tmp_path, cores):
# The true adapter sequence in the illumina.fastq.gz data set is
# GCCTAACTTCTTAGACTGCCTTAAGGACGT (fourth base is different from the sequence shown here)
- info_path = str(tmpdir.join("info.txt"))
+ info_path = tmp_path / "info.txt"
run(["--cores", str(cores), "--info-file", info_path, "-a", "adapt=GCCGAACTTCTTAGACTGCCTTAAGGACGT"],
"illumina.fastq", "illumina.fastq.gz")
- assert_files_equal(cutpath("illumina.info.txt"), info_path)
+ assert_files_equal(cutpath("illumina.info.txt"), info_path, ignore_trailing_space=True)
-def test_info_file_times(run, tmpdir, cores):
- info_path = str(tmpdir.join("info.txt"))
+def test_info_file_times(run, tmp_path, cores):
+ info_path = tmp_path / "info.txt"
run(["--cores", str(cores), "--info-file", info_path, "--times", "2", "-a", "adapt=GCCGAACTTCTTA",
"-a", "adapt2=GACTGCCTTAAGGACGT"], "illumina5.fastq", "illumina5.fastq")
- assert_files_equal(cutpath('illumina5.info.txt'), info_path)
+ assert_files_equal(cutpath('illumina5.info.txt'), info_path, ignore_trailing_space=True)
-def test_info_file_fasta(run, tmpdir, cores):
- info_path = str(tmpdir.join("info.txt"))
+def test_info_file_fasta(run, tmp_path, cores):
+ info_path = tmp_path / "info.txt"
# Just make sure that it runs
run(["--cores", str(cores), "--info-file", info_path, "-a", "TTAGACATAT", "-g", "GAGATTGCCA", "--no-indels"],
"no_indels.fasta", "no_indels.fasta")
+def test_info_file_revcomp(run, tmp_path):
+ info_path = tmp_path / "info-rc.txt"
+ main([
+ "--info-file", str(info_path),
+ "-a", "adapt=GAGTCG",
+ "--revcomp",
+ "--rename={header}",
+ "-o", str(tmp_path / "out.fasta"),
+ datapath("info-rc.fasta")
+ ])
+ assert_files_equal(cutpath("info-rc.txt"), info_path)
+
+
def test_named_adapter(run):
run("-a MY_ADAPTER=GCCGAACTTCTTAGACTGCCTTAAGGACGT", "illumina.fastq", "illumina.fastq.gz")
@@ -439,11 +468,14 @@ def test_unconditional_cut_invalid_number():
main(["-u", "a,b", datapath("small.fastq")])
-def test_untrimmed_output(run, cores, tmpdir):
- path = str(tmpdir.join("untrimmed.fastq"))
- run(["--cores", str(cores), "-a", "TTAGACATATCTCCGTCG", "--untrimmed-output", path],
+def test_untrimmed_output(run, cores, tmp_path):
+ path = tmp_path / "untrimmed.fastq"
+ stats = run(["--cores", str(cores), "-a", "TTAGACATATCTCCGTCG", "--untrimmed-output", path],
"small.trimmed.fastq", "small.fastq")
assert_files_equal(cutpath("small.untrimmed.fastq"), path)
+ assert stats.with_adapters[0] == 2
+ assert stats.written == 2
+ assert stats.written_bp[0] == 46
def test_adapter_file(run):
@@ -528,7 +560,7 @@ def test_quiet_is_quiet():
try:
sys.stdout = captured_standard_output
sys.stderr = captured_standard_error
- main(['-o', '/dev/null', '--quiet', datapath('small.fastq')])
+ main(['-o', os.devnull, '--quiet', datapath('small.fastq')])
finally:
sys.stdout = old_stdout
sys.stderr = old_stderr
@@ -539,7 +571,7 @@ def test_quiet_is_quiet():
def test_x_brace_notation():
- main(['-o', '/dev/null', '--quiet', '-a', 'X{5}', datapath('small.fastq')])
+ main(['-o', os.devnull, '--quiet', '-a', 'X{5}', datapath('small.fastq')])
def test_nextseq(run):
@@ -575,11 +607,11 @@ def test_linked_lowercase(run):
'linked-lowercase.fasta', 'linked.fasta')
-def test_linked_info_file(tmpdir):
- info_path = str(tmpdir.join('info.txt'))
- main(['-a linkedadapter=^AAAAAAAAAA...TTTTTTTTTT', '--info-file', info_path,
- '-o', str(tmpdir.join('out.fasta')), datapath('linked.fasta')])
- assert_files_equal(cutpath('linked-info.txt'), info_path)
+def test_linked_info_file(tmp_path):
+ info_path = tmp_path / 'info.txt'
+ main(['-a linkedadapter=^AAAAAAAAAA...TTTTTTTTTT', '--info-file', str(info_path),
+ '-o', str(tmp_path / 'out.fasta'), datapath('linked.fasta')])
+ assert_files_equal(cutpath('linked-info.txt'), info_path, ignore_trailing_space=True)
def test_linked_anywhere():
@@ -614,14 +646,19 @@ def test_negative_length(run):
@pytest.mark.timeout(0.5)
-def test_issue_296(tmpdir):
+def test_issue_296(tmp_path):
# Hang when using both --no-trim and --info-file together
- info_path = str(tmpdir.join('info.txt'))
- reads_path = str(tmpdir.join('reads.fasta'))
- out_path = str(tmpdir.join('out.fasta'))
- with open(reads_path, 'w') as f:
- f.write('>read\nCACAAA\n')
- main(['--info-file', info_path, '--no-trim', '-g', 'TTTCAC', '-o', out_path, reads_path])
+ info_path = tmp_path / 'info.txt'
+ reads_path = tmp_path / 'reads.fasta'
+ out_path = tmp_path / 'out.fasta'
+ reads_path.write_text(">read\nCACAAA\n")
+ main([
+ "--info-file", str(info_path),
+ "--no-trim",
+ "-g", "TTTCAC",
+ "-o", str(out_path),
+ str(reads_path),
+ ])
# Output should be unchanged because of --no-trim
assert_files_equal(reads_path, out_path)
@@ -649,8 +686,8 @@ def test_cores_autodetect(run):
run('--cores 0 -b TTAGACATATCTCCGTCG', 'small.fastq', 'underscore_fastq.gz')
-def test_write_compressed_fastq(cores, tmpdir):
- main(['--cores', str(cores), '-o', str(tmpdir.join('out.fastq.gz')), datapath('small.fastq')])
+def test_write_compressed_fastq(cores, tmp_path):
+ main(['--cores', str(cores), '-o', str(tmp_path / 'out.fastq.gz'), datapath('small.fastq')])
def test_minimal_report(run):
@@ -667,9 +704,9 @@ def test_empty_read_with_wildcard_in_adapter(run):
run("-g CWC", "empty.fastq", "empty.fastq")
-def test_print_progress_to_tty(tmpdir, mocker):
+def test_print_progress_to_tty(tmp_path, mocker):
mocker.patch("cutadapt.utils.sys.stderr").isatty.return_value = True
- main(["-o", str(tmpdir.join("out.fastq")), datapath("small.fastq")])
+ main(["-o", str(tmp_path / "out.fastq"), datapath("small.fastq")])
def test_adapter_order(run):
@@ -677,6 +714,24 @@ def test_adapter_order(run):
run("-a CCGGG -g ^AAACC", "adapterorder-ag.fasta", "adapterorder.fasta")
+def test_reverse_complement_no_rc_suffix(run, tmp_path):
+ out_path = tmp_path / "out.fastq"
+ main([
+ "-o", str(out_path),
+ "--revcomp",
+ "--no-index",
+ "--rename", "{header}",
+ "-g", "^TTATTTGTCT",
+ "-g", "^TCCGCACTGG",
+ datapath("revcomp.1.fastq")
+ ])
+ with dnaio.open(out_path) as f:
+ reads = list(f)
+ assert len(reads) == 6
+ assert reads[1].name == "read2/1"
+ assert reads[1].sequence == "ACCATCCGATATGTCTAATGTGGCCTGTTG"
+
+
def test_reverse_complement_normalized(run):
stats = run(
"--revcomp --no-index -g ^TTATTTGTCT -g ^TCCGCACTGG",
@@ -718,12 +773,12 @@ def test_max_expected_errors(run, cores):
def test_max_expected_errors_fasta(tmp_path):
path = tmp_path / "input.fasta"
path.write_text(">read\nACGTACGT\n")
- main(["--max-ee=0.001", "-o", "/dev/null", str(path)])
+ main(["--max-ee=0.001", "-o", os.devnull, str(path)])
def test_warn_if_en_dashes_used():
with pytest.raises(SystemExit):
- main(["–q", "25", "-o", "/dev/null", "in.fastq"])
+ main(["–q", "25", "-o", os.devnull, "in.fastq"])
@pytest.mark.parametrize("opt", ["-y", "--suffix"])
@@ -735,7 +790,7 @@ def test_suffix(opt, run):
@pytest.mark.parametrize("opt", ["--prefix", "--suffix"])
def test_rename_cannot_be_combined_with_other_renaming_options(opt):
with pytest.raises(SystemExit):
- main([opt, "something", "--rename='{id} {comment} extrainfo'", "-o", "/dev/null", datapath("empty.fastq")])
+ main([opt, "something", "--rename='{id} {comment} extrainfo'", "-o", os.devnull, datapath("empty.fastq")])
def test_rename(run):
=====================================
tests/test_main.py
=====================================
@@ -42,5 +42,5 @@ def test_parse_lengths():
def test_setup_logging():
import logging
logger = logging.getLogger(__name__)
- setup_logging(logger, stdout=True, quiet=False, minimal=False, debug=False)
+ setup_logging(logger, log_to_stderr=False, quiet=False, minimal=False, debug=False)
logger.info("Log message")
=====================================
tests/test_modifiers.py
=====================================
@@ -206,6 +206,16 @@ class TestRenamer:
info.cut_suffix = "TTAAGG"
assert renamer(read, info).name == "theid_TTAAGG thecomment"
+ def test_rc_template_varialbe(self):
+ renamer = Renamer("{id} rc={rc} {comment}")
+ read = Sequence("theid thecomment", "ACGT")
+ info = ModificationInfo(read)
+ assert renamer(read, info).name == "theid rc= thecomment"
+
+ read = Sequence("theid thecomment", "ACGT")
+ info.is_rc = True
+ assert renamer(read, info).name == "theid rc=rc thecomment"
+
class TestPairedEndRenamer:
def test_invalid_template_variable(self):
=====================================
tests/test_paired.py
=====================================
@@ -1,3 +1,4 @@
+import os
import os.path
import shutil
from itertools import product
@@ -121,14 +122,14 @@ def test_explicit_format_with_paired(tmpdir, run_paired):
def test_no_trimming_legacy():
# make sure that this doesn"t divide by zero
main([
- "-a", "XXXXX", "-o", "/dev/null", "-p", "/dev/null",
+ "-a", "XXXXX", "-o", os.devnull, "-p", os.devnull,
datapath("paired.1.fastq"), datapath("paired.2.fastq")])
def test_no_trimming():
# make sure that this doesn"t divide by zero
main([
- "-a", "XXXXX", "-A", "XXXXX", "-o", "/dev/null", "-p", "/dev/null",
+ "-a", "XXXXX", "-A", "XXXXX", "-o", os.devnull, "-p", os.devnull,
datapath("paired.1.fastq"), datapath("paired.2.fastq")])
@@ -147,7 +148,7 @@ def test_first_too_short(tmpdir, cores):
with pytest.raises(SystemExit):
main([
- "-o", "/dev/null",
+ "-o", os.devnull,
"--paired-output", str(tmpdir.join("out.fastq")),
"--cores", str(cores),
str(trunc1), datapath("paired.2.fastq")
@@ -164,7 +165,7 @@ def test_second_too_short(tmpdir, cores):
with pytest.raises(SystemExit):
main([
- "-o", "/dev/null",
+ "-o", os.devnull,
"--paired-output", str(tmpdir.join("out.fastq")),
"--cores", str(cores),
datapath("paired.1.fastq"), str(trunc2)
@@ -192,7 +193,7 @@ def test_unmatched_read_names(tmpdir, cores):
def test_p_without_o(cores):
"""Option -p given but -o missing"""
with pytest.raises(SystemExit):
- main("-a XX -p /dev/null".split()
+ main(["-a", "XX", "-p", os.devnull]
+ ["--cores", str(cores)]
+ [datapath("paired.1.fastq"), datapath("paired.2.fastq")])
@@ -200,7 +201,7 @@ def test_p_without_o(cores):
def test_paired_but_only_one_input_file(cores):
"""Option -p given but only one input file"""
with pytest.raises(SystemExit):
- main("-a XX -o /dev/null -p /dev/null".split()
+ main(["-a", "XX", "-o", os.devnull, "-p", os.devnull]
+ ["--cores", str(cores)]
+ [datapath("paired.1.fastq")])
=====================================
tests/test_parser.py
=====================================
@@ -66,6 +66,15 @@ def test_parse_not_linked():
assert p('a_name=ADAPT', 'front') == AdapterSpecification('a_name', None, 'ADAPT', {}, 'front')
+ at pytest.mark.parametrize("where", ("front", "back"))
+ at pytest.mark.parametrize("reqopt", ("required", "optional"))
+def test_parse_invalid_adapter_specific_parameter(where, reqopt):
+ parser = AdapterParser()
+ with pytest.raises(ValueError) as e:
+ parser._parse_not_linked("A;{}".format(reqopt), "name", where)
+ assert "can only be used within linked adapters" in e.value.args[0]
+
+
def test_parse_invalid_cmdline_type():
with pytest.raises(ValueError) as e:
AdapterSpecification._parse('A', 'invalid_type')
=====================================
tests/utils.py
=====================================
@@ -1,3 +1,4 @@
+import sys
import os.path
import subprocess
@@ -14,9 +15,18 @@ class FilesDifferent(Exception):
pass
-def assert_files_equal(path1, path2):
+def assert_files_equal(path1, path2, ignore_trailing_space: bool = False):
+ cmd = ["diff", "-u"]
+ if sys.platform == "win32":
+ cmd.append("--strip-trailing-cr")
+ if ignore_trailing_space:
+ if sys.platform == "darwin":
+ # Ignores too much, but macOS doesn’t have the option below
+ cmd.append("-b")
+ else:
+ cmd.append("--ignore-trailing-space")
try:
- subprocess.check_output(['diff', '-u', path1, path2], stderr=subprocess.STDOUT)
+ subprocess.check_output(cmd + [path1, path2], stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
raise FilesDifferent('\n' + e.output.decode()) from None
=====================================
tox.ini
=====================================
@@ -16,7 +16,7 @@ commands =
coverage xml
[testenv:docs]
-basepython = python3.6
+basepython = python3.7
changedir = doc
deps =
sphinx
@@ -24,13 +24,13 @@ deps =
commands = sphinx-build -W -b html -d {envtmpdir}/doctrees . {envtmpdir}/html
[testenv:flake8]
-basepython = python3.6
+basepython = python3.7
deps = flake8
skip_install = true
commands = flake8 src/ tests/
[testenv:mypy]
-basepython = python3.6
+basepython = python3.7
deps = mypy
commands = mypy src/
View it on GitLab: https://salsa.debian.org/med-team/python-cutadapt/-/commit/a93b33f441d39ebc55c7c5a4de421ba368c63be1
--
View it on GitLab: https://salsa.debian.org/med-team/python-cutadapt/-/commit/a93b33f441d39ebc55c7c5a4de421ba368c63be1
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20210421/2f602c48/attachment-0001.htm>
More information about the debian-med-commit
mailing list