[med-svn] [Git][med-team/python-dnaio][master] 4 commits: New upstream version 0.9.1
Andreas Tille (@tille)
gitlab at salsa.debian.org
Thu Aug 25 17:14:47 BST 2022
Andreas Tille pushed to branch master at Debian Med / python-dnaio
Commits:
7a5dc96e by Andreas Tille at 2022-08-25T18:13:18+02:00
New upstream version 0.9.1
- - - - -
057965fe by Andreas Tille at 2022-08-25T18:13:18+02:00
routine-update: New upstream version
- - - - -
0ce96a36 by Andreas Tille at 2022-08-25T18:13:18+02:00
Update upstream source from tag 'upstream/0.9.1'
Update to upstream version '0.9.1'
with Debian dir 04fc18c67b9e4b8f8a8dabb34daf41712b56ed11
- - - - -
1a54f793 by Andreas Tille at 2022-08-25T18:14:03+02:00
routine-update: Ready to upload to unstable
- - - - -
16 changed files:
- .github/workflows/ci.yml
- CHANGES.rst
- README.rst
- debian/changelog
- doc/api.rst
- doc/conf.py
- doc/tutorial.rst
- pyproject.toml
- setup.cfg
- src/dnaio/__init__.py
- src/dnaio/_core.pyx
- src/dnaio/interfaces.py
- src/dnaio/pairedend.py
- src/dnaio/readers.py
- src/dnaio/singleend.py
- src/dnaio/writers.py
Changes:
=====================================
.github/workflows/ci.yml
=====================================
@@ -65,7 +65,7 @@ jobs:
timeout-minutes: 15
strategy:
matrix:
- os: [ubuntu-20.04, windows-2019]
+ os: [ubuntu-20.04, windows-2019, macos-10.15]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout at v2
@@ -74,10 +74,11 @@ jobs:
- name: Build wheels
uses: pypa/cibuildwheel at v2.3.1
env:
- CIBW_BUILD: "cp*-manylinux_x86_64 cp3*-win_amd64"
+ CIBW_BUILD: "cp*-manylinux_x86_64 cp3*-win_amd64 cp3*-macosx_x86_64"
CIBW_ENVIRONMENT: "CFLAGS=-g0"
CIBW_TEST_REQUIRES: "pytest"
CIBW_TEST_COMMAND_LINUX: "cd {project} && pytest tests"
+ CIBW_TEST_COMMAND_MACOS: "cd {project} && pytest tests"
CIBW_TEST_COMMAND_WINDOWS: "cd /d {project} && pytest tests"
- uses: actions/upload-artifact at v2
with:
=====================================
CHANGES.rst
=====================================
@@ -2,6 +2,13 @@
Changelog
=========
+v0.9.1 (2022-08-01)
+-------------------
+
+* :pr:`85`: macOS wheels are now also built as part of the release procedure.
+* :pr:`81`: API documentation improvements and minor code refactors for
+ readability.
+
v0.9.0 (2022-05-17)
-------------------
=====================================
README.rst
=====================================
@@ -30,7 +30,8 @@ The main interface is the `dnaio.open <https://dnaio.readthedocs.io/en/latest/ap
bp += len(record)
print(f"The input file contains {bp/1E6:.1f} Mbp")
-See the `documentation <https://dnaio.readthedocs.io/>`_ for more.
+For more, see the `tutorial <https://dnaio.readthedocs.io/en/latest/tutorial.html>`_ and
+`API documentation <https://dnaio.readthedocs.io/en/latest/api.html>`_.
Features and supported file types
=================================
@@ -43,14 +44,12 @@ Features and supported file types
- Files with DOS/Windows linebreaks can be read
- FASTQ files with a second header line (after the ``+``) are supported
-
Limitations
===========
- Multi-line FASTQ files are not supported.
- FASTQ parsing is the focus of this library. The FASTA parser is not as optimized.
-
Links
=====
=====================================
debian/changelog
=====================================
@@ -1,3 +1,10 @@
+python-dnaio (0.9.1-1) unstable; urgency=medium
+
+ * Team upload.
+ * New upstream version
+
+ -- Andreas Tille <tille at debian.org> Thu, 25 Aug 2022 18:13:25 +0200
+
python-dnaio (0.9.0-1) unstable; urgency=medium
* Team upload.
=====================================
doc/api.rst
=====================================
@@ -4,6 +4,7 @@ The dnaio API
.. module:: dnaio
+
The open function
-----------------
@@ -20,40 +21,52 @@ The ``SequenceRecord`` class
.. automethod:: __init__(name: str, sequence: str, qualities: Optional[str] = None)
+Reader and writer interfaces
+----------------------------
+
+.. autoclass:: SingleEndReader
+ :members: __iter__
+
+.. autoclass:: PairedEndReader
+ :members: __iter__
+
+.. autoclass:: SingleEndWriter
+ :members: write
+
+.. autoclass:: PairedEndWriter
+ :members: write
+
+
Reader and writer classes
-------------------------
+The `dnaio.open` function returns an instance of one of the following classes.
+They can also be used directly if needed.
+
+
.. autoclass:: FastaReader
+ :show-inheritance:
.. autoclass:: FastaWriter
- :members: write
+ :show-inheritance:
.. autoclass:: FastqReader
+ :show-inheritance:
.. autoclass:: FastqWriter
- :members: writeseq
-
- .. py:method:: write(record: SequenceRecord) -> None:
-
- Write a SequenceRecord to the FASTQ file.
+ :show-inheritance:
.. autoclass:: TwoFilePairedEndReader
+ :show-inheritance:
.. autoclass:: TwoFilePairedEndWriter
+ :show-inheritance:
.. autoclass:: InterleavedPairedEndReader
+ :show-inheritance:
.. autoclass:: InterleavedPairedEndWriter
- :members: write
-
-.. autoclass:: SingleEndReader
-
-.. autoclass:: PairedEndReader
-
-.. autoclass:: SingleEndWriter
-
-.. autoclass:: PairedEndWriter
- :members: write
+ :show-inheritance:
Chunked reading of sequence records
@@ -68,6 +81,12 @@ processed there.
.. autofunction:: read_paired_chunks
+Functions
+---------
+
+.. autofunction:: records_are_mates
+
+
Exceptions
----------
=====================================
doc/conf.py
=====================================
@@ -36,3 +36,6 @@ default_role = "obj" # (or "any")
issues_uri = "https://github.com/marcelm/dnaio/issues/{issue}"
issues_pr_uri = "https://github.com/marcelm/dnaio/pull/{pr}"
+
+autodoc_typehints = "description"
+python_use_unqualified_type_names = True
=====================================
doc/tutorial.rst
=====================================
@@ -3,7 +3,7 @@ Tutorial
This should get you started with using ``dnaio``.
The only essential concepts to know about are
-the `dnaio.open` function and the `SequenceRecord` object.
+the `dnaio.open` function and the `~dnaio.SequenceRecord` object.
Reading
@@ -35,7 +35,7 @@ A ``SequenceRecord`` has the attributes ``name``, ``sequence``
and ``qualities``. All of these are ``str`` objects.
The ``qualities`` attribute is ``None`` when reading FASTA files.
-This program uses the ``name`` attribute
+The following program uses the ``name`` attribute
to check whether any sequence names are duplicated in a FASTA file::
import dnaio
@@ -59,12 +59,13 @@ pass the ``mode="w"`` argument to ``dnaio.open``::
writer.write(dnaio.SequenceRecord("name", "ACGT", "#B!#"))
Here, a `~dnaio.FastqWriter` object is returned by ``dnaio.open``,
-which has a ``~dnaio.FastqWriter.write()`` method that accepts a ``SequenceRecord``.
+which has a `~dnaio.FastqWriter.write()` method that accepts a ``SequenceRecord``.
Instead of constructing a single record from scratch,
-it may be more realistic to take input reads,
-process them somehow and write them to a new output file.
-The following program truncates all reads in the input file to a length of 30 nt
+in practice it is more realistic to take input reads,
+process them, and write them to a new output file.
+The following example program shows how that can be done.
+It truncates all reads in the input file to a length of 30 nt
and writes them to another file::
import dnaio
@@ -76,7 +77,7 @@ and writes them to another file::
This also shows that `~dnaio.SequenceRecord` objects support slicing:
``record[:30]`` returns a new ``SequenceRecord`` object with the sequence and qualities
-trimmed to the first 30 characters (leaving the name unchanged).
+trimmed to the first 30 characters, leaving the name unchanged.
Paired-end data
@@ -100,7 +101,7 @@ In this example, ``dnaio.open`` returns a `~dnaio.TwoFilePairedEndReader`.
It also supports iteration, but instead of a single ``SequenceRecord``,
it returns a pair of them.
-To read from interleaved paired-end data, the only change needed is to
+To read from interleaved paired-end data,
pass ``interleaved=True`` to ``dnaio.open`` instead of a second file name::
...
@@ -110,11 +111,11 @@ pass ``interleaved=True`` to ``dnaio.open`` instead of a second file name::
The ``PairedEndReader`` classes check whether the input files are properly paired,
that is, whether they have the same number of reads in both inputs and whether the
read names match.
-For this reason, always use a single call to ``dnaio.open`` to open paired-end files.
-(Avoid opening them as two single-end files.)
+For this reason, always use a single call to ``dnaio.open`` to open paired-end files
+(that is, avoid opening them as two single-end files.)
To demonstrate how to write paired-end data,
-we show a program that reads from a single-end FASTQ file and converts them to
+we show a program that reads from a single-end FASTQ file and converts the records to
simulated paired-end reads by writing the first 30 nt to R1 and the last 30 nt
to R2::
=====================================
pyproject.toml
=====================================
@@ -1,6 +1,9 @@
[build-system]
-requires = ["setuptools >= 52", "wheel", "setuptools_scm >= 6.2", "Cython >= 0.29.20"]
+requires = ["setuptools >= 52", "setuptools_scm >= 6.2", "Cython >= 0.29.20"]
build-backend = "setuptools.build_meta"
[tool.setuptools_scm]
write_to = "src/dnaio/_version.py"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
=====================================
setup.cfg
=====================================
@@ -5,7 +5,7 @@ author_email = marcel.martin at scilifelab.se
url = https://dnaio.readthedocs.io/
description = Read and write FASTA and FASTQ files efficiently
long_description = file: README.rst
-long_description_content_type = text/markdown
+long_description_content_type = text/x-rst
license = MIT
project_urls =
Changelog = https://dnaio.readthedocs.io/en/latest/changes.html
=====================================
src/dnaio/__init__.py
=====================================
@@ -41,6 +41,7 @@ from .readers import FastaReader, FastqReader
from .writers import FastaWriter, FastqWriter
from .singleend import _open_single
from .pairedend import (
+ _open_paired,
TwoFilePairedEndReader,
TwoFilePairedEndWriter,
InterleavedPairedEndReader,
@@ -77,7 +78,7 @@ def open(
opener=xopen
) -> Union[SingleEndReader, PairedEndReader, SingleEndWriter, PairedEndWriter]:
"""
- Open sequence files in FASTA or FASTQ format for reading or writing.
+ Open one or two files in FASTA or FASTQ format for reading or writing.
Parameters:
@@ -104,14 +105,15 @@ def open(
qualities:
When mode is ``'w'`` and fileformat is *None*, this can be set
to *True* or *False* to specify whether the written sequences will have
- quality values. This is is used in two ways:
+ quality values. This is used in two ways:
- If the output format cannot be determined (unrecognized extension
- etc), no exception is raised, but fasta or fastq format is chosen
+ etc.), no exception is raised, but FASTA or FASTQ format is chosen
appropriately.
- When False (no qualities available), an exception is raised when the
auto-detected output format is FASTQ.
+
opener: A function that is used to open file1 and file2 if they are not
already open file-like objects. By default, ``xopen`` is used, which can
also open compressed file formats.
@@ -122,41 +124,19 @@ def open(
"""
if mode not in ("r", "rb", "w", "a"):
raise ValueError("Mode must be 'r', 'rb', 'w' or 'a'")
- if interleaved and file2 is not None:
- raise ValueError("When interleaved is set, file2 must be None")
-
- if file2 is not None:
- if mode in "wa" and file1 == file2:
- raise ValueError("The paired-end output files are identical")
- if "r" in mode:
- return TwoFilePairedEndReader(
- file1, file2, fileformat=fileformat, opener=opener, mode=mode
- )
- append = mode == "a"
- return TwoFilePairedEndWriter(
+ elif interleaved and file2 is not None:
+ raise ValueError("When interleaved is True, file2 must be None")
+ elif interleaved or file2 is not None:
+ return _open_paired(
file1,
- file2,
- fileformat=fileformat,
- qualities=qualities,
+ file2=file2,
opener=opener,
- append=append,
- )
- if interleaved:
- if "r" in mode:
- return InterleavedPairedEndReader(
- file1, fileformat=fileformat, opener=opener, mode=mode
- )
- append = mode == "a"
- return InterleavedPairedEndWriter(
- file1,
fileformat=fileformat,
+ interleaved=interleaved,
+ mode=mode,
qualities=qualities,
- opener=opener,
- append=append,
)
-
- # The multi-file options have been dealt with, delegate rest to the
- # single-file function.
- return _open_single(
- file1, opener=opener, fileformat=fileformat, mode=mode, qualities=qualities
- )
+ else:
+ return _open_single(
+ file1, opener=opener, fileformat=fileformat, mode=mode, qualities=qualities
+ )
=====================================
src/dnaio/_core.pyx
=====================================
@@ -239,10 +239,7 @@ cdef class SequenceRecord:
def fastq_bytes_two_headers(self):
- """
- Return this record in FASTQ format as a bytes object where the header (after the @) is
- repeated on the third line.
- """
+ # Deprecated, use ``.fastq_bytes(two_headers=True)`` instead.
return self.fastq_bytes(two_headers=True)
def is_mate(self, SequenceRecord other):
@@ -619,11 +616,11 @@ cdef inline bint record_ids_match(char *header1,
size_t header2_length,
bint id1_ends_with_number):
"""
- Check whether the ASCII-encoded IDs match.
-
+ Check whether the ASCII-encoded IDs match.
+
header1, header2 pointers to the ASCII-encoded headers
id1_length, the length of header1 before the first whitespace
- header2_length, the full length of header2.
+ header2_length, the full length of header2.
id1_ends_with_number, whether id1 ends with a 1,2 or 3.
"""
@@ -644,19 +641,26 @@ cdef inline bint record_ids_match(char *header1,
return memcmp(<void *>header1, <void *>header2, id1_length) == 0
-def records_are_mates(*args):
+def records_are_mates(*args) -> bool:
"""
- Check if provided SequenceRecord objects are all mates of each other by
+ Check if the provided `SequenceRecord` objects are all mates of each other by
comparing their record IDs.
- Accepts two or more SequenceRecord objects.
-
- Example usage:
- for records in zip(*all_my_fastq_readers):
- if not records_are_mates(*records):
- raise MateError(f"Ids do not match for {records}")
-
+ Accepts two or more `SequenceRecord` objects.
+
+ This is the same as `SequenceRecord.is_mate` in the case of only two records,
+ but allows for for cases where information is split into three records or more
+ (such as UMI, R1, R2 or index, R1, R2).
+
+ If there are only two records to check, prefer `SequenceRecord.is_mate`.
+
+ Example usage::
+
+ for records in zip(*all_my_fastq_readers):
+ if not records_are_mates(*records):
+ raise MateError(f"IDs do not match for {records}")
+
Args:
- *args: two or more SequenceRecord objects
+ *args: two or more `~dnaio.SequenceRecord` objects
Returns: True or False
"""
=====================================
src/dnaio/interfaces.py
=====================================
@@ -7,22 +7,36 @@ from dnaio import SequenceRecord
class SingleEndReader(ABC):
@abstractmethod
def __iter__(self) -> Iterator[SequenceRecord]:
- pass
+ """Yield the records in the input as `SequenceRecord` objects."""
class PairedEndReader(ABC):
@abstractmethod
def __iter__(self) -> Iterator[Tuple[SequenceRecord, SequenceRecord]]:
- pass
+ """
+ Yield the records in the paired-end input as pairs of `SequenceRecord` objects.
+
+ Raises a `FileFormatError` if reads are improperly paired, that is,
+ if there are more reads in one file than the other or if the record IDs
+ do not match (according to `SequenceRecord.is_mate`).
+ """
class SingleEndWriter(ABC):
@abstractmethod
def write(self, record: SequenceRecord) -> None:
- pass
+ """Write a `SequenceRecord` to the output."""
class PairedEndWriter(ABC):
@abstractmethod
def write(self, record1: SequenceRecord, record2: SequenceRecord) -> None:
- pass
+ """
+ Write a pair of `SequenceRecord` objects to the paired-end output.
+
+ This method does not verify that both records have matching IDs
+ because this was already done at parsing time. If it is possible
+ that the record IDs no longer match, check that
+ ``record1.is_mate(record2)`` returns True before calling
+ this function.
+ """
=====================================
src/dnaio/pairedend.py
=====================================
@@ -12,12 +12,56 @@ from .writers import FastaWriter, FastqWriter
from .singleend import _open_single
+def _open_paired(
+ file1: Union[str, PathLike, BinaryIO],
+ *,
+ file2: Optional[Union[str, PathLike, BinaryIO]] = None,
+ fileformat: Optional[str] = None,
+ interleaved: bool = False,
+ mode: str = "r",
+ qualities: Optional[bool] = None,
+ opener=xopen,
+) -> Union[PairedEndReader, PairedEndWriter]:
+ """
+ Open paired-end reads
+ """
+ if interleaved and file2 is not None:
+ raise ValueError("When interleaved is True, file2 must be None")
+ if file2 is not None:
+ if mode in "wa" and file1 == file2:
+ raise ValueError("The paired-end output files are identical")
+ if "r" in mode:
+ return TwoFilePairedEndReader(
+ file1, file2, fileformat=fileformat, opener=opener, mode=mode
+ )
+ append = mode == "a"
+ return TwoFilePairedEndWriter(
+ file1,
+ file2,
+ fileformat=fileformat,
+ qualities=qualities,
+ opener=opener,
+ append=append,
+ )
+ if interleaved:
+ if "r" in mode:
+ return InterleavedPairedEndReader(
+ file1, fileformat=fileformat, opener=opener, mode=mode
+ )
+ append = mode == "a"
+ return InterleavedPairedEndWriter(
+ file1,
+ fileformat=fileformat,
+ qualities=qualities,
+ opener=opener,
+ append=append,
+ )
+ assert False
+
+
class TwoFilePairedEndReader(PairedEndReader):
"""
Read paired-end reads from two files.
-
- Wraps two BinaryFileReader instances, making sure that reads are properly
- paired.
"""
paired = True
=====================================
src/dnaio/readers.py
=====================================
@@ -4,6 +4,7 @@ Classes for reading FASTA and FASTQ files
__all__ = ["FastaReader", "FastqReader"]
import io
+from os import PathLike
from typing import Union, BinaryIO, Optional, Iterator, List
from xopen import xopen
@@ -25,7 +26,7 @@ class BinaryFileReader:
def __init__(
self,
- file: Union[str, BinaryIO],
+ file: Union[PathLike, str, BinaryIO],
*,
opener=xopen,
_close_file: Optional[bool] = None,
@@ -67,7 +68,7 @@ class FastaReader(BinaryFileReader, SingleEndReader):
def __init__(
self,
- file: Union[str, BinaryIO],
+ file: Union[PathLike, str, BinaryIO],
*,
keep_linebreaks: bool = False,
sequence_class=SequenceRecord,
@@ -89,7 +90,7 @@ class FastaReader(BinaryFileReader, SingleEndReader):
def __iter__(self) -> Iterator[SequenceRecord]:
"""
- Read next entry from the file (single entry at a time).
+ Iterate over the records in this FASTA file.
"""
name = None
seq: List[str] = []
@@ -128,7 +129,7 @@ class FastqReader(BinaryFileReader, SingleEndReader):
def __init__(
self,
- file: Union[str, BinaryIO],
+ file: Union[PathLike, str, BinaryIO],
*,
sequence_class=SequenceRecord,
buffer_size: int = 128 * 1024, # Buffer size used by cat, pigz etc.
@@ -165,6 +166,7 @@ class FastqReader(BinaryFileReader, SingleEndReader):
raise
def __iter__(self) -> Iterator[SequenceRecord]:
+ """Iterate over the records in this FASTQ file."""
return self._iter
@property
=====================================
src/dnaio/singleend.py
=====================================
@@ -16,7 +16,7 @@ def _open_single(
qualities: Optional[bool] = None,
) -> Union[FastaReader, FastaWriter, FastqReader, FastqWriter]:
"""
- Open a single sequence file. See description of open() above.
+ Open a single sequence file.
"""
if mode not in ("r", "w", "a"):
raise ValueError("Mode must be 'r', 'w' or 'a'")
=====================================
src/dnaio/writers.py
=====================================
@@ -140,9 +140,10 @@ class FastqWriter(FileWriter, SingleEndWriter):
def write(self, record: SequenceRecord) -> None:
"""
- Dummy method to make it possible to instantiate this class.
- The correct write method is assigned in the constructor.
+ Write a record to the FASTQ file.
"""
+ # The 'write' attribute is overwritten in the constructor with the correct
+ # write method (_write or _write_two_headers)
assert False
def _write(self, record: SequenceRecord) -> None:
@@ -159,4 +160,5 @@ class FastqWriter(FileWriter, SingleEndWriter):
self._file.write(record.fastq_bytes(two_headers=True))
def writeseq(self, name: str, sequence: str, qualities: str) -> None:
+ # Deprecated
self._file.write(f"@{name:s}\n{sequence:s}\n+\n{qualities:s}\n".encode("ascii"))
View it on GitLab: https://salsa.debian.org/med-team/python-dnaio/-/compare/453b6fe9e2effe50d8b649ef879bb843b503c6a9...1a54f793b27fd9bd9ac6454887e69372e2bfafee
--
View it on GitLab: https://salsa.debian.org/med-team/python-dnaio/-/compare/453b6fe9e2effe50d8b649ef879bb843b503c6a9...1a54f793b27fd9bd9ac6454887e69372e2bfafee
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20220825/14a4c82a/attachment-0001.htm>
More information about the debian-med-commit
mailing list