[med-svn] [Git][med-team/changeo][upstream] New upstream version 1.3.5
Andreas Tille (@tille)
gitlab at salsa.debian.org
Sun Nov 15 21:03:44 GMT 2026
Andreas Tille pushed to branch upstream at Debian Med / changeo
Commits:
421d5388 by Andreas Tille at 2026-11-15T17:50:36+01:00
New upstream version 1.3.5
- - - - -
30 changed files:
- INSTALL.rst
- MANIFEST.in
- NEWS.rst
- PKG-INFO
- README.rst
- bin/AssignGenes.py
- bin/BuildTrees.py
- bin/ConvertDb.py
- bin/CreateGermlines.py
- bin/DefineClones.py
- bin/MakeDb.py
- bin/ParseDb.py
- changeo.egg-info/PKG-INFO
- changeo.egg-info/SOURCES.txt
- changeo.egg-info/requires.txt
- changeo/Applications.py
- changeo/Commandline.py
- changeo/Distance.py
- changeo/Gene.py
- changeo/IO.py
- changeo/Multiprocessing.py
- changeo/Version.py
- + pyproject.toml
- requirements.txt
- setup.py
- + tests/test_DefineClones.py
- + tests/test_Distance.py
- + tests/test_IO.py
- + tests/test_ParseDb.py
- + tests/test_Receptor.py
Changes:
=====================================
INSTALL.rst
=====================================
@@ -7,7 +7,7 @@ The simplest way to install the latest stable release of Change-O is via pip::
The current development build can be installed using pip and git in similar fashion::
- > pip3 install git+https://bitbucket.org/kleinstein/changeo@master --user
+ > pip3 install git+https://github.com/immcantation/changeo@master --user
If you currently have a development version installed, then you will likely
need to add the arguments ``--upgrade --no-deps --force-reinstall`` to the
@@ -18,14 +18,17 @@ Requirements
The minimum dependencies for installation are:
-+ `Python 3.4.0 <http://python.org>`__
-+ `setuptools 2.0 <http://bitbucket.org/pypa/setuptools>`__
-+ `NumPy 1.8 <http://numpy.org>`__
-+ `SciPy 0.14 <http://scipy.org>`__
-+ `pandas 0.24 <http://pandas.pydata.org>`__
-+ `Biopython 1.77 <http://biopython.org>`__
-+ `presto 0.7.0 <http://presto.readthedocs.io>`__
++ `Python 3.10.0 <http://python.org>`__
++ `setuptools 65.5 <http://bitbucket.org/pypa/setuptools>`__
++ `NumPy 1.23.2 <http://numpy.org>`__
++ `SciPy 1.9.3 <http://scipy.org>`__
++ `pandas 1.5.0 <http://pandas.pydata.org>`__
++ `Biopython 1.81 <http://biopython.org>`__
++ `presto 0.7.1 <http://presto.readthedocs.io>`__
+ `airr 1.3.1 <https://docs.airr-community.org>`__
++ `PyYAML 6.0 <http://pyyaml.org>`__
++ `packaging 21.3 <https://packaging.pypa.io>`__
++ `importlib-resoures 6.4.0 <https://pypi.org/project/importlib-resources>`__
Some tools wrap external applications that are not required for installation.
Those tools require minimum versions of:
@@ -34,7 +37,7 @@ Those tools require minimum versions of:
+ ConvertDb-genbank requires `tbl2asn <https://www.ncbi.nlm.nih.gov/genbank/tbl2asn2>`__
+ AssignGenes requires `IgBLAST 1.6 <https://ncbi.github.io/igblast>`__, but
version 1.11 or higher is strongly recommended.
-+ BuildTrees requires `IgPhyML 1.0.5 <https://bitbucket.org/kbhoehn/igphyml>`_
++ BuildTrees requires `IgPhyML 1.0.5 <https://github.com/immcantation/igphyml>`_
Linux
--------------------------------------------------------------------------------
@@ -45,9 +48,10 @@ Linux
Biopython according to its
`instructions <http://biopython.org/DIST/docs/install/Installation.html>`__.
-2. Install `presto 0.6.2 <http://presto.readthedocs.io>`__ or greater.
+2. Install `presto 0.7.0 <http://presto.readthedocs.io>`__ or greater.
-3. Download the Change-O bundle and run::
+3. Download the `Change-O bundle <https://github.com/immcantation/changeo/tags>`__
+ and run::
> pip3 install changeo-x.y.z.tar.gz --user
@@ -86,10 +90,10 @@ Mac OS X
> pip3 install numpy scipy pandas biopython
-8. Install `presto 0.6.2 <http://presto.readthedocs.io>`__ or greater.
+8. Install `presto 0.7.0 <http://presto.readthedocs.io>`__ or greater.
-9. Download the Change-O bundle, open a terminal window, change directories
- to the download folder, and run::
+9. Download the `Change-O bundle <https://github.com/immcantation/changeo/tags>`__,
+ open a terminal window, change directories to the download folder, and run::
> pip3 install changeo-x.y.z.tar.gz
@@ -104,10 +108,10 @@ Windows
`Unofficial Windows binary <http://www.lfd.uci.edu/~gohlke/pythonlibs>`__
collection.
-3. Install `presto 0.6.2 <http://presto.readthedocs.io>`__ or greater.
+3. Install `presto 0.7.0 <http://presto.readthedocs.io>`__ or greater.
-4. Download the Change-O bundle, open a Command Prompt, change directories to
- the download folder, and run::
+4. Download the `Change-O bundle <https://github.com/immcantation/changeo/tags>`__,
+ open a Command Prompt, change directories to the download folder, and run::
> pip install changeo-x.y.z.tar.gz
=====================================
MANIFEST.in
=====================================
@@ -2,4 +2,5 @@ include requirements.txt
include README.rst
include INSTALL.rst
include NEWS.rst
-include LICENSE
\ No newline at end of file
+include LICENSE
+exclude announcements.html
\ No newline at end of file
=====================================
NEWS.rst
=====================================
@@ -1,6 +1,86 @@
Release Notes
===============================================================================
+Version 1.3.5: September 2, 2026
+-------------------------------------------------------------------------------
+
+General:
+
++ Added support for gzip compressed (``.gz``) repertoire files. Compressed input
+ is detected automatically, and output is compressed to match whenever the
+ input is compressed. A new ``--gzip-output`` argument forces gzip compressed
+ output regardless of the input. Supported by ``AssignGenes.py``,
+ ``BuildTrees.py``, ``ConvertDb.py``, ``CreateGermlines.py``,
+ ``DefineClones.py``, ``MakeDb.py`` and ``ParseDb.py``. ``AssignGenes.py``
+ supports compressed input only, as its output is written directly by the
+ wrapped IgBLAST/IgBLAST-aa executables.
+
+CreateGermlines:
+
++ ``CreateGermlines.py`` now validates the sequence field and raises an
+ informative error if amino acid sequences are detected. Amino acid sequences
+ are not supported; a nucleotide sequence field must be provided.
+
+MakeDb:
+
++ ``MakeDb.py igblast-aa`` now reports all top-scoring ``v_call`` hits in the
+ event of ties. Previously, only the first top-scoring hit was reported. The
+ new behavior aligns with that of the ``igblast`` subcommand.
+
+Gene:
+
++ Added support for dual-locus ``TRA/TRD`` gene detection. ``getLocus`` now
+ correctly identifies genes with the ``TRAV.../DV...`` naming convention
+ (e.g., ``TRAV14/DV4``) and returns ``TRA/TRD`` as the locus.
++ Fixed a bug where ``getLocus`` would raise a ``TypeError`` when the gene
+ call was ``None``.
+
+ParseDb:
+
++ ``ParseDb.py merge`` now accepts ``-f`` and ``-u`` arguments to annotate
+ merged records with a sample identifier field, with one value per input
+ file specified via ``-d``.
++ ``ParseDb.py merge`` now issues a warning when ``-f`` specifies a field that
+ already exists in one or more input files, as its values will be
+ overwritten, and when ``-u`` values are provided without a ``-f`` field to
+ assign them to, in which case the values are ignored.
+
+
+Version 1.3.4: July 31, 2025
+-------------------------------------------------------------------------------
+
+MakeDb:
+
++ ``MakeDb.py`` is now parallelized across input files. The ``--nproc`` argument
+ can be used to restrict the resources used.
++ Added the flag ``--partial`` to ``MakeDb igblast-aa`` to pass incomplete alignment
+ results. As ``igblastp`` (as at igblast 1.22.0) only uses the V germline
+ database, all sequences will be missing the ``junction`` and ``j_call`` fields
+ and be considered incomplete. Specifying ``--partial`` will allow these
+ sequences to be processed ignoring the missing fields.
+
+Documentation:
+
++ Added a "Contributing" section to the documentation menu for community guidelines.
++ Updated "Contact".
+
+Version 1.3.3: May 14, 2025
+-------------------------------------------------------------------------------
+
++ Updated dependencies to address deprecation warnings. Replaced ``pkg_resources``
+ with ``packaging`` and ``importlib``.
++ Bumped minimum Python version to 3.10.0 and updated requirements to: numpy>=1.23.2,
+ scipy>=1.9.3, pandas>=1.5.0, biopython>=1.81, PyYAML>=6.0, setuptools>=65.5.0,
+ presto>=0.7.1, airr>=1.3.1, packaging>=21.3, importlib-resources>=6.4.0.
+
+Version 1.3.1: March 27, 2025
+-------------------------------------------------------------------------------
+
++ Active development has moved from Bitbucket to GitHub (https://github.com/immcantation/changeo).
++ Documentation updates.
++ Various updates to internals to avoid deprecation warnings.
++ Updates in requirements: Python 3.7.0, biopython>=1.81 and packaging>=23.2.
+
Version 1.3.0: December 11, 2022
-------------------------------------------------------------------------------
@@ -58,6 +138,7 @@ Version 1.1.0: June 21, 2021
+ Updated dependencies to biopython >= v1.77, airr >= v1.3.1, PyYAML>=5.1.
MakeDb:
+
+ Added the ``--imgt-id-len`` argument to accommodate changes introduced in how
IMGT/HighV-QUEST truncates sequence identifiers as of v1.8.3 (May 7, 2021).
The header lines in the fasta files are now truncated to 49 characters. In
=====================================
PKG-INFO
=====================================
@@ -1,9 +1,9 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
Name: changeo
-Version: 1.3.0
+Version: 1.3.5
Summary: A bioinformatics toolkit for processing high-throughput lymphocyte receptor sequencing data.
Home-page: http://changeo.readthedocs.io
-Download-URL: https://bitbucket.org/kleinstein/changeo/downloads
+Download-URL: https://github.com/immcantation/changeo/tags
Author: Namita Gupta, Jason Anthony Vander Heiden
Author-email: immcantation at googlegroups.com
License: GNU Affero General Public License 3 (AGPL-3)
@@ -13,9 +13,30 @@ Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3.4
+Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
License-File: LICENSE
+Requires-Dist: numpy>=1.23.2
+Requires-Dist: scipy>=1.9.3
+Requires-Dist: pandas>=1.5.0
+Requires-Dist: biopython>=1.81
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: setuptools>=65.5.0
+Requires-Dist: presto>=0.7.1
+Requires-Dist: airr>=1.3.1
+Requires-Dist: packaging>=21.3
+Requires-Dist: importlib-resources>=6.4.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary
.. image:: https://img.shields.io/pypi/dm/changeo
:target: https://pypi.org/project/changeo
@@ -28,7 +49,7 @@ Change-O - Repertoire clonal assignment toolkit
Change-O is a collection of tools for processing the output of V(D)J alignment
tools, assigning clonal clusters to immunoglobulin (Ig) sequences, and
reconstructing germline sequences.
-
+
Dramatic improvements in high-throughput sequencing technologies now enable
large-scale characterization of Ig repertoires, defined as the collection of
trans-membrane antigen-receptor proteins located on the surface of B cells and
@@ -37,3 +58,6 @@ Ig and TCR sequences following germline segment assignment. Change-O
handles output from IMGT/HighV-QUEST and IgBLAST, and provides a wide variety of
clustering methods for assigning clonal groups to Ig sequences. Record sorting,
grouping, and various database manipulation operations are also included.
+
+**IMPORTANT!**
+Active development of Change-O has moved to https://github.com/immcantation/changeo
=====================================
README.rst
=====================================
@@ -9,7 +9,7 @@ Change-O - Repertoire clonal assignment toolkit
Change-O is a collection of tools for processing the output of V(D)J alignment
tools, assigning clonal clusters to immunoglobulin (Ig) sequences, and
reconstructing germline sequences.
-
+
Dramatic improvements in high-throughput sequencing technologies now enable
large-scale characterization of Ig repertoires, defined as the collection of
trans-membrane antigen-receptor proteins located on the surface of B cells and
@@ -18,3 +18,6 @@ Ig and TCR sequences following germline segment assignment. Change-O
handles output from IMGT/HighV-QUEST and IgBLAST, and provides a wide variety of
clustering methods for assigning clonal groups to Ig sequences. Record sorting,
grouping, and various database manipulation operations are also included.
+
+**IMPORTANT!**
+Active development of Change-O has moved to https://github.com/immcantation/changeo
=====================================
bin/AssignGenes.py
=====================================
@@ -11,7 +11,7 @@ import os
import shutil
from argparse import ArgumentParser
from collections import OrderedDict
-from pkg_resources import parse_version
+from packaging.version import parse as parse_version
from textwrap import dedent
from time import time
import re
@@ -91,8 +91,8 @@ def assignIgBLAST(seq_file, amino_acid=False, igdata=default_igdata, loci='ig',
out_file = getOutputName(seq_file, out_label='igblast', out_dir=out_args['out_dir'],
out_name=out_args['out_name'], out_type=out_type)
# convert to FASTA if needed
- infile = open(seq_file, 'r')
- test = infile.read()[0]
+ with open(seq_file, 'r') as infile:
+ test = infile.read()[0]
if test == "@":
printMessage("Running conversion from FASTQ to FASTA")
fasta_out_dir, filename = os.path.split(out_file)
@@ -129,7 +129,7 @@ def assignIgBLAST(seq_file, amino_acid=False, igdata=default_igdata, loci='ig',
while f.read(1) != b'\n':
f.seek(-2, os.SEEK_CUR)
pass_info = f.readline().decode()
- num_seqs_match = re.search('(# BLAST processed )(\d+)( .*)', pass_info)
+ num_seqs_match = re.search(r'(# BLAST processed )(\d+)( .*)', pass_info)
num_sequences = num_seqs_match.group(2)
else:
f = open(out_file, 'rb')
@@ -183,7 +183,11 @@ def getArgParser():
subparsers.required = True
# Parent parser
- parent_parser = getCommonArgParser(db_in=False, log=False, failed=False, format=False, multiproc=True)
+ # gzip_output is intentionally excluded: assignIgBLAST() writes its output through the
+ # igblastn/igblastp executable directly, not through getOutputHandle/openFile, so the
+ # option would otherwise be silently accepted and ignored.
+ parent_parser = getCommonArgParser(db_in=False, log=False, failed=False, format=False,
+ multiproc=True, gzip_output=False)
# Subparser to run igblastn
parser_igblast = subparsers.add_parser('igblast', parents=[parent_parser],
=====================================
bin/BuildTrees.py
=====================================
@@ -23,7 +23,8 @@ from functools import partial
from presto.Defaults import default_out_args
from presto.IO import printLog, printMessage, printWarning, printError, printDebug
from changeo.Defaults import default_format
-from changeo.IO import splitName, getDbFields, getFormatOperators, getOutputHandle, getOutputName
+from changeo.IO import splitName, getDbFields, getFormatOperators, getOutputHandle, getOutputName, \
+ openFile
from changeo.Alignment import RegionDefinition
from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgParser, parseCommonArgs
@@ -1054,7 +1055,8 @@ def buildTrees(db_file, meta_data=None, target_clones=None, collapse=False, ncdr
out_label=out_label,
out_dir=out_args["out_dir"],
out_name= out_args["out_name"],
- out_type="tsv")
+ out_type="tsv",
+ gzip_output=out_args.get("gzip_output"))
igphyml_out = None
if igphyml:
@@ -1084,7 +1086,7 @@ def buildTrees(db_file, meta_data=None, target_clones=None, collapse=False, ncdr
out_fields = getDbFields(db_file, reader=reader)
# open input file
- handle = open(db_file, "r")
+ handle = openFile(db_file, "r")
records = reader(handle)
fail_handle, fail_writer = None, None
@@ -1093,7 +1095,8 @@ def buildTrees(db_file, meta_data=None, target_clones=None, collapse=False, ncdr
out_label="lineages-fail",
out_dir=out_args["out_dir"],
out_name=out_args["out_name"],
- out_type=out_args["out_type"])
+ out_type=out_args["out_type"],
+ gzip_output=out_args.get("gzip_output"))
fail_writer = writer(fail_handle, fields=out_fields)
cloneseqs = {}
=====================================
bin/ConvertDb.py
=====================================
@@ -29,9 +29,9 @@ from changeo.Defaults import default_id_field, default_seq_field, default_germ_f
default_csv_size, default_format, default_out_args
from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgParser, parseCommonArgs
from changeo.Gene import getCGene
-from changeo.IO import countDbFile, getFormatOperators, getOutputHandle, AIRRReader, AIRRWriter, \
- ChangeoReader, ChangeoWriter, TSVReader, ReceptorData, readGermlines, \
- checkFields, yamlDict
+from changeo.IO import countDbFile, getFormatOperators, getOutputHandle, gzipOutputName, openFile, \
+ AIRRReader, AIRRWriter, ChangeoReader, ChangeoWriter, TSVReader, ReceptorData, \
+ readGermlines, checkFields, yamlDict
from changeo.Receptor import AIRRSchema, ChangeoSchema
# System settings
@@ -101,7 +101,7 @@ def convertToAIRR(db_file, format=default_format,
printError('Invalid format %s.' % format)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = reader(db_handle)
# Set output fields replacing length with end fields
@@ -116,10 +116,11 @@ def convertToAIRR(db_file, format=default_format,
# Open output writer
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='airr', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=AIRRSchema.out_type)
+ out_name=out_args['out_name'], out_type=AIRRSchema.out_type,
+ gzip_output=out_args.get('gzip_output'))
pass_writer = AIRRWriter(pass_handle, fields=out_fields)
# Count records
@@ -169,7 +170,7 @@ def convertToChangeo(db_file, out_file=None, out_args=default_out_args):
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = AIRRReader(db_handle)
# Set output fields replacing length with end fields
@@ -184,10 +185,11 @@ def convertToChangeo(db_file, out_file=None, out_args=default_out_args):
# Open output writer
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='changeo', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=ChangeoSchema.out_type)
+ out_name=out_args['out_name'], out_type=ChangeoSchema.out_type,
+ gzip_output=out_args.get('gzip_output'))
pass_writer = ChangeoWriter(pass_handle, fields=out_fields)
# Count records
@@ -252,16 +254,17 @@ def convertToBaseline(db_file, id_field=default_id_field, seq_field=default_seq_
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
result_count = countDbFile(db_file)
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='sequences', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type='clip')
+ out_name=out_args['out_name'], out_type='clip',
+ gzip_output=out_args.get('gzip_output'))
# Iterate over records
start_time = time()
rec_count, germ_count, pass_count, fail_count = 0, 0, 0, 0
@@ -347,16 +350,17 @@ def convertToFasta(db_file, id_field=default_id_field, seq_field=default_seq_fie
# Open input
out_type = 'fasta'
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
result_count = countDbFile(db_file)
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='sequences', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_type)
+ out_name=out_args['out_name'], out_type=out_type,
+ gzip_output=out_args.get('gzip_output'))
# Iterate over records
start_time = time()
@@ -693,7 +697,7 @@ def convertToGenbank(db_file, inference=None, db_xref=None, molecule=default_mol
printError('Invalid format %s.' % format)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = reader(db_handle)
# Check for required columns
=====================================
bin/CreateGermlines.py
=====================================
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""
-Reconstructs germline sequences from alignment data
+Reconstructs germline sequences from nucleotide alignment data
+
+Note: Amino acid sequences are not currently supported.
"""
# Info
@@ -24,7 +26,7 @@ from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgPars
setDefaultFields
from changeo.Gene import buildGermline, buildClonalGermline
from changeo.IO import countDbFile, getDbFields, getFormatOperators, getOutputHandle, readGermlines, \
- checkFields
+ gzipOutputName, openFile, checkFields
# Defaults
default_germ_types = ['dmask']
@@ -40,7 +42,7 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
Arguments:
db_file : input tab-delimited database file.
references : folders and/or files containing germline repertoire data in FASTA format.
- seq_field : field in which to look for sequence.
+ seq_field : field in which to look for sequence. Must be a nucleotide sequence field.
v_field : field in which to look for V call.
d_field : field in which to look for D call.
j_field : field in which to look for J call.
@@ -53,6 +55,9 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
Returns:
dict: names of the 'pass' and 'fail' output files.
+
+ Note:
+ Amino acid sequences are not currently supported. The seq_field must contain nucleotide sequences.
"""
# Print parameter info
log = OrderedDict()
@@ -92,7 +97,7 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
# Get repertoire and open Db reader
reference_dict = readGermlines(references)
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = reader(db_handle)
# Check for required columns
@@ -115,6 +120,32 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
if f not in db_iter.fields:
printError('%s field does not exist in input database file.' % f)
+ # Check for amino acid sequences
+ aa_fields = ['sequence_aa', 'sequence_alignment_aa', 'SEQUENCE_AA', 'SEQUENCE_ALIGNMENT_AA']
+ if seq_field in aa_fields:
+ printError('Reconstruction of germlines from amino acid sequences is not currently supported. '
+ 'Please use a nucleotide sequence field (e.g., sequence_alignment or SEQUENCE_IMGT).')
+
+ # Also check actual sequence content to detect amino acids
+ # Sample first few sequences to detect amino acid-specific characters
+ aa_chars = set('EFIKLMPQWY') # Amino acid characters not valid in nucleotide sequences
+ sample_count = 0
+ max_samples = 10
+ for rec in db_iter:
+ seq = rec.getField(seq_field)
+ if seq and any(c in aa_chars for c in seq.upper().replace('.', '').replace('-', '')):
+ printError('Amino acid sequence detected in %s field. '
+ 'Reconstruction of germlines from amino acid sequences is not currently supported. '
+ 'Please use a nucleotide sequence field.' % seq_field)
+ sample_count += 1
+ if sample_count >= max_samples:
+ break
+
+ # Reset the file handle after sampling
+ db_handle.close()
+ db_handle = openFile(db_file, 'rt')
+ db_iter = reader(db_handle)
+
# Translate to Receptor attribute names
v_field = schema.toReceptor(v_field)
d_field = schema.toReceptor(d_field)
@@ -186,13 +217,14 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
except AttributeError:
# Create output file handle and writer
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file,
out_label='germ-pass',
out_dir=out_args['out_dir'],
out_name=out_args['out_name'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = writer(pass_handle, fields=out_fields)
for r in records:
r.setDict(annotations)
@@ -207,7 +239,8 @@ def createGermlines(db_file, references, seq_field=default_seq_field, v_field=de
out_label='germ-fail',
out_dir=out_args['out_dir'],
out_name=out_args['out_name'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
fail_writer = writer(fail_handle, fields=out_fields)
fail_writer.writeReceptor(records)
@@ -299,7 +332,8 @@ def getArgParser():
germline call used for the entire clone within the
germline_v_call, germline_d_call and germline_j_call fields.''')
group.add_argument('--sf', action='store', dest='seq_field', default=None,
- help='''Field containing the aligned sequence.
+ help='''Field containing the aligned sequence. Must be a nucleotide sequence field.
+ Amino acid sequences are not currently supported.
Defaults to sequence_alignment (airr) or SEQUENCE_IMGT (changeo).''')
group.add_argument('--vf', action='store', dest='v_field', default=None,
help='''Field containing the germline V segment call.
=====================================
bin/DefineClones.py
=====================================
@@ -28,7 +28,7 @@ from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgPars
setDefaultFields
from changeo.Distance import distance_models, calcDistances, formClusters
from changeo.IO import countDbFile, getDbFields, getFormatOperators, getOutputHandle, \
- AIRRWriter, checkFields
+ gzipOutputName, openFile, AIRRWriter, checkFields
from changeo.Multiprocessing import DbResult, feedDbQueue, processDbQueue
# Defaults
@@ -299,7 +299,7 @@ def distanceClones(result, seq_field=default_junction_field, model=default_dista
seq_map = {}
for rec in result.data_pass:
seq = rec.getField(seq_field)
- seq = re.sub('[\.-]', 'N', seq)
+ seq = re.sub(r'[\.-]', 'N', seq)
# Translate sequence for amino acid model
if model == 'aa':
# Check for valid translation
@@ -367,13 +367,14 @@ def collectQueue(alive, result_queue, collect_queue, db_file, fields,
# Wrapper for opening handles and writers
def _open(x, f, writer=writer, out_file=out_file):
if out_file is not None and x == 'pass':
- handle = open(out_file, 'w')
+ handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
handle = getOutputHandle(db_file,
out_label='clone-%s' % x,
out_dir=out_args['out_dir'],
out_name=out_args['out_name'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
return handle, writer(handle, fields=f)
# Open log file
=====================================
bin/MakeDb.py
=====================================
@@ -16,6 +16,8 @@ from collections import OrderedDict
from textwrap import dedent
from time import time
from Bio import SeqIO
+from concurrent.futures import ProcessPoolExecutor
+from concurrent.futures import as_completed
# Presto and changeo imports
from presto.Annotation import parseAnnotation
@@ -25,8 +27,8 @@ from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgPars
from changeo.Alignment import RegionDefinition, gapV
from changeo.Gene import buildGermline
from changeo.IO import countDbFile, extractIMGT, readGermlines, getFormatOperators, getOutputHandle, \
- AIRRWriter, ChangeoWriter, IgBLASTReader, IgBLASTReaderAA, IMGTReader, IHMMuneReader, \
- checkFields
+ gzipOutputName, openFile, AIRRWriter, ChangeoWriter, IgBLASTReader, IgBLASTReaderAA, \
+ IMGTReader, IHMMuneReader, checkFields
from changeo.Receptor import ChangeoSchema, AIRRSchema
# 10X Receptor attributes
@@ -121,9 +123,9 @@ def getIDforIMGT(seq_file, imgt_id_len=default_imgt_id_len):
id_key = rec.description
else: # truncate and replace characters
if imgt_id_len == 49: # 28 September 2021 (version 1.8.4)
- id_key = re.sub('\s|\t', '_', rec.description[:imgt_id_len])
+ id_key = re.sub(r'\s|\t', '_', rec.description[:imgt_id_len])
else: # older versions
- id_key = re.sub('\||\s|!|&|\*|<|>|\?', '_', rec.description[:imgt_id_len])
+ id_key = re.sub(r'\||\s|!|&|\*|<|>|\?', '_', rec.description[:imgt_id_len])
ids.update({id_key: rec.description})
return ids
@@ -233,13 +235,14 @@ def writeDb(records, fields, aligner_file, total_count, id_dict=None, annotation
# Wrapper for opening handles and writers
def _open(x, f, writer=writer, out_file=out_file):
if out_file is not None and x == 'pass':
- handle = open(out_file, 'w')
+ handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
handle = getOutputHandle(aligner_file,
out_label='db-%s' % x,
out_dir=out_args['out_dir'],
out_name=out_args['out_name'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
return handle, writer(handle, fields=f)
# Function to convert fasta header annotations to changeo columns
@@ -423,7 +426,7 @@ def writeDb(records, fields, aligner_file, total_count, id_dict=None, annotation
def parseIMGT(aligner_file, seq_file=None, repo=None, cellranger_file=None, validate='strict', asis_id=True,
extended=False, format=default_format, out_file=None, out_args=default_out_args,
- imgt_id_len=default_imgt_id_len):
+ imgt_id_len=default_imgt_id_len, nproc=None):
"""
Main for IMGT aligned sample sequences.
@@ -438,6 +441,7 @@ def parseIMGT(aligner_file, seq_file=None, repo=None, cellranger_file=None, vali
out_file (str): output file name. Automatically generated from the input file if None.
out_args (dict): common output argument dictionary from parseCommonArgs.
imgt_id_len (int): maximum character length of sequence identifiers reported by IMGT/HighV-QUEST.
+ nproc (int): number of processes to use for parallel processing.
Returns:
dict: names of the 'pass' and 'fail' output files.
@@ -448,6 +452,7 @@ def parseIMGT(aligner_file, seq_file=None, repo=None, cellranger_file=None, vali
log['COMMAND'] = 'imgt'
log['ALIGNER_FILE'] = aligner_file
log['SEQ_FILE'] = os.path.basename(seq_file) if seq_file else ''
+ log['NPROC'] = nproc
log['ASIS_ID'] = asis_id
log['VALIDATE'] = validate
log['EXTENDED'] = extended
@@ -518,7 +523,7 @@ def parseIMGT(aligner_file, seq_file=None, repo=None, cellranger_file=None, vali
def parseIgBLAST(aligner_file, seq_file, repo, amino_acid=False, cellranger_file=None, validate='strict',
asis_id=True, asis_calls=False, extended=False, regions='default', infer_junction=False,
- format='changeo', out_file=None, out_args=default_out_args):
+ format='changeo', out_file=None, out_args=default_out_args, nproc=None):
"""
Main for IgBLAST aligned sample sequences.
@@ -536,6 +541,7 @@ def parseIgBLAST(aligner_file, seq_file, repo, amino_acid=False, cellranger_file
format (str): output format. one of 'changeo' or 'airr'.
out_file (str): output file name. Automatically generated from the input file if None.
out_args (dict): common output argument dictionary from parseCommonArgs.
+ nproc (int): number of processes to use for parallel processing.
Returns:
dict: names of the 'pass' and 'fail' output files.
@@ -546,6 +552,7 @@ def parseIgBLAST(aligner_file, seq_file, repo, amino_acid=False, cellranger_file
log['COMMAND'] = 'igblast-aa' if amino_acid else 'igblast'
log['ALIGNER_FILE'] = os.path.basename(aligner_file)
log['SEQ_FILE'] = os.path.basename(seq_file)
+ log['NPROC'] = nproc
log['ASIS_ID'] = asis_id
log['ASIS_CALLS'] = asis_calls
log['VALIDATE'] = validate
@@ -600,7 +607,7 @@ def parseIgBLAST(aligner_file, seq_file, repo, amino_acid=False, cellranger_file
fields.extend(custom)
# Parse and write output
- with open(aligner_file, 'r') as f:
+ with openFile(aligner_file, 'r') as f:
parse_iter = parser(f, seq_dict, references, regions=regions, asis_calls=asis_calls, infer_junction=infer_junction)
germ_iter = (addGermline(x, references, amino_acid=amino_acid) for x in parse_iter)
output = writeDb(germ_iter, fields=fields, aligner_file=aligner_file, total_count=total_count,
@@ -611,7 +618,7 @@ def parseIgBLAST(aligner_file, seq_file, repo, amino_acid=False, cellranger_file
def parseIHMM(aligner_file, seq_file, repo, cellranger_file=None, validate='strict', asis_id=True,
- extended=False, format=default_format, out_file=None, out_args=default_out_args):
+ extended=False, format=default_format, out_file=None, out_args=default_out_args, nproc=None):
"""
Main for iHMMuneAlign aligned sample sequences.
@@ -625,6 +632,7 @@ def parseIHMM(aligner_file, seq_file, repo, cellranger_file=None, validate='stri
format (str): output format. One of 'changeo' or 'airr'.
out_file (str): output file name. Automatically generated from the input file if None.
out_args (dict): common output argument dictionary from parseCommonArgs.
+ nproc (int): number of processes to use for parallel processing.
Returns:
dict: names of the 'pass' and 'fail' output files.
@@ -635,6 +643,7 @@ def parseIHMM(aligner_file, seq_file, repo, cellranger_file=None, validate='stri
log['COMMAND'] = 'ihmm'
log['ALIGNER_FILE'] = os.path.basename(aligner_file)
log['SEQ_FILE'] = os.path.basename(seq_file)
+ log['NPROC'] = nproc
log['ASIS_ID'] = asis_id
log['VALIDATE'] = validate
log['EXTENDED'] = extended
@@ -679,7 +688,7 @@ def parseIHMM(aligner_file, seq_file, repo, cellranger_file=None, validate='stri
fields.extend(custom)
# Parse and write output
- with open(aligner_file, 'r') as f:
+ with openFile(aligner_file, 'r') as f:
parse_iter = IHMMuneReader(f, seq_dict, references)
germ_iter = (addGermline(x, references) for x in parse_iter)
output = writeDb(germ_iter, fields=fields, aligner_file=aligner_file, total_count=total_count,
@@ -691,7 +700,7 @@ def parseIHMM(aligner_file, seq_file, repo, cellranger_file=None, validate='stri
def numberAIRR(aligner_file, repo=None, format=default_format,
- out_file=None, out_args=default_out_args):
+ out_file=None, out_args=default_out_args, nproc=None):
"""
Inserts IMGT numbering into V fields
@@ -701,6 +710,7 @@ def numberAIRR(aligner_file, repo=None, format=default_format,
format (str): output format.
out_file (str): output file name. Automatically generated from the input file if None.
out_args (dict): common output argument dictionary from parseCommonArgs.
+ nproc (int): number of processes to use for parallel processing.
Returns:
str: output file name.
@@ -709,6 +719,7 @@ def numberAIRR(aligner_file, repo=None, format=default_format,
log['START'] = 'MakeDb'
log['COMMAND'] = 'number'
log['ALIGNER_FILE'] = os.path.basename(aligner_file)
+ log['NPROC'] = nproc
printLog(log)
# Define format operators
@@ -718,7 +729,7 @@ def numberAIRR(aligner_file, repo=None, format=default_format,
printError('Invalid format %s.' % format)
# Open input
- db_handle = open(aligner_file, 'rt')
+ db_handle = openFile(aligner_file, 'rt')
db_iter = reader(db_handle)
# Define log handle
@@ -743,15 +754,17 @@ def numberAIRR(aligner_file, repo=None, format=default_format,
# Open output writer
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(aligner_file, out_label='db-pass', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=schema.out_type)
+ out_name=out_args['out_name'], out_type=schema.out_type,
+ gzip_output=out_args.get('gzip_output'))
pass_writer = writer(pass_handle, fields=db_iter.fields)
if out_args['failed']:
fail_handle = getOutputHandle(aligner_file, out_label='db-fail', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=schema.out_type)
+ out_name=out_args['out_name'], out_type=schema.out_type,
+ gzip_output=out_args.get('gzip_output'))
fail_writer = writer(fail_handle, fields=db_iter.fields)
# Count records
@@ -859,7 +872,7 @@ def getArgParser():
subparsers.required = True
# Parent parser
- parser_parent = getCommonArgParser(db_in=False)
+ parser_parent = getCommonArgParser(db_in=False, multiproc=True)
# igblastn output parser
parser_igblast = subparsers.add_parser('igblast', parents=[parser_parent],
@@ -952,6 +965,10 @@ def getArgParser():
group_igblast_aa.add_argument('--regions', action='store', dest='regions',
choices=('default', 'rhesus-igl'), default='default',
help='''IMGT CDR and FWR boundary definition to use.''')
+ # Add mutually exclusive group for validation, including --partial
+ group_igblast_aa_validate = group_igblast_aa.add_mutually_exclusive_group(required=False)
+ group_igblast_aa_validate.add_argument('--partial', action='store_const', const='partial', dest='validate',
+ help='''If specified, include incomplete V(D)J alignments in the pass file instead of the fail file. An incomplete alignment is defined as a record that is missing a V gene assignment, J gene assignment, junction region, or productivity call.''')
parser_igblast_aa.set_defaults(func=parseIgBLAST, amino_acid=True, validate='strict')
@@ -1090,6 +1107,10 @@ if __name__ == "__main__":
if 'seq_files' in args_dict and not args_dict['seq_files']:
args_dict['asis_id'] = True
+ # Extract nproc before deleting, use parser default if not specified
+ nproc_default = parser.get_default('nproc')
+ nproc = args_dict.get('nproc', nproc_default)
+
# Delete
if 'aligner_files' in args_dict: del args_dict['aligner_files']
if 'seq_files' in args_dict: del args_dict['seq_files']
@@ -1097,16 +1118,30 @@ if __name__ == "__main__":
if 'out_files' in args_dict: del args_dict['out_files']
if 'command' in args_dict: del args_dict['command']
if 'func' in args_dict: del args_dict['func']
+ if 'nproc' in args_dict: del args_dict['nproc']
+
+
+ ds=[]
- # Call main
+ # initialize a dict for each execution, and add to list
for i, f in enumerate(args.__dict__['aligner_files']):
- args_dict['aligner_file'] = f
- args_dict['out_file'] = args.__dict__['out_files'][i] \
+ tmp_dict=args_dict.copy()
+ tmp_dict['aligner_file'] = f
+ tmp_dict['nproc'] = nproc
+ tmp_dict['out_file'] = args.__dict__['out_files'][i] \
if args.__dict__['out_files'] else None
if 'seq_files' in args.__dict__:
- args_dict['seq_file'] = args.__dict__['seq_files'][i] \
+ tmp_dict['seq_file'] = args.__dict__['seq_files'][i] \
if args.__dict__['seq_files'] else None
if 'cellranger_files' in args.__dict__:
- args_dict['cellranger_file'] = args.__dict__['cellranger_files'][i] \
+ tmp_dict['cellranger_file'] = args.__dict__['cellranger_files'][i] \
if args.__dict__['cellranger_files'] else None
- args.func(**args_dict)
+ ds.append(tmp_dict)
+
+ # now do the loop in parallel
+ with ProcessPoolExecutor(max_workers=nproc) as executor:
+ futures = [executor.submit(args.func, **d) for d in ds]
+ for future in as_completed(futures):
+ result = future.result()
+ # original code threw away return values, so I am too...
+
=====================================
bin/ParseDb.py
=====================================
@@ -18,10 +18,11 @@ from textwrap import dedent
from time import time
# Presto and changeo imports
-from presto.IO import printLog, printProgress, printMessage
+from presto.IO import printLog, printProgress, printMessage, printWarning
from changeo.Defaults import default_csv_size, default_out_args
from changeo.Commandline import CommonHelpFormatter, checkArgs, getCommonArgParser, parseCommonArgs
-from changeo.IO import checkFields, countDbFile, getOutputHandle, splitName, TSVReader, TSVWriter
+from changeo.IO import checkFields, countDbFile, getOutputHandle, gzipOutputName, openFile, \
+ splitName, TSVReader, TSVWriter
# System settings
csv.field_size_limit(default_csv_size)
@@ -54,7 +55,7 @@ def splitDbFile(db_file, field, num_split=None, out_args=default_out_args):
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
out_fields = db_iter.fields
__, __, out_args['out_type'] = splitName(db_file)
@@ -73,13 +74,13 @@ def splitDbFile(db_file, field, num_split=None, out_args=default_out_args):
# Sort records into files based on textual field
if num_split is None:
# Create set of unique field tags
- with open(db_file, 'rt') as tmp_handle:
+ with openFile(db_file, 'rt') as tmp_handle:
tmp_iter = TSVReader(tmp_handle)
tag_list = list(set([row[field] for row in tmp_iter]))
# Forbidden characters in filename and replacements
- no_good = {'\/':'f','\\':'b','?':'q','\%':'p','*':'s',':':'c',
- '\|':'pi','\"':'dq','\'':'sq','<':'gt','>':'lt',' ':'_'}
+ no_good = {r'\/':'f',r'\\':'b',r'?':'q',r'\%':'p',r'*':'s',r':':'c',
+ r'\|':'pi',r'\"':'dq',r'\'':'sq',r'<':'gt',r'>':'lt',r' ':'_'}
# Replace forbidden characters in tag_list
tag_dict = {}
for tag in tag_list:
@@ -92,7 +93,8 @@ def splitDbFile(db_file, field, num_split=None, out_args=default_out_args):
out_label='%s-%s' % (field, label),
out_name=out_args['out_name'],
out_dir=out_args['out_dir'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
for tag, label in tag_dict.items()}
# Create Db writer instances
@@ -116,12 +118,14 @@ def splitDbFile(db_file, field, num_split=None, out_args=default_out_args):
out_label='under-%.1f' % num_split,
out_name=out_args['out_name'],
out_dir=out_args['out_dir'],
- out_type=out_args['out_type']),
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output')),
'atleast': getOutputHandle(db_file,
out_label='atleast-%.1f' % num_split,
out_name=out_args['out_name'],
out_dir=out_args['out_dir'],
- out_type=out_args['out_type'])}
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))}
# Create Db writer instances
writers_dict = {'under': TSVWriter(handles_dict['under'], fields=out_fields),
@@ -178,7 +182,7 @@ def addDbFile(db_file, fields, values, out_file=None, out_args=default_out_args)
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
__, __, out_args['out_type'] = splitName(db_file)
@@ -188,10 +192,11 @@ def addDbFile(db_file, fields, values, out_file=None, out_args=default_out_args)
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-add', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -247,7 +252,7 @@ def indexDbFile(db_file, field=default_index_field, out_file=None, out_args=defa
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
__, __, out_args['out_type'] = splitName(db_file)
@@ -257,10 +262,11 @@ def indexDbFile(db_file, field=default_index_field, out_file=None, out_args=defa
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-index', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -314,7 +320,7 @@ def dropDbFile(db_file, fields, out_file=None, out_args=default_out_args):
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
__, __, out_args['out_type'] = splitName(db_file)
@@ -329,10 +335,11 @@ def dropDbFile(db_file, fields, out_file=None, out_args=default_out_args):
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-drop', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -400,7 +407,7 @@ def deleteDbFile(db_file, fields, values, logic='any', regex=False,
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
out_fields = db_iter.fields
__, __, out_args['out_type'] = splitName(db_file)
@@ -413,10 +420,11 @@ def deleteDbFile(db_file, fields, values, logic='any', regex=False,
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-delete', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -479,7 +487,7 @@ def renameDbFile(db_file, fields, names, out_file=None, out_args=default_out_arg
printLog(log)
# Open file handles
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
__, __, out_args['out_type'] = splitName(db_file)
@@ -497,10 +505,11 @@ def renameDbFile(db_file, fields, names, out_file=None, out_args=default_out_arg
# Open writer
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-rename', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -575,7 +584,7 @@ def selectDbFile(db_file, fields, values, logic='any', regex=False,
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
out_fields = db_iter.fields
__, __, out_args['out_type'] = splitName(db_file)
@@ -588,10 +597,11 @@ def selectDbFile(db_file, fields, values, logic='any', regex=False,
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-select', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -659,7 +669,7 @@ def sortDbFile(db_file, field, numeric=False, descend=False,
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
out_fields = db_iter.fields
__, __, out_args['out_type'] = splitName(db_file)
@@ -672,10 +682,11 @@ def sortDbFile(db_file, field, numeric=False, descend=False,
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-sort', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Store all records in a dictionary
@@ -741,7 +752,7 @@ def updateDbFile(db_file, field, values, updates, out_file=None, out_args=defaul
printLog(log)
# Open input
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = TSVReader(db_handle)
out_fields = db_iter.fields
__, __, out_args['out_type'] = splitName(db_file)
@@ -754,10 +765,11 @@ def updateDbFile(db_file, field, values, updates, out_file=None, out_args=defaul
# Open output
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
pass_handle = getOutputHandle(db_file, out_label='parse-update', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Count records
@@ -796,28 +808,40 @@ def updateDbFile(db_file, field, values, updates, out_file=None, out_args=defaul
return pass_handle.name
-def mergeDbFiles(db_files, drop=False, out_file=None, out_args=default_out_args):
+def mergeDbFiles(db_files, drop=False, field=None, values=None,
+ out_file=None, out_args=default_out_args):
"""
Updates field and value pairs to a database file
Arguments:
db_files : list of database file names.
drop : if True drop columns not present in all files.
+ field : name of the field to add annotating the source file of each record;
+ if None do not add a sample identifier field.
+ values : list of sample identifiers, one per file in db_files, assigned to
+ field for each record; ignored if field is None.
out_file : output file name. Automatically generated from the input file if None.
out_args : common output argument dictionary from parseCommonArgs.
Returns:
str : output file name.
"""
+ if field is None and values is not None:
+ printWarning('Values (-u) were specified without a field (-f) to assign '
+ 'them to and will be ignored.')
+
log = OrderedDict()
log['START'] = 'ParseDb'
log['COMMAND'] = 'merge'
log['FILES'] = ','.join([os.path.basename(f) for f in db_files])
log['DROP'] = drop
+ if field is not None:
+ log['FIELD'] = field
+ log['VALUES'] = ','.join(values)
printLog(log)
# Open input
- db_handles = [open(f, 'rt') for f in db_files]
+ db_handles = [openFile(f, 'rt') for f in db_files]
db_iters = [TSVReader(x) for x in db_handles]
result_count = sum([countDbFile(f) for f in db_files])
@@ -829,25 +853,39 @@ def mergeDbFiles(db_files, drop=False, out_file=None, out_args=default_out_args)
field_set = set.union(*map(set, field_list))
field_order = OrderedDict([(f, None) for f in chain(*field_list)])
out_fields = [f for f in field_order if f in field_set]
+ if field is not None:
+ # Check against all input headers (field_order), not just the post-drop
+ # output header (out_fields), so a field present in only some input files
+ # still triggers the overwrite warning when --drop removes it from out_fields.
+ if field in field_order:
+ printWarning('Field %s already exists in one or more input files and its '
+ 'existing values will be overwritten.' % field)
+ if field not in out_fields:
+ out_fields.append(field)
# Open output file
if out_file is not None:
- pass_handle = open(out_file, 'w')
+ pass_handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
__, __, out_args['out_type'] = splitName(db_files[0])
pass_handle = getOutputHandle(db_files[0], out_label='parse-merge', out_dir=out_args['out_dir'],
- out_name=out_args['out_name'], out_type=out_args['out_type'])
+ out_name=out_args['out_name'], out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
pass_writer = TSVWriter(pass_handle, out_fields)
# Iterate over records
start_time = time()
rec_count = 0
- for db in db_iters:
+ for i, db in enumerate(db_iters):
for rec in db:
# Print progress for previous iteration
printProgress(rec_count, result_count, 0.05, start_time=start_time)
rec_count += 1
+ # Annotate sample identifier
+ if field is not None:
+ rec[field] = values[i]
+
# Write records
pass_writer.writeDict(rec)
@@ -1036,6 +1074,14 @@ def getArgParser():
help='''If specified, drop fields that do not exist in all input files.
Otherwise, include all columns in all files and fill missing data
with empty strings.''')
+ group_merge.add_argument('-f', action='store', dest='field', default=None,
+ help='''Name of the field to add to the output file annotating each
+ record with a sample identifier. If not specified, no sample
+ identifier field is added.''')
+ group_merge.add_argument('-u', nargs='+', action='store', dest='values', default=None,
+ help='''Sample identifiers to assign to the field specified by -f, one
+ per input file (-d) and in the same order. Ignored if -f is not
+ specified.''')
parser_merge.set_defaults(func=mergeDbFiles)
# Subparser to partition files by annotation values
@@ -1078,6 +1124,11 @@ if __name__ == '__main__':
parser.error('You must specify exactly one new name (-k) per field (-f)')
elif args.command == 'update' and len(args_dict['values']) != len(args_dict['updates']):
parser.error('You must specify exactly one value (-u) per replacement (-t)')
+ elif args.command == 'merge' and args_dict['field'] is not None:
+ if args_dict['values'] is None:
+ parser.error('You must specify sample identifiers (-u) when using -f')
+ elif len(args_dict['values']) != len(args_dict['db_files']):
+ parser.error('You must specify exactly one sample identifier (-u) per input file (-d)')
# Call parser function for each database file
if args.command == 'merge':
=====================================
changeo.egg-info/PKG-INFO
=====================================
@@ -1,9 +1,9 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
Name: changeo
-Version: 1.3.0
+Version: 1.3.5
Summary: A bioinformatics toolkit for processing high-throughput lymphocyte receptor sequencing data.
Home-page: http://changeo.readthedocs.io
-Download-URL: https://bitbucket.org/kleinstein/changeo/downloads
+Download-URL: https://github.com/immcantation/changeo/tags
Author: Namita Gupta, Jason Anthony Vander Heiden
Author-email: immcantation at googlegroups.com
License: GNU Affero General Public License 3 (AGPL-3)
@@ -13,9 +13,30 @@ Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3.4
+Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
License-File: LICENSE
+Requires-Dist: numpy>=1.23.2
+Requires-Dist: scipy>=1.9.3
+Requires-Dist: pandas>=1.5.0
+Requires-Dist: biopython>=1.81
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: setuptools>=65.5.0
+Requires-Dist: presto>=0.7.1
+Requires-Dist: airr>=1.3.1
+Requires-Dist: packaging>=21.3
+Requires-Dist: importlib-resources>=6.4.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary
.. image:: https://img.shields.io/pypi/dm/changeo
:target: https://pypi.org/project/changeo
@@ -28,7 +49,7 @@ Change-O - Repertoire clonal assignment toolkit
Change-O is a collection of tools for processing the output of V(D)J alignment
tools, assigning clonal clusters to immunoglobulin (Ig) sequences, and
reconstructing germline sequences.
-
+
Dramatic improvements in high-throughput sequencing technologies now enable
large-scale characterization of Ig repertoires, defined as the collection of
trans-membrane antigen-receptor proteins located on the surface of B cells and
@@ -37,3 +58,6 @@ Ig and TCR sequences following germline segment assignment. Change-O
handles output from IMGT/HighV-QUEST and IgBLAST, and provides a wide variety of
clustering methods for assigning clonal groups to Ig sequences. Record sorting,
grouping, and various database manipulation operations are also included.
+
+**IMPORTANT!**
+Active development of Change-O has moved to https://github.com/immcantation/changeo
=====================================
changeo.egg-info/SOURCES.txt
=====================================
@@ -3,6 +3,7 @@ LICENSE
MANIFEST.in
NEWS.rst
README.rst
+pyproject.toml
requirements.txt
setup.py
bin/AlignRecords.py
@@ -35,4 +36,9 @@ changeo/data/hh_s5f_dist.tsv
changeo/data/hs1f_compat_dist.tsv
changeo/data/m1n_compat_dist.tsv
changeo/data/mk_rs1nf_dist.tsv
-changeo/data/mk_rs5nf_dist.tsv
\ No newline at end of file
+changeo/data/mk_rs5nf_dist.tsv
+tests/test_DefineClones.py
+tests/test_Distance.py
+tests/test_IO.py
+tests/test_ParseDb.py
+tests/test_Receptor.py
\ No newline at end of file
=====================================
changeo.egg-info/requires.txt
=====================================
@@ -1,8 +1,10 @@
-numpy>=1.8
-scipy>=0.14
-pandas>=0.24
-biopython>=1.77
-PyYAML>=5.1
-setuptools>=2.0
-presto>=0.7.0
+numpy>=1.23.2
+scipy>=1.9.3
+pandas>=1.5.0
+biopython>=1.81
+PyYAML>=6.0
+setuptools>=65.5.0
+presto>=0.7.1
airr>=1.3.1
+packaging>=21.3
+importlib-resources>=6.4.0
=====================================
changeo/Applications.py
=====================================
@@ -8,7 +8,7 @@ __author__ = 'Jason Anthony Vander Heiden'
# Imports
import os
import re
-from pkg_resources import parse_version
+from packaging.version import parse as parse_version
from subprocess import check_output, STDOUT, CalledProcessError
# Presto and changeo imports
@@ -280,7 +280,7 @@ def getIgBLASTVersion(exec=default_igblastn_exec):
printError('Running command: %s\n%s' % (' '.join(cmd), e.output))
# Extract version number
- match = re.search('(?<=Package: igblast )(\d+\.\d+\.\d+)', stdout_str)
+ match = re.search(r'(?<=Package: igblast )(\d+\.\d+\.\d+)', stdout_str)
version = match.group(0)
return version
\ No newline at end of file
=====================================
changeo/Commandline.py
=====================================
@@ -16,6 +16,7 @@ from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter, \
# Changeo imports
from presto.IO import printWarning, printError
from changeo.Defaults import choices_format, default_format
+from changeo.IO import gzipOutputName
from changeo.Receptor import AIRRSchema, ChangeoSchema
@@ -45,7 +46,7 @@ class CommonHelpFormatter(RawDescriptionHelpFormatter, ArgumentDefaultsHelpForma
def getCommonArgParser(db_in=True, db_out=True, out_file=True, failed=True, log=True,
- format=True, multiproc=False, add_help=True):
+ format=True, multiproc=False, add_help=True, gzip_output=True):
"""
Defines an ArgumentParser object with common pRESTO arguments
@@ -57,6 +58,11 @@ def getCommonArgParser(db_in=True, db_out=True, out_file=True, failed=True, log=
log (bool): if True include log arguments.
format (bool): input and output type arguments.
multiproc (bool): if True include multiprocessing arguments.
+ gzip_output (bool): if True include the --gzip-output argument. Should be
+ False for tools whose output is not produced through
+ getOutputHandle/openFile (e.g. wrapped external
+ executables), since the flag would otherwise be
+ silently ignored.
Returns:
argparse.ArgumentParser : an argument parser.
@@ -96,6 +102,11 @@ def getCommonArgParser(db_in=True, db_out=True, out_file=True, failed=True, log=
help='''Changes the prefix of the successfully processed output file
to the string specified. May not be specified with multiple
input files.''')
+ if gzip_output:
+ group.add_argument('--gzip-output', action='store_true', dest='gzip_output', default=None,
+ help='''Specify to force gzip compressed output files. Output is
+ automatically gzip compressed when the input file(s) are
+ themselves gzip compressed (.gz), even without this argument.''')
# Log arguments
if log:
@@ -119,11 +130,35 @@ def getCommonArgParser(db_in=True, db_out=True, out_file=True, failed=True, log=
if multiproc:
group.add_argument('--nproc', action='store', dest='nproc', type=int, default=mp.cpu_count(),
help='''The number of simultaneous computational processes to execute
- (CPU cores to utilized).''')
+ (CPU cores to utilize).''')
return parser
+def _samePath(path_a, path_b):
+ """
+ Determines whether two paths refer to the same file
+
+ Arguments:
+ path_a (str): first file path.
+ path_b (str): second file path.
+
+ Returns:
+ bool: True if the paths resolve to the same file. Uses os.path.samefile
+ (which follows symlinks and normalizes case/aliasing per the
+ filesystem) when both paths exist; otherwise falls back to
+ comparing normalized absolute paths, since one of the paths
+ (typically the output) may not exist yet.
+ """
+ if os.path.exists(path_a) and os.path.exists(path_b):
+ try:
+ return os.path.samefile(path_a, path_b)
+ except OSError:
+ pass
+
+ return os.path.normpath(os.path.abspath(path_a)) == os.path.normpath(os.path.abspath(path_b))
+
+
def parseCommonArgs(args, in_arg=None, in_types=None, in_list=False):
"""
Checks common arguments from getCommonArgParser and transforms output options to a dictionary
@@ -172,8 +207,10 @@ def parseCommonArgs(args, in_arg=None, in_types=None, in_list=False):
for f in args_dict['db_files']:
if not os.path.isfile(f):
printError('Database file %s does not exist.' % f)
- if os.path.splitext(f)[-1].lower() not in db_types:
- printError('Database file %s is not a supported type. Must be one: %s.' \
+ # Strip a trailing .gz extension before checking the underlying file type
+ f_check = f[:-3] if f.lower().endswith('.gz') else f
+ if os.path.splitext(f_check)[-1].lower() not in db_types:
+ printError('Database file %s is not a supported type. Must be one: %s (optionally gzip compressed with a .gz extension).' \
% (f, ', '.join(db_types)))
# Verify non-standard input files
@@ -198,17 +235,22 @@ def parseCommonArgs(args, in_arg=None, in_types=None, in_list=False):
if args_dict.get('out_files', None) is not None:
if len(args_dict['out_files']) != input_count:
printError('The -o argument requires one output file name per input file.')
- for f in args_dict['out_files']:
- if f in input_files:
+ # Resolve the .gz suffix that gzipOutputName will add at write time, so the
+ # collision and overwrite checks below see the file name that will actually be opened.
+ resolved_out_files = [gzipOutputName(f, args_dict.get('gzip_output', None)) \
+ for f in args_dict['out_files']]
+ for f in resolved_out_files:
+ if any(_samePath(f, i) for i in input_files):
printError('Output files and input files cannot have the same names.')
- for f in args_dict['out_files']:
+ for f in resolved_out_files:
if os.path.isfile(f):
printWarning('Output file %s already exists and will be overwritten.' % f)
if args_dict.get('out_file', None) is not None:
- if args_dict['out_file'] in input_files:
+ resolved_out_file = gzipOutputName(args_dict['out_file'], args_dict.get('gzip_output', None))
+ if any(_samePath(resolved_out_file, i) for i in input_files):
printError('Output files and input files cannot have the same names.')
- if os.path.isfile(args_dict['out_file']):
- printWarning('Output file %s already exists and will be overwritten.' % args_dict['out_file'])
+ if os.path.isfile(resolved_out_file):
+ printWarning('Output file %s already exists and will be overwritten.' % resolved_out_file)
# Exit if output names or log files are specified with multiple input files
if args_dict.get('out_name', None) is not None \
@@ -224,7 +266,7 @@ def parseCommonArgs(args, in_arg=None, in_types=None, in_list=False):
printError('Path %s exists but it is not a directory.' % args_dict['out_dir'])
# Redefine common output options as out_args dictionary
- out_args = ['log_file', 'out_dir', 'out_name', 'out_type', 'failed']
+ out_args = ['log_file', 'out_dir', 'out_name', 'out_type', 'failed', 'gzip_output']
args_dict['out_args'] = {k:args_dict.setdefault(k, None) for k in out_args}
for k in out_args: del args_dict[k]
=====================================
changeo/Distance.py
=====================================
@@ -9,7 +9,7 @@ __author__ = 'Jason Anthony Vander Heiden, Namita Gupta'
import numpy as np
import pandas as pd
from itertools import combinations, product, zip_longest
-from pkg_resources import resource_stream
+from importlib_resources import open_text
from scipy.cluster.hierarchy import fcluster, linkage
from scipy.spatial.distance import squareform
@@ -220,22 +220,22 @@ aa_model = getAADistMatrix(mask_dist=0, gap_dist=0)
ham_model = getDNADistMatrix(mask_dist=0, gap_dist=0)
# Load model data
-with resource_stream(__name__, 'data/hh_s1f_dist.tsv') as f:
+with open_text('changeo', 'data/hh_s1f_dist.tsv') as f:
hh_s1f_model = pd.read_csv(f, sep='\t', index_col=0)
-with resource_stream(__name__, 'data/hh_s5f_dist.tsv') as f:
+with open_text('changeo', 'data/hh_s5f_dist.tsv') as f:
hh_s5f_model = pd.read_csv(f, sep='\t', index_col=0)
-with resource_stream(__name__, 'data/mk_rs1nf_dist.tsv') as f:
+with open_text('changeo', 'data/mk_rs1nf_dist.tsv') as f:
mk_rs1nf_model = pd.read_csv(f, sep='\t', index_col=0)
-with resource_stream(__name__, 'data/mk_rs5nf_dist.tsv') as f:
+with open_text('changeo', 'data/mk_rs5nf_dist.tsv') as f:
mk_rs5nf_model = pd.read_csv(f, sep='\t', index_col=0)
-with resource_stream(__name__, 'data/m1n_compat_dist.tsv') as f:
+with open_text('changeo', 'data/m1n_compat_dist.tsv') as f:
m1n_compat_model = pd.read_csv(f, sep='\t', index_col=0)
-with resource_stream(__name__, 'data/hs1f_compat_dist.tsv') as f:
+with open_text('changeo', 'data/hs1f_compat_dist.tsv') as f:
hs1f_compat_model = pd.read_csv(f, sep='\t', index_col=0)
distance_models = {'ham': ham_model,
=====================================
changeo/Gene.py
=====================================
@@ -18,6 +18,7 @@ allele_regex = re.compile(r'((IG[HLK]|TR[ABGD])([VDJ][A-R0-9]+[-/\w]*[-\*][\.\w]
gene_regex = re.compile(r'((IG[HLK]|TR[ABGD])([VDJ][A-R0-9]+[-/\w]*))')
family_regex = re.compile(r'((IG[HLK]|TR[ABGD])([VDJ][A-R0-9]+))')
locus_regex = re.compile(r'(IG[HLK]|TR[ABGD])')
+multi_locus_tra_trd = re.compile(r'TRAV\d+(?:-\d+)?/DV\d+')
v_allele_regex = re.compile(r'((IG[HLK]|TR[ABGD])V[A-R0-9]+[-/\w]*[-\*][\.\w]+)')
d_allele_regex = re.compile(r'((IG[HLK]|TR[ABGD])D[A-R0-9]+[-/\w]*[-\*][\.\w]+)')
@@ -115,6 +116,16 @@ def getLocus(gene, action='first'):
str: String of the first locus call when action is 'first'.
tuple: Tuple of locus calls for 'set' or 'list' actions.
"""
+ # Handle None input
+ if gene is None:
+ return None
+
+ if multi_locus_tra_trd.search(gene):
+ dual = ['TRA', 'TRD']
+ if action == 'first':
+ return 'TRA/TRD'
+ elif action in ('set', 'list'):
+ return tuple(dual)
return parseGeneCall(gene, locus_regex, action=action)
=====================================
changeo/IO.py
=====================================
@@ -6,6 +6,7 @@ __author__ = 'Namita Gupta, Jason Anthony Vander Heiden'
# Imports
import csv
+import gzip
import os
import re
import tarfile
@@ -29,6 +30,40 @@ from changeo.Alignment import decodeBTOP, encodeCIGAR, padAlignment, gapV, infer
csv.field_size_limit(default_csv_size)
+def openFile(filename, mode='r'):
+ """
+ Opens a file with automatic gzip support based on the .gz extension
+
+ Arguments:
+ filename (str): path to the file.
+ mode (str): file opening mode ('r' or 'w'; a trailing 't'/'b' is ignored).
+
+ Returns:
+ file : file handle opened in text mode.
+ """
+ if filename.lower().endswith('.gz'):
+ return gzip.open(filename, mode[0] + 't')
+ else:
+ return open(filename, mode)
+
+
+def gzipOutputName(file, gzip_output):
+ """
+ Appends a .gz extension to a file name if compressed output is requested
+
+ Arguments:
+ file (str): file name.
+ gzip_output (bool): if True force a .gz extension onto file.
+
+ Returns:
+ str: file name, with a .gz extension added if needed.
+ """
+ if gzip_output and not file.lower().endswith('.gz'):
+ file += '.gz'
+
+ return file
+
+
class TSVReader:
"""
Simple csv.DictReader wrapper to read format agnostic TSV files.
@@ -526,8 +561,8 @@ class IMGTReader:
Returns:
dict : database entries for gene calls.
"""
- clean_regex = re.compile('(,)|(\(see comment\))')
- delim_regex = re.compile('\sor\s')
+ clean_regex = re.compile(r'(,)|(\(see comment\))')
+ delim_regex = re.compile(r'\sor\s')
# Gene calls
v_str = summary['V-GENE and allele']
@@ -953,7 +988,7 @@ class IgBLASTReader:
# Extract column names from comments
f = next((x for x in chunk if x.startswith('# V-(D)-J rearrangement summary')))
- f = re.search('summary for query sequence \((.+)\)\.', f).group(1)
+ f = re.search(r'summary for query sequence \((.+)\)\.', f).group(1)
columns = [summary_map[x.strip()] for x in f.split(',')]
# Extract first row as a list
@@ -987,7 +1022,7 @@ class IgBLASTReader:
# Extract column names from comments
f = next((x for x in chunk if x.startswith('# Sub-region sequence details')))
- f = re.search('sequence details \((.+)\)', f).group(1)
+ f = re.search(r'sequence details \((.+)\)', f).group(1)
columns = [cdr3_map[x.strip()] for x in f.split(',')]
# Extract first CDR3 as a list and remove the CDR3 label
@@ -1608,13 +1643,37 @@ class IgBLASTReaderAA(IgBLASTReader):
result = {}
seq_vdj = db['sequence_aa_vdj']
seq_trim = db['sequence_aa_trim']
- v_hit = next(x for x in hits if x['segment'] == 'V')
-
+
+ # Get all V hits
+ v_hits = [x for x in hits if x['segment'] == 'V']
+
+ # Find the top score
+ top_score = max(float(hit['bit score']) for hit in v_hits)
+
+ # Get all hits with the top score
+ top_v_hits = [hit for hit in v_hits if float(hit['bit score']) == top_score]
+
+ # Sort top-scoring hits by subject id and
+ # use the first hit for positional information and sequence building
+ top_v_hits = sorted(top_v_hits, key=lambda hit: hit['subject id'])
+ v_hit = top_v_hits[0]
+
# Alignment positions
result.update(self._parseVHitPos(v_hit))
- # Assign V gene and update VDJ sequence with and without removing insertions
- result['v_call'] = v_hit['subject id']
+ # Assign V gene calls from all top-scoring hits (comma-separated) and update VDJ sequence with and without removing insertions
+ if not self.asis_calls:
+ # Parse gene calls and join multiple alleles with commas (consistent with nucleotide parser)
+ v_calls = []
+ for hit in top_v_hits:
+ v_call = getVAllele(hit['subject id'], action='list')
+ if v_call:
+ v_calls.extend(v_call)
+ result['v_call'] = ','.join(set(v_calls)) if v_calls else None
+ else:
+ # Join raw subject IDs with commas
+ result['v_call'] = ','.join(set(hit['subject id'] for hit in top_v_hits))
+
result['sequence_aa_vdj'] = self._appendSeq(seq_vdj, v_hit, 0, trim=False)
result['sequence_aa_trim'] = self._appendSeq(seq_trim, v_hit, 0, trim=True)
@@ -2296,7 +2355,7 @@ def countDbFile(file):
"""
# Count records and check file
try:
- with open(file, 'rt') as db_handle:
+ with openFile(file, 'rt') as db_handle:
db_records = csv.reader(db_handle, dialect='excel-tab')
for i, __ in enumerate(db_records): pass
db_count = i
@@ -2324,7 +2383,7 @@ def getDbFields(file, add=None, exclude=None, reader=TSVReader):
list : list of field names
"""
try:
- with open(file, 'rt') as handle:
+ with openFile(file, 'rt') as handle:
fields = reader(handle).fields
except IOError:
printError('File %s cannot be read.' % file)
@@ -2384,16 +2443,22 @@ def splitName(file):
file (str): file name.
Returns:
- tuple : tuple of the file directory, basename and extension.
+ tuple : tuple of the file directory, basename and extension. A trailing
+ .gz extension is stripped from file before the extension is
+ determined, so a gzip compressed file reports the extension of
+ its underlying (uncompressed) type.
"""
directory, filename = os.path.split(file)
+ if filename.lower().endswith('.gz'):
+ filename = filename[:-3]
basename, extension = os.path.splitext(filename)
extension = extension.lower().lstrip('.')
return directory, basename, extension
-def getOutputName(file, out_label=None, out_dir=None, out_name=None, out_type=None):
+def getOutputName(file, out_label=None, out_dir=None, out_name=None, out_type=None,
+ gzip_output=None):
"""
Creates and output filename from an existing filename
@@ -2407,11 +2472,15 @@ def getOutputName(file, out_label=None, out_dir=None, out_name=None, out_type=No
if None use directory of input file
out_name : the short filename to use for the output file;
if None use input file short name.
+ gzip_output : if True force a .gz extension onto the output file name;
+ if None add a .gz extension only when file is itself
+ gzip compressed (preserve compression by default).
Returns:
str: file name.
"""
# Get filename components
+ was_gzipped = os.path.split(file)[1].lower().endswith('.gz')
directory, basename, extension = splitName(file)
# Define output directory
@@ -2431,11 +2500,16 @@ def getOutputName(file, out_label=None, out_dir=None, out_name=None, out_type=No
else:
out_file = os.path.join(out_dir, '%s_%s.%s' % (out_name, out_label, out_type))
+ # Preserve or force gzip compression of the output file name
+ if gzip_output or (gzip_output is None and was_gzipped):
+ out_file += '.gz'
+
# Return file name
return out_file
-def getOutputHandle(file, out_label=None, out_dir=None, out_name=None, out_type=None):
+def getOutputHandle(file, out_label=None, out_dir=None, out_name=None, out_type=None,
+ gzip_output=None):
"""
Opens an output file handle
@@ -2449,16 +2523,20 @@ def getOutputHandle(file, out_label=None, out_dir=None, out_name=None, out_type=
if None use directory of input file
out_name : the short filename to use for the output file;
if None use input file short name.
+ gzip_output : if True force a .gz extension onto the output file name
+ and open it as gzip compressed; if None preserve
+ compression when file is itself gzip compressed.
Returns:
file : File handle
"""
out_file = getOutputName(file, out_label=out_label, out_dir=out_dir,
- out_name=out_name, out_type=out_type)
+ out_name=out_name, out_type=out_type,
+ gzip_output=gzip_output)
# Open and return handle
try:
- return open(out_file, mode='w')
+ return openFile(out_file, mode='w')
except:
printError('File %s cannot be opened.' % out_file)
=====================================
changeo/Multiprocessing.py
=====================================
@@ -14,7 +14,7 @@ from time import time
# Presto and changeo imports
from presto.IO import printProgress, printLog, printError, printWarning
from changeo.Defaults import default_out_args
-from changeo.IO import countDbFile, getOutputHandle, AIRRReader, AIRRWriter
+from changeo.IO import countDbFile, getOutputHandle, gzipOutputName, openFile, AIRRReader, AIRRWriter
from changeo.Receptor import Receptor
@@ -113,7 +113,7 @@ def feedDbQueue(alive, data_queue, db_file, reader=AIRRReader, group_func=None,
# Open input file and perform grouping
try:
# Iterate over records and assign groups
- db_handle = open(db_file, 'rt')
+ db_handle = openFile(db_file, 'rt')
db_iter = reader(db_handle)
if group_func is not None:
# import cProfile
@@ -224,13 +224,14 @@ def collectDbQueue(alive, result_queue, collect_queue, db_file, label, fields,
# Wrapper for opening handles and writers
def _open(x, f, writer=writer, label=label, out_file=out_file):
if out_file is not None and x == 'pass':
- handle = open(out_file, 'w')
+ handle = openFile(gzipOutputName(out_file, out_args.get('gzip_output')), 'w')
else:
handle = getOutputHandle(db_file,
out_label='%s-%s' % (label, x),
out_dir=out_args['out_dir'],
out_name=out_args['out_name'],
- out_type=out_args['out_type'])
+ out_type=out_args['out_type'],
+ gzip_output=out_args.get('gzip_output'))
return handle, writer(handle, fields=f)
try:
=====================================
changeo/Version.py
=====================================
@@ -3,7 +3,7 @@ Version and authorship information
"""
__author__ = 'Namita Gupta, Jason Anthony Vander Heiden'
-__copyright__ = 'Copyright 2021 Kleinstein Lab, Yale University. All rights reserved.'
+__copyright__ = 'Copyright 2026 Kleinstein Lab, Yale University. All rights reserved.'
__license__ = 'GNU Affero General Public License 3 (AGPL-3)'
-__version__ = '1.3.0'
-__date__ = '2022.12.11'
+__version__ = '1.3.5'
+__date__ = '2026.09.02'
=====================================
pyproject.toml
=====================================
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=40.8.0", "wheel"]
+build-backend = "setuptools.build_meta:__legacy__"
=====================================
requirements.txt
=====================================
@@ -1,8 +1,10 @@
-numpy>=1.8
-scipy>=0.14
-pandas>=0.24
-biopython>=1.77
-PyYAML>=5.1
-setuptools>=2.0
-presto>=0.7.0
+numpy>=1.23.2
+scipy>=1.9.3
+pandas>=1.5.0
+biopython>=1.81
+PyYAML>=6.0
+setuptools>=65.5.0
+presto>=0.7.1
airr>=1.3.1
+packaging>=21.3
+importlib-resources>=6.4.0
=====================================
setup.py
=====================================
@@ -7,8 +7,8 @@ import os
import sys
# Check setup requirements
-if sys.version_info < (3,4,0):
- sys.exit('At least Python 3.4.0 is required.\n')
+if sys.version_info < (3,10,0):
+ sys.exit('At least Python 3.10.0 is required.\n')
try:
from setuptools import setup
@@ -63,7 +63,7 @@ setup(name='changeo',
zip_safe=False,
license=__license__,
url='http://changeo.readthedocs.io',
- download_url='https://bitbucket.org/kleinstein/changeo/downloads',
+ download_url='https://github.com/immcantation/changeo/tags',
keywords=['bioinformatics', 'sequencing', 'immunology', 'adaptive immunity',
'immunoglobulin', 'AIRR-seq', 'Rep-Seq',
'B cell repertoire analysis', 'adaptive immune receptor repertoires'],
@@ -77,5 +77,5 @@ setup(name='changeo',
'Intended Audience :: Science/Research',
'Natural Language :: English',
'Operating System :: OS Independent',
- 'Programming Language :: Python :: 3.4',
+ 'Programming Language :: Python :: 3.10',
'Topic :: Scientific/Engineering :: Bio-Informatics'])
=====================================
tests/test_DefineClones.py
=====================================
@@ -0,0 +1,253 @@
+"""
+Unit tests for DefineClones
+"""
+# Info
+__author__ = 'Jason Anthony Vander Heiden'
+from changeo import __version__, __date__
+
+# Imports
+import os
+import sys
+import time
+import unittest
+from copy import deepcopy
+from itertools import chain
+
+# Presto and changeo imports
+from changeo.Receptor import Receptor
+from changeo.Multiprocessing import DbResult
+
+# Paths
+test_path = os.path.dirname(os.path.realpath(__file__))
+data_path = os.path.join(test_path, 'data')
+
+# Import script
+sys.path.append(os.path.join(test_path, os.pardir, 'bin'))
+import DefineClones
+
+
+class Test_DefineClones(unittest.TestCase):
+ def setUp(self):
+ print('-> %s()' % self._testMethodName)
+
+ self.mg_ig_file = os.path.join(data_path, 'AB0RF_MG67U_functional-heavy_parse-select_filter-collapse.tab')
+
+ # Define list of preclone properties
+ group_list = [{'V_CALL': 'IGHV1-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GA',
+ 'FIELD2': '1'},
+ {'V_CALL': 'IGHV2-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GA',
+ 'FIELD2': '1'},
+ {'V_CALL': 'IGHV3-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GA',
+ 'FIELD2': '2'},
+ {'V_CALL': 'IGHV1-1*01, IGHV2-1*01, IGHV3-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GA',
+ 'FIELD2': '2'},
+ {'V_CALL': 'IGHV4-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GB',
+ 'FIELD2': '1'},
+ {'V_CALL': 'IGHV2-1*01, IGHV4-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GB',
+ 'FIELD2': '2'},
+ {'V_CALL': 'IGHV5-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': 'GB',
+ 'FIELD2': None},
+ {'V_CALL': 'IGHV5-1*01, IGHV6-1*01',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*01',
+ 'JUNCTION_LENGTH': 48,
+ 'FIELD1': '',
+ 'FIELD2': '1'}]
+
+ # Define unique sequences
+ seq_list = [{'SEQUENCE_ID': 'A1',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG',
+ 'JUNCTION': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG'},
+ {'SEQUENCE_ID': 'A2',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG',
+ 'JUNCTION': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG'},
+ {'SEQUENCE_ID': 'A3',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG',
+ 'JUNCTION': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGTCTGG'},
+ {'SEQUENCE_ID': 'A4',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGNNNNN',
+ 'JUNCTION': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGNNNNN'},
+ {'SEQUENCE_ID': 'B1',
+ 'SEQUENCE_INPUT': 'TGTGCAAGATATAGCAGCAGCTACTACTACTACGGTATGGACGTCTGG',
+ 'JUNCTION': 'TGTGCAAGATATAGCAGCAGCTACTACTACTACGGTATGGACGTCTGG'},
+ {'SEQUENCE_ID': 'B2',
+ 'SEQUENCE_INPUT': 'TGTGNAAGATNTAGCAGCAGCTACTACTACTACGGTATNGACGTCTGG',
+ 'JUNCTION': 'TGTGNAAGATNTAGCAGCAGCTACTACTACTACGGTATNGACGTCTGG'},
+ {'SEQUENCE_ID': 'B3',
+ 'SEQUENCE_INPUT': 'TGTGCAAGATATAGCAGCAGCTACTACTACTACGGTATGGACGTCTGG',
+ 'JUNCTION': 'TGTGCAAGATATAGCAGCAGCTACTACTACTACGGTATGGACGTCTGG'},
+ {'SEQUENCE_ID': 'B4',
+ 'SEQUENCE_INPUT': 'TGTGNAAGATNTAGCAGCAGCTACTACTACTACGGTATNGACGTCTGG',
+ 'JUNCTION': 'TGTGNAAGATNTAGCAGCAGCTACTACTACTACGGTATNGACGTCTGG'}]
+
+ # Build preclone Receptor list with unambiguous gene calls
+ seq_copy = deepcopy(seq_list)
+ for x in seq_copy: x.update(deepcopy(group_list[1]))
+ self.unambig_records = [Receptor(x) for x in seq_copy]
+ self.unambig_data = DbResult('unambig', self.unambig_records)
+ self.unambig_clones = {('A1', 'A2', 'A3', 'A4'),
+ ('B1', 'B2', 'B3', 'B4')}
+
+ # Build db iterator with ambiguous assignments
+ group_copy = deepcopy(group_list)
+ seq_copy = deepcopy(seq_list)
+ for i, x in enumerate(group_copy): x.update(seq_copy[i])
+ self.ambig_records = [Receptor(x) for x in group_copy]
+ self.ambig_data = DbResult('ambig', self.ambig_records)
+ self.first_nofields = {('IGHV1-1', 'IGHJ6', '48'): ['A1','A4'],
+ ('IGHV2-1', 'IGHJ6', '48'): ['A2','B2'],
+ ('IGHV3-1', 'IGHJ6', '48'): ['A3'],
+ ('IGHV4-1', 'IGHJ6', '48'): ['B1'],
+ ('IGHV5-1', 'IGHJ6', '48'): ['B3','B4']}
+ self.set_nofields = {('48', ('IGHJ6',), ('IGHV1-1', 'IGHV2-1', 'IGHV3-1', 'IGHV4-1')):
+ ['A1', 'A2', 'A3', 'A4', 'B1', 'B2'],
+ ('48', ('IGHJ6',), ('IGHV5-1', 'IGHV6-1')):
+ ['B3', 'B4']}
+ self.first_fields = {('IGHV1-1', 'IGHJ6', '48', 'GA', '1'): ['A1'],
+ ('IGHV2-1', 'IGHJ6', '48', 'GA', '1'): ['A2'],
+ ('IGHV3-1', 'IGHJ6', '48', 'GA', '2'): ['A3'],
+ ('IGHV1-1', 'IGHJ6', '48', 'GA', '2'): ['A4'],
+ ('IGHV4-1', 'IGHJ6', '48', 'GB', '1'): ['B1'],
+ ('IGHV2-1', 'IGHJ6', '48', 'GB', '2'): ['B2'],
+ None: ['B3','B4']}
+ self.set_fields = {('48', 'GA', '1', ('IGHJ6',), ('IGHV1-1',)): ['A1'],
+ ('48', 'GA', '1', ('IGHJ6',), ('IGHV2-1',)): ['A2'],
+ ('48', 'GA', '2', ('IGHJ6',), ('IGHV1-1', 'IGHV2-1', 'IGHV3-1')): ['A3', 'A4'],
+ ('48', 'GB', '1', ('IGHJ6',), ('IGHV4-1',)): ['B1'],
+ ('48', 'GB', '2', ('IGHJ6',), ('IGHV2-1', 'IGHV4-1')): ['B2'],
+ None: ['B3','B4']}
+ self.start = time.time()
+
+ def tearDown(self):
+ t = time.time() - self.start
+ print("<- %s() %.3f" % (self._testMethodName, t))
+
+ # @unittest.skip("-> groupByGene() skipped\n")
+ def test_groupByGene(self):
+ # Test first grouping without fields
+ results = DefineClones.groupByGene(self.ambig_records, mode='gene', action='first')
+ # Extract nested keys and group lengths for comparison
+ results_dict = dict()
+ print('FIRST>')
+ for k, v in results.items():
+ nest_key = tuple([tuple(sorted(chain(x))) if isinstance(x, tuple) else str(x) for x in k])
+ results_dict[nest_key] = sorted([x.sequence_id for x in v])
+ print(' GROUP>', nest_key, ':', results_dict[nest_key])
+ print('')
+ self.assertDictEqual(self.first_nofields, results_dict)
+
+ # Test ambiguous grouping without fields
+ results = DefineClones.groupByGene(self.ambig_records, mode='gene', action='set')
+ # Extract nested keys and group lengths for comparison
+ results_dict = dict()
+ print('SET>')
+ for k, v in results.items():
+ nest_key = tuple([tuple(sorted(chain(x))) if isinstance(x, tuple) else str(x) for x in k])
+ results_dict[nest_key] = sorted([x.sequence_id for x in v])
+ print(' GROUP>', nest_key, ':', results_dict[nest_key])
+ print('')
+ self.assertDictEqual(self.set_nofields, results_dict)
+
+ # Test first grouping with fields
+ results = DefineClones.groupByGene(self.ambig_records, group_fields=['FIELD1', 'FIELD2'],
+ mode='gene', action='first')
+ # Extract nested keys and group lengths for comparison
+ results_dict = dict()
+ print('FIRST>')
+ for k, v in results.items():
+ nest_key = tuple([tuple(sorted(chain(x))) if isinstance(x, tuple) else str(x) for x in k]) if k is not None else None
+ results_dict[nest_key] = sorted([x.sequence_id for x in v])
+ print(' GROUP>', nest_key, ':', results_dict[nest_key])
+ print('')
+ print('RRRR>', self.first_fields)
+ print('QQQQ>', results_dict)
+
+ self.assertDictEqual(self.first_fields, results_dict)
+
+ # Test ambiguous grouping with fields
+ results = DefineClones.groupByGene(self.ambig_records, group_fields=['FIELD1', 'FIELD2'],
+ mode='gene', action='set')
+ # Extract nested keys and group lengths for comparison
+ results_dict = dict()
+ print('SET>')
+ for k, v in results.items():
+ nest_key = tuple([tuple(sorted(chain(x))) if isinstance(x, tuple) else str(x) for x in k]) if k is not None else None
+ results_dict[nest_key] = sorted([x.sequence_id for x in v])
+ print(' GROUP>', nest_key, ':', results_dict[nest_key])
+ print('')
+ self.assertDictEqual(self.set_fields, results_dict)
+
+
+ # @unittest.skip("-> distanceClones() skipped\n")
+ def test_distanceClones(self):
+ # import cProfile
+ # prof = cProfile.Profile()
+ # results = prof.runcall(DefineClones.distanceClones, self.unambig_records, model='hs5f', distance=1.0, dist_mat=self.dist_mat)
+ # prof.dump_stats('hs5f-unit-test-dict.prof')
+
+ # ham model
+ results = DefineClones.distanceClones(self.unambig_data, model='ham', distance=9.0, norm='none')
+ results_set = set([tuple(sorted([x.sequence_id for x in v])) for v in results.results.values()])
+ print('MODEL> ham')
+ for i, x in enumerate(results_set): print(' CLONE-%i> %s' % (i + 1, x))
+ self.assertSetEqual(results_set, self.unambig_clones)
+
+ # m1n_compat model
+ results = DefineClones.distanceClones(self.unambig_data, model='m1n_compat', distance=10.0, norm='none')
+ results_set = set([tuple(sorted([x.sequence_id for x in v])) for v in results.results.values()])
+ print('MODEL> m1n_compat')
+ for i, x in enumerate(results_set): print(' CLONE-%i> %s' % (i + 1, x))
+ self.assertSetEqual(results_set, self.unambig_clones)
+
+ # hs1f_compat model
+ results = DefineClones.distanceClones(self.unambig_data, model='hs1f_compat', distance=0.25)
+ results_set = set([tuple(sorted([x.sequence_id for x in v])) for v in results.results.values()])
+ print('MODEL> hs1f_compat')
+ for i, x in enumerate(results_set): print(' CLONE-%i> %s' % (i + 1, x))
+ self.assertSetEqual(results_set, self.unambig_clones)
+
+ # hh_s5f model
+ results = DefineClones.distanceClones(self.unambig_data, model='hh_s5f', distance=0.1)
+ results_set = set([tuple(sorted([x.sequence_id for x in v])) for v in results.results.values()])
+ print('MODEL> hh_s5f')
+ for i, x in enumerate(results_set): print(' CLONE-%i> %s' % (i + 1, x))
+ self.assertSetEqual(results_set, self.unambig_clones)
+
+ # aa model
+ results = DefineClones.distanceClones(self.unambig_data, model='aa', distance=3.0, norm='none')
+ results_set = set([tuple(sorted([x.sequence_id for x in v])) for v in results.results.values()])
+ print('MODEL> aa')
+ for i, x in enumerate(results_set): print(' CLONE-%i> %s' % (i + 1, x))
+ self.assertSetEqual(results_set, self.unambig_clones)
+
+
+if __name__ == '__main__':
+ unittest.main()
\ No newline at end of file
=====================================
tests/test_Distance.py
=====================================
@@ -0,0 +1,188 @@
+"""
+Unit tests for Distance module
+"""
+# Info
+__author__ = 'Jason Anthony Vander Heiden'
+from changeo import __version__, __date__
+
+# Imports
+import os
+import time
+import unittest
+import numpy as np
+from Bio.Seq import Seq
+
+# Presto and changeo imports
+import changeo.Distance as Distance
+
+# Paths
+test_path = os.path.dirname(os.path.realpath(__file__))
+data_path = os.path.join(test_path, 'data')
+
+
+class Test_Distance(unittest.TestCase):
+ def setUp(self):
+ print('-> %s()' % self._testMethodName)
+
+ self.nt_seq = ['TGTGCAAGGGGGCCA',
+ 'TGTGCAAGGGGGCCA',
+ 'TGTATTTGGGGGCCA',
+ 'ACACTTGCCACTGAT',
+ 'NNNNNNNNNNNNTGA',
+ 'NNNNNNNNNNNNNNN']
+ self.nt_seq_short = ['TGTGCAAGG',
+ 'TGTGCAAGG',
+ 'TGTATTTGG',
+ 'NNNNNNNNN']
+ self.aa_seq = [str(Seq(x).translate()) for x in self.nt_seq]
+
+
+ # Amino acid distance from sequence 0 to sequence 0,..,5
+ self.aa_len = 5.0
+ self.aa_ham = np.array([0.0, 0.0, 2.0, 5.0, 1.0, 0.0])
+
+ # Nucleotide distances sequence 0 to sequence 0,..,5
+ self.nt_len = 15.0
+ # 0 vs 2
+ # A-C = 0; A-G = 1; A-T = 2;
+ # C-G = 0; C-T = 1; G-T = 0
+ # 0 vs 3
+ # A-C = 1; A-G = 2; A-T = 4;
+ # C-G = 6; C-T = 1; G-T = 1
+ # 0 vs 4
+ # A-C = 0; A-G = 0; A-T = 0;
+ # C-G = 1; C-T = 1; G-T = 0
+ self.nt_ham = np.array([0.0, 0.0, 4.0, 15.0, 2.0, 0.0])
+ self.nt_hh_s1f = np.array([0.0,
+ 0.0,
+ 0.0 + 1*0.64 + 2*1.16 + 0.0 + 1*0.64 + 0.0,
+ 1*1.21 + 2*0.64 + 4*1.16 + 6*1.16 + 1*0.64 + 1*1.21,
+ 0.0 + 0.0 + 0.0 + 0.0 + 1*1.16 + 1*0.64 + 0.0,
+ 0.0])
+ self.nt_mk_rs1nf = np.array([0.0,
+ 0.0,
+ 0.0 + 1*0.32 + 2*1.17 + 0.0 + 1*0.32 + 0.0,
+ 1*1.51 + 2*0.32 + 4*1.17 + 6*1.17 + 1*0.32 + 1*1.51,
+ 0.0 + 0.0 + 0.0 + 0.0 + 1*1.17 + 1*0.32 + 0.0,
+ 0.0])
+
+ # 5-mer models use shorter sequence
+ # For sequence 0 to sequence 0,..,3
+ self.nt_short_len = 9.0
+ # hh_hs5f
+ # GTGCA-GTATT = [0.97, 0.84]
+ # TGCAA-TATTT = [0.93, 0.83]
+ # GCAAG-ATTTG = [1.08, 1.16]
+ # CAAGG-TTTGG = [0.91, 1.07]
+ self.nt_hh_s5f_avg = np.array([0.0,
+ 0.0,
+ np.mean([0.97, 0.84]) + np.mean([0.93, 0.83]) +
+ np.mean([1.08, 1.16]) + np.mean([0.91, 1.07]),
+ 0.0])
+ self.nt_hh_s5f_min = np.array([0.0,
+ 0.0,
+ np.min([0.97, 0.84]) + np.min([0.93, 0.83]) +
+ np.min([1.08, 1.16]) + np.min([0.91, 1.07]),
+ 0.0])
+ # mk_rs5nf
+ # GTGCA-GTATT = [0.71, 0.77]
+ # TGCAA-TATTT = [0.71, 0.93]
+ # GCAAG-ATTTG = [1.05, 1.03]
+ # CAAGG-TTTGG = [1.08, 1.13]
+ self.nt_mk_rs5nf_avg = np.array([0.0,
+ 0.0,
+ np.mean([0.71, 0.77]) + np.mean([0.71, 0.93]) +
+ np.mean([1.05, 1.03]) + np.mean([1.08, 1.13]),
+ 0.0])
+ self.nt_mk_rs5nf_min = np.array([0.0,
+ 0.0,
+ np.min([0.71, 0.77]) + np.min([0.71, 0.93]) +
+ np.min([1.05, 1.03]) + np.min([1.08, 1.13]),
+ 0.0])
+
+ self.start = time.time()
+
+ def tearDown(self):
+ t = time.time() - self.start
+ print("<- %s() %.3f" % (self._testMethodName, t))
+
+ def test_calcDistances(self):
+ # aa
+ print(' AA>', self.aa_seq)
+ result = Distance.calcDistances(self.aa_seq, n=1, dist_mat=Distance.aa_model, norm=None)
+ print(' AA>', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.aa_ham)
+
+ result = Distance.calcDistances(self.aa_seq, n=1, dist_mat=Distance.aa_model, norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.aa_ham / self.aa_len)
+
+ # ham
+ print(' NT>', self.nt_seq)
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.ham_model, norm=None)
+ print('HAM>', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_ham)
+
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.ham_model, norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_ham / self.nt_len)
+
+ # hh_s1f
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.hh_s1f_model, norm=None)
+ print('HH_S1F>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s1f)
+
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.hh_s1f_model, norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s1f / self.nt_len)
+
+ # mk_rs1nf
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.mk_rs1nf_model, norm=None)
+ print('MK_RS1NF>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs1nf)
+
+ result = Distance.calcDistances(self.nt_seq, n=1, dist_mat=Distance.mk_rs1nf_model, norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs1nf / self.nt_len)
+
+ # hh_s5f
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.hh_s5f_model, sym='avg', norm=None)
+ print('HH_S5F-AVG>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s5f_avg)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.hh_s5f_model, sym='avg', norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s5f_avg / self.nt_short_len)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.hh_s5f_model, sym='min', norm=None)
+ print('HH_S5F-MIN>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s5f_min)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.hh_s5f_model, sym='min', norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_hh_s5f_min / self.nt_short_len)
+
+ # mk_rs5nf
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.mk_rs5nf_model, sym='avg', norm=None)
+ print('MK_RS5NF-AVG>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs5nf_avg)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.mk_rs5nf_model, sym='avg', norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs5nf_avg / self.nt_short_len)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.mk_rs5nf_model, sym='min', norm=None)
+ print('MK_RS5NF-MIN>\n', result[1, :])
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs5nf_min)
+
+ result = Distance.calcDistances(self.nt_seq_short, n=5, dist_mat=Distance.mk_rs5nf_model, sym='min', norm='len')
+ np.testing.assert_almost_equal(result[1, :], self.nt_mk_rs5nf_min / self.nt_short_len)
+
+ # Raise error on length mismatch
+ with self.assertRaises(IndexError):
+ Distance.calcDistances(['ATGCAA', 'ATGC'], n=1, dist_mat=Distance.ham_model)
+
+
+ @unittest.skip("-> loadModels() skipped\n")
+ def test_loadModels(self):
+ print('MK_RS1NF> ', Distance.mk_rs1nf_model)
+ print(' HH_S1F> ', Distance.hh_s5f_model)
+
+ self.fail('TODO')
+
+
+if __name__ == '__main__':
+ unittest.main()
=====================================
tests/test_IO.py
=====================================
@@ -0,0 +1,264 @@
+"""
+Unit tests for MakeDb
+"""
+# Info
+__author__ = 'Jason Anthony Vander Heiden'
+from changeo import __version__, __date__
+
+# Imports
+import gzip
+import os
+import shutil
+import sys
+import tempfile
+import time
+import unittest
+from Bio import SeqIO
+
+# Presto and changeo imports
+from changeo.Commandline import getCommonArgParser, parseCommonArgs
+from changeo.IO import getDbFields, extractIMGT, readGermlines, ChangeoReader, \
+ IgBLASTReader, IgBLASTReaderAA, IHMMuneReader, IMGTReader, openFile
+
+# Paths
+test_path = os.path.dirname(os.path.realpath(__file__))
+data_path = os.path.join(test_path, 'data')
+
+# Import script
+sys.path.append(os.path.join(test_path, os.pardir, 'bin'))
+import MakeDb
+
+
+class Test_MakeDb(unittest.TestCase):
+ def setUp(self):
+ print('-> %s()' % self._testMethodName)
+
+ # Germline files
+ self.repo_ig = '/usr/local/share/germlines/imgt/human/vdj'
+ self.repo_ig_aa = '/usr/local/share/germlines/imgt/human/vdj_aa'
+ # Read files
+ self.reads_ig = os.path.join(data_path, 'reads_ig.fasta')
+ self.reads_ig_aa = os.path.join(data_path, 'reads_ig_aa.fasta')
+ # IMGT output
+ self.imgt_ig = os.path.join(data_path, 'imgt_ig.txz')
+ # IgBLAST output
+ self.igblast_ig = os.path.join(data_path, 'igblast1.7_ig.fmt7')
+ self.igblast_ig_aa = os.path.join(data_path, 'igblast1.14_ig_aa.fmt7')
+ # iHMMune-Align output
+ self.ihmmune_ig = os.path.join(data_path, 'ihmmune_ig.csv')
+ # Change-O files
+ self.db_ig = os.path.join(data_path, 'imgt_ig_db-pass.tsv')
+
+ self.start = time.time()
+
+ def tearDown(self):
+ t = time.time() - self.start
+ print("<- %s() %.3f" % (self._testMethodName, t))
+
+ @unittest.skip("-> ChangeoReader() skipped\n")
+ def test_getDbFields(self):
+ # Get fields
+ x = getDbFields(self.db_ig)
+ print(x)
+
+ # Add fields
+ x = getDbFields(self.db_ig, add=['A', 'B', 'C'])
+ print(x)
+
+ # Exclude fields
+ x = getDbFields(self.db_ig, exclude=['V_SCORE', 'V_IDENTITY', 'J_SCORE', 'J_IDENTITY'])
+ print(x)
+
+ # Add and exclude fields
+ x = getDbFields(self.db_ig, add=['A', 'B', 'C'],
+ exclude=['V_SCORE', 'V_IDENTITY', 'J_SCORE', 'J_IDENTITY'])
+ print(x)
+
+ self.fail('TODO')
+
+ @unittest.skip("-> ChangeoReader() skipped\n")
+ def test_ChangeoReader(self):
+ # Parse
+ with open(self.db_ig, 'r') as f:
+ result = ChangeoReader(f)
+ for x in result: print(x.toDict())
+
+ self.fail('TODO')
+
+ @unittest.skip("-> IMGTReader() skipped\n")
+ def test_IMGTReader(self):
+ # Extract IMGT files
+ temp_dir, files = extractIMGT(self.imgt_ig)
+
+ # Parse
+ with open(files['summary'], 'r') as summary, \
+ open(files['gapped'], 'r') as gapped, \
+ open(files['ntseq'], 'r') as ntseq, \
+ open(files['junction'], 'r') as junction:
+ result = IMGTReader(summary, gapped, ntseq, junction, receptor=False)
+ for x in result: print(x)
+
+ # Remove IMGT temporary directory
+ temp_dir.cleanup()
+
+ self.fail('TODO')
+
+ @unittest.skip("-> IgBLASTReader() skipped\n")
+ def test_IgBLASTReader(self):
+ # Load germlines and sequences
+ seq_dict = MakeDb.getSeqDict(self.reads_ig_aa)
+ repo_dict = readGermlines([self.repo_ig])
+
+ # Parse
+ with open(self.igblast_ig, 'r') as f:
+ result = IgBLASTReader(f, seq_dict, repo_dict, receptor=False)
+ for x in result: print(x)
+
+ self.fail('TODO')
+
+ @unittest.skip("-> IgBLASTAAReader() skipped\n")
+ def test_IgBLASTAAReader(self):
+ # Load germlines and sequences
+ seq_dict = MakeDb.getSeqDict(self.reads_ig_aa)
+ repo_dict = readGermlines([self.repo_ig_aa])
+
+ # Parse
+ with open(self.igblast_ig_aa, 'r') as f:
+ result = IgBLASTReaderAA(f, seq_dict, repo_dict, receptor=False)
+ for x in result: print(x)
+
+ # self.fail('TODO')
+
+ @unittest.skip("-> IHMMReader() skipped\n")
+ def test_IHMMReader(self):
+ # Load germlines and sequences
+ seq_dict = MakeDb.getSeqDict(self.reads_ig)
+ repo_dict = readGermlines([self.repo_ig])
+
+ # Parse
+ with open(self.ihmmune_ig, 'r') as f:
+ result = IHMMuneReader(f, seq_dict, repo_dict, receptor=False)
+ for x in result: print(x)
+
+ self.fail('TODO')
+
+ @unittest.skip("-> igblast-aa partial execution test skipped\n")
+ def test_igblast_aa_partial_execution(self):
+ """Test actual execution of MakeDb.py with and without --partial"""
+ import subprocess, tempfile, shutil
+ temp_dir = tempfile.mkdtemp()
+ try:
+ strict_pass = os.path.join(temp_dir, 'strict_db-pass.tsv')
+ strict_fail = os.path.join(temp_dir, 'strict_db-fail.tsv')
+ partial_pass = os.path.join(temp_dir, 'partial_db-pass.tsv')
+ partial_fail = os.path.join(temp_dir, 'partial_db-fail.tsv')
+ bin_path = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))), 'bin')
+ # Run MakeDb.py with strict validation (default)
+ strict_cmd = [
+ os.path.join(bin_path, 'MakeDb.py'),
+ 'igblast-aa',
+ '-i', self.igblast_ig_aa,
+ '-s', self.reads_ig_aa,
+ '-r', self.repo_ig_aa,
+ '--outname', os.path.join(temp_dir, 'strict')
+ ]
+ subprocess.run(strict_cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ # Run MakeDb.py with partial validation
+ partial_cmd = [
+ os.path.join(bin_path, 'MakeDb.py'),
+ 'igblast-aa',
+ '-i', self.igblast_ig_aa,
+ '-s', self.reads_ig_aa,
+ '-r', self.repo_ig_aa,
+ '--partial',
+ '--outname', os.path.join(temp_dir, 'partial')
+ ]
+ subprocess.run(partial_cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ # Output files may not exist if there are no records for that category
+ for f in [strict_pass, strict_fail, partial_pass, partial_fail]:
+ if os.path.exists(f):
+ self.assertTrue(os.path.isfile(f))
+ def count_records(filepath):
+ if os.path.isfile(filepath) and os.path.getsize(filepath) > 0:
+ return sum(1 for line in open(filepath) if not line.startswith('#'))
+ return 0
+ strict_pass_count = count_records(strict_pass)
+ strict_fail_count = count_records(strict_fail)
+ partial_pass_count = count_records(partial_pass)
+ partial_fail_count = count_records(partial_fail)
+ self.assertGreaterEqual(partial_pass_count + partial_fail_count,
+ strict_pass_count + strict_fail_count,
+ "Total records should be the same or greater with partial validation")
+ if strict_fail_count > 0:
+ self.assertTrue(
+ partial_pass_count > strict_pass_count or partial_fail_count < strict_fail_count,
+ "Partial validation should pass more records or fail fewer records than strict validation"
+ )
+ finally:
+ shutil.rmtree(temp_dir)
+
+
+class Test_Gzip(unittest.TestCase):
+ def setUp(self):
+ self.temp_dir = tempfile.mkdtemp()
+
+ def tearDown(self):
+ shutil.rmtree(self.temp_dir)
+
+ def test_openFile_writes_real_gzip(self):
+ """openFile('w') on a .gz name produces an actual gzip file, verified
+ independently of openFile itself (via the stdlib gzip module and the
+ gzip magic bytes), not just round-tripped through the same function."""
+ file_name = os.path.join(self.temp_dir, 'records.tsv.gz')
+ content = 'sequence_id\tproductive\nseq1\tT\n'
+
+ with openFile(file_name, 'w') as handle:
+ handle.write(content)
+
+ with open(file_name, 'rb') as handle:
+ self.assertEqual(handle.read(2), b'\x1f\x8b') # gzip magic bytes
+ with gzip.open(file_name, 'rt') as handle:
+ self.assertEqual(handle.read(), content)
+
+ def test_openFile_reads_real_gzip(self):
+ """openFile('r') on a .gz name transparently decompresses a file
+ written independently by the stdlib gzip module."""
+ file_name = os.path.join(self.temp_dir, 'records.tsv.gz')
+ content = 'sequence_id\tproductive\nseq1\tT\n'
+
+ with gzip.open(file_name, 'wt') as handle:
+ handle.write(content)
+
+ with openFile(file_name, 'r') as handle:
+ self.assertEqual(handle.read(), content)
+
+ def test_gzip_output_cannot_overwrite_input(self):
+ """The final gzip output name may not resolve to the input file."""
+ input_file = os.path.join(self.temp_dir, 'records.tsv.gz')
+ with gzip.open(input_file, 'wt') as handle:
+ handle.write('sequence_id\nseq1\n')
+
+ parser = getCommonArgParser()
+ args = parser.parse_args(['-d', input_file, '-o', input_file[:-3], '--gzip-output'])
+ with self.assertRaises(SystemExit):
+ parseCommonArgs(args)
+
+ def test_gzip_output_cannot_overwrite_input_via_path_alias(self):
+ """A -o that is only path-equivalent to the input (e.g. via a redundant
+ './' segment, as in -d ./input.tsv.gz -o input.tsv) must still be caught,
+ even though the two names are not string-identical."""
+ input_file = os.path.join(self.temp_dir, 'records.tsv.gz')
+ with gzip.open(input_file, 'wt') as handle:
+ handle.write('sequence_id\nseq1\n')
+
+ alias_out = os.path.join(self.temp_dir, '.', 'records.tsv')
+ self.assertNotEqual(alias_out, input_file[:-3])
+
+ parser = getCommonArgParser()
+ args = parser.parse_args(['-d', input_file, '-o', alias_out, '--gzip-output'])
+ with self.assertRaises(SystemExit):
+ parseCommonArgs(args)
+
+
+if __name__ == '__main__':
+ unittest.main()
=====================================
tests/test_ParseDb.py
=====================================
@@ -0,0 +1,171 @@
+"""
+Unit tests for ParseDb
+"""
+
+# Info
+__author__ = 'Jason Anthony Vander Heiden'
+from changeo import __version__, __date__
+
+# Imports
+import csv
+import gzip
+import io
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+from contextlib import redirect_stderr
+
+# Paths
+test_path = os.path.dirname(os.path.realpath(__file__))
+data_path = os.path.join(test_path, 'data')
+bin_path = os.path.join(test_path, os.pardir, 'bin')
+
+# Import script
+sys.path.append(bin_path)
+import ParseDb
+
+
+class Test_ParseDb(unittest.TestCase):
+ def setUp(self):
+ print('-> %s()' % self._testMethodName)
+ self.start = time.time()
+ self.tmp_dir = tempfile.mkdtemp()
+
+ def tearDown(self):
+ shutil.rmtree(self.tmp_dir)
+ t = time.time() - self.start
+ print("<- %s() %.3f" % (self._testMethodName, t))
+
+ def _write_tsv(self, name, rows):
+ path = os.path.join(self.tmp_dir, name)
+ with open(path, 'w', newline='') as f:
+ writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()), delimiter='\t')
+ writer.writeheader()
+ writer.writerows(rows)
+ return path
+
+ def _read_tsv(self, path):
+ with open(path, 'r', newline='') as f:
+ return list(csv.DictReader(f, delimiter='\t'))
+
+ # @unittest.skip("-> mergeDbFiles() skipped\n")
+ def test_mergeDbFiles(self):
+ file1 = self._write_tsv('sample1.tsv', [{'SEQUENCE_ID': 'A1', 'V_CALL': 'IGHV1'},
+ {'SEQUENCE_ID': 'A2', 'V_CALL': 'IGHV2'}])
+ file2 = self._write_tsv('sample2.tsv', [{'SEQUENCE_ID': 'B1', 'V_CALL': 'IGHV3'}])
+ out_file = os.path.join(self.tmp_dir, 'merged.tsv')
+
+ # Merge without sample annotation
+ ParseDb.mergeDbFiles([file1, file2], out_file=out_file)
+ records = self._read_tsv(out_file)
+ self.assertEqual(len(records), 3)
+ self.assertNotIn('SAMPLE', records[0])
+
+ # Merge with sample annotation
+ ParseDb.mergeDbFiles([file1, file2], field='SAMPLE', values=['sample1', 'sample2'],
+ out_file=out_file)
+ records = self._read_tsv(out_file)
+ results = {r['SEQUENCE_ID']: r['SAMPLE'] for r in records}
+ self.assertDictEqual(results, {'A1': 'sample1', 'A2': 'sample1', 'B1': 'sample2'})
+
+ def test_mergeDbFiles_values_without_field(self):
+ file1 = self._write_tsv('sample1.tsv', [{'SEQUENCE_ID': 'A1', 'V_CALL': 'IGHV1'}])
+ file2 = self._write_tsv('sample2.tsv', [{'SEQUENCE_ID': 'B1', 'V_CALL': 'IGHV3'}])
+ out_file = os.path.join(self.tmp_dir, 'merged.tsv')
+
+ # Values without a field are ignored, with a warning
+ stderr = io.StringIO()
+ with redirect_stderr(stderr):
+ ParseDb.mergeDbFiles([file1, file2], values=['sample1', 'sample2'], out_file=out_file)
+ self.assertIn('WARNING', stderr.getvalue())
+
+ records = self._read_tsv(out_file)
+ self.assertEqual(len(records), 2)
+ self.assertNotIn('SAMPLE', records[0])
+
+ def test_mergeDbFiles_field_overwrites_existing_column(self):
+ file1 = self._write_tsv('sample1.tsv', [{'SEQUENCE_ID': 'A1', 'SAMPLE': 'original'}])
+ file2 = self._write_tsv('sample2.tsv', [{'SEQUENCE_ID': 'B1', 'SAMPLE': 'original'}])
+ out_file = os.path.join(self.tmp_dir, 'merged.tsv')
+
+ # Reusing an existing column name overwrites its values, with a warning
+ stderr = io.StringIO()
+ with redirect_stderr(stderr):
+ ParseDb.mergeDbFiles([file1, file2], field='SAMPLE', values=['sample1', 'sample2'],
+ out_file=out_file)
+ self.assertIn('WARNING', stderr.getvalue())
+
+ records = self._read_tsv(out_file)
+ results = {r['SEQUENCE_ID']: r['SAMPLE'] for r in records}
+ self.assertDictEqual(results, {'A1': 'sample1', 'B1': 'sample2'})
+
+ def test_mergeDbFiles_field_overwrites_partial_column_with_drop(self):
+ # Only file1 has a SAMPLE column; with drop=True it is excluded from
+ # out_fields before the annotation field is considered, so the warning
+ # must be based on the input headers rather than the post-drop output.
+ file1 = self._write_tsv('sample1.tsv', [{'SEQUENCE_ID': 'A1', 'SAMPLE': 'original'}])
+ file2 = self._write_tsv('sample2.tsv', [{'SEQUENCE_ID': 'B1', 'V_CALL': 'IGHV1'}])
+ out_file = os.path.join(self.tmp_dir, 'merged.tsv')
+
+ stderr = io.StringIO()
+ with redirect_stderr(stderr):
+ ParseDb.mergeDbFiles([file1, file2], drop=True, field='SAMPLE',
+ values=['sample1', 'sample2'], out_file=out_file)
+ self.assertIn('WARNING', stderr.getvalue())
+
+ records = self._read_tsv(out_file)
+ results = {r['SEQUENCE_ID']: r['SAMPLE'] for r in records}
+ self.assertDictEqual(results, {'A1': 'sample1', 'B1': 'sample2'})
+
+class Test_ParseDbGzip(unittest.TestCase):
+ """Tests for compressed input and output."""
+ def setUp(self):
+ self.temp_dir = tempfile.mkdtemp()
+ source_file = os.path.join(data_path, 'imgt_ig_db-pass.tsv')
+ self.plain_file = os.path.join(self.temp_dir, 'input.tsv')
+ self.gzip_file = '%s.gz' % self.plain_file
+ shutil.copyfile(source_file, self.plain_file)
+ with open(self.plain_file, 'rb') as source, gzip.open(self.gzip_file, 'wb') as target:
+ shutil.copyfileobj(source, target)
+
+ with open(self.plain_file, 'r') as handle:
+ self.expected_lines = handle.read().splitlines()[:2]
+ self.sequence_id = self.expected_lines[1].split('\t')[0]
+
+ def tearDown(self):
+ shutil.rmtree(self.temp_dir)
+
+ def _select(self, input_file, gzip_output=False):
+ command = [sys.executable, os.path.join(bin_path, 'ParseDb.py'), 'select',
+ '-d', input_file, '-f', 'SEQUENCE_ID', '-u', self.sequence_id,
+ '--outdir', self.temp_dir]
+ if gzip_output:
+ command.append('--gzip-output')
+ result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ self.assertEqual(result.returncode, 0, result.stdout.decode('utf-8'))
+
+ input_names = {os.path.basename(self.plain_file), os.path.basename(self.gzip_file)}
+ output_names = [name for name in os.listdir(self.temp_dir) if name not in input_names]
+ self.assertEqual(len(output_names), 1)
+ return os.path.join(self.temp_dir, output_names[0])
+
+ def _assert_selected_gzip_output(self, output_file):
+ self.assertTrue(output_file.endswith('.gz'))
+ with gzip.open(output_file, 'rt') as handle:
+ self.assertEqual(handle.read().splitlines(), self.expected_lines)
+
+ def test_compressed_input_preserves_compressed_output(self):
+ output_file = self._select(self.gzip_file)
+ self._assert_selected_gzip_output(output_file)
+
+ def test_gzip_output_compresses_plain_input_result(self):
+ output_file = self._select(self.plain_file, gzip_output=True)
+ self._assert_selected_gzip_output(output_file)
+
+
+if __name__ == '__main__':
+ unittest.main()
=====================================
tests/test_Receptor.py
=====================================
@@ -0,0 +1,220 @@
+"""
+Unit tests for MakeDb
+"""
+
+# Imports
+import os
+import time
+import unittest
+
+# Presto and changeo imports
+from changeo.Receptor import Receptor
+from changeo.Alignment import decodeBTOP, decodeCIGAR, encodeCIGAR, padAlignment, alignmentPositions
+
+# Paths
+test_path = os.path.dirname(os.path.realpath(__file__))
+data_path = os.path.join(test_path, 'data')
+
+
+class Test_Receptor(unittest.TestCase):
+ def setUp(self):
+ print('-> %s()' % self._testMethodName)
+
+ row = {'SEQUENCE_ID': 'TEST1',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGNNNNN',
+ 'V_CALL': 'IGHV6-1*01,IGHV6-1*02',
+ 'D_CALL': 'IGHD6-6*01',
+ 'J_CALL': 'IGHJ6*02',}
+ self.ig_rec = Receptor(row)
+
+ row = {'SEQUENCE_ID': 'TEST2',
+ 'SEQUENCE_INPUT': 'TGTGCAAGGGGGCCATTGGACTACTTCTACTACGGTGTGGACGNNNNN',
+ 'V_CALL': 'TRAV6-1*01,TRAV6-1*02',
+ 'D_CALL': 'TRAD6-6*01',
+ 'J_CALL': 'TRAJ6*02',}
+ self.tr_rec = Receptor(row)
+
+ row = {'SEQUENCE_ID': 'TEST3',
+ 'SEQUENCE_INPUT': '',
+ 'V_CALL': '',
+ 'D_CALL': None,
+ 'J_CALL': 'princess,banana,hammock',}
+ self.bad_rec = Receptor(row)
+
+ # CIGAR strings
+ self.cigar_string = ['30M1I69M3D', '4S16N30M1I69M3D']
+ self.cigar_decoded = [[('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 4), ('N', 16), ('M', 30), ('I', 1), ('M', 69), ('D', 3)]]
+ self.cigar_len = [{'q_start': 0, 'q_length': 30 + 1 + 69 + 0, 'r_start': 0, 'r_length': 30 + 0 + 69 + 3},
+ {'q_start': 4, 'q_length': 30 + 1 + 69 + 0, 'r_start': 16, 'r_length': 30 + 0 + 69 + 3}]
+
+ self.cigar_pos = [(0, 0), (5, 0), (0, 3), (5, 3)]
+ self.cigar_pad_1 = [[('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 5), ('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('N', 3), ('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 5), ('N', 3), ('M', 30), ('I', 1), ('M', 69), ('D', 3)]]
+ self.cigar_pad_2 = [[('S', 4), ('N', 16), ('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 9), ('N', 16), ('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 4), ('N', 19), ('M', 30), ('I', 1), ('M', 69), ('D', 3)],
+ [('S', 9), ('N', 19), ('M', 30), ('I', 1), ('M', 69), ('D', 3)]]
+
+ # BTOP strings
+ self.btop_string = ['7AGAC39',
+ '7A-39',
+ '6-G-A41',
+ 'AG8-GC-CTCT']
+ self.btop_decoded_full = [[('=', 7), ('X', 2), ('=', 39)],
+ [('=', 7), ('D', 1), ('=', 39)],
+ [('=', 6), ('I', 2), ('=', 41)],
+ [('X', 1), ('=', 8), ('I', 1), ('D', 1), ('X', 2)]]
+
+ self.start = time.time()
+
+ def tearDown(self):
+ t = time.time() - self.start
+ print('<- %s() %.3f' % (self._testMethodName, t))
+
+ @unittest.skip("-> Receptor() skipped\n")
+ def test_Receptor(self):
+ print('IG>')
+ print(self.ig_rec.getAlleleCalls(['v','d','j'], action='first'))
+ print(self.ig_rec.getAlleleCalls(['v','j'], action='first'))
+ print(self.ig_rec.getAlleleCalls(['j','v'], action='first'))
+ print(self.ig_rec.getGeneCalls(['v','d','j'], action='first'))
+ print(self.ig_rec.getGeneCalls(['v','j'], action='first'))
+ print(self.ig_rec.getGeneCalls(['j','v'], action='first'))
+ print(self.ig_rec.getFamilyCalls(['v','d','j'], action='first'))
+ print(self.ig_rec.getFamilyCalls(['v','j'], action='first'))
+ print(self.ig_rec.getFamilyCalls(['j','v'], action='first'))
+
+ print(self.ig_rec.getAlleleCalls(['v','d','j'], action='set'))
+ print(self.ig_rec.getAlleleCalls(['v','j'], action='set'))
+ print(self.ig_rec.getAlleleCalls(['j','v'], action='set'))
+ print(self.ig_rec.getGeneCalls(['v','d','j'], action='set'))
+ print(self.ig_rec.getGeneCalls(['v','j'], action='set'))
+ print(self.ig_rec.getGeneCalls(['j','v'], action='set'))
+ print(self.ig_rec.getFamilyCalls(['v','d','j'], action='set'))
+ print(self.ig_rec.getFamilyCalls(['v','j'], action='set'))
+ print(self.ig_rec.getFamilyCalls(['j','v'], action='set'))
+
+ print(self.ig_rec.getAlleleCalls(['v','d','j'], action='list'))
+ print(self.ig_rec.getAlleleCalls(['v','j'], action='list'))
+ print(self.ig_rec.getAlleleCalls(['j','v'], action='list'))
+ print(self.ig_rec.getGeneCalls(['v','d','j'], action='list'))
+ print(self.ig_rec.getGeneCalls(['v','j'], action='list'))
+ print(self.ig_rec.getGeneCalls(['j','v'], action='list'))
+ print(self.ig_rec.getFamilyCalls(['v','d','j'], action='list'))
+ print(self.ig_rec.getFamilyCalls(['v','j'], action='list'))
+ print(self.ig_rec.getFamilyCalls(['j','v'], action='list'))
+
+ print('TR>')
+ print(self.tr_rec.getAlleleCalls(['v','d','j'], action='first'))
+ print(self.tr_rec.getAlleleCalls(['v','j'], action='first'))
+ print(self.tr_rec.getAlleleCalls(['j','v'], action='first'))
+ print(self.tr_rec.getGeneCalls(['v','d','j'], action='first'))
+ print(self.tr_rec.getGeneCalls(['v','j'], action='first'))
+ print(self.tr_rec.getGeneCalls(['j','v'], action='first'))
+ print(self.tr_rec.getFamilyCalls(['v','d','j'], action='first'))
+ print(self.tr_rec.getFamilyCalls(['v','j'], action='first'))
+ print(self.tr_rec.getFamilyCalls(['j','v'], action='first'))
+
+ print(self.tr_rec.getAlleleCalls(['v','d','j'], action='set'))
+ print(self.tr_rec.getAlleleCalls(['v','j'], action='set'))
+ print(self.tr_rec.getAlleleCalls(['j','v'], action='set'))
+ print(self.tr_rec.getGeneCalls(['v','d','j'], action='set'))
+ print(self.tr_rec.getGeneCalls(['v','j'], action='set'))
+ print(self.tr_rec.getGeneCalls(['j','v'], action='set'))
+ print(self.tr_rec.getFamilyCalls(['v','d','j'], action='set'))
+ print(self.tr_rec.getFamilyCalls(['v','j'], action='set'))
+ print(self.tr_rec.getFamilyCalls(['j','v'], action='set'))
+
+ print(self.tr_rec.getAlleleCalls(['v','d','j'], action='list'))
+ print(self.tr_rec.getAlleleCalls(['v','j'], action='list'))
+ print(self.tr_rec.getAlleleCalls(['j','v'], action='list'))
+ print(self.tr_rec.getGeneCalls(['v','d','j'], action='list'))
+ print(self.tr_rec.getGeneCalls(['v','j'], action='list'))
+ print(self.tr_rec.getGeneCalls(['j','v'], action='list'))
+ print(self.tr_rec.getFamilyCalls(['v','d','j'], action='list'))
+ print(self.tr_rec.getFamilyCalls(['v','j'], action='list'))
+ print(self.tr_rec.getFamilyCalls(['j','v'], action='list'))
+
+ print('JUNK>')
+ print(self.bad_rec.getAlleleCalls(['v','d','j'], action='first'))
+ print(self.bad_rec.getAlleleCalls(['v','j'], action='first'))
+ print(self.bad_rec.getAlleleCalls(['j','v'], action='first'))
+ print(self.bad_rec.getGeneCalls(['v','d','j'], action='first'))
+ print(self.bad_rec.getGeneCalls(['v','j'], action='first'))
+ print(self.bad_rec.getGeneCalls(['j','v'], action='first'))
+ print(self.bad_rec.getFamilyCalls(['v','d','j'], action='first'))
+ print(self.bad_rec.getFamilyCalls(['v','j'], action='first'))
+ print(self.bad_rec.getFamilyCalls(['j','v'], action='first'))
+
+ print(self.bad_rec.getAlleleCalls(['v','d','j'], action='set'))
+ print(self.bad_rec.getAlleleCalls(['v','j'], action='set'))
+ print(self.bad_rec.getAlleleCalls(['j','v'], action='set'))
+ print(self.bad_rec.getGeneCalls(['v','d','j'], action='set'))
+ print(self.bad_rec.getGeneCalls(['v','j'], action='set'))
+ print(self.bad_rec.getGeneCalls(['j','v'], action='set'))
+ print(self.bad_rec.getFamilyCalls(['v','d','j'], action='set'))
+ print(self.bad_rec.getFamilyCalls(['v','j'], action='set'))
+ print(self.bad_rec.getFamilyCalls(['j','v'], action='set'))
+
+ print(self.bad_rec.getAlleleCalls(['v','d','j'], action='list'))
+ print(self.bad_rec.getAlleleCalls(['v','j'], action='list'))
+ print(self.bad_rec.getAlleleCalls(['j','v'], action='list'))
+ print(self.bad_rec.getGeneCalls(['v','d','j'], action='list'))
+ print(self.bad_rec.getGeneCalls(['v','j'], action='list'))
+ print(self.bad_rec.getGeneCalls(['j','v'], action='list'))
+ print(self.bad_rec.getFamilyCalls(['v','d','j'], action='list'))
+ print(self.bad_rec.getFamilyCalls(['v','j'], action='list'))
+ print(self.bad_rec.getFamilyCalls(['j','v'], action='list'))
+
+ self.fail('TODO')
+
+ #@unittest.skip("-> decodeCIGAR() skipped\n")
+ def test_decodeCIGAR(self):
+ for cigar, truth in zip(self.cigar_string, self.cigar_decoded):
+ result = decodeCIGAR(cigar)
+ print(result)
+ self.assertListEqual(truth, result)
+
+ #@unittest.skip("-> decodeBTOP() skipped\n")
+ def test_decodeBTOP(self):
+ for btop, truth in zip(self.btop_string, self.btop_decoded_full):
+ result = decodeBTOP(btop)
+ print('FULL> ', result)
+ self.assertListEqual(truth, result)
+
+ #@unittest.skip("-> encodeCIGAR() skipped\n")
+ def test_encodeCIGAR(self):
+ for align, truth in zip(self.cigar_decoded, self.cigar_string):
+ result = encodeCIGAR(align)
+ print(result)
+ self.assertEqual(truth, result)
+
+ #@unittest.skip("-> padAlignment() skipped\n")
+ def test_padAlignment(self):
+ cigar = self.cigar_decoded[0]
+ for (s, r), truth in zip(self.cigar_pos, self.cigar_pad_1):
+ #print('POS>', s, r)
+ result = padAlignment(cigar, s, r)
+ print('PAD>', '(%i, %i) =' % (s, r), result)
+ self.assertEqual(truth, result)
+
+ cigar = self.cigar_decoded[1]
+ for (s, r), truth in zip(self.cigar_pos, self.cigar_pad_2):
+ #print('POS>', s, r)
+ result = padAlignment(cigar, s, r)
+ print('PAD>', '(%i, %i) =' % (s, r), result)
+ self.assertEqual(truth, result)
+
+ #@unittest.skip("-> alignmentPositions() skipped\n")
+ def test_alignmentPositions(self):
+ for align, truth in zip(self.cigar_decoded, self.cigar_len):
+ result = alignmentPositions(align)
+ print('POS>', result)
+ self.assertDictEqual(truth, result)
+
+
+if __name__ == '__main__':
+ unittest.main()
\ No newline at end of file
View it on GitLab: https://salsa.debian.org/med-team/changeo/-/commit/421d53889231d8172b75c692ba7dbe41bd215704
--
View it on GitLab: https://salsa.debian.org/med-team/changeo/-/commit/421d53889231d8172b75c692ba7dbe41bd215704
You're receiving this email because of your account on salsa.debian.org. Manage all notifications: https://salsa.debian.org/-/profile/notifications | Help: https://salsa.debian.org/help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20261115/33492ffe/attachment-0001.htm>
More information about the debian-med-commit
mailing list