[med-svn] [Git][med-team/pycoqc][master] 8 commits: Add autopkgtests
Nilesh Patra
gitlab at salsa.debian.org
Wed Aug 19 15:50:26 BST 2020
Nilesh Patra pushed to branch master at Debian Med / pycoqc
Commits:
e3788918 by Nilesh Patra at 2020-08-19T20:07:10+05:30
Add autopkgtests
- - - - -
45dcaaa5 by Nilesh Patra at 2020-08-19T20:07:33+05:30
Clean up rules file
- - - - -
ac755881 by Nilesh Patra at 2020-08-19T20:11:21+05:30
Fix lintian
- - - - -
e3cca96c by Nilesh Patra at 2020-08-19T20:12:51+05:30
Update changelog
- - - - -
4fceccfd by Nilesh Patra at 2020-08-19T20:14:07+05:30
routine-update: New upstream version
- - - - -
ac4408fc by Nilesh Patra at 2020-08-19T20:14:09+05:30
New upstream version 2.5.0.23+dfsg
- - - - -
de662d96 by Nilesh Patra at 2020-08-19T20:14:55+05:30
Update upstream source from tag 'upstream/2.5.0.23+dfsg'
Update to upstream version '2.5.0.23+dfsg'
with Debian dir baac6f2c09bf23fcd534533b6f7e45705ecc9ac7
- - - - -
34e01da0 by Nilesh Patra at 2020-08-19T20:14:55+05:30
routine-update: debhelper-compat 13
- - - - -
13 changed files:
- README.md
- debian/changelog
- debian/control
- debian/rules
- debian/source/lintian-overrides
- + debian/tests/control
- + debian/tests/run-unit-test
- docs/index.md
- meta.yaml
- pycoQC/Fast5_to_seq_summary.py
- pycoQC/__init__.py
- pycoQC/templates/spectre.html.j2
- setup.py
Changes:
=====================================
README.md
=====================================
@@ -8,9 +8,13 @@
[](https://badge.fury.io/py/pycoQC)
[](https://pepy.tech/project/pycoqc)
+
[](https://anaconda.org/aleg/pycoqc)
[](https://anaconda.org/aleg/pycoqc)
+[](http://bioconda.github.io/recipes/pycoqc/README.html)
+[](https://anaconda.org/bioconda/pycoqc)
+
[](https://travis-ci.com/a-slide/pycoQC)
[](https://www.codacy.com/app/a-slide/pycoQC?utm_source=github.com&utm_medium=referral&utm_content=a-slide/pycoQC&utm_campaign=Badge_Grade)
---
@@ -23,6 +27,8 @@
PycoQC relies on the *sequencing_summary.txt* file generated by Albacore and Guppy, but if needed it can also generates a summary file from basecalled fast5 files. The package supports 1D and 1D2 runs generated with Minion, Gridion and Promethion devices and basecalled with Albacore 1.2.1+ or Guppy 2.1.3+. PycoQC is written in pure Python3. **Python 2 is not supported**.
+Great tutorial with detailed explanations by [Tim Kahlke](https://github.com/timkahlke) available at https://timkahlke.github.io/LongRead_tutorials/QC_P.html
+
## Gallery
=====================================
debian/changelog
=====================================
@@ -1,3 +1,23 @@
+pycoqc (2.5.0.23+dfsg-1) UNRELEASED; urgency=medium
+
+ * Team Upload.
+ [ Andreas Tille ]
+ * Remove empty reference data
+ * Not available in conda
+ * Spelling
+ * Fix conda ID
+
+ [ Steffen Möller ]
+ * Update metadata - only bioconda knows this tool
+
+ [ Nilesh Patra ]
+ * Add autopkgtests
+ * Clean up rules file
+ * Fix lintian
+ * debhelper-compat 13 (routine-update)
+
+ -- Nilesh Patra <npatra974 at gmail.com> Wed, 19 Aug 2020 20:11:31 +0530
+
pycoqc (2.5.0.21+dfsg-3) unstable; urgency=medium
* Remove totally useless test template
=====================================
debian/control
=====================================
@@ -3,7 +3,7 @@ Maintainer: Debian Med Packaging Team <debian-med-packaging at lists.alioth.debian.
Uploaders: Andreas Tille <tille at debian.org>
Section: science
Priority: optional
-Build-Depends: debhelper-compat (= 12),
+Build-Depends: debhelper-compat (= 13),
dh-python,
python3,
python3-setuptools
=====================================
debian/rules
=====================================
@@ -4,28 +4,7 @@
export LC_ALL=C.UTF-8
include /usr/share/dpkg/default.mk
-# this provides:
-# DEB_SOURCE: the source package name
-# DEB_VERSION: the full version of the package (epoch + upstream vers. + revision)
-# DEB_VERSION_EPOCH_UPSTREAM: the package's version without the Debian revision
-# DEB_VERSION_UPSTREAM_REVISION: the package's version without the Debian epoch
-# DEB_VERSION_UPSTREAM: the package's upstream version
-# DEB_DISTRIBUTION: the distribution(s) listed in the current entry of debian/changelog
-# SOURCE_DATE_EPOCH: the source release date as seconds since the epoch, as
-# specified by <https://reproducible-builds.org/specs/source-date-epoch/>
-
-# for hardening you might like to uncomment this:
-# export DEB_BUILD_MAINT_OPTIONS=hardening=+all
+export PYBUILD_NAME=pycoqc
%:
dh $@ --with python3 --buildsystem=pybuild
-
-### When overriding auto_test make sure DEB_BUILD_OPTIONS will be respected
-#override_dh_auto_test:
-#ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
-# do_stuff_for_testing
-#endif
-
-### If you **really** can not use uscan (even not with mode=git) use a debian/get-orig-script
-#get-orig-source:
-# . debian/get-orig-source
=====================================
debian/source/lintian-overrides
=====================================
@@ -1,2 +1,4 @@
# The rendered sample results are no JS without source
pycoqc source: source-is-missing docs/pycoQC/results*
+pycoqc source: very-long-line-length-in-source-file docs/pycoQC/results*
+pycoqc source: source-contains-prebuilt-javascript-object docs/pycoQC/results*
=====================================
debian/tests/control
=====================================
@@ -0,0 +1,4 @@
+Tests: run-unit-test
+Depends: @
+Restrictions: allow-stderr
+
=====================================
debian/tests/run-unit-test
=====================================
@@ -0,0 +1,139 @@
+#!/bin/bash
+set -e
+
+pkg=pycoqc
+CUR_DIR=`pwd`
+
+if [ "${AUTOPKGTEST_TMP}" = "" ] ; then
+ AUTOPKGTEST_TMP=$(mktemp -d /tmp/${pkg}-test.XXXXXX)
+ trap "rm -rf ${AUTOPKGTEST_TMP}" 0 INT QUIT ABRT PIPE TERM
+fi
+
+cp ${CUR_DIR}/docs/Barcode_split/data/* -a "${AUTOPKGTEST_TMP}"
+cp ${CUR_DIR}/docs/Fast5_to_seq_summary/data/* -a "${AUTOPKGTEST_TMP}"
+cp ${CUR_DIR}/docs/pycoQC/data/* -a "${AUTOPKGTEST_TMP}"
+
+cd "${AUTOPKGTEST_TMP}"
+gunzip -r *
+rm -rf data
+mkdir -p datafast5
+mv *.fast5 datafast5
+mv summary_sequencing.tsv datafast5
+
+echo 'Test 1'
+pycoQC \
+ -f Albacore-1.2.1_basecall-1D-DNA_sequencing_summary.txt \
+ -o results/Albacore-1.2.1_basecall-1D-DNA.html \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 2'
+pycoQC \
+ -f Guppy-2.1.3_basecall-1D-RNA_sequencing_summary.txt \
+ -o results/Guppy-2.1.3_basecall-1D_RNA.html \
+ -j results/Guppy-2.1.3_basecall-1D_RNA.json \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 3'
+pycoQC \
+ -f Guppy-2.1.3_basecall-1D-DNA_sequencing_summary.txt \
+ -b Guppy-2.1.3_basecall-1D_DNA_barcoding_summary.txt \
+ -o results/Guppy-2.1.3_basecall-1D_DNA_barcode.html \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 4'
+pycoQC \
+ -f Albacore*RNA* \
+ -o results/Albacore_all_RNA.html \
+ --report_title "All RNA runs" \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 5'
+pycoQC \
+ -f Albacore-2.1.10_basecall-1D-DNA_sequencing_summary.txt\
+ -o results/Albacore-2.1.10_basecall-1D-DNA.html \
+ --min_pass_qual 8 \
+ --min_pass_len 200 \
+ --filter_calibration \
+ --min_barcode_percent 10 \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 6'
+pycoQC --default_config > test6
+diff test6 pycoQC_config.json
+echo 'PASS'
+echo
+
+echo 'Test 7'
+pycoQC \
+ -f Albacore-1.7.0_basecall-1D-DNA_sequencing_summary.txt\
+ -o results/Albacore-1.7.0_basecall-1D-DNA.html \
+ --config pycoQC_config.json \
+ --quiet
+echo 'PASS'
+echo
+
+echo 'Test 8'
+Fast5_to_seq_summary \
+ -f datafast5/ \
+ -s results/summary_sequencing.tsv
+
+head results/summary_sequencing.tsv
+echo 'PASS'
+echo
+
+echo 'Test 9'
+Fast5_to_seq_summary \
+ -f datafast5/ \
+ -s results/custom_summary_sequencing.tsv \
+ --verbose_level 1 \
+ --fields mean_qscore_template called_events duration strand_score
+
+head results/custom_summary_sequencing.tsv
+echo 'PASS'
+echo
+
+echo 'Test 10'
+Fast5_to_seq_summary \
+ -f datafast5/ \
+ -s results/fn_summary_sequencing.tsv \
+ --verbose_level 1 \
+ --include_path
+
+head results/fn_summary_sequencing.tsv
+echo 'PASS'
+echo
+
+echo 'Test 11'
+Barcode_split \
+ -f 'Guppy-2.2.4-basecall-1D-DNA_sequencing_summary+barcode.txt' \
+ -o "results/"
+echo 'PASS'
+echo
+
+echo 'Test 12'
+Barcode_split \
+ -f "Guppy-basecall-1D-DNA_sequencing_summary.txt" \
+ -b "Guppy-basecall-1D-DNA_deepbinner_barcoding_summary.txt" \
+ -o "results/" \
+ -v
+echo 'PASS'
+echo
+
+echo "Check if files really exist"
+for f in results/*
+do
+ [ -s $f ] || exit 1
+done
+echo "PASS"
+echo
+
=====================================
docs/index.md
=====================================
@@ -6,6 +6,7 @@
PycoQC relies on the *sequencing_summary.txt* file generated by Albacore and Guppy, but if needed it can also generates a summary file from basecalled fast5 files. The package supports 1D and 1D2 runs generated with Minion, Gridion and Promethion devices and basecalled with Albacore 1.2.1+ or Guppy 2.1.3+. PycoQC is written in pure Python3. **Python 2 is not supported**.
+Great tutorial with detailed explanations by [Tim Kahlke](https://github.com/timkahlke) available at https://timkahlke.github.io/LongRead_tutorials/QC_P.html
## Gallery
=====================================
meta.yaml
=====================================
@@ -1,4 +1,4 @@
-{% set version = "2.5.0.21" %}
+{% set version = "2.5.0.23" %}
{% set name = "pycoQC" %}
package:
@@ -23,7 +23,7 @@ requirements:
- pip>=19.2.1
- ripgrep>=11.0.1
run:
- - python=3.6
+ - python>=3.6
- numpy=1.17.1
- scipy=1.3.1
- pandas=0.25.1
@@ -45,5 +45,9 @@ test:
about:
home: "https://github.com/a-slide/pycoQC"
- license: "GPLv3"
+ license: "GNU General Public v3 (GPLv3)"
+ license_file: LICENSE
+ license_family: GPL
summary: "PycoQC computes metrics and generates interactive QC plots for Oxford Nanopore technologies sequencing data"
+ doc_url: "https://a-slide.github.io/pycoQC/"
+ dev_url: ""
=====================================
pycoQC/Fast5_to_seq_summary.py
=====================================
@@ -197,54 +197,73 @@ class Fast5_to_seq_summary ():
for fast5_fn in iter(in_q.get, None):
- # Try to extract data from the fast5 file
- d = OrderedDict()
with h5py.File(fast5_fn, "r") as h5_fp:
- # Define group names for current read
- grp_dict = {
- "raw_read" : "/Raw/Reads/{}/".format(list(h5_fp["/Raw/Reads"].keys())[0]),
- "summary_basecall" : "/Analyses/Basecall_1D_{:03}/Summary/basecall_1d_template/".format(self.basecall_id),
- "summary_calibration" : "/Analyses/Calibration_Strand_Detection_{:03}/Summary/calibration_strand_template/".format(self.basecall_id),
- "summary_barcoding" : "/Analyses/Barcoding_{:03}/Summary/barcoding/".format(self.basecall_id),
- "tracking_id" : "UniqueGlobalKey/tracking_id",
- "channel_id" : "UniqueGlobalKey/channel_id"}
-
- # Fetch required fields is available
- for field in self.fields:
-
- # Special case for start time
- if field == "start_time":
- start_time = self._get_h5_attrs (fp=h5_fp,
- grp=grp_dict[self.attrs_grp_dict["start_time"]["grp"]],
- attrs=self.attrs_grp_dict["start_time"]["attrs"])
- sampling_rate = self._get_h5_attrs (fp=h5_fp,
- grp=grp_dict[self.attrs_grp_dict["channel_sampling_rate"]["grp"]],
- attrs=self.attrs_grp_dict["channel_sampling_rate"]["attrs"])
- if start_time and sampling_rate:
- d[field] = int(start_time/sampling_rate)
- c["fields_found"][field] +=1
- else:
- c["fields_not_found"][field] +=1
- # Everything else
- else:
- v = self._get_h5_attrs (
- fp=h5_fp, grp=grp_dict[self.attrs_grp_dict[field]["grp"]], attrs=self.attrs_grp_dict[field]["attrs"])
- if v:
- d[field] = v
- c["fields_found"][field] +=1
- else:
- c["fields_not_found"][field] +=1
+ multi_read = 'file_type' in h5_fp.attrs.keys() and h5_fp.attrs['file_type'] == b'multi-read'
+
+ if multi_read:
+ read_ids = list(h5_fp["/"].keys())
+ else:
+ read_ids = list(h5_fp["/Raw/Reads"].keys())
- if self.include_path:
- d["path"] = os.path.abspath(fast5_fn)
+ for read_id in read_ids:
+ # Try to extract data from the fast5 file
+ d = OrderedDict()
- # Put read data in queue
- if d:
- out_q.put(d)
- c["overall"]["valid files"] += 1
- else:
- c["overall"]["invalid files"] += 1
+ if multi_read:
+ grp_dict = {
+ "raw_read" : "/{}/Raw/".format(read_id),
+ "summary_basecall" : "/{read_id}/Analyses/Basecall_1D_{bc_id:03}/Summary/basecall_1d_template/".format(read_id=read_id, bc_id=self.basecall_id),
+ "summary_calibration" : "/{read_id}/Analyses/Calibration_Strand_Detection_{bc_id:03}/Summary/calibration_strand_template/".format(read_id=read_id, bc_id=self.basecall_id),
+ "summary_barcoding" : "/{read_id}/Analyses/Barcoding_{bc_id:03}/Summary/barcoding/".format(read_id=read_id, bc_id=self.basecall_id),
+ "tracking_id" : "/{}/tracking_id".format(read_id),
+ "channel_id" : "/{}/channel_id".format(read_id)}
+
+ else:
+ # Define group names for current read
+ grp_dict = {
+ "raw_read" : "/Raw/Reads/{}/".format(read_id),
+ "summary_basecall" : "/Analyses/Basecall_1D_{:03}/Summary/basecall_1d_template/".format(self.basecall_id),
+ "summary_calibration" : "/Analyses/Calibration_Strand_Detection_{:03}/Summary/calibration_strand_template/".format(self.basecall_id),
+ "summary_barcoding" : "/Analyses/Barcoding_{:03}/Summary/barcoding/".format(self.basecall_id),
+ "tracking_id" : "UniqueGlobalKey/tracking_id",
+ "channel_id" : "UniqueGlobalKey/channel_id"}
+
+ # Fetch required fields is available
+ for field in self.fields:
+
+ # Special case for start time
+ if field == "start_time":
+ start_time = self._get_h5_attrs (fp=h5_fp,
+ grp=grp_dict[self.attrs_grp_dict["start_time"]["grp"]],
+ attrs=self.attrs_grp_dict["start_time"]["attrs"])
+ sampling_rate = self._get_h5_attrs (fp=h5_fp,
+ grp=grp_dict[self.attrs_grp_dict["channel_sampling_rate"]["grp"]],
+ attrs=self.attrs_grp_dict["channel_sampling_rate"]["attrs"])
+ if start_time and sampling_rate:
+ d[field] = int(start_time/sampling_rate)
+ c["fields_found"][field] +=1
+ else:
+ c["fields_not_found"][field] +=1
+ # Everything else
+ else:
+ v = self._get_h5_attrs (
+ fp=h5_fp, grp=grp_dict[self.attrs_grp_dict[field]["grp"]], attrs=self.attrs_grp_dict[field]["attrs"])
+ if v:
+ d[field] = v
+ c["fields_found"][field] +=1
+ else:
+ c["fields_not_found"][field] +=1
+
+ if self.include_path:
+ d["path"] = os.path.abspath(fast5_fn)
+
+ # Put read data in queue
+ if d:
+ out_q.put(d)
+ c["overall"]["valid files"] += 1
+ else:
+ c["overall"]["invalid files"] += 1
# Put counter in counter queue
counter_q.put(c)
=====================================
pycoQC/__init__.py
=====================================
@@ -1,4 +1,4 @@
# -*- coding: utf-8 -*-
-__version__ = '2.5.0.21'
+__version__ = '2.5.0.23'
__all__ = ["pycoQC", "Fast5_to_seq_summary", "Barcode_split", "common"]
=====================================
pycoQC/templates/spectre.html.j2
=====================================
@@ -68,10 +68,6 @@
{{ src_files }}
</div>
</div>
-
- <!-- <div class="column col-12 col-mx-auto text-center">
- <h6>SHA256: {{ summary_file_hash }}</h6>
- </div> -->
</div>
</div>
</body>
=====================================
setup.py
=====================================
@@ -5,7 +5,7 @@ from setuptools import setup
# Define package info
name = "pycoQC"
-version = "2.5.0.21"
+version = "2.5.0.23"
description = "PycoQC computes metrics and generates interactive QC plots for Oxford Nanopore technologies sequencing data"
with open("README.md", "r") as fh:
long_description = fh.read()
View it on GitLab: https://salsa.debian.org/med-team/pycoqc/-/compare/bd4675360e91d6c48c45ed7eba5124982f08552f...34e01da06c8d88086746b98b637c951bddedf0ff
--
View it on GitLab: https://salsa.debian.org/med-team/pycoqc/-/compare/bd4675360e91d6c48c45ed7eba5124982f08552f...34e01da06c8d88086746b98b637c951bddedf0ff
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20200819/a21c0a5a/attachment-0001.html>
More information about the debian-med-commit
mailing list