[med-svn] [Git][med-team/python-pysam][ariba] Replace faidx-stdout.patch with the upstream fix
Andreas Tille (@tille)
gitlab at salsa.debian.org
Wed Nov 4 10:07:59 GMT 2026
Andreas Tille pushed to branch ariba at Debian Med / python-pysam
Commits:
b1e79724 by Andreas Tille at 2026-11-04T10:44:08+01:00
Replace faidx-stdout.patch with the upstream fix
The previous version worked around the empty pysam.faidx() result by
injecting a synthetic -o option, which needed a heuristic to decide
whether a faidx invocation writes sequences at all, and which clobbered
a -o option given by the caller.
Upstream fixed the cause in commit 6261249228: faidx now uses
bgzf_open(samtools_stdout_fn, ...) like calmd, so pysam's stdout capture
works again, the synthetic -o option is prepended rather than appended so
that the caller's -o wins, and the obsolete "view -c" special case is
dropped. Take that commit as a patch and enable it, since it is now a
proper upstream backport rather than an unreviewed workaround.
- - - - -
3 changed files:
- debian/changelog
- debian/patches/faidx-stdout.patch
- debian/patches/series
Changes:
=====================================
debian/changelog
=====================================
@@ -1,5 +1,10 @@
python-pysam (0.24.1+ds-3) UNRELEASED; urgency=medium
+ * New patch faidx-stdout.patch, taken from upstream: fix pysam.faidx()
+ returning an empty string with the bundled samtools 1.22 and later,
+ which write their output with bgzf_open() instead of through the
+ replaced stdout FILE*. This broke faidx sequence retrieval in ariba
+ (see https://github.com/sanger-pathogens/ariba/issues/349).
* skip-32bit-tests.patch: skip the 64-bit object-size checks
and the PileupColumn access-after-iterator test on 32-bit platforms.
Closes: #1152973
=====================================
debian/patches/faidx-stdout.patch
=====================================
@@ -1,151 +1,130 @@
-Description: Fix pysam.faidx() returning an empty string with samtools >= 1.22
- Since the bundled samtools was updated to 1.22 in pysam 0.24.0, the faidx
- subcommand writes its output via bgzf_open("-") instead of the patched
- stdout FILE*, so the sequence fetched with pysam.faidx(fa, region) was no
- longer captured and the call returned an empty string while the sequence
- leaked to the process' real stdout.
- Capture the output with -o for faidx, like the other samtools commands
- that support it, and route save_stdout requests the same way. Index
- building invocations (no regions) write nothing to stdout, so for those
- the -o option is deliberately not added to avoid redirecting the index
- file into the capture file.
- Add regression tests for the returned wrapped sequence, multiple
- regions, and index-only invocations.
-Author: Andreas Tille <tille at debian.org>
-Assisted-By: https://inference.debian.net/ Model DeepSeek V4 Flash 0731
-Forwarded: no
-Last-Update: 2026-09-22
-Comment: Needs review, not applied yet
+Description: Fix pysam.samtools.faidx() output capture with samtools >= 1.22
+ Since samtools 1.22 the bundled faidx writes its output, like calmd, via
+ bgzf_open() to standard output rather than through the stdout FILE* that
+ pysam replaces, so pysam.faidx(fa, region) returned an empty string and
+ the fetched sequence leaked to the process' real stdout. This broke
+ ariba (sanger-pathogens/ariba#349).
+ Redirect bgzf_open("-") through samtools_stdout_fn the same way calmd
+ does, and map faidx to the pysam_stdout_fn capture mechanism instead of
+ a synthetic -o option.
+ Also prepend rather than append the synthetic -o option, so that a
+ user-supplied -o takes precedence instead of being clobbered, and drop
+ the special case for "view -c", which has printed to stdout rather than
+ through sam_open_format() since samtools 1.4.
+Origin: upstream, https://github.com/pysam-developers/pysam/commit/6261249228
+Bug: https://github.com/sanger-pathogens/ariba/issues/349
+Forwarded: not-needed
+Last-Update: 2026-11-04
+diff --git a/devtools/import.py b/devtools/import.py
+index 2f41163..4e83bff 100644
+--- a/devtools/import.py
++++ b/devtools/import.py
+@@ -107,6 +107,9 @@ def _update_pysam_files(cf, destdir):
+ "bam_md.c": (
+ 'sam_open_format("-", mode_w',
+ 'sam_open_format({}_stdout_fn, mode_w'.format(basename)),
++ "faidx.c": (
++ 'bgzf_open("-", "wu"',
++ 'bgzf_open({}_stdout_fn, "wu"'.format(basename)),
+ "phase.c": (
+ 'putc("ACGT"[f->seq[j] == 1? (c&3, {}_stdout) : (c>>16&3)]);'.format(basename),
+ 'putc("ACGT"[f->seq[j] == 1? (c&3) : (c>>16&3)], {}_stdout);'.format(basename)),
+diff --git a/pysam/libcutils.pyx b/pysam/libcutils.pyx
+index f8c13aa..cf6cd3c 100644
--- a/pysam/libcutils.pyx
+++ b/pysam/libcutils.pyx
-@@ -300,6 +300,57 @@ cdef int libc_whence_from_io(int whence):
- return whence # Otherwise likely invalid, but let HTSlib or OS report it
-
-
-+# ``samtools faidx`` options that consume the following argument as
-+# their value. Used to count positional arguments (input file plus
-+# regions) when deciding whether a faidx call has output to capture.
-+_FAIDX_VALUE_OPTIONS = frozenset((
-+ "-o", "-n", "-r", "-@",
-+ "--output", "--length", "--region-file", "--fai-idx", "--gzi-idx",
-+ "--mark-strand", "--input-fmt", "--input-fmt-option", "--output-fmt",
-+ "--output-fmt-option", "--reference", "--threads", "--verbosity",
-+))
-+
-+
-+def _faidx_fetches_to_stdout(args):
-+ """Return True if a ``samtools faidx`` invocation writes sequences to
-+ stdout, i.e. it fetches regions and does not pass an explicit -o.
-+
-+ faidx builds the index when given a single positional argument (the
-+ input file) and no regions; such calls write nothing to stdout.
-+ Fetching sequences is done by giving regions as further positional
-+ arguments or via -r/--region-file. Only in these cases is output
-+ produced, and only if the caller did not already provide -o.
-+ """
-+ n_positional = 0
-+ skip_next = False
-+ has_output = False
-+ has_regions = False
-+ for arg in args:
-+ if skip_next:
-+ skip_next = False
-+ continue
-+ if arg == "-o" or arg.startswith("-o"):
-+ has_output = True
-+ skip_next = True
-+ continue
-+ if arg == "--output" or arg.startswith("--output="):
-+ has_output = True
-+ continue
-+ if arg == "-r" or arg.startswith("-r"):
-+ has_regions = True
-+ skip_next = True
-+ continue
-+ if arg == "--region-file" or arg.startswith("--region-file="):
-+ has_regions = True
-+ continue
-+ if arg.startswith("-") and arg != "-":
-+ if arg in _FAIDX_VALUE_OPTIONS:
-+ skip_next = True
-+ continue
-+ n_positional += 1
-+ return not has_output and (has_regions or n_positional > 1)
-+
-+
- def _pysam_dispatch(collection,
- method,
- args=None,
-@@ -352,6 +403,12 @@ def _pysam_dispatch(collection,
- stdout_h = c_open(stdout_f_bytes, O_WRONLY|O_CREAT|O_TRUNC, 0o666)
- if stdout_h == -1:
- raise OSError_from_errno("Could not redirect standard output", stdout_f)
-+ # samtools faidx writes via bgzf_open("-") to the real stdout
-+ # rather than the patched stdout FILE*, so route it to the
-+ # requested file with -o instead
-+ if collection == "samtools" and method == "faidx" \
-+ and _faidx_fetches_to_stdout(args):
-+ args.extend(["-o", stdout_f])
-
- elif catch_stdout:
- stdout_h, stdout_f = tempfile.mkstemp()
-@@ -361,6 +418,9 @@ def _pysam_dispatch(collection,
+@@ -361,6 +361,7 @@ def _pysam_dispatch(collection,
"mpileup": "-o {}",
"depad": "-o {}",
"calmd": "", # uses pysam_stdout_fn
-+ # since samtools 1.22, faidx writes via bgzf_open("-") rather
-+ # than the patched stdout FILE*, so capture with -o instead
-+ "faidx": "-o {}",
++ "faidx": "", # uses pysam_stdout_fn
},
"bcftools": {}
}
-@@ -375,6 +435,11 @@ def _pysam_dispatch(collection,
- # special case - samtools view -c outputs on stdout
- if not(method == "view" and "-c" in args):
- stdout_option = MAP_STDOUT_OPTIONS[collection][method]
-+ # special case - samtools faidx only produces output when
-+ # fetching regions; when it merely builds the index, or when
-+ # the caller already passes -o, nothing should be captured
-+ if method == "faidx" and not _faidx_fetches_to_stdout(args):
-+ stdout_option = None
+@@ -372,14 +373,12 @@ def _pysam_dispatch(collection,
+ if method not in ("head", "index", "roh", "stats"):
+ stdout_option = "-o {}"
+ elif method in MAP_STDOUT_OPTIONS[collection]:
+- # special case - samtools view -c outputs on stdout
+- if not(method == "view" and "-c" in args):
+- stdout_option = MAP_STDOUT_OPTIONS[collection][method]
++ stdout_option = MAP_STDOUT_OPTIONS[collection][method]
if stdout_option is not None and not is_usage:
os.close(stdout_h)
---- a/tests/test_samtools_python.py
-+++ b/tests/test_samtools_python.py
-@@ -48,3 +48,34 @@ def test_bedcov():
- assert len(fields) in [4, 5], \
- ("bedcov should give tab delimited output with 4 or 5 fields. "
- "Split line (%s) gives %d fields." % (fields, len(fields)))
+ stdout_f_bytes = force_bytes(stdout_f)
+- args.extend(stdout_option.format(stdout_f).split(" "))
++ args[0:0] = stdout_option.format(stdout_f).split(maxsplit=1)
+ stdout_h = c_open(b"/dev/null", O_WRONLY)
+ else:
+ stdout_f_bytes = None
+diff --git a/samtools/faidx.c.pysam.c b/samtools/faidx.c.pysam.c
+index 014d7b6..08c6948 100644
+--- a/samtools/faidx.c.pysam.c
++++ b/samtools/faidx.c.pysam.c
+@@ -613,7 +613,7 @@ int faidx_core(int argc, char *argv[], enum fai_format_options format)
+ }
+ }
+ } else {
+- if (!(out.bgzf_fp = bgzf_open("-", "wu"))) {
++ if (!(out.bgzf_fp = bgzf_open(samtools_stdout_fn, "wu"))) {
+ fprintf(samtools_stderr,"[faidx] Cannot open output for writing :%s.\n", strerror(errno) );
+ goto exit2;
+ }
+diff --git a/tests/samtools_test.py b/tests/samtools_test.py
+index 050ac27..cb99fd6 100644
+--- a/tests/samtools_test.py
++++ b/tests/samtools_test.py
+@@ -1,4 +1,5 @@
+ import warnings
++import json
+ import os
+ import re
+ import glob
+@@ -329,6 +330,45 @@ class TestStdout:
+ assert len(r) > 0
+
+
++class TestSamtoolsCapture:
++ """Test that each subcommand's standard output is correctly captured.
++ """
++ @pytest.mark.skipif(pysam.__version__ < "0.24.0", reason="calmd redirection broken")
++ def test_calmd(self):
++ text = pysam.samtools.calmd("-Q", os.path.join(BAM_DATADIR, "ex1.bam"), os.path.join(BAM_DATADIR, "ex1.fa"))
++ assert "EAS56_57:6:190:289:82" in text
+
++ def test_depad(self):
++ text = pysam.samtools.depad("-s", "-T", os.path.join(BAM_DATADIR, "ex1.fa"), os.path.join(BAM_DATADIR, "ex5.bam"))
++ assert "read_28833_29006_6945" in text
+
-+def _faidx_filename():
-+ return os.path.join(BAM_DATADIR, "ex1.fa")
++ def test_faidx(self):
++ seq = pysam.samtools.faidx(os.path.join(BAM_DATADIR, "ex1.fa"), "chr2:101-120")
++ assert seq == ">chr2:101-120\nGAACTTGAAGACAAGTCTCT\n"
+
++ def test_flags(self):
++ text = pysam.samtools.flags("16")
++ assert text == "0x10\t16\tREVERSE\n"
+
-+def test_faidx_fetch_returns_sequence():
-+ # Regression test: since pysam 0.24.0 (bundled samtools 1.22) faidx
-+ # writes via bgzf_open() instead of the patched stdout FILE*, so the
-+ # fetched sequence must be captured with -o. Before the fix this
-+ # returned an empty string.
-+ seq = pysam.faidx(_faidx_filename(), "chr1")
-+ assert seq.startswith(">chr1\n")
-+ lines = seq.splitlines()
-+ # samtools faidx line-wraps at 60 characters
-+ assert len(lines[1]) == 60
-+ assert all(len(l) == 60 for l in lines[1:-1])
-+ with pysam.FastaFile(_faidx_filename()) as fasta:
-+ assert "".join(lines[1:]) == fasta.fetch("chr1")
++ def test_flagstat(self):
++ stats = json.loads(pysam.samtools.flagstat("-Ojson", os.path.join(BAM_DATADIR, "ex1.bam")))
++ assert stats["QC-passed reads"]["properly paired"] == 3124
+
++ def test_mpileup(self):
++ text = pysam.samtools.mpileup("--region", "chr2:500-501", os.path.join(BAM_DATADIR, "ex1.bam"))
++ assert "chr2\t500\tN\t44" in text
+
-+def test_faidx_fetch_multiple_regions():
-+ out = pysam.faidx(_faidx_filename(), "chr2", "chr1")
-+ assert [l for l in out.splitlines() if l.startswith(">")] == [">chr2", ">chr1"]
++ def test_view(self):
++ line = pysam.samtools.view(os.path.join(BAM_DATADIR, "ex1.bam"), "chr2:500-501", split_lines=True)
++ assert len(line) == 47
++ assert "EAS1_93:7:319:280:57" in line[0]
++ assert "EAS192_3:6:45:183:25" in line[-1]
+
++ def test_view_count(self):
++ line = pysam.samtools.view("--count", os.path.join(BAM_DATADIR, "ex1.bam"), "chr2:500-501")
++ assert line.rstrip() == "47"
+
-+def test_faidx_index_only_returns_empty():
-+ # Building the index writes no sequences to stdout, so nothing should
-+ # be captured (and no stray index file should be left behind).
-+ assert pysam.faidx(_faidx_filename()) == ""
+
+ class TestPysam(TestSamtools):
+ """check access to samtools command in the pysam
+ main package.
=====================================
debian/patches/series
=====================================
@@ -1,5 +1,4 @@
# skip_test_remote.patch
clean_less
-# faidx-stdout.patch # needs human review first
+faidx-stdout.patch
skip-32bit-tests.patch
-
View it on GitLab: https://salsa.debian.org/med-team/python-pysam/-/commit/b1e797242081171d6ab29b29401437cbf1fa7d4d
--
View it on GitLab: https://salsa.debian.org/med-team/python-pysam/-/commit/b1e797242081171d6ab29b29401437cbf1fa7d4d
You're receiving this email because of your account on salsa.debian.org. Manage all notifications: https://salsa.debian.org/-/profile/notifications | Help: https://salsa.debian.org/help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20261104/990dcc51/attachment-0001.htm>
More information about the debian-med-commit
mailing list