[med-svn] [Git][med-team/conda-package-streaming][upstream] New upstream version 0.8.0
Andreas Tille (@tille)
gitlab at salsa.debian.org
Tue Jul 11 16:59:21 BST 2023
Andreas Tille pushed to branch upstream at Debian Med / conda-package-streaming
Commits:
4375d90f by Andreas Tille at 2023-07-11T17:36:38+02:00
New upstream version 0.8.0
- - - - -
21 changed files:
- − .github/workflows/main.yml
- + .github/workflows/tests.yml
- .pre-commit-config.yaml
- + CHANGELOG.md
- + CODE_OF_CONDUCT.md
- README.md
- conda.recipe/meta.yaml
- conda_package_streaming/__init__.py
- conda_package_streaming/extract.py
- conda_package_streaming/lazy_wheel.py
- conda_package_streaming/transmute.py
- conda_package_streaming/url.py
- + docs/changelog.md
- docs/index.md
- pyproject.toml
- − renovate.json
- requirements.txt
- tests/conftest.py
- tests/server.py
- tests/test_transmute.py
- tests/test_url.py
Changes:
=====================================
.github/workflows/main.yml deleted
=====================================
@@ -1,18 +0,0 @@
-name: Pre-commit
-on:
- pull_request:
-
-jobs:
- pre_commit:
- runs-on: ubuntu-latest
- steps:
- - name: Retrieve the source code
- uses: actions/checkout at 2541b1294d2704b0964813337f33b291d3f8596b # tag=v3
- - name: Install build dependencies
- run: |
- source $CONDA/bin/activate
- conda install -y conda-forge::pre-commit
- - name: Run pre-commit
- run: |
- source $CONDA/bin/activate
- pre-commit run --all-files
=====================================
.github/workflows/tests.yml
=====================================
@@ -0,0 +1,100 @@
+name: Tests
+
+on:
+ # NOTE: github.event context is push payload:
+ # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push
+ push:
+ branches:
+ - main
+ - feature/**
+
+ # NOTE: github.event context is pull_request payload:
+ # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request
+ pull_request:
+
+concurrency:
+ # Concurrency group that uses the workflow name and PR number if available
+ # or commit SHA as a fallback. If a new build is triggered under that
+ # concurrency group while a previous build is running it will be canceled.
+ # Repeated pushes to a PR will cancel all previous builds, while multiple
+ # merges to main will not cancel.
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+ cancel-in-progress: true
+
+jobs:
+ linux:
+ runs-on: ubuntu-latest
+ defaults:
+ run:
+ shell: bash -l {0}
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ['3.10', '3.11']
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout at v2
+ with:
+ fetch-depth: 0
+
+ - uses: actions/setup-python at v4
+ with:
+ python-version: ${{ matrix.python-version }}
+ architecture: "x64"
+ cache: "pip"
+
+ - name: Setup Miniconda
+ uses: conda-incubator/setup-miniconda at v2
+ with:
+ python-version: ${{ matrix.python-version }}
+ channels: defaults
+ activate-environment: test_env
+ auto-update-conda: false
+ auto-activate-base: false
+ show-channel-urls: true
+
+ - name: Source Scripts
+ run: |
+ set -x
+ # conda is our test dependency but can't be pip installed
+ conda install --quiet conda pip
+ pip install -e .[test]
+ conda info --json
+ echo "condarc"
+ cat ~/.condarc
+ echo "conda_pkgs_dir"
+ ls /home/runner/conda_pkgs_dir
+ echo "miniconda/pkgs"
+ ls /usr/share/miniconda/pkgs
+ echo "test_env"
+ ls /usr/share/miniconda/envs/test_env
+ pytest
+
+ analyze:
+ name: Analyze test results
+ needs: [linux]
+ if: always()
+ runs-on: ubuntu-latest
+ steps:
+ - name: Download test results
+ uses: actions/download-artifact at v3
+
+ - name: Upload combined test results
+ # provides one downloadable archive of all .coverage/test-report.xml files
+ # of all matrix runs for further analysis.
+ uses: actions/upload-artifact at v3
+ with:
+ name: test-results-${{ github.sha }}-all
+ path: test-results-${{ github.sha }}-*
+ retention-days: 90 # default: 90
+
+ - name: Test Summary
+ uses: test-summary/action at v2
+ with:
+ paths: ./test-results-${{ github.sha }}-**/test-report*.xml
+
+ - name: Decide whether the needed jobs succeeded or failed
+ uses: re-actors/alls-green at release/v1
+ with:
+ jobs: ${{ toJSON(needs) }}
=====================================
.pre-commit-config.yaml
=====================================
@@ -1,7 +1,9 @@
-exclude: ^(versioneer.py|conda_pack/_version.py|build)
+# disable autofixing PRs, commenting "pre-commit.ci autofix" on a pull request triggers a autofix
+ci:
+ autofix_prs: false
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v4.3.0
+ rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-ast
@@ -17,17 +19,20 @@ repos:
- id: trailing-whitespace
- id: check-yaml
exclude: conda.recipe/meta.yaml
- - id: check-merge-conflict
- repo: https://github.com/asottile/pyupgrade
- rev: v2.34.0
+ rev: v3.3.1
hooks:
- id: pyupgrade
args: ["--py37-plus"]
- repo: https://github.com/PyCQA/isort
- rev: 5.10.1
+ rev: 5.12.0
hooks:
- id: isort
+ - repo: https://github.com/psf/black
+ rev: 23.1.0
+ hooks:
+ - id: black
- repo: https://github.com/PyCQA/flake8
- rev: 4.0.1
+ rev: 6.0.0
hooks:
- id: flake8
=====================================
CHANGELOG.md
=====================================
@@ -0,0 +1,8 @@
+[//]: # (current developments)
+
+## 0.8.0 (2023-05)
+
+* Update transmute to use SpooledTemporaryFile instead of streaming directly to
+ zip [(#57)](https://github.com/conda/conda-package-streaming/issues/57). This
+ can reduce zstd memory usage during decompression.
+* `transmute` returns Path to transmuted package instead of `None`.
=====================================
CODE_OF_CONDUCT.md
=====================================
@@ -0,0 +1,20 @@
+# Conda Organization Code of Conduct
+
+> **Note**
+> Below is the short version of our CoC, see the long version [here](https://github.com/conda-incubator/governance/blob/main/CODE_OF_CONDUCT.md).
+
+# The Short Version
+
+Be kind to others. Do not insult or put down others. Behave professionally. Remember that harassment and sexist, racist, or exclusionary jokes are not appropriate for the conda Organization.
+
+All communication should be appropriate for a professional audience including people of many different backgrounds. Sexual language and imagery is not appropriate.
+
+The conda Organization is dedicated to providing a harassment-free community for everyone, regardless of gender, sexual orientation, gender identity and expression, disability, physical appearance, body size, race, or religion. We do not tolerate harassment of community members in any form.
+
+Thank you for helping make this a welcoming, friendly community for all.
+
+## Report an Incident
+
+* Report a code of conduct incident [using a form](https://form.jotform.com/221527028480048).
+* Report a code of conduct incident via email: [conduct at conda.org](mailto:conduct at conda.org).
+* Contact [an individual committee member](#committee-membership) or [CoC event representative](#coc-representatives) to report an incident in confidence.
=====================================
README.md
=====================================
@@ -1,5 +1,10 @@
# conda-package-streaming
+[](https://results.pre-commit.ci/latest/github/conda/conda-package-streaming/main)
+
+An efficient library to read from new and old format .conda and .tar.bz2 conda
+packages.
+
Download conda metadata from packages without transferring entire file. Get
metadata from local `.tar.bz2` packages without reading entire files.
@@ -64,6 +69,10 @@ with closing(conda):
break
```
+If you need the entire package, download it first and use the file-based APIs.
+The URL-based APIs are more efficient if you only need to access package
+metadata.
+
# Package goals
* Extract conda packages (both formats)
=====================================
conda.recipe/meta.yaml
=====================================
@@ -39,8 +39,9 @@ test:
about:
home: https://github.com/conda/conda-package-streaming
- summary: Download metadata from conda packages without transferring entire file.
+ summary: An efficient library to read from new and old format .conda and .tar.bz2 conda packages.
license: BSD-3-Clause
+ license_family: BSD
license_file: LICENSE
doc_url: https://conda.github.io/conda-package-streaming/
dev_url: https://github.com/conda/conda-package-streaming
=====================================
conda_package_streaming/__init__.py
=====================================
@@ -1 +1 @@
-__version__ = "0.7.0"
+__version__ = "0.8.0"
=====================================
conda_package_streaming/extract.py
=====================================
@@ -35,7 +35,6 @@ def extract_stream(
return prefix == dest_dir
for tar_file, _ in stream:
-
# careful not to seek backwards
def checked_members():
# from conda_package_handling
=====================================
conda_package_streaming/lazy_wheel.py
=====================================
@@ -38,7 +38,6 @@ class LazyZipOverHTTP:
def __init__(
self, url: str, session: Session, chunk_size: int = CONTENT_CHUNK_SIZE
) -> None:
-
# if CONTENT_CHUNK_SIZE is bigger than the file:
# In [8]: response.headers["Content-Range"]
# Out[8]: 'bytes 0-3133374/3133375'
=====================================
conda_package_streaming/transmute.py
=====================================
@@ -13,11 +13,13 @@ the `ZipFile`, instead of the first for normal conda packages.
from __future__ import annotations
-import io
import json
import os
+import shutil
import tarfile
+import tempfile
import zipfile
+from pathlib import Path
from typing import Callable
import zstandard
@@ -25,11 +27,14 @@ import zstandard
# streams everything in .tar.bz2 mode
from .package_streaming import CondaComponent, stream_conda_component
-# increase to reduce speed and increase compression (22 = conda's default)
-ZSTD_COMPRESS_LEVEL = 22
+# increase to reduce speed and increase compression (levels above 19 use much
+# more memory)
+ZSTD_COMPRESS_LEVEL = 19
# increase to reduce compression and increase speed
ZSTD_COMPRESS_THREADS = 1
+CONDA_PACKAGE_FORMAT_VERSION = 2
+
def transmute(
package,
@@ -41,7 +46,7 @@ def transmute(
level=ZSTD_COMPRESS_LEVEL, threads=ZSTD_COMPRESS_THREADS
),
is_info: Callable[[str], bool] = lambda filename: filename.startswith("info/"),
-):
+) -> Path:
"""
Convert .tar.bz2 conda :package to .conda-format under path.
@@ -54,33 +59,21 @@ def transmute(
(not this package ``conda-package-streaming``) uses a set of regular
expressions to keep expected items in the info- component, while other
items starting with ``info/`` wind up in the pkg- component.
+
+ :return: Path to transmuted package.
"""
assert package.endswith(".tar.bz2"), "can only convert .tar.bz2 to .conda"
assert os.path.isdir(path)
file_id = os.path.basename(package)[: -len(".tar.bz2")]
-
- # x to not append to existing
- with zipfile.ZipFile(
- os.path.join(path, f"{file_id}.conda"), "x", compresslevel=zipfile.ZIP_STORED
- ) as conda_file:
-
- info_compress = compressor()
- data_compress = compressor()
-
- # in theory, info_tar could grow uncomfortably big, in which case we would
- # rather swap it to disk
- info_io = io.BytesIO()
- info_stream = info_compress.stream_writer(info_io, closefd=False)
- info_tar = tarfile.TarFile(fileobj=info_stream, mode="w")
-
- conda_file.writestr(
- "metadata.json", json.dumps({"conda_pkg_format_version": 2})
- )
-
- with conda_file.open(f"pkg-{file_id}.tar.zst", "w") as pkg_file:
- pkg_stream = data_compress.stream_writer(pkg_file, closefd=False)
- pkg_tar = tarfile.TarFile(fileobj=pkg_stream, mode="w")
-
+ output_path = Path(path, f"{file_id}.conda")
+
+ with tempfile.SpooledTemporaryFile() as info_file, tempfile.SpooledTemporaryFile() as pkg_file:
+ with tarfile.TarFile(fileobj=info_file, mode="w") as info_tar, tarfile.TarFile(
+ fileobj=pkg_file, mode="w"
+ ) as pkg_tar:
+ # If we wanted to compress these at a low setting to save temporary
+ # space, we could insert a file object that counts bytes written in
+ # front of a zstd (level between 1..3) compressor.
stream = iter(stream_conda_component(package))
for tar, member in stream:
tar_get = info_tar if is_info(member.name) else pkg_tar
@@ -89,20 +82,47 @@ def transmute(
else:
tar_get.addfile(member)
+ info_tar.close()
pkg_tar.close()
- pkg_stream.close()
- info_tar.close()
- info_stream.close()
+ info_size = info_file.tell()
+ pkg_size = pkg_file.tell()
- with conda_file.open(f"info-{file_id}.tar.zst", "w") as info_file:
- info_file.write(info_io.getvalue())
+ info_file.seek(0)
+ pkg_file.seek(0)
+
+ with zipfile.ZipFile(
+ output_path,
+ "x", # x to not append to existing
+ compresslevel=zipfile.ZIP_STORED,
+ ) as conda_file:
+ # Use a maximum of one Zstd compressor, stream_writer at a time to save memory.
+ data_compress = compressor()
+
+ pkg_metadata = {"conda_pkg_format_version": CONDA_PACKAGE_FORMAT_VERSION}
+ conda_file.writestr("metadata.json", json.dumps(pkg_metadata))
+
+ with conda_file.open(
+ f"pkg-{file_id}.tar.zst", "w"
+ ) as pkg_file_zip, data_compress.stream_writer(
+ pkg_file_zip, size=pkg_size, closefd=False
+ ) as pkg_stream:
+ shutil.copyfileobj(pkg_file._file, pkg_stream)
+
+ with conda_file.open(
+ f"info-{file_id}.tar.zst", "w"
+ ) as info_file_zip, data_compress.stream_writer(
+ info_file_zip, size=info_size, closefd=False
+ ) as info_stream:
+ shutil.copyfileobj(info_file._file, info_stream)
+
+ return output_path
def transmute_tar_bz2(
- package,
+ package: str,
path,
-):
+) -> Path:
"""
Convert .conda :package to .tar.bz2 format under path.
@@ -110,6 +130,8 @@ def transmute_tar_bz2(
:param package: path to `.conda` or `.tar.bz2` package.
:param path: destination path for transmuted package.
+
+ :return: Path to transmuted package.
"""
assert package.endswith((".tar.bz2", ".conda")), "Unknown extension"
assert os.path.isdir(path)
@@ -125,9 +147,9 @@ def transmute_tar_bz2(
# .tar.bz2 doesn't filter by component
components = [CondaComponent.pkg]
- with open(package, "rb") as fileobj, tarfile.open(
- os.path.join(path, f"{file_id}.tar.bz2"), "x:bz2"
- ) as pkg_tar:
+ output_path = Path(path, f"{file_id}.tar.bz2")
+
+ with open(package, "rb") as fileobj, tarfile.open(output_path, "x:bz2") as pkg_tar:
for component in components:
stream = iter(stream_conda_component(package, fileobj, component=component))
for tar, member in stream:
@@ -135,3 +157,5 @@ def transmute_tar_bz2(
pkg_tar.addfile(member, tar.extractfile(member))
else:
pkg_tar.addfile(member)
+
+ return output_path
=====================================
conda_package_streaming/url.py
=====================================
@@ -2,6 +2,11 @@
Fetch metadata from remote .conda or .tar.bz2 package.
Try to fetch less than the whole file if possible.
+
+This module should only be used to make *partial* reads against a remote
+package, typically just the ``info`` portion. If a full ``.conda`` format
+package is needed, it is more efficient to download locally first and then use
+the file-based API.
"""
import logging
@@ -32,7 +37,7 @@ def extract_conda_info(url, destdir, checklist=METADATA_CHECKLIST, session=sessi
"""
checklist = set(checklist)
stream = stream_conda_info(url, session=session)
- for (tar, member) in stream:
+ for tar, member in stream:
if member.name in checklist:
tar.extract(member, destdir)
checklist.remove(member.name)
=====================================
docs/changelog.md
=====================================
@@ -0,0 +1,3 @@
+# Changelog
+```{include} ../CHANGELOG.md
+```
=====================================
docs/index.md
=====================================
@@ -30,6 +30,7 @@ zstd-compressed streams.
:caption: 'Contents:'
:maxdepth: 2
modules
+changelog
```
# Indices and tables
=====================================
pyproject.toml
=====================================
@@ -13,7 +13,7 @@ name = "conda_package_streaming"
authors = [
{ name = "Anaconda, Inc. & Contributors", email = "conda at continuum.io" },
]
-description = "Download metadata from conda packages without transferring entire file."
+description = "An efficient library to read from new and old format .conda and .tar.bz2 conda packages."
license = { file = "LICENSE" }
readme = "README.md"
classifiers = [
@@ -34,6 +34,8 @@ test = [
"boto3",
"boto3-stubs[essential]",
"bottle",
+ "conda",
+ "conda-package-handling >=2",
]
docs = ["furo", "sphinx", "myst-parser", "mdit-py-plugins>=0.3.0"]
=====================================
renovate.json deleted
=====================================
@@ -1,6 +0,0 @@
-{
- "$schema": "https://docs.renovatebot.com/renovate-schema.json",
- "extends": [
- "github>anaconda/renovate-config"
- ]
-}
=====================================
requirements.txt
=====================================
@@ -7,6 +7,7 @@ pytest-mock
boto3
boto3-stubs[essential]
bottle
+conda
# docs
furo
sphinx
=====================================
tests/conftest.py
=====================================
@@ -1,21 +1,101 @@
+import json
+import logging
+import os.path
+import shutil
+import subprocess
from pathlib import Path
import pytest
import server
+from conda_package_streaming.transmute import transmute_tar_bz2
+
+log = logging.getLogger(__name__)
+
+
+LIMIT_TEST_PACKAGES = 16
+
+
+def find_packages_dirs() -> Path:
+ """
+ Ask conda for package directories.
+ """
+ conda_info = json.loads(
+ subprocess.run(
+ [os.environ["CONDA_EXE"], "info", "--json"],
+ stdout=subprocess.PIPE,
+ check=True,
+ ).stdout
+ )
+
+ # XXX can run individual environment's conda (base conda is more likely to
+ # have useful cached packages)
+ pkgs_dirs = conda_info["pkgs_dirs"] + [os.path.expanduser("~/miniconda3/pkgs")]
+
+ log.debug("search %s", pkgs_dirs)
+
+ first_pkg_dir = next(path for path in pkgs_dirs if os.path.exists(path))
+
+ return Path(first_pkg_dir)
+
+
+ at pytest.fixture(scope="session")
+def pkgs_dir(tmp_path_factory):
+ """
+ Dedicated test package directory.
+ """
+ return tmp_path_factory.mktemp("pkgs")
+
@pytest.fixture(scope="session")
-def package_server():
- thread = server.get_server_thread()
+def package_server(pkgs_dir, conda_paths):
+ thread = server.get_server_thread(pkgs_dir)
thread.start()
return thread
@pytest.fixture(scope="session")
-def conda_paths(package_server):
- pkgs_dir = Path(package_server.app.pkgs_dir)
+def conda_paths(pkgs_dir: Path):
+ found_packages = find_packages_dirs()
conda_paths = []
- for path in pkgs_dir.iterdir():
+ for path in found_packages.iterdir():
if path.name.endswith((".tar.bz2", ".conda")):
conda_paths.append(path)
- return conda_paths
+
+ return add_tar_bz2s(conda_paths, pkgs_dir)
+
+
+def add_tar_bz2s(paths: list[Path], pkgs_dir: Path):
+ """
+ If there aren't enough .tar.bz2's available, create some from available
+ .conda's. Return paths.
+ """
+ conda_paths: list[Path] = []
+ tarbz2_paths: list[Path] = []
+ output_paths: list[Path] = []
+
+ assert isinstance(pkgs_dir, Path)
+
+ for path in paths:
+ if path.name.endswith(".tar.bz2"):
+ tarbz2_paths.append(path)
+ elif path.name.endswith(".conda"):
+ conda_paths.append(path)
+
+ tarbz2_path: Path = pkgs_dir
+
+ medium_conda_paths = []
+ for path in conda_paths:
+ if 1 << 20 < path.stat().st_size < 1 << 22:
+ medium_conda_paths.append(path)
+ medium_conda_paths = medium_conda_paths[:LIMIT_TEST_PACKAGES]
+
+ # this ignores existing .tar.bz2 for simplicity (.tar.bz2 is missing in CI)
+ for conda in set(medium_conda_paths + conda_paths[:10]):
+ shutil.copy(conda, tarbz2_path)
+ transmute_tar_bz2(str(conda), tarbz2_path)
+
+ output_paths.extend(tarbz2_path.glob("*.tar.bz2"))
+ output_paths.extend(tarbz2_path.glob("*.conda"))
+
+ return sorted(output_paths) # sort interleaves .tar.bz2 and .conda
=====================================
tests/server.py
=====================================
@@ -1,51 +1,25 @@
"""
Test web server.
"""
-import json
import logging
-import os
-import os.path
-import subprocess
import threading
import wsgiref.simple_server
+from pathlib import Path
from typing import Any
import bottle
+import conftest
log = logging.getLogger(__name__)
-def find_packages_dirs():
- """
- Ask conda for package directories.
- """
- conda_info = json.loads(
- subprocess.run(
- [os.environ["CONDA_EXE"], "info", "--json"],
- stdout=subprocess.PIPE,
- check=True,
- ).stdout
- )
-
- # XXX can run individual environment's conda (base conda is more likely to
- # have useful cached packages)
- pkgs_dirs = conda_info["pkgs_dirs"] + [os.path.expanduser("~/miniconda3/pkgs")]
-
- log.debug("search %s", pkgs_dirs)
-
- first_pkg_dir = next(path for path in pkgs_dirs if os.path.exists(path))
-
- return first_pkg_dir
-
-
-def get_app():
+def get_app(pkgs_dir):
"""
Bottle conveniently supports Range requests.
Server may block if browser etc. keeps connection open.
"""
app = bottle.Bottle()
- pkgs_dir = find_packages_dirs()
app.pkgs_dir = pkgs_dir
def serve_file(filename):
@@ -65,7 +39,7 @@ def selftest():
"""
Run server in a thread that will die when the application exits.
"""
- t = get_server_thread()
+ t = get_server_thread(conftest.find_packages_dirs())
t.start()
import time
@@ -78,13 +52,13 @@ class ServerThread(threading.Thread):
app: Any
-def get_server_thread():
+def get_server_thread(pkgs_dir: Path):
"""
Return test server thread with additional .server, .app properties.
Call .start() to serve in the background.
"""
- app = get_app()
+ app = get_app(pkgs_dir)
server = wsgiref.simple_server.make_server("127.0.0.1", 0, app)
log.info(f"serving {app.pkgs_dir} on {server.server_address}/pkgs")
t = ServerThread(daemon=True, target=server.serve_forever)
=====================================
tests/test_transmute.py
=====================================
@@ -4,8 +4,10 @@ import os
import tarfile
import time
from pathlib import Path
+from zipfile import ZipFile
import pytest
+from conda_package_handling.validate import validate_converted_files_match_streaming
from conda_package_streaming.package_streaming import (
CondaComponent,
@@ -38,8 +40,7 @@ def timeme(message: str = ""):
print(f"{message}{end-begin:0.2f}s")
-def test_transmute(conda_paths, tmpdir):
-
+def test_transmute(conda_paths: list[Path], tmpdir):
tarbz_packages = []
for path in conda_paths:
path = str(path)
@@ -49,17 +50,33 @@ def test_transmute(conda_paths, tmpdir):
assert tarbz_packages, "no medium-sized .tar.bz2 packages found"
+ metadata_checks = 0
+
for packages in (conda_packages, tarbz_packages):
for package in packages:
with timeme(f"{package} took "):
- transmute(package, tmpdir)
+ out = transmute(package, tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, package, strict=True
+ )
+ assert missing == mismatched == []
+ if out.name.endswith(".conda"):
+ with ZipFile(out) as zf:
+ metadata_checks += 1
+ assert "metadata.json" in zf.namelist()
+
+ assert metadata_checks > 0
def test_transmute_symlink(tmpdir, testtar_bytes):
testtar = Path(tmpdir, "test.tar.bz2")
testtar.write_bytes(testtar_bytes)
- transmute(str(testtar), tmpdir)
+ out = transmute(str(testtar), tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
def test_transmute_info_filter(tmpdir, testtar_bytes):
@@ -83,7 +100,6 @@ def test_transmute_info_filter(tmpdir, testtar_bytes):
def test_transmute_backwards(tmpdir, conda_paths):
-
tarbz_packages = []
for path in conda_paths:
path = str(path)
@@ -96,7 +112,11 @@ def test_transmute_backwards(tmpdir, conda_paths):
for packages in (conda_packages, tarbz_packages):
for package in packages:
with timeme(f"{package} took "):
- transmute_tar_bz2(package, tmpdir)
+ out = transmute_tar_bz2(package, tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, package, strict=True
+ )
+ assert missing == mismatched == []
def test_transmute_tarbz2_to_tarbz2(tmpdir, testtar_bytes):
@@ -104,4 +124,8 @@ def test_transmute_tarbz2_to_tarbz2(tmpdir, testtar_bytes):
testtar.write_bytes(testtar_bytes)
outdir = Path(tmpdir, "output")
outdir.mkdir()
- transmute_tar_bz2(str(testtar), outdir)
+ out = transmute_tar_bz2(str(testtar), outdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
=====================================
tests/test_url.py
=====================================
@@ -32,6 +32,7 @@ def package_urls(package_server, package_url):
pkgs_dir = Path(package_server.app.pkgs_dir)
conda = []
tar_bz2 = []
+
for path in pkgs_dir.iterdir():
if len(conda) > LIMIT and len(tar_bz2) > LIMIT:
break
@@ -102,7 +103,9 @@ def test_lazy_wheel(package_urls):
if lazy_tests <= 0:
break
else:
- raise LookupError("not enough .conda packages found")
+ raise LookupError(
+ "not enough .conda packages found %d %s" % (lazy_tests, package_urls)
+ )
with pytest.raises(HTTPError):
conda_reader_for_url(package_urls[0] + ".404.conda")
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/commit/4375d90f0ee36e4dbb09e5d1079b3669804751e9
--
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/commit/4375d90f0ee36e4dbb09e5d1079b3669804751e9
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20230711/9bbf33f5/attachment-0001.htm>
More information about the debian-med-commit
mailing list