[med-svn] [Git][med-team/conda-package-streaming][master] 9 commits: routine-update: New upstream version
Andreas Tille (@tille)
gitlab at salsa.debian.org
Tue Jul 11 16:59:05 BST 2023
Andreas Tille pushed to branch master at Debian Med / conda-package-streaming
Commits:
eae2d11e by Andreas Tille at 2023-07-11T17:36:37+02:00
routine-update: New upstream version
- - - - -
4375d90f by Andreas Tille at 2023-07-11T17:36:38+02:00
New upstream version 0.8.0
- - - - -
2c7355dd by Andreas Tille at 2023-07-11T17:36:38+02:00
Update upstream source from tag 'upstream/0.8.0'
Update to upstream version '0.8.0'
with Debian dir bdc8c7ddbd6cae6445babcc5fdaec272a3a78a5e
- - - - -
738381dc by Andreas Tille at 2023-07-11T17:36:41+02:00
routine-update: Drop ancient X-Python-Version field
- - - - -
d31c9ec8 by Andreas Tille at 2023-07-11T17:36:42+02:00
Add missing build dependency on python3:any | python3-all:any | python3-dev:any | python3-all-dev:any | dh-sequence-python3 for addon python3.
Changes-By: lintian-brush
Fixes: lintian: missing-build-dependency-for-dh-addon
See-also: https://lintian.debian.org/tags/missing-build-dependency-for-dh-addon.html
- - - - -
9f8f7bad by Andreas Tille at 2023-07-11T17:48:54+02:00
Update patch
- - - - -
bf1d34df by Andreas Tille at 2023-07-11T17:53:04+02:00
Deactivate compression patch since upstream went beyond our patch
- - - - -
ad842089 by Andreas Tille at 2023-07-11T17:56:58+02:00
Build-Depends: conda-package-handling <!nocheck>
- - - - -
51ca7f82 by Andreas Tille at 2023-07-11T17:58:10+02:00
Upload to unstable
- - - - -
26 changed files:
- − .github/workflows/main.yml
- + .github/workflows/tests.yml
- .pre-commit-config.yaml
- + CHANGELOG.md
- + CODE_OF_CONDUCT.md
- README.md
- conda.recipe/meta.yaml
- conda_package_streaming/__init__.py
- conda_package_streaming/extract.py
- conda_package_streaming/lazy_wheel.py
- conda_package_streaming/transmute.py
- conda_package_streaming/url.py
- debian/changelog
- debian/control
- debian/patches/32-bits.patch
- debian/patches/ignore_tests_requiring_conda_exe.patch
- debian/patches/series
- + docs/changelog.md
- docs/index.md
- pyproject.toml
- − renovate.json
- requirements.txt
- tests/conftest.py
- tests/server.py
- tests/test_transmute.py
- tests/test_url.py
Changes:
=====================================
.github/workflows/main.yml deleted
=====================================
@@ -1,18 +0,0 @@
-name: Pre-commit
-on:
- pull_request:
-
-jobs:
- pre_commit:
- runs-on: ubuntu-latest
- steps:
- - name: Retrieve the source code
- uses: actions/checkout at 2541b1294d2704b0964813337f33b291d3f8596b # tag=v3
- - name: Install build dependencies
- run: |
- source $CONDA/bin/activate
- conda install -y conda-forge::pre-commit
- - name: Run pre-commit
- run: |
- source $CONDA/bin/activate
- pre-commit run --all-files
=====================================
.github/workflows/tests.yml
=====================================
@@ -0,0 +1,100 @@
+name: Tests
+
+on:
+ # NOTE: github.event context is push payload:
+ # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push
+ push:
+ branches:
+ - main
+ - feature/**
+
+ # NOTE: github.event context is pull_request payload:
+ # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request
+ pull_request:
+
+concurrency:
+ # Concurrency group that uses the workflow name and PR number if available
+ # or commit SHA as a fallback. If a new build is triggered under that
+ # concurrency group while a previous build is running it will be canceled.
+ # Repeated pushes to a PR will cancel all previous builds, while multiple
+ # merges to main will not cancel.
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+ cancel-in-progress: true
+
+jobs:
+ linux:
+ runs-on: ubuntu-latest
+ defaults:
+ run:
+ shell: bash -l {0}
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ['3.10', '3.11']
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout at v2
+ with:
+ fetch-depth: 0
+
+ - uses: actions/setup-python at v4
+ with:
+ python-version: ${{ matrix.python-version }}
+ architecture: "x64"
+ cache: "pip"
+
+ - name: Setup Miniconda
+ uses: conda-incubator/setup-miniconda at v2
+ with:
+ python-version: ${{ matrix.python-version }}
+ channels: defaults
+ activate-environment: test_env
+ auto-update-conda: false
+ auto-activate-base: false
+ show-channel-urls: true
+
+ - name: Source Scripts
+ run: |
+ set -x
+ # conda is our test dependency but can't be pip installed
+ conda install --quiet conda pip
+ pip install -e .[test]
+ conda info --json
+ echo "condarc"
+ cat ~/.condarc
+ echo "conda_pkgs_dir"
+ ls /home/runner/conda_pkgs_dir
+ echo "miniconda/pkgs"
+ ls /usr/share/miniconda/pkgs
+ echo "test_env"
+ ls /usr/share/miniconda/envs/test_env
+ pytest
+
+ analyze:
+ name: Analyze test results
+ needs: [linux]
+ if: always()
+ runs-on: ubuntu-latest
+ steps:
+ - name: Download test results
+ uses: actions/download-artifact at v3
+
+ - name: Upload combined test results
+ # provides one downloadable archive of all .coverage/test-report.xml files
+ # of all matrix runs for further analysis.
+ uses: actions/upload-artifact at v3
+ with:
+ name: test-results-${{ github.sha }}-all
+ path: test-results-${{ github.sha }}-*
+ retention-days: 90 # default: 90
+
+ - name: Test Summary
+ uses: test-summary/action at v2
+ with:
+ paths: ./test-results-${{ github.sha }}-**/test-report*.xml
+
+ - name: Decide whether the needed jobs succeeded or failed
+ uses: re-actors/alls-green at release/v1
+ with:
+ jobs: ${{ toJSON(needs) }}
=====================================
.pre-commit-config.yaml
=====================================
@@ -1,7 +1,9 @@
-exclude: ^(versioneer.py|conda_pack/_version.py|build)
+# disable autofixing PRs, commenting "pre-commit.ci autofix" on a pull request triggers a autofix
+ci:
+ autofix_prs: false
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v4.3.0
+ rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-ast
@@ -17,17 +19,20 @@ repos:
- id: trailing-whitespace
- id: check-yaml
exclude: conda.recipe/meta.yaml
- - id: check-merge-conflict
- repo: https://github.com/asottile/pyupgrade
- rev: v2.34.0
+ rev: v3.3.1
hooks:
- id: pyupgrade
args: ["--py37-plus"]
- repo: https://github.com/PyCQA/isort
- rev: 5.10.1
+ rev: 5.12.0
hooks:
- id: isort
+ - repo: https://github.com/psf/black
+ rev: 23.1.0
+ hooks:
+ - id: black
- repo: https://github.com/PyCQA/flake8
- rev: 4.0.1
+ rev: 6.0.0
hooks:
- id: flake8
=====================================
CHANGELOG.md
=====================================
@@ -0,0 +1,8 @@
+[//]: # (current developments)
+
+## 0.8.0 (2023-05)
+
+* Update transmute to use SpooledTemporaryFile instead of streaming directly to
+ zip [(#57)](https://github.com/conda/conda-package-streaming/issues/57). This
+ can reduce zstd memory usage during decompression.
+* `transmute` returns Path to transmuted package instead of `None`.
=====================================
CODE_OF_CONDUCT.md
=====================================
@@ -0,0 +1,20 @@
+# Conda Organization Code of Conduct
+
+> **Note**
+> Below is the short version of our CoC, see the long version [here](https://github.com/conda-incubator/governance/blob/main/CODE_OF_CONDUCT.md).
+
+# The Short Version
+
+Be kind to others. Do not insult or put down others. Behave professionally. Remember that harassment and sexist, racist, or exclusionary jokes are not appropriate for the conda Organization.
+
+All communication should be appropriate for a professional audience including people of many different backgrounds. Sexual language and imagery is not appropriate.
+
+The conda Organization is dedicated to providing a harassment-free community for everyone, regardless of gender, sexual orientation, gender identity and expression, disability, physical appearance, body size, race, or religion. We do not tolerate harassment of community members in any form.
+
+Thank you for helping make this a welcoming, friendly community for all.
+
+## Report an Incident
+
+* Report a code of conduct incident [using a form](https://form.jotform.com/221527028480048).
+* Report a code of conduct incident via email: [conduct at conda.org](mailto:conduct at conda.org).
+* Contact [an individual committee member](#committee-membership) or [CoC event representative](#coc-representatives) to report an incident in confidence.
=====================================
README.md
=====================================
@@ -1,5 +1,10 @@
# conda-package-streaming
+[](https://results.pre-commit.ci/latest/github/conda/conda-package-streaming/main)
+
+An efficient library to read from new and old format .conda and .tar.bz2 conda
+packages.
+
Download conda metadata from packages without transferring entire file. Get
metadata from local `.tar.bz2` packages without reading entire files.
@@ -64,6 +69,10 @@ with closing(conda):
break
```
+If you need the entire package, download it first and use the file-based APIs.
+The URL-based APIs are more efficient if you only need to access package
+metadata.
+
# Package goals
* Extract conda packages (both formats)
=====================================
conda.recipe/meta.yaml
=====================================
@@ -39,8 +39,9 @@ test:
about:
home: https://github.com/conda/conda-package-streaming
- summary: Download metadata from conda packages without transferring entire file.
+ summary: An efficient library to read from new and old format .conda and .tar.bz2 conda packages.
license: BSD-3-Clause
+ license_family: BSD
license_file: LICENSE
doc_url: https://conda.github.io/conda-package-streaming/
dev_url: https://github.com/conda/conda-package-streaming
=====================================
conda_package_streaming/__init__.py
=====================================
@@ -1 +1 @@
-__version__ = "0.7.0"
+__version__ = "0.8.0"
=====================================
conda_package_streaming/extract.py
=====================================
@@ -35,7 +35,6 @@ def extract_stream(
return prefix == dest_dir
for tar_file, _ in stream:
-
# careful not to seek backwards
def checked_members():
# from conda_package_handling
=====================================
conda_package_streaming/lazy_wheel.py
=====================================
@@ -38,7 +38,6 @@ class LazyZipOverHTTP:
def __init__(
self, url: str, session: Session, chunk_size: int = CONTENT_CHUNK_SIZE
) -> None:
-
# if CONTENT_CHUNK_SIZE is bigger than the file:
# In [8]: response.headers["Content-Range"]
# Out[8]: 'bytes 0-3133374/3133375'
=====================================
conda_package_streaming/transmute.py
=====================================
@@ -13,11 +13,13 @@ the `ZipFile`, instead of the first for normal conda packages.
from __future__ import annotations
-import io
import json
import os
+import shutil
import tarfile
+import tempfile
import zipfile
+from pathlib import Path
from typing import Callable
import zstandard
@@ -25,11 +27,14 @@ import zstandard
# streams everything in .tar.bz2 mode
from .package_streaming import CondaComponent, stream_conda_component
-# increase to reduce speed and increase compression (22 = conda's default)
-ZSTD_COMPRESS_LEVEL = 22
+# increase to reduce speed and increase compression (levels above 19 use much
+# more memory)
+ZSTD_COMPRESS_LEVEL = 19
# increase to reduce compression and increase speed
ZSTD_COMPRESS_THREADS = 1
+CONDA_PACKAGE_FORMAT_VERSION = 2
+
def transmute(
package,
@@ -41,7 +46,7 @@ def transmute(
level=ZSTD_COMPRESS_LEVEL, threads=ZSTD_COMPRESS_THREADS
),
is_info: Callable[[str], bool] = lambda filename: filename.startswith("info/"),
-):
+) -> Path:
"""
Convert .tar.bz2 conda :package to .conda-format under path.
@@ -54,33 +59,21 @@ def transmute(
(not this package ``conda-package-streaming``) uses a set of regular
expressions to keep expected items in the info- component, while other
items starting with ``info/`` wind up in the pkg- component.
+
+ :return: Path to transmuted package.
"""
assert package.endswith(".tar.bz2"), "can only convert .tar.bz2 to .conda"
assert os.path.isdir(path)
file_id = os.path.basename(package)[: -len(".tar.bz2")]
-
- # x to not append to existing
- with zipfile.ZipFile(
- os.path.join(path, f"{file_id}.conda"), "x", compresslevel=zipfile.ZIP_STORED
- ) as conda_file:
-
- info_compress = compressor()
- data_compress = compressor()
-
- # in theory, info_tar could grow uncomfortably big, in which case we would
- # rather swap it to disk
- info_io = io.BytesIO()
- info_stream = info_compress.stream_writer(info_io, closefd=False)
- info_tar = tarfile.TarFile(fileobj=info_stream, mode="w")
-
- conda_file.writestr(
- "metadata.json", json.dumps({"conda_pkg_format_version": 2})
- )
-
- with conda_file.open(f"pkg-{file_id}.tar.zst", "w") as pkg_file:
- pkg_stream = data_compress.stream_writer(pkg_file, closefd=False)
- pkg_tar = tarfile.TarFile(fileobj=pkg_stream, mode="w")
-
+ output_path = Path(path, f"{file_id}.conda")
+
+ with tempfile.SpooledTemporaryFile() as info_file, tempfile.SpooledTemporaryFile() as pkg_file:
+ with tarfile.TarFile(fileobj=info_file, mode="w") as info_tar, tarfile.TarFile(
+ fileobj=pkg_file, mode="w"
+ ) as pkg_tar:
+ # If we wanted to compress these at a low setting to save temporary
+ # space, we could insert a file object that counts bytes written in
+ # front of a zstd (level between 1..3) compressor.
stream = iter(stream_conda_component(package))
for tar, member in stream:
tar_get = info_tar if is_info(member.name) else pkg_tar
@@ -89,20 +82,47 @@ def transmute(
else:
tar_get.addfile(member)
+ info_tar.close()
pkg_tar.close()
- pkg_stream.close()
- info_tar.close()
- info_stream.close()
+ info_size = info_file.tell()
+ pkg_size = pkg_file.tell()
- with conda_file.open(f"info-{file_id}.tar.zst", "w") as info_file:
- info_file.write(info_io.getvalue())
+ info_file.seek(0)
+ pkg_file.seek(0)
+
+ with zipfile.ZipFile(
+ output_path,
+ "x", # x to not append to existing
+ compresslevel=zipfile.ZIP_STORED,
+ ) as conda_file:
+ # Use a maximum of one Zstd compressor, stream_writer at a time to save memory.
+ data_compress = compressor()
+
+ pkg_metadata = {"conda_pkg_format_version": CONDA_PACKAGE_FORMAT_VERSION}
+ conda_file.writestr("metadata.json", json.dumps(pkg_metadata))
+
+ with conda_file.open(
+ f"pkg-{file_id}.tar.zst", "w"
+ ) as pkg_file_zip, data_compress.stream_writer(
+ pkg_file_zip, size=pkg_size, closefd=False
+ ) as pkg_stream:
+ shutil.copyfileobj(pkg_file._file, pkg_stream)
+
+ with conda_file.open(
+ f"info-{file_id}.tar.zst", "w"
+ ) as info_file_zip, data_compress.stream_writer(
+ info_file_zip, size=info_size, closefd=False
+ ) as info_stream:
+ shutil.copyfileobj(info_file._file, info_stream)
+
+ return output_path
def transmute_tar_bz2(
- package,
+ package: str,
path,
-):
+) -> Path:
"""
Convert .conda :package to .tar.bz2 format under path.
@@ -110,6 +130,8 @@ def transmute_tar_bz2(
:param package: path to `.conda` or `.tar.bz2` package.
:param path: destination path for transmuted package.
+
+ :return: Path to transmuted package.
"""
assert package.endswith((".tar.bz2", ".conda")), "Unknown extension"
assert os.path.isdir(path)
@@ -125,9 +147,9 @@ def transmute_tar_bz2(
# .tar.bz2 doesn't filter by component
components = [CondaComponent.pkg]
- with open(package, "rb") as fileobj, tarfile.open(
- os.path.join(path, f"{file_id}.tar.bz2"), "x:bz2"
- ) as pkg_tar:
+ output_path = Path(path, f"{file_id}.tar.bz2")
+
+ with open(package, "rb") as fileobj, tarfile.open(output_path, "x:bz2") as pkg_tar:
for component in components:
stream = iter(stream_conda_component(package, fileobj, component=component))
for tar, member in stream:
@@ -135,3 +157,5 @@ def transmute_tar_bz2(
pkg_tar.addfile(member, tar.extractfile(member))
else:
pkg_tar.addfile(member)
+
+ return output_path
=====================================
conda_package_streaming/url.py
=====================================
@@ -2,6 +2,11 @@
Fetch metadata from remote .conda or .tar.bz2 package.
Try to fetch less than the whole file if possible.
+
+This module should only be used to make *partial* reads against a remote
+package, typically just the ``info`` portion. If a full ``.conda`` format
+package is needed, it is more efficient to download locally first and then use
+the file-based API.
"""
import logging
@@ -32,7 +37,7 @@ def extract_conda_info(url, destdir, checklist=METADATA_CHECKLIST, session=sessi
"""
checklist = set(checklist)
stream = stream_conda_info(url, session=session)
- for (tar, member) in stream:
+ for tar, member in stream:
if member.name in checklist:
tar.extract(member, destdir)
checklist.remove(member.name)
=====================================
debian/changelog
=====================================
@@ -1,3 +1,11 @@
+conda-package-streaming (0.8.0-1) unstable; urgency=medium
+
+ * New upstream version
+ * Drop ancient X-Python-Version field (routine-update)
+ * Build-Depends: conda-package-handling <!nocheck>
+
+ -- Andreas Tille <tille at debian.org> Tue, 11 Jul 2023 17:57:03 +0200
+
conda-package-streaming (0.7.0-4) unstable; urgency=medium
* Team upload.
=====================================
debian/control
=====================================
@@ -11,19 +11,20 @@ Build-Depends: debhelper-compat (= 13),
# FIXME: restore python3-all support when python3.10 is removed.
python3,
python3-setuptools,
+ conda-package-handling <!nocheck>,
python3-pytest <!nocheck>,
python3-pytest-cov <!nocheck>,
python3-pytest-mock <!nocheck>,
python3-boto3 <!nocheck>,
python3-bottle <!nocheck>,
python3-zstandard <!nocheck>,
- flit
+ flit,
+ python3:any | python3-all:any | python3-dev:any | python3-all-dev:any | dh-sequence-python3
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/med-team/conda-package-streaming
Vcs-Git: https://salsa.debian.org/med-team/conda-package-streaming.git
Homepage: https://github.com/conda/conda-package-streaming
Rules-Requires-Root: no
-X-Python3-Version: >= 3.11
Package: python3-conda-package-streaming
Architecture: all
=====================================
debian/patches/32-bits.patch
=====================================
@@ -1,6 +1,8 @@
Description: Set ZSTD_COMPRESS_LEVEL to 21 instead, as it consumes a lot of mem on 32 bit machines
Author: Nilesh Patra <nilesh at debian.org>
-Last-Update: 2022-12-31
+Last-Update: 2023-07-11
+Comment: Deactivated since upstream has now set level 19
+
--- a/conda_package_streaming/transmute.py
+++ b/conda_package_streaming/transmute.py
@@ -26,7 +26,7 @@
=====================================
debian/patches/ignore_tests_requiring_conda_exe.patch
=====================================
@@ -71,27 +71,26 @@ Description: Skip test that are testing conda executable
print(package)
--- a/tests/test_transmute.py
+++ b/tests/test_transmute.py
-@@ -37,7 +37,7 @@ def timeme(message: str = ""):
- end = time.time()
+@@ -40,6 +40,7 @@ def timeme(message: str = ""):
print(f"{message}{end-begin:0.2f}s")
--
-+ at pytest.mark.skip(reason="Conda is not yet available for Debian")
- def test_transmute(conda_paths, tmpdir):
++ at pytest.mark.skip(reason="Conda is not yet available for Debian")
+ def test_transmute(conda_paths: list[Path], tmpdir):
tarbz_packages = []
-@@ -81,7 +81,7 @@ def test_transmute_info_filter(tmpdir, t
+ for path in conda_paths:
+@@ -98,7 +99,7 @@ def test_transmute_info_filter(tmpdir, t
items = stream_conda_component("test.conda", fileobj, component)
assert {member.name for tar, member in items} == expected, items
-
+ at pytest.mark.skip(reason="Conda is not yet available for Debian")
def test_transmute_backwards(tmpdir, conda_paths):
-
tarbz_packages = []
+ for path in conda_paths:
--- a/tests/test_url.py
+++ b/tests/test_url.py
-@@ -46,7 +46,7 @@ def package_urls(package_server, package
+@@ -47,7 +47,7 @@ def package_urls(package_server, package
urls.extend(pair)
return urls
@@ -100,7 +99,7 @@ Description: Skip test that are testing conda executable
def test_stream_url(package_urls):
with pytest.raises(ValueError):
next(stream_conda_info("https://localhost/notaconda.rar"))
-@@ -60,13 +60,13 @@ def test_stream_url(package_urls):
+@@ -61,13 +61,13 @@ def test_stream_url(package_urls):
else:
pytest.fail("info/index.json not found")
=====================================
debian/patches/series
=====================================
@@ -1,2 +1,2 @@
ignore_tests_requiring_conda_exe.patch
-32-bits.patch
+# 32-bits.patch
=====================================
docs/changelog.md
=====================================
@@ -0,0 +1,3 @@
+# Changelog
+```{include} ../CHANGELOG.md
+```
=====================================
docs/index.md
=====================================
@@ -30,6 +30,7 @@ zstd-compressed streams.
:caption: 'Contents:'
:maxdepth: 2
modules
+changelog
```
# Indices and tables
=====================================
pyproject.toml
=====================================
@@ -13,7 +13,7 @@ name = "conda_package_streaming"
authors = [
{ name = "Anaconda, Inc. & Contributors", email = "conda at continuum.io" },
]
-description = "Download metadata from conda packages without transferring entire file."
+description = "An efficient library to read from new and old format .conda and .tar.bz2 conda packages."
license = { file = "LICENSE" }
readme = "README.md"
classifiers = [
@@ -34,6 +34,8 @@ test = [
"boto3",
"boto3-stubs[essential]",
"bottle",
+ "conda",
+ "conda-package-handling >=2",
]
docs = ["furo", "sphinx", "myst-parser", "mdit-py-plugins>=0.3.0"]
=====================================
renovate.json deleted
=====================================
@@ -1,6 +0,0 @@
-{
- "$schema": "https://docs.renovatebot.com/renovate-schema.json",
- "extends": [
- "github>anaconda/renovate-config"
- ]
-}
=====================================
requirements.txt
=====================================
@@ -7,6 +7,7 @@ pytest-mock
boto3
boto3-stubs[essential]
bottle
+conda
# docs
furo
sphinx
=====================================
tests/conftest.py
=====================================
@@ -1,21 +1,101 @@
+import json
+import logging
+import os.path
+import shutil
+import subprocess
from pathlib import Path
import pytest
import server
+from conda_package_streaming.transmute import transmute_tar_bz2
+
+log = logging.getLogger(__name__)
+
+
+LIMIT_TEST_PACKAGES = 16
+
+
+def find_packages_dirs() -> Path:
+ """
+ Ask conda for package directories.
+ """
+ conda_info = json.loads(
+ subprocess.run(
+ [os.environ["CONDA_EXE"], "info", "--json"],
+ stdout=subprocess.PIPE,
+ check=True,
+ ).stdout
+ )
+
+ # XXX can run individual environment's conda (base conda is more likely to
+ # have useful cached packages)
+ pkgs_dirs = conda_info["pkgs_dirs"] + [os.path.expanduser("~/miniconda3/pkgs")]
+
+ log.debug("search %s", pkgs_dirs)
+
+ first_pkg_dir = next(path for path in pkgs_dirs if os.path.exists(path))
+
+ return Path(first_pkg_dir)
+
+
+ at pytest.fixture(scope="session")
+def pkgs_dir(tmp_path_factory):
+ """
+ Dedicated test package directory.
+ """
+ return tmp_path_factory.mktemp("pkgs")
+
@pytest.fixture(scope="session")
-def package_server():
- thread = server.get_server_thread()
+def package_server(pkgs_dir, conda_paths):
+ thread = server.get_server_thread(pkgs_dir)
thread.start()
return thread
@pytest.fixture(scope="session")
-def conda_paths(package_server):
- pkgs_dir = Path(package_server.app.pkgs_dir)
+def conda_paths(pkgs_dir: Path):
+ found_packages = find_packages_dirs()
conda_paths = []
- for path in pkgs_dir.iterdir():
+ for path in found_packages.iterdir():
if path.name.endswith((".tar.bz2", ".conda")):
conda_paths.append(path)
- return conda_paths
+
+ return add_tar_bz2s(conda_paths, pkgs_dir)
+
+
+def add_tar_bz2s(paths: list[Path], pkgs_dir: Path):
+ """
+ If there aren't enough .tar.bz2's available, create some from available
+ .conda's. Return paths.
+ """
+ conda_paths: list[Path] = []
+ tarbz2_paths: list[Path] = []
+ output_paths: list[Path] = []
+
+ assert isinstance(pkgs_dir, Path)
+
+ for path in paths:
+ if path.name.endswith(".tar.bz2"):
+ tarbz2_paths.append(path)
+ elif path.name.endswith(".conda"):
+ conda_paths.append(path)
+
+ tarbz2_path: Path = pkgs_dir
+
+ medium_conda_paths = []
+ for path in conda_paths:
+ if 1 << 20 < path.stat().st_size < 1 << 22:
+ medium_conda_paths.append(path)
+ medium_conda_paths = medium_conda_paths[:LIMIT_TEST_PACKAGES]
+
+ # this ignores existing .tar.bz2 for simplicity (.tar.bz2 is missing in CI)
+ for conda in set(medium_conda_paths + conda_paths[:10]):
+ shutil.copy(conda, tarbz2_path)
+ transmute_tar_bz2(str(conda), tarbz2_path)
+
+ output_paths.extend(tarbz2_path.glob("*.tar.bz2"))
+ output_paths.extend(tarbz2_path.glob("*.conda"))
+
+ return sorted(output_paths) # sort interleaves .tar.bz2 and .conda
=====================================
tests/server.py
=====================================
@@ -1,51 +1,25 @@
"""
Test web server.
"""
-import json
import logging
-import os
-import os.path
-import subprocess
import threading
import wsgiref.simple_server
+from pathlib import Path
from typing import Any
import bottle
+import conftest
log = logging.getLogger(__name__)
-def find_packages_dirs():
- """
- Ask conda for package directories.
- """
- conda_info = json.loads(
- subprocess.run(
- [os.environ["CONDA_EXE"], "info", "--json"],
- stdout=subprocess.PIPE,
- check=True,
- ).stdout
- )
-
- # XXX can run individual environment's conda (base conda is more likely to
- # have useful cached packages)
- pkgs_dirs = conda_info["pkgs_dirs"] + [os.path.expanduser("~/miniconda3/pkgs")]
-
- log.debug("search %s", pkgs_dirs)
-
- first_pkg_dir = next(path for path in pkgs_dirs if os.path.exists(path))
-
- return first_pkg_dir
-
-
-def get_app():
+def get_app(pkgs_dir):
"""
Bottle conveniently supports Range requests.
Server may block if browser etc. keeps connection open.
"""
app = bottle.Bottle()
- pkgs_dir = find_packages_dirs()
app.pkgs_dir = pkgs_dir
def serve_file(filename):
@@ -65,7 +39,7 @@ def selftest():
"""
Run server in a thread that will die when the application exits.
"""
- t = get_server_thread()
+ t = get_server_thread(conftest.find_packages_dirs())
t.start()
import time
@@ -78,13 +52,13 @@ class ServerThread(threading.Thread):
app: Any
-def get_server_thread():
+def get_server_thread(pkgs_dir: Path):
"""
Return test server thread with additional .server, .app properties.
Call .start() to serve in the background.
"""
- app = get_app()
+ app = get_app(pkgs_dir)
server = wsgiref.simple_server.make_server("127.0.0.1", 0, app)
log.info(f"serving {app.pkgs_dir} on {server.server_address}/pkgs")
t = ServerThread(daemon=True, target=server.serve_forever)
=====================================
tests/test_transmute.py
=====================================
@@ -4,8 +4,10 @@ import os
import tarfile
import time
from pathlib import Path
+from zipfile import ZipFile
import pytest
+from conda_package_handling.validate import validate_converted_files_match_streaming
from conda_package_streaming.package_streaming import (
CondaComponent,
@@ -38,8 +40,7 @@ def timeme(message: str = ""):
print(f"{message}{end-begin:0.2f}s")
-def test_transmute(conda_paths, tmpdir):
-
+def test_transmute(conda_paths: list[Path], tmpdir):
tarbz_packages = []
for path in conda_paths:
path = str(path)
@@ -49,17 +50,33 @@ def test_transmute(conda_paths, tmpdir):
assert tarbz_packages, "no medium-sized .tar.bz2 packages found"
+ metadata_checks = 0
+
for packages in (conda_packages, tarbz_packages):
for package in packages:
with timeme(f"{package} took "):
- transmute(package, tmpdir)
+ out = transmute(package, tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, package, strict=True
+ )
+ assert missing == mismatched == []
+ if out.name.endswith(".conda"):
+ with ZipFile(out) as zf:
+ metadata_checks += 1
+ assert "metadata.json" in zf.namelist()
+
+ assert metadata_checks > 0
def test_transmute_symlink(tmpdir, testtar_bytes):
testtar = Path(tmpdir, "test.tar.bz2")
testtar.write_bytes(testtar_bytes)
- transmute(str(testtar), tmpdir)
+ out = transmute(str(testtar), tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
def test_transmute_info_filter(tmpdir, testtar_bytes):
@@ -83,7 +100,6 @@ def test_transmute_info_filter(tmpdir, testtar_bytes):
def test_transmute_backwards(tmpdir, conda_paths):
-
tarbz_packages = []
for path in conda_paths:
path = str(path)
@@ -96,7 +112,11 @@ def test_transmute_backwards(tmpdir, conda_paths):
for packages in (conda_packages, tarbz_packages):
for package in packages:
with timeme(f"{package} took "):
- transmute_tar_bz2(package, tmpdir)
+ out = transmute_tar_bz2(package, tmpdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, package, strict=True
+ )
+ assert missing == mismatched == []
def test_transmute_tarbz2_to_tarbz2(tmpdir, testtar_bytes):
@@ -104,4 +124,8 @@ def test_transmute_tarbz2_to_tarbz2(tmpdir, testtar_bytes):
testtar.write_bytes(testtar_bytes)
outdir = Path(tmpdir, "output")
outdir.mkdir()
- transmute_tar_bz2(str(testtar), outdir)
+ out = transmute_tar_bz2(str(testtar), outdir)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
=====================================
tests/test_url.py
=====================================
@@ -32,6 +32,7 @@ def package_urls(package_server, package_url):
pkgs_dir = Path(package_server.app.pkgs_dir)
conda = []
tar_bz2 = []
+
for path in pkgs_dir.iterdir():
if len(conda) > LIMIT and len(tar_bz2) > LIMIT:
break
@@ -102,7 +103,9 @@ def test_lazy_wheel(package_urls):
if lazy_tests <= 0:
break
else:
- raise LookupError("not enough .conda packages found")
+ raise LookupError(
+ "not enough .conda packages found %d %s" % (lazy_tests, package_urls)
+ )
with pytest.raises(HTTPError):
conda_reader_for_url(package_urls[0] + ".404.conda")
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/compare/8975b6a9280861e84253943b1e1e048d1a611064...51ca7f820923975fce82839e8b7af4c37b35e0a0
--
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/compare/8975b6a9280861e84253943b1e1e048d1a611064...51ca7f820923975fce82839e8b7af4c37b35e0a0
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20230711/5b9692e0/attachment-0001.htm>
More information about the debian-med-commit
mailing list