[med-svn] [Git][med-team/conda-package-streaming][upstream] New upstream version 0.13.0
Alexandre Detiste (@detiste-guest)
gitlab at salsa.debian.org
Fri Sep 11 20:29:05 BST 2026
Alexandre Detiste pushed to branch upstream at Debian Med / conda-package-streaming
Commits:
daa11965 by Alexandre Detiste at 2026-09-11T21:15:29+02:00
New upstream version 0.13.0
- - - - -
25 changed files:
- .github/workflows/cla.yml
- .github/workflows/issues.yml
- .github/workflows/labels.yml
- .github/workflows/lock.yml
- .github/workflows/project.yml
- + .github/workflows/pypi.yml
- .github/workflows/sphinx.yml
- .github/workflows/stale.yml
- .github/workflows/tests.yml
- .github/workflows/update.yml
- .pre-commit-config.yaml
- CHANGELOG.md
- HOW_WE_USE_GITHUB.md
- README.md
- conda.recipe/meta.yaml
- conda_package_streaming/__init__.py
- conda_package_streaming/create.py
- conda_package_streaming/package_streaming.py
- conda_package_streaming/transmute.py
- pyproject.toml
- tests/requirements.txt
- tests/test_degraded.py
- tests/test_extract.py
- tests/test_streaming.py
- tests/test_transmute.py
Changes:
=====================================
.github/workflows/cla.yml
=====================================
@@ -6,6 +6,11 @@ on:
- created
pull_request_target:
+permissions:
+ contents: read
+ pull-requests: write
+ statuses: write
+
jobs:
check:
if: >-
@@ -15,10 +20,10 @@ jobs:
&& github.event.comment.body == '@conda-bot check'
|| github.event_name == 'pull_request_target'
)
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- name: Check CLA
- uses: conda/actions/check-cla at eb545bb8ab48d499b31c057a6df3cf46753fdbcb # v25.3.1
+ uses: conda/actions/check-cla at 7f6830b1428a9bd47f0b068892c77eae95207037 # v26.1.0
with:
# [required]
# A token with ability to comment, label, and modify the commit status
=====================================
.github/workflows/issues.yml
=====================================
@@ -6,6 +6,10 @@ on:
issue_comment:
types: [created]
+permissions:
+ contents: read
+ issues: write
+
env:
FEEDBACK_LBL: pending::feedback
SUPPORT_LBL: pending::support
@@ -20,7 +24,7 @@ jobs:
!github.event.repository.fork
&& !github.event.issue.pull_request
&& contains(github.event.issue.labels.*.name, 'pending::feedback')
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
# remove [pending::feedback]
- uses: actions-ecosystem/action-remove-labels at 2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1.3.0
=====================================
.github/workflows/labels.yml
=====================================
@@ -15,18 +15,26 @@ on:
default: false
type: boolean
+permissions:
+ contents: read
+
jobs:
sync:
if: '!github.event.repository.fork'
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
+ permissions:
+ contents: read
+ issues: write
env:
GLOBAL: https://raw.githubusercontent.com/conda/infra/main/.github/global.yml
LOCAL: .github/labels.yml
steps:
- - uses: actions/checkout at 11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+ - uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ persist-credentials: false
- id: has_local
- uses: andstor/file-existence-action at 076e0072799f4942c8bc574a82233e1e4d13e9d6 # v3.0.0
+ uses: andstor/file-existence-action at 558493d6c74bf472d87c84eab196434afc2fa029 # v3.1.0
with:
files: ${{ env.LOCAL }}
=====================================
.github/workflows/lock.yml
=====================================
@@ -15,9 +15,9 @@ permissions:
jobs:
lock:
if: '!github.event.repository.fork'
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- - uses: dessant/lock-threads at 1bf7ec25051fe7c00bdd17e6a7cf3d7bfb7dc771 # v5.0.1
+ - uses: dessant/lock-threads at 7266a7ce5c1df01b1c6db85bf8cd86c737dadbe7 # v6.0.0
with:
# Number of days of inactivity before a closed issue is locked
issue-inactive-days: 180
=====================================
.github/workflows/project.yml
=====================================
@@ -1,21 +1,20 @@
name: Add to Project
on:
- issues:
- types:
- - opened
pull_request_target:
types:
- opened
+permissions:
+ contents: read
+
jobs:
add_to_project:
if: '!github.event.repository.fork'
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
steps:
- - uses: actions/add-to-project at 244f685bbc3b7adfa8466e08b698b5577571133e # v1.0.2
+ - uses: actions/add-to-project at 5afcf98fcd03f1c2f92c3c83f58ae24323cc57fd # v2.0.0
with:
- # issues are added to the Planning project
# PRs are added to the Review project
- project-url: https://github.com/orgs/conda/projects/${{ github.event_name == 'issues' && 2 || 16 }}
+ project-url: https://github.com/orgs/conda/projects/16
github-token: ${{ secrets.PROJECT_TOKEN }}
=====================================
.github/workflows/pypi.yml
=====================================
@@ -0,0 +1,93 @@
+name: Publish Python 🐍 distribution 📦 to PyPI
+
+on:
+ pull_request:
+ push:
+ branches:
+ - main
+ release:
+ types:
+ - published
+
+permissions:
+ contents: read
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ build:
+ name: Build distribution 📦
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ with:
+ fetch-depth: 0
+ persist-credentials: false
+
+ - name: Set up Python
+ uses: actions/setup-python at a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+ with:
+ python-version: "3.13"
+
+ - name: Build sdist and wheel
+ run: pipx run --spec build==1.5.0 pyproject-build --sdist --wheel --outdir dist
+
+ - name: Check distributions
+ working-directory: dist
+ run: |
+ ls -alh
+ python -m pip install *.whl
+ pipx run --spec twine==6.2.0 twine check *
+
+ - name: Upload distributions 📦
+ uses: actions/upload-artifact at 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+ with:
+ name: dist
+ path: dist/
+
+ publish:
+ name: Publish Python 🐍 distribution 📦 to PyPI
+ runs-on: ubuntu-latest
+ if: github.event_name == 'release'
+ needs: [build]
+ permissions:
+ id-token: write
+ environment:
+ name: pypi
+ url: https://pypi.org/p/conda-package-streaming
+
+ steps:
+ - name: Retrieve distributions
+ uses: actions/download-artifact at 3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+ with:
+ name: dist
+ path: dist/
+
+ - name: Publish distribution 📦 to PyPI
+ uses: pypa/gh-action-pypi-publish at cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
+ with:
+ packages-dir: dist/
+
+ upload-release-assets:
+ name: Upload release assets
+ runs-on: ubuntu-latest
+ if: github.event_name == 'release'
+ needs: [publish]
+ permissions:
+ contents: write
+
+ steps:
+ - name: Retrieve distributions
+ uses: actions/download-artifact at 3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+ with:
+ name: dist
+ path: dist/
+
+ - name: Upload assets to GitHub Release
+ env:
+ GH_TOKEN: ${{ github.token }}
+ GH_REPO: ${{ github.repository }}
+ TAG_NAME: ${{ github.event.release.tag_name }}
+ run: gh release upload "$TAG_NAME" dist/* --repo "$GH_REPO"
=====================================
.github/workflows/sphinx.yml
=====================================
@@ -14,8 +14,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout at 11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- - uses: actions/setup-python at 8d9ed9ac5c53483de85588cdf95a591a75ab9f55 # v5.5.0
+ - uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+ - uses: actions/setup-python at a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.x"
architecture: "x64"
@@ -25,7 +25,7 @@ jobs:
pip install -e .[docs]
make html
- name: Upload artifact
- uses: actions/upload-pages-artifact at 56afc609e74202658d3ffba0e8f6dda462b719fa # v3.0.1
+ uses: actions/upload-pages-artifact at fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
# Upload entire repository
path: 'build/html'
@@ -49,4 +49,4 @@ jobs:
steps:
- name: Deploy to GitHub Pages
id: deployment
- uses: actions/deploy-pages at d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4.0.5
+ uses: actions/deploy-pages at cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0
=====================================
.github/workflows/stale.yml
=====================================
@@ -21,7 +21,7 @@ permissions:
jobs:
stale:
if: '!github.event.repository.fork'
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
strategy:
matrix:
include:
@@ -33,12 +33,12 @@ jobs:
days-before-issue-stale: 90
days-before-issue-close: 21
steps:
- - uses: conda/actions/read-yaml at eb545bb8ab48d499b31c057a6df3cf46753fdbcb # v25.3.1
+ - uses: conda/actions/read-yaml at 7f6830b1428a9bd47f0b068892c77eae95207037 # v26.1.0
id: read_yaml
with:
path: https://raw.githubusercontent.com/conda/infra/main/.github/messages.yml
- - uses: actions/stale at 5bef64f19d7facfb25b37b414482c7164d639639 # v9.1.0
+ - uses: actions/stale at b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0
id: stale
with:
# Only issues with these labels are checked whether they are stale
=====================================
.github/workflows/tests.yml
=====================================
@@ -30,16 +30,16 @@ jobs:
strategy:
fail-fast: false
matrix:
- python-version: ['3.9', '3.10', '3.11', '3.12']
+ python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
steps:
- name: Checkout repository
- uses: actions/checkout at 11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+ uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
- name: Setup Miniconda
- uses: conda-incubator/setup-miniconda at 505e6394dae86d6a5c7fbb6e3fb8938e3e863830 # v3.1.1
+ uses: conda-incubator/setup-miniconda at 8ee1f361103df19b6f8c8655fd3967a8ecb162d5 # v4.0.1
with:
python-version: ${{ matrix.python-version }}
channels: defaults
@@ -72,19 +72,19 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Download test results
- uses: actions/download-artifact at d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+ uses: actions/download-artifact at 3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
- name: Upload combined test results
# provides one downloadable archive of all .coverage/test-report.xml files
# of all matrix runs for further analysis.
- uses: actions/upload-artifact at ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ uses: actions/upload-artifact at 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: test-results-${{ github.sha }}-all
path: test-results-${{ github.sha }}-*
retention-days: 90 # default: 90
- name: Test Summary
- uses: test-summary/action at 31493c76ec9e7aa675f1585d3ed6f1da69269a86 # v2.4
+ uses: test-summary/action at 37b508cfee6d4d080eedd00b5bb240a6a784a6a5 # v2.6
with:
paths: ./test-results-${{ github.sha }}-**/test-report*.xml
=====================================
.github/workflows/update.yml
=====================================
@@ -8,43 +8,18 @@ on:
workflow_dispatch:
- issue_comment:
- types:
- - created
+permissions:
+ contents: read
jobs:
update:
- if: >-
- !github.event.repository.fork
- && (
- github.event_name == 'schedule'
- || github.event_name == 'workflow_dispatch'
- || (
- github.event_name == 'issue_comment'
- && github.event.issue.pull_request
- && (
- github.event.comment.body == '@conda-bot render'
- || github.event.comment.body == '@conda-bot recreate'
- )
- )
- )
- runs-on: ubuntu-latest
+ runs-on: ubuntu-slim
+ permissions:
+ contents: write
+ pull-requests: write
+ issues: write
steps:
- - if: github.event_name == 'issue_comment'
- uses: peter-evans/create-or-update-comment at 71345be0265236311c031f5c7866368bd1eff043 # v4.0.0
- with:
- comment-id: ${{ github.event.comment.id }}
- reactions: eyes
- reactions-edit-mode: replace
- token: ${{ secrets.SYNC_TOKEN }}
-
- - if: github.event.comment.body == '@conda-bot render'
- name: Configure git origin
- run: |
- echo REPOSITORY=$(curl --silent ${{ github.event.issue.pull_request.url }} | jq --raw-output '.head.repo.full_name') >> $GITHUB_ENV
- echo REF=$(curl --silent ${{ github.event.issue.pull_request.url }} | jq --raw-output '.head.ref') >> $GITHUB_ENV
-
- - uses: actions/checkout at 11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+ - uses: actions/checkout at df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
repository: ${{ env.REPOSITORY || github.repository }}
ref: ${{ env.REF || '' }}
@@ -55,11 +30,11 @@ jobs:
git config --global user.name 'Conda Bot'
git config --global user.email '18747875+conda-bot at users.noreply.github.com'
- - uses: conda/actions/combine-durations at eb545bb8ab48d499b31c057a6df3cf46753fdbcb # v25.3.1
+ - uses: conda/actions/combine-durations at 7f6830b1428a9bd47f0b068892c77eae95207037 # v26.1.0
id: durations
continue-on-error: true
- - uses: conda/actions/template-files at eb545bb8ab48d499b31c057a6df3cf46753fdbcb # v25.3.1
+ - uses: conda/actions/template-files at 7f6830b1428a9bd47f0b068892c77eae95207037 # v26.1.0
id: templates
continue-on-error: true
@@ -70,17 +45,13 @@ jobs:
git add .
git commit --message "🤖 updated file(s)"
- - if: github.event.comment.body != '@conda-bot render'
- name: Create fork
+ - name: Create fork
# no-op if the repository is already forked
run: echo FORK=$(gh repo fork --clone=false --default-branch-only 2>&1 | awk '{print $1}') >> $GITHUB_ENV
env:
GH_TOKEN: ${{ secrets.SYNC_TOKEN }}
- - if: github.event.comment.body != '@conda-bot render'
- id: create
- # no-op if no commits were made
- uses: peter-evans/create-pull-request at 271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
+ - uses: peter-evans/create-pull-request at 5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
push-to-fork: ${{ env.FORK }}
token: ${{ secrets.SYNC_TOKEN }}
@@ -98,27 +69,4 @@ jobs:
This PR was triggered by @${{ github.triggering_actor }} via ${{ github.event_name }}.
- <details>
- <summary>Commands</summary>
-
- Trigger actions by commenting on this PR:
-
- - `@conda-bot render` will run rendering workflows and commit and push any changes to this PR
- - `@conda-bot recreate` will recreate this PR, overwriting any edits that have been made to it
-
- </details>
-
###### Auto-generated by the [`update.yml`][update.yml] workflow, see ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}.
-
- - if: github.event.comment.body == '@conda-bot render'
- id: update
- name: Push changes
- run: git push --force-with-lease
-
- - if: always() && github.event_name == 'issue_comment'
- uses: peter-evans/create-or-update-comment at 71345be0265236311c031f5c7866368bd1eff043 # v4.0.0
- with:
- comment-id: ${{ github.event.comment.id }}
- reactions: ${{ (steps.create.conclusion == 'success' || steps.update.conclusion == 'success') && 'hooray' || 'confused' }}
- reactions-edit-mode: replace
- token: ${{ secrets.SYNC_TOKEN }}
=====================================
.pre-commit-config.yaml
=====================================
@@ -2,14 +2,9 @@
ci:
autofix_prs: false
repos:
- - repo: https://github.com/astral-sh/ruff-pre-commit
- rev: v0.12.0
- hooks:
- - id: ruff
- args: [ --fix ]
- - id: ruff-format
+ # generic verification and formatting
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v5.0.0
+ rev: v6.0.0
hooks:
# ensure syntaxes are valid
- id: check-toml
@@ -34,16 +29,27 @@ repos:
- id: check-shebang-scripts-are-executable
- id: debug-statements
- id: detect-private-key
+ - repo: https://github.com/adamchainz/blacken-docs
+ rev: 1.20.0
+ hooks:
+ # auto format Python codes within docstrings
+ - id: blacken-docs
+ - repo: https://github.com/astral-sh/ruff-pre-commit
+ rev: v0.15.15
+ hooks:
+ # lint & attempt to correct failures (e.g. pyupgrade)
+ - id: ruff-check
+ args: [--fix]
+ # compatible replacement for black
+ - id: ruff-format
+ - repo: https://github.com/python-jsonschema/check-jsonschema
+ rev: 0.37.2
+ hooks:
+ # verify github syntaxes
+ - id: check-github-workflows
+ - id: check-dependabot
- repo: meta
# see https://pre-commit.com/#meta-hooks
hooks:
- id: check-hooks-apply
- id: check-useless-excludes
- - repo: local
- hooks:
- - id: git-diff
- name: git diff
- entry: git diff --exit-code
- language: system
- pass_filenames: false
- always_run: true
=====================================
CHANGELOG.md
=====================================
@@ -1,5 +1,12 @@
[//]: # (current developments)
+## 0.13.0 (2026-06)
+
+* Use `compression.zstd` or `backports.zstd` (before / after Python 3.14), instead of
+ `zstandard / python-zstandard`. (#137)
+* Set minimum Python version to 3.10 (#137)
+* Accept pre-opened `ZipFile` via `zf=` kwarg on `stream_conda_component` (#173)
+
## 0.12.0 (2025-06)
* Skip setting permissions if `tarinfo.mode` is `None`. (#140)
=====================================
HOW_WE_USE_GITHUB.md
=====================================
@@ -1,23 +1,19 @@
<!-- edit this in https://github.com/conda/infrastructure -->
+# How We Use GitHub
<!-- absolute URLs -->
[conda-org]: https://github.com/conda
-[sub-team]: https://github.com/conda-incubator/governance#sub-teams
-[project-planning]: https://github.com/orgs/conda/projects/2/views/11
-[project-sorting]: https://github.com/orgs/conda/projects/2/views/11
-[project-support]: https://github.com/orgs/conda/projects/2/views/12
-[project-backlog]: https://github.com/orgs/conda/projects/2/views/13
-[project-in-progress]: https://github.com/orgs/conda/projects/2/views/14
+[project-refinement]: https://github.com/orgs/conda/projects/22/views/14
+[project-backlog]: https://github.com/orgs/conda/projects/22/views/2
+[project-current-sprint]: https://github.com/orgs/conda/projects/22/views/10
+[project-review]: https://github.com/orgs/conda/projects/16
[docs-toc]: https://github.blog/changelog/2021-04-13-table-of-contents-support-in-markdown-files/
-[docs-actions]: https://docs.github.com/en/actions
[docs-saved-reply]: https://docs.github.com/en/get-started/writing-on-github/working-with-saved-replies/creating-a-saved-reply
-[docs-commit-signing]: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits
[infrastructure]: https://github.com/conda/infrastructure
[workflow-sync]: https://github.com/conda/infrastructure/blob/main/.github/workflows/sync.yml
-[workflow-update]: https://github.com/conda/conda-package-streaming/blob/main/.github/workflows/update.yml
[labels-global]: https://github.com/conda/infrastructure/blob/main/.github/global.yml
<!-- relative URLs -->
@@ -30,21 +26,20 @@
[labels-local]: https://github.com/conda/conda-package-streaming/blob/main/.github/labels.yml
[labels-page]: https://github.com/conda/conda-package-streaming/labels
-# How We Use GitHub
-
This document seeks to outline how we as a community use GitHub Issues to track bugs and feature requests while still catering to development practices & project management (_e.g._, release cycles, feature planning, priority sorting, etc.).
<!-- only include high-level topics or particularly noteworthy sections here -->
**Topics:**
- - [What is "Issue Sorting"?](#what-is-issue-sorting)
- - [Issue Sorting Procedures](#issue-sorting-procedures)
- - [Commit Signing](#commit-signing)
- - [Types of Issues](#types-of-issues)
- - [Standard Issue](#standard-issue)
- - [Epics](#epics)
- - [Spikes](#spikes)
- - [Working on Issues](#working-on-issues)
+- [What is "Issue Sorting"?](#what-is-issue-sorting)
+- [Labeling](#labeling)
+- [Types of Issues](#types-of-issues)
+ - [Standard Issue](#standard-issue)
+ - [Epics](#epics)
+ - [Spikes](#spikes)
+- [Working on Issues](#working-on-issues)
+- [Development Processes](#development-processes)
+- [Code Review and Merging](#code-review-and-merging)
> [!NOTE]
> This document is written in the style of an FAQ. For easier navigation, use [GitHub's table of contents feature][docs-toc].
@@ -58,31 +53,27 @@ This document seeks to outline how we as a community use GitHub Issues to track
```mermaid
flowchart LR
- subgraph flow_sorting [Issue Sorting]
- board_sorting{{Sorting}}
- board_support{{Support}}
-
- board_sorting<-->board_support
+ subgraph flow_sorting [Issue Sorting in Issue Tracker]
+ state_sorting{{Maintainer sorting}}
end
- subgraph flow_refinement [Refinement]
+ subgraph flow_roadmap [Roadmap Board]
+ board_refinement{{Refinement}}
board_backlog{{Backlog}}
- board_backlog-- refine -->board_backlog
- end
-
- subgraph flow_progress [In Progress]
- board_progress{{In Progress}}
+ board_refinement-->board_backlog
+ board_backlog-- reprioritize -->board_backlog
+ board_progress{{Current Sprint - In Progress}}
end
state_new(New Issues)
state_closed(Closed)
- state_new-->board_sorting
- board_sorting-- investigated -->board_backlog
- board_sorting-- duplicates, off-topic -->state_closed
- board_support-- resolved, unresponsive -->state_closed
+ state_new-->state_sorting
+ state_sorting-- accepted for work -->board_refinement
+ state_sorting-- duplicate, off-topic, support resolved -->state_closed
board_backlog-- pending work -->board_progress
+ board_refinement-- not actionable -->state_closed
board_backlog-- resolved, irrelevant -->state_closed
board_progress-- resolved -->state_closed
```
@@ -98,109 +89,51 @@ At the most basic "bird's eye view" level, sorted issues will fall into the cate
At its core, sorting enables new issues to be placed into these four categories, which helps to ensure that they will be processed at a velocity similar to or exceeding the rate at which new issues are coming in. One of the benefits of actively sorting issues is to avoid engineer burnout and to make necessary work sustainable; this is done by eliminating a never-ending backlog that has not been reviewed by any maintainers.
-There will always be broad-scope design and architecture implementations that the maintainers will be interested in pursuing; by actively organizing issues, the sorting engineers will be able to more easily track and tackle both specific and big-picture goals.
+There will always be broad-scope design and architecture implementations that the maintainers will be interested in pursuing; by actively organizing issues, they will be able to more easily track and tackle both specific and big-picture goals.
### Who does the sorting?
-Sorting engineers are a conda governance [sub-team][sub-team]; they are a group of community members who are responsible for making decisions regarding closing issues and setting feature work priorities, among other sorting-related tasks.
-
-### How do items show up for sorting?
+Core maintainers help with sorting issues, making decisions regarding closing issues and setting feature work priorities, among other sorting-related tasks.
-New issues that are opened in any of the repositories in the [conda GitHub organization][conda-org] will show up in the "Sorting" tab of the [Planning project][project-planning]. There are two [GitHub Actions][docs-actions] workflows utilized for this purpose; [`.github/workflows/issues.yml`][workflow-issues] and [`.github/workflows/project.yml`][workflow-project].
+### How does issue sorting and board intake work?
-The GitHub workflows in the [`conda/infrastructure`][infrastructure] repository are viewed as canonical; the [`.github/workflows/sync.yml` workflow][workflow-sync] pushes any modifications to other repositories from there and individual repositories can pull additional files using the [`.github/workflows/update.yml`][workflow-update] workflow.
-
-### What is done about the issues in the "Sorting" tab?
-
-Issues in the ["Sorting" tab of the project board][project-sorting] are considered ready for the following procedures:
+New issues that are opened in any of the repositories in the [conda GitHub organization][conda-org] are reviewed in the repository issue tracker first. During sorting in the issue tracker, issues are reviewed for the following outcomes:
- Mitigation via short-term workarounds and fixes
- Redirection to the correct project
- Determining if support can be provided for errors and questions
- Closing out of any duplicate/off-topic issues
-The sorting engineers on rotation are not seeking to _resolve_ issues that arise. Instead, the goal is to understand the issue and to determine whether it is legitimate, and then to collect as much relevant information as possible so that the maintainers can make an informed decision about the appropriate resolution schedule.
-
-Issues will remain in the ["Sorting" tab][project-sorting] as long as the issue is in an investigatory phase (_e.g._, querying the user for more details, asking the user to attempt other workarounds, other debugging efforts, etc.) and are likely to remain in this state the longest, but should still be progressing over the course of 1-2 weeks.
-
-For more information on the sorting process, see [Issue Sorting Procedures](#issue-sorting-procedures).
+The core maintainers are not seeking to _resolve_ issues that arise. Instead, the goal is to understand the issue and to determine whether it is legitimate, and then to collect as much relevant information as possible so that the maintainers can make an informed decision about the appropriate resolution schedule.
-### When do items move out of the "Sorting" tab?
+Issues can remain in this investigatory phase (_e.g._, querying the user for more details, asking the user to attempt other workarounds, other debugging efforts, etc.) and are likely to remain in this state the longest, but should still be progressing over the course of 1-2 weeks.
-Items move out of the ["Sorting" tab][project-sorting] once the investigatory phase described in [What is done about the issues in the "Sorting" tab?](#what-is-done-about-the-issues-in-the-sorting-tab) has concluded and the sorting engineer has enough information to make a decision about the appropriate resolution schedule for the issue. The additional tabs in the project board that the issues can be moved to include the following:
+Items are added to the [Refinement tab of the Roadmap Board][project-refinement] once sorting has concluded and the core maintainer has enough information to make a decision about the appropriate resolution schedule for the issue. Newly opened pull requests are automatically added to the [Review board][project-review] by [`.github/workflows/project.yml`][workflow-project].
-- **"Support"** - Any issue in the ["Support" tab of the Planning board][project-support] is a request for support and is not a feature request or a bug report. Add the https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport label to move an issue to this tab.
-- **"Backlog"** - The issue has revealed a bug or feature request. We have collected enough details to understand the problem/request and to reproduce it on our own. These issues have been moved into the [Backlog tab of the Planning board][project-backlog] at the end of the sorting rotation during Refinement. Add the https://github.com/conda/conda-package-streaming/labels/backlog label to move an issue to this tab.
-- **"Closed"** - The issue was closed due to being a duplicate, being redirected to a different project, was a user error, a question that has been resolved, etc.
+Issues that are not accepted for planned work are closed instead (_e.g._ duplicates, redirects, user errors, resolved support questions, etc.).
### Where do work issues go after being sorted?
-Once issues are deemed ready to be worked on, they will be moved to the ["Backlog" tab of the Planning board][project-backlog]. Once actively in progress, the issues will be moved to the ["In Progress" tab of the Planning board][project-in-progress] and then closed out once the work is complete.
+Once issues are accepted for work, they are added to the ["Refinement" tab of the Roadmap Board][project-refinement]. After refinement and prioritization, issues move to ["Backlog"][project-backlog] and then to ["Current Sprint"][project-current-sprint] when actively being worked. Issues are closed once the work is complete.
### What is the purpose of having a "Backlog"?
-Issues are "backlogged" when they have been sorted but not yet earmarked for an upcoming release.
+Issues are "backlogged" when they have been accepted and refined but are not yet planned into the current sprint.
### What automation procedures are currently in place?
Global automation procedures synced out from the [`conda/infrastructure`][infrastructure] repo include:
-- [Marking of issues and pull requests as stale][workflow-stale], resulting in:
- - issues marked as https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport being labeled stale after 21 days of inactivity and being closed after 7 further days of inactivity (that is, closed after 30 inactive days total)
- - all other inactive issues (not labeled as https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport being labeled stale after 365 days of inactivity and being closed after 30 further days of inactivity (that is, closed after an approximate total of 1 year and 1 month of inactivity)
- - all inactive pull requests being labeled stale after 365 days of inactivity and being closed after 30 further days of inactivity (that is, closed after an approximate total of 1 year and 1 month of inactivity)
+- [Marking/Closing stale issues and pull requests][workflow-stale]:
+ - https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport issues are labeled as stale after 21 days of inactivity and are closed after 7 more days of inactivity (that is, closed after 30 inactive days total)
+ - non https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport issues are labeled as stale after 365 days of inactivity and are closed after 30 more days of inactivity (that is, closed after an approximate total of 1 year and 1 month of inactivity)
+ - all pull requests are labeled as stale after 365 days of inactivity and are closed after 30 more days of inactivity (that is, closed after an approximate total of 1 year and 1 month of inactivity)
- [Locking of closed issues and pull requests with no further activity][workflow-lock] after 365 days
-- [Adding new issues and pull requests to the respective project boards][workflow-project]
-- [Indicating an issue is ready for the sorting engineer's attention][workflow-issues] by toggling https://github.com/conda/conda-package-streaming/labels/pending%3A%3Afeedback with https://github.com/conda/conda-package-streaming/labels/pending%3A%3Asupport after a contributor leaves a comment
-- [Verifying that contributors have signed the CLA][workflow-cla] before allowing pull requests to be merged; if the contributor hasn't signed the CLA previously, merging is be blocked until a manual review can be done
+- [Adding new pull requests to the Review board][workflow-project]
+- [Indicating an issue is ready for a maintainer's attention][workflow-issues] by toggling https://github.com/conda/conda-package-streaming/labels/pending%3A%3Afeedback with https://github.com/conda/conda-package-streaming/labels/pending%3A%3Asupport after a contributor leaves a comment
+- [Verifying that contributors have signed the CLA][workflow-cla] before allowing pull requests to be merged; if the contributor hasn't signed the CLA previously, merging is blocked until a manual review can be done
- [Syncing out templates, labels, workflows, and documentation][workflow-sync] from [`conda/infrastructure`][infrastructure] to the other repositories
-## Issue Sorting Procedures
-
-### How are issues sorted?
-
-Issues in the ["Sorting" tab of the Planning board][project-sorting] are reviewed by issue sorting engineers, who take rotational sorting shifts. In the process of sorting issues, engineers label the issues and move them to the other tabs of the project board for further action.
-
-Issues that require input from multiple members of the sorting team will be brought up during refinement meetings in order to understand how those particular issues fit into the short- and long-term roadmap. These meetings enable the sorting engineers to get together to collectively prioritize issues, earmark feature requests for specific future releases (versus a more open-ended backlog), tag issues as ideal for first-time contributors, as well as whether or not to close/reject specific feature requests.
-
-### How does labeling work?
-
-Labeling is a very important means for sorting engineers to keep track of the current state of an issue with regards to the asynchronous nature of communicating with users. Utilizing the proper labels helps to identify the severity of the issue as well as to quickly understand the current state of a discussion.
-
-Each label has an associated description that clarifies how the label should be used. Hover on the label to see its description. Label colors are used to distinguish labels by category.
-
-Generally speaking, labels with the same category are considered mutually exclusive, but in some cases labels sharing the same category can occur concurrently, as they indicate qualifiers as opposed to types. For example, we may have the following types, https://github.com/conda/conda-package-streaming/labels/type%3A%3Abug, https://github.com/conda/conda-package-streaming/labels/type%3A%3Afeature, and https://github.com/conda/conda-package-streaming/labels/type%3A%3Adocumentation, where for any one issue there would be _at most_ **one** of these to be defined (_i.e._ an issue should not be a bug _and_ a feature request at the same time). Alternatively, with issues involving specific operating systems (_i.e._, https://github.com/conda/conda-package-streaming/labels/os%3A%3Alinux, https://github.com/conda/conda-package-streaming/labels/os%3A%3Amacos, and https://github.com/conda/conda-package-streaming/labels/os%3A%3Awindows), an issue could be labeled with one or more, depending on the system(s) the issue occurs on.
-
-Please note that there are also automation policies in place that are affected by labeling. For example, if an issue is labeled as https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport, that issue will be marked https://github.com/conda/conda-package-streaming/labels/stale after 21 days of inactivity and auto-closed after seven more days without activity (30 inactive days total), which is earlier than issues without this label. See [What automation procedures are currently in place?](#what-automation-procedures-are-currently-in-place) for more details.
-
-### What labels are required for each issue?
-
-At minimum, both `type` and `source` labels should be specified on each issue before moving it from the "Sorting" tab to the "Backlog" tab. All issues that are bugs should also be tagged with a `severity` label.
-
-The `type` labels are exclusive of each other: each sorted issue should have exactly one `type` label. These labels give high-level information on the issue's classification (_e.g._, bug, feature, tech debt, etc.)
-
-The `source` labels are exclusive of each other: each sorted issue should have exactly one `source` label. These labels give information on the sub-group to which the issue's author belongs (_e.g._, a partner, a frequent contributor, the wider community, etc.). Through these labels, maintainers gain insight into how well we're meeting the needs of various groups.
-
-The `severity` labels are exclusive of each other and, while required for the https://github.com/conda/conda-package-streaming/labels/type%3A%bug label, they can also be applied to other types to indicate demand or need. These labels help us to prioritize our work. Severity is not the only factor for work prioritization, but it is an important consideration.
-
-Please review the descriptions of the `type`, `source`, and `severity` labels on the [labels page][labels-page] prior to use.
-
-### How are new labels defined?
-
-Labels are defined using a scoped syntax with an optional high-level category (_e.g._, `source`, `tag`, `type`, etc.) and a specific topic, much like the following:
-
-- `[topic]`
-- `[category::topic]`
-- `[category::topic-phrase]`
-
-This syntax helps with issue sorting enforcement, as it helps to ensure that sorted issues are, at minimum, categorized by type and source.
-
-There are a number of labels that have been defined for the different repositories. In order to create a streamlined sorting process, label terminologies are standardized using similar (if not the same) labels.
-
-### How are new labels added?
-
-New **global** labels (_i.e._, labels that apply equally to all repositories within the conda GitHub organization) are added to [`conda/infrastructure`][infrastructure]'s [`.github/global.yml` file][labels-global]; new **local** labels (_i.e._, labels specific to particular repositories) are added to each repository's [`.github/labels.yml` file][labels-local]. All new labels should follow the labeling syntax described in ["How are new labels defined?"](#how-are-new-labels-defined). Global labels are combined with any local labels and these aggregated labels are used by the [`.github/workflows/labels.yml` workflow][workflow-labels] to synchronize the labels available for the repository.
-
### Are there any templates to use as responses for commonly-seen issues?
Some of the same types of issues appear regularly (_e.g._, issues that are duplicates of others, issues that should be filed in the Anaconda issue tracker, errors that are due to a user's specific setup/environment, etc.).
@@ -262,24 +195,59 @@ Community support can be found elsewhere, though, and we encourage you to explor
</details>
-
In order to not have to manually type or copy/paste the above repeatedly, note that it's possible to add text for the most commonly-used responses via [GitHub's "Add Saved Reply" option][docs-saved-reply].
-## Commit Signing
+## Labeling
+
+This section covers the labels and conventions used during issue sorting.
-For all maintainers, we require commit signing and strongly recommend it for all others wishing to contribute. More information about how to set this up within GitHub can be found here:
+### How does labeling work?
-- [GitHub's signing commits docs][docs-commit-signing]
+Labeling is a very important means for core maintainers to keep track of the current state of an issue with regards to the asynchronous nature of communicating with users. Utilizing the proper labels helps to identify the severity of the issue as well as to quickly understand the current state of a discussion.
+
+Each label has an associated description that clarifies how the label should be used. Hover on the label to see its description. Label colors are used to distinguish labels by category.
+
+Generally speaking, labels with the same category are considered mutually exclusive, but in some cases labels sharing the same category can occur concurrently, as they indicate qualifiers as opposed to types. For example, we may have the following types, https://github.com/conda/conda-package-streaming/labels/type%3A%3Abug, https://github.com/conda/conda-package-streaming/labels/type%3A%3Afeature, and https://github.com/conda/conda-package-streaming/labels/type%3A%3Adocumentation, where for any one issue there would be _at most_ **one** of these to be defined (_i.e._ an issue should not be a bug _and_ a feature request at the same time). Alternatively, with issues involving specific operating systems (_i.e._, https://github.com/conda/conda-package-streaming/labels/os%3A%3Alinux, https://github.com/conda/conda-package-streaming/labels/os%3A%3Amacos, and https://github.com/conda/conda-package-streaming/labels/os%3A%3Awindows), an issue could be labeled with one or more, depending on the system(s) the issue occurs on.
+
+Please note that there are also automation policies in place that are affected by labeling. For example, if an issue is labeled as https://github.com/conda/conda-package-streaming/labels/type%3A%3Asupport, that issue will be marked https://github.com/conda/conda-package-streaming/labels/stale after 21 days of inactivity and auto-closed after seven more days without activity (30 inactive days total), which is earlier than issues without this label. See [What automation procedures are currently in place?](#what-automation-procedures-are-currently-in-place) for more details.
+
+### What labels are required for each issue?
+
+At minimum, both `type` and `source` labels should be specified on each issue before adding it to the "Refinement" tab of the Roadmap Board. All issues that are bugs should also be tagged with a `severity` label.
+
+The `type` labels are exclusive of each other: each sorted issue should have exactly one `type` label. These labels give high-level information on the issue's classification (_e.g._, bug, feature, tech debt, etc.)
+
+The `source` labels are exclusive of each other: each sorted issue should have exactly one `source` label. These labels give information on the sub-group to which the issue's author belongs (_e.g._, a partner, a frequent contributor, the wider community, etc.). Through these labels, maintainers gain insight into how well we're meeting the needs of various groups.
+
+The `severity` labels are exclusive of each other and, while required for the https://github.com/conda/conda-package-streaming/labels/type%3A%3Abug label, they can also be applied to other types to indicate demand or need. These labels help us to prioritize our work. Severity is not the only factor for work prioritization, but it is an important consideration.
+
+Please review the descriptions of the `type`, `source`, and `severity` labels on the [labels page][labels-page] prior to use.
+
+### How are new labels defined?
+
+Labels are defined using a scoped syntax with an optional high-level category (_e.g._, `source`, `tag`, `type`, etc.) and a specific topic, much like the following:
+
+- `[topic]`
+- `[category::topic]`
+- `[category::topic-phrase]`
+
+This syntax helps with issue sorting enforcement, as it helps to ensure that sorted issues are, at minimum, categorized by type and source.
+
+There are a number of labels that have been defined for the different repositories. In order to create a streamlined sorting process, label terminologies are standardized using similar (if not the same) labels.
+
+### How are new labels added?
+
+New **global** labels (_i.e._, labels that apply equally to all repositories within the conda GitHub organization) are added to [`conda/infrastructure`][infrastructure]'s [`.github/global.yml` file][labels-global]; new **local** labels (_i.e._, labels specific to particular repositories) are added to each repository's [`.github/labels.yml` file][labels-local]. All new labels should follow the labeling syntax described in ["How are new labels defined?"](#how-are-new-labels-defined). Global labels are combined with any local labels and these aggregated labels are used by the [`.github/workflows/labels.yml` workflow][workflow-labels] to synchronize the labels available for the repository.
## Types of Issues
### Standard Issue
-TODO
+Standard issues represent typical bug reports, feature requests, or other work items that have a clear definition and expected outcome.
### Epics
-TODO
+Epics are large work items that can be broken down into smaller, more manageable issues. They typically represent major features or significant changes that span multiple iterations or releases. Relate the smaller issues to the epic using the sub-issues feature in GitHub.
### Spikes
@@ -314,3 +282,76 @@ If you do **not** have permissions, please indicate that you are working on an i
If you are assigned to an issue but will not be able to continue work on it, please comment to indicate that you will no longer be working on it and press `unassign me` next to your username in the `Assignees` section of the issue page (top right).
If you **do** have permissions, please assign yourself to the issue by pressing `assign myself` under the `Assignees` section of the issue page (top right).
+
+## Development Processes
+
+The following are practices the conda organization encourages for feature
+development. While we recommend projects under the conda organization adopt
+these practices, they are not strictly required.
+
+### How should we approach feature development?
+
+For new features, first open an issue if one doesn’t exist. Once the feature request
+has been accepted (indicated by the issue's status transitioning from "Sorting" to
+"Refinement"), create a specification to gather early feedback. This can include
+mockups, API/command references, a written plan in the issue, and sample CLI
+arguments (without functionality).
+
+### What is our change process?
+
+For larger features, break down the work into smaller, manageable issues
+that are added to the backlog. As long as a feature remains on the roadmap
+or backlog, do not create long-lived feature branches that span multiple
+pull requests. Instead, you should integrate small slices of an overall
+feature directly into the main branch to avoid complex integration challenges.
+
+### Should we make unrelated changes at the same time?
+
+When making changes, try to follow the Campsite Rule to leave things better
+than when you found them. You should enhance the code you encounter, even if
+primary goal is unrelated. This could involve refactoring small sections,
+improving readability, or fixing minor bugs.
+
+## Code Review and Merging
+
+### What are the review requirements?
+
+#### Standard Review
+
+Most code changes require one reviewer from someone on the maintainer team for
+the repository. Instead of waiting for someone on the team to review it,
+directly requesting a review from the person you previously identified to work
+with is preferred to optimize teamwork. If you paired with them during
+development, continuous review counts as this requirement.
+
+#### Second Review
+
+Required only when the code author or the first reviewer feels like it is
+necessary to get another set of eyes on a proposed change. In this case, they
+add someone specific through GitHub's Request Review feature with a comment on
+what they want the person to look for.
+
+### What are the code review best practices?
+
+If you are conducting a review, adhere to these best practices:
+
+- Provide comprehensive feedback in the first review to minimize review rounds
+- Reserve Request Changes for blocking issues (bugs or other major problems) —
+ Select Comment for suggestions and improvements
+- Follow-up reviews should focus on whether requested changes resolve original
+ comments
+- Code should be production-ready and maintainable when merged, but doesn't
+ need to be perfect
+- If providing feedback outside the core review focus (nitpicks, tips,
+ suggestions), clearly mark these as non-blocking comments that don't need to
+ be addressed before merging.
+
+### How do we merge code?
+
+If you are the approving reviewer (typically the first reviewer, or the second
+reviewer when needed) and you have completed your review and approved the
+changes, you should merge the code immediately to maintain development
+velocity.
+
+Normally, we use squash and merge to keep a clean git history. If you are
+merging a pull request, help ensure that the pull request title is updated.
=====================================
README.md
=====================================
@@ -25,6 +25,7 @@ caller to decide when to stop reading.
From a url,
```python
from conda_package_streaming.url import stream_conda_info
+
# url = (ends with .conda or .tar.bz2)
for tar, member in stream_conda_info(url):
if member.name == "info/index.json":
@@ -36,6 +37,7 @@ From s3,
```python
client = boto3.client("s3")
from conda_package_streaming.s3 import stream_conda_info
+
# key = (ends with .conda or .tar.bz2)
for tar, member in stream_conda_info(client, bucket, key):
if member.name == "info/index.json":
@@ -46,6 +48,7 @@ for tar, member in stream_conda_info(client, bucket, key):
From a filename,
```python
from conda_package_streaming import package_streaming
+
# filename = (ends with .conda or .tar.bz2)
for tar, member in package_streaming.stream_conda_info(filename):
if member.name == "info/index.json":
@@ -59,6 +62,7 @@ from contextlib import closing
from conda_package_streaming.url import conda_reader_for_url
from conda_package_streaming.package_streaming import stream_conda_component
+
filename, conda = conda_reader_for_url(url)
# file object must be seekable for `.conda` format, but merely readable for `.tar.bz2`
=====================================
conda.recipe/meta.yaml
=====================================
@@ -21,11 +21,11 @@ build:
requirements:
host:
- flit-core
- - python >=3.7
+ - python >=3.10
- pip
run:
- - zstandard >=0.15
- - python >=3.7
+ - backports.zstd
+ - python >=3.10
# allow optional 'requests'
test:
=====================================
conda_package_streaming/__init__.py
=====================================
@@ -1 +1 @@
-__version__ = "0.12.0"
+__version__ = "0.13.0"
=====================================
conda_package_streaming/create.py
=====================================
@@ -21,14 +21,32 @@ import zipfile
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
-from typing import Callable
+from typing import TYPE_CHECKING
-import zstandard
+try:
+ import compression.zstd as zstd
+except ImportError:
+ import backports.zstd as zstd
+
+if TYPE_CHECKING:
+ from collections.abc import Callable
+ from typing import BinaryIO, Protocol
+
+ class LegacyCompressor(Protocol):
+ def stream_writer(
+ self,
+ writer: BinaryIO,
+ *,
+ size: int,
+ closefd: bool,
+ ) -> BinaryIO: ...
+
+ LegacyCompressorOrFactory = LegacyCompressor | Callable[[], LegacyCompressor]
# increase to reduce speed and increase compression (levels above 19 use much
# more memory)
ZSTD_COMPRESS_LEVEL = 19
-# increase to reduce compression and increase speed
+# increase for greater speed
ZSTD_COMPRESS_THREADS = 1
CONDA_PACKAGE_FORMAT_VERSION = 2
@@ -90,16 +108,65 @@ class CondaTarFile(tarfile.TarFile):
return super().addfile(tarinfo, fileobj)
+class _ZstdFile(zstd.ZstdFile):
+ """
+ ZstdFile adding pledged_input_size.
+
+ For the older zstandard binding, a pledged size was important for
+ decompression memory usage. Check if that is still true in compression.zstd.
+ """
+
+ def __init__(self, *args, pledged_input_size=0, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._compressor.set_pledged_input_size(pledged_input_size)
+
+
+def _open_writer(
+ output,
+ *,
+ pledged_input_size: int,
+ compressor: LegacyCompressorOrFactory | None,
+ compression_level: int | None = None,
+ compression_threads: int | None = None,
+):
+ if compressor is not None:
+ if compression_level is not None or compression_threads is not None:
+ msg = "`compressor` overrides `compression_level` and `compression_threads`"
+ raise ValueError(msg)
+ legacy_compressor = compressor() if callable(compressor) else compressor
+ if not hasattr(legacy_compressor, "stream_writer"):
+ msg = "compressor must provide stream_writer(...)"
+ raise TypeError(msg)
+ return legacy_compressor.stream_writer(
+ output,
+ size=pledged_input_size,
+ closefd=False,
+ )
+
+ if compression_level is None:
+ compression_level = ZSTD_COMPRESS_LEVEL
+ if compression_threads is None:
+ compression_threads = ZSTD_COMPRESS_THREADS
+
+ return _ZstdFile(
+ output,
+ mode="w",
+ pledged_input_size=pledged_input_size,
+ options={
+ zstd.CompressionParameter.compression_level: compression_level,
+ zstd.CompressionParameter.nb_workers: compression_threads,
+ },
+ )
+
+
@contextmanager
def conda_builder(
stem,
path,
*,
- compressor: Callable[
- [], zstandard.ZstdCompressor
- ] = lambda: zstandard.ZstdCompressor(
- level=ZSTD_COMPRESS_LEVEL, threads=ZSTD_COMPRESS_THREADS
- ),
+ compressor: LegacyCompressorOrFactory | None = None,
+ compression_level: int | None = None,
+ compression_threads: int | None = None,
is_info: Callable[[str], bool] = lambda filename: filename.startswith("info/"),
encoding="utf-8",
) -> Iterator[CondaTarFile]:
@@ -114,8 +181,19 @@ def conda_builder(
Args:
stem: output filename without extension
- path: destination path for transmuted .conda package compressor: A
- function that creates instances of ``zstandard.ZstdCompressor()``.
+ path: destination path for transmuted .conda package.
+
+ compressor: Legacy ``zstandard`` compressor object (or factory
+ returning one) with ``stream_writer(...)``. Mutually exclusive with
+ ``compression_level`` and ``compression_threads``.
+
+ compression_level: zstd compression level for ``compression.zstd`` or
+ ``backports.zstd`` code path. Defaults to ``ZSTD_COMPRESS_LEVEL`` if not
+ specified and ``compressor`` is None.
+
+ compression_threads: Number of zstd worker threads for
+ ``compression.zstd`` or ``backports.zstd`` code path. Defaults to
+ ``ZSTD_COMPRESS_THREADS`` if not specified and ``compressor`` is None.
encoding: passed to TarFile constructor. Keep default "utf-8" for valid
.conda.
@@ -157,10 +235,6 @@ def conda_builder(
"x", # x to not append to existing
compresslevel=zipfile.ZIP_STORED,
) as conda_file:
- # Use a maximum of one Zstd compressor, stream_writer at a time to save
- # # memory.
- data_compress = compressor()
-
pkg_metadata = {"conda_pkg_format_version": CONDA_PACKAGE_FORMAT_VERSION}
conda_file.writestr("metadata.json", json.dumps(pkg_metadata))
@@ -170,8 +244,12 @@ def conda_builder(
"w",
force_zip64=(pkg_size > CONDA_ZIP64_LIMIT),
) as pkg_file_zip,
- data_compress.stream_writer(
- pkg_file_zip, size=pkg_size, closefd=False
+ _open_writer(
+ pkg_file_zip,
+ pledged_input_size=pkg_size,
+ compressor=compressor,
+ compression_level=compression_level,
+ compression_threads=compression_threads,
) as pkg_stream,
):
shutil.copyfileobj(pkg_file._file, pkg_stream)
@@ -182,10 +260,12 @@ def conda_builder(
"w",
force_zip64=(info_size > CONDA_ZIP64_LIMIT),
) as info_file_zip,
- data_compress.stream_writer(
+ _open_writer(
info_file_zip,
- size=info_size,
- closefd=False,
+ pledged_input_size=info_size,
+ compressor=compressor,
+ compression_level=compression_level,
+ compression_threads=compression_threads,
) as info_stream,
):
shutil.copyfileobj(info_file._file, info_stream)
=====================================
conda_package_streaming/package_streaming.py
=====================================
@@ -18,13 +18,18 @@ UMASK = os.umask(0)
os.umask(UMASK)
try:
- import zstandard
+ import compression.zstd as zstd # Python 3.14+
except ImportError:
- import warnings
+ try:
+ import backports.zstd as zstd
+ except ImportError:
+ import warnings
- warnings.warn("zstandard could not be imported. Running without .conda support.")
+ warnings.warn(
+ "zstd module could not be imported. Running without .conda support."
+ )
- zstandard = None
+ zstd = None
class CondaComponent(Enum):
@@ -117,6 +122,7 @@ def stream_conda_component(
component: CondaComponent | str = CondaComponent.pkg,
*,
encoding="utf-8",
+ zf: zipfile.ZipFile | None = None,
) -> Generator[tuple[tarfile.TarFile, tarfile.TarInfo]]:
"""
Yield members from .conda's embedded {component}- tarball. "info" or "pkg".
@@ -130,12 +136,28 @@ def stream_conda_component(
first result and then ignore the rest of this generator. ``extractall`` takes
care of some directory permissions/mtime issues, compared to ``extract`` or
writing out the file objects yourself.
+
+ Args:
+ filename: path or file-like object to the .conda or .tar.bz2 file
+
+ component: "pkg" or "info" component to extract
+
+ encoding: "utf-8" passed to TarFile.open(); can be changed for testing.
+
+ zf: optional pre-opened ``zipfile.ZipFile`` for .conda archives.
+ When provided, reuses the already-opened ZipFile instead of
+ parsing the central directory again. Callers that stream both
+ the ``pkg`` and ``info`` components can open the zip once and
+ pass it in both times, avoiding duplicate parsing. See #173.
"""
if str(filename).endswith(".conda"):
- if zstandard is None:
- raise RuntimeError("Cannot unpack `.conda` without zstandard")
+ if zstd is None:
+ raise RuntimeError(
+ "Cannot unpack `.conda` without `backports.zstd` or `compression.zstd`"
+ )
- zf = zipfile.ZipFile(fileobj or filename)
+ if zf is None:
+ zf = zipfile.ZipFile(fileobj or filename)
stem, _, _ = os.path.basename(filename).rpartition(".")
component_name = f"{component}-{stem}"
component_filename = [
@@ -144,9 +166,7 @@ def stream_conda_component(
if not component_filename:
raise LookupError(f"didn't find {component_name} component in {filename}")
assert len(component_filename) == 1
- reader = zstandard.ZstdDecompressor().stream_reader(
- zf.open(component_filename[0])
- )
+ reader = zstd.open(zf.open(component_filename[0]))
elif str(filename).endswith(".tar.bz2"):
reader = bz2.open(fileobj or filename, mode="rb")
else:
=====================================
conda_package_streaming/transmute.py
=====================================
@@ -13,25 +13,26 @@ import os
import tarfile
from collections.abc import Iterator
from pathlib import Path
-from typing import Callable
+from typing import TYPE_CHECKING
-import zstandard
-
-from .create import ZSTD_COMPRESS_LEVEL, ZSTD_COMPRESS_THREADS, conda_builder
+from .create import conda_builder
# streams everything in .tar.bz2 mode
from .package_streaming import CondaComponent, stream_conda_component
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+ from .create import LegacyCompressorOrFactory
+
def transmute(
package,
path,
*,
- compressor: Callable[
- [], zstandard.ZstdCompressor
- ] = lambda: zstandard.ZstdCompressor(
- level=ZSTD_COMPRESS_LEVEL, threads=ZSTD_COMPRESS_THREADS
- ),
+ compressor: LegacyCompressorOrFactory | None = None,
+ compression_level: int | None = None,
+ compression_threads: int | None = None,
is_info: Callable[[str], bool] = lambda filename: filename.startswith("info/"),
) -> Path:
"""
@@ -39,8 +40,15 @@ def transmute(
:param package: path to .tar.bz2 conda package
:param path: destination path for transmuted .conda package
- :param compressor: A function that creates instances of
- ``zstandard.ZstdCompressor()`` to override defaults.
+ :param compressor: Legacy ``zstandard`` compressor object (or factory
+ returning one) with ``stream_writer(...)``. Mutually exclusive with
+ ``compression_level`` and ``compression_threads``.
+ :param compression_level: zstd compression level for ``compression.zstd`` or
+ ``backports.zstd`` code path. Defaults to ``ZSTD_COMPRESS_LEVEL`` if not
+ specified and ``compressor`` is None.
+ :param compression_threads: Number of zstd worker threads for
+ ``compression.zstd`` or ``backports.zstd`` code path. Defaults to
+ ``ZSTD_COMPRESS_THREADS`` if not specified and ``compressor`` is None.
:param is_info: A function that returns True if a file belongs in the
``info`` component of a `.conda` package. ``conda-package-handling``
(not this package ``conda-package-streaming``) uses a set of regular
@@ -58,6 +66,8 @@ def transmute(
stem,
path,
compressor=compressor,
+ compression_level=compression_level,
+ compression_threads=compression_threads,
is_info=is_info,
package_stream=package_stream,
)
@@ -67,11 +77,9 @@ def transmute_stream(
stem,
path,
*,
- compressor: Callable[
- [], zstandard.ZstdCompressor
- ] = lambda: zstandard.ZstdCompressor(
- level=ZSTD_COMPRESS_LEVEL, threads=ZSTD_COMPRESS_THREADS
- ),
+ compressor: LegacyCompressorOrFactory | None = None,
+ compression_level: int | None = None,
+ compression_threads: int | None = None,
is_info: Callable[[str], bool] = lambda filename: filename.startswith("info/"),
package_stream: Iterator[tuple[tarfile.TarFile, tarfile.TarInfo]],
):
@@ -84,20 +92,28 @@ def transmute_stream(
.. code-block:: python
- transmute_stream(..., package_stream=itertools.chain(
- stream_conda_component("package.conda",
- component=CondaComponent.pkg),
- stream_conda_component("package.conda",
- component=CondaComponent.info),
- ))
+ transmute_stream(
+ ...,
+ package_stream=itertools.chain(
+ stream_conda_component("package.conda", component=CondaComponent.pkg),
+ stream_conda_component("package.conda", component=CondaComponent.info),
+ ),
+ )
This example could move files between the ``pkg-`` and ``info-`` components
depending on the ``is_info`` function.
:param stem: output filename without extension
:param path: destination path for transmuted .conda package
- :param compressor: A function that creates instances of
- ``zstandard.ZstdCompressor()`` to override defaults.
+ :param compressor: Legacy ``zstandard`` compressor object (or factory
+ returning one) with ``stream_writer(...)``. Mutually exclusive with
+ ``compression_level`` and ``compression_threads``.
+ :param compression_level: zstd compression level for ``compression.zstd`` or
+ ``backports.zstd`` code path. Defaults to ``ZSTD_COMPRESS_LEVEL`` if not
+ specified and ``compressor`` is None.
+ :param compression_threads: Number of zstd worker threads for
+ ``compression.zstd`` or ``backports.zstd`` code path. Defaults to
+ ``ZSTD_COMPRESS_THREADS`` if not specified and ``compressor`` is None.
:param is_info: A function that returns True if a file belongs in the
``info`` component of a `.conda` package. ``conda-package-handling``
(not this package ``conda-package-streaming``) uses a set of regular
@@ -108,7 +124,14 @@ def transmute_stream(
:return: Path to transmuted package.
"""
output_path = Path(path, f"{stem}.conda")
- with conda_builder(stem, path, compressor=compressor, is_info=is_info) as conda_tar:
+ with conda_builder(
+ stem,
+ path,
+ compressor=compressor,
+ compression_level=compression_level,
+ compression_threads=compression_threads,
+ is_info=is_info,
+ ) as conda_tar:
for tar, member in package_stream:
if member.isfile():
conda_tar.addfile(member, tar.extractfile(member))
=====================================
pyproject.toml
=====================================
@@ -17,8 +17,8 @@ classifiers = [
"Programming Language :: Python :: Implementation :: PyPy",
]
dynamic = ["version"]
-requires-python = ">=3.9"
-dependencies = ["requests", "zstandard >=0.15"]
+requires-python = ">=3.10"
+dependencies = ["requests", "backports.zstd ; python_version<'3.14'"]
[project.optional-dependencies]
test = [
@@ -38,17 +38,20 @@ docs = ["furo", "sphinx", "myst-parser", "mdit-py-plugins>=0.3.0"]
Home = "https://github.com/conda/conda-package-streaming"
Documentation = "https://conda.github.io/conda-package-streaming/"
+[tool.coverage.report]
+exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]
+
+[tool.coverage.run]
+source = ["conda_package_streaming/", "tests/"]
+
# pyproject.toml
[tool.pytest.ini_options]
minversion = "7.0"
addopts = "--cov=conda_package_streaming"
testpaths = ["tests"]
-[tool.coverage.report]
-exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]
-
-[tool.coverage.run]
-source = ["conda_package_streaming/", "tests/"]
+[tool.ruff]
+show-fixes = true
[tool.ruff.lint]
select = [
=====================================
tests/requirements.txt
=====================================
@@ -6,4 +6,3 @@ pytest >=7
pytest-cov
pytest-mock
requests
-zstandard >=0.15
=====================================
tests/test_degraded.py
=====================================
@@ -1,6 +1,6 @@
"""
-Allow conda_package_streaming to work in .tar.bz2-only mode if zstandard is not
-available (please immediately install zstandard if this is the case).
+Allow conda_package_streaming to work in .tar.bz2-only mode if zstd support is
+not available.
"""
import importlib
@@ -14,7 +14,8 @@ import pytest
def test_degraded(tmpdir):
try:
- sys.modules["zstandard"] = None # type: ignore
+ sys.modules["backports.zstd"] = None # type: ignore
+ sys.modules["compression.zstd"] = None # type: ignore
import conda_package_streaming.extract
import conda_package_streaming.package_streaming
@@ -48,9 +49,10 @@ def test_degraded(tmpdir):
conda_package_streaming.extract.extract(testconda, tmpdir)
finally:
- sys.modules.pop("zstandard", None)
+ sys.modules.pop("compression.zstd", None)
+ sys.modules.pop("backports.zstd", None)
import conda_package_streaming.package_streaming
importlib.reload(conda_package_streaming.package_streaming)
- assert conda_package_streaming.package_streaming.zstandard
+ assert conda_package_streaming.package_streaming.zstd
=====================================
tests/test_extract.py
=====================================
@@ -125,7 +125,7 @@ def test_slip(tmp_path):
extract.extract_stream(stream(tar2), tmp_path)
-def test_chown(conda_paths, tmp_path, mocker):
+def test_chown(conda_paths, tmp_path):
for package in conda_paths[:2]:
print(package)
with open(package, "rb") as fileobj:
@@ -142,7 +142,7 @@ def test_chown(conda_paths, tmp_path, mocker):
"tar_filter",
(pytest.param(None, id="no tar filter"), pytest.param("data", id="data_filter")),
)
-def test_umask(tmp_path, mocker, tar_filter):
+def test_umask(tmp_path, monkeypatch, tar_filter):
"""
Demonstrate that umask-respecting tar implementation works.
@@ -153,7 +153,9 @@ def test_umask(tmp_path, mocker, tar_filter):
try:
MOCK_UMASK = 0o022
current_umask = os.umask(MOCK_UMASK)
- mocker.patch("conda_package_streaming.package_streaming.UMASK", new=MOCK_UMASK)
+ monkeypatch.setattr(
+ "conda_package_streaming.package_streaming.UMASK", MOCK_UMASK
+ )
assert (
package_streaming.TarfileNoSameOwner(
@@ -191,7 +193,9 @@ def test_umask(tmp_path, mocker, tar_filter):
# specifically forbid that stat bit
MOCK_UMASK |= stat_check
- mocker.patch("conda_package_streaming.package_streaming.UMASK", new=MOCK_UMASK)
+ monkeypatch.setattr(
+ "conda_package_streaming.package_streaming.UMASK", MOCK_UMASK
+ )
os.umask(MOCK_UMASK)
root_path = tmp_path / "cps"
=====================================
tests/test_streaming.py
=====================================
@@ -35,7 +35,7 @@ def test_early_exit(conda_paths):
assert found, f"index.json not found in {package}"
-def test_chmod_error(tmp_path, mocker):
+def test_chmod_error(tmp_path, monkeypatch):
"""
Coverage for os.chmod() error handling.
"""
@@ -43,7 +43,10 @@ def test_chmod_error(tmp_path, mocker):
member = tarfile.TarInfo(name="file")
tar.addfile(member, io.BytesIO())
- mocker.patch("os.chmod", side_effect=OSError)
+ def not_chmod(*args):
+ raise OSError()
+
+ monkeypatch.setattr("os.chmod", not_chmod)
with pytest.raises(tarfile.ExtractError):
# only logs a debug message if errorlevel<=1
with package_streaming.TarfileNoSameOwner(
=====================================
tests/test_transmute.py
=====================================
@@ -8,9 +8,13 @@ from pathlib import Path
from zipfile import ZipFile
import pytest
-import zstandard
from conda_package_handling.validate import validate_converted_files_match_streaming
+try:
+ import compression.zstd as zstd
+except ImportError:
+ import backports.zstd as zstd
+
from conda_package_streaming.create import anonymize
from conda_package_streaming.package_streaming import (
CondaComponent,
@@ -141,7 +145,7 @@ def test_transmute_tarbz2_to_tarbz2(tmpdir, testtar_bytes):
assert missing == mismatched == []
-def test_transmute_conditional_zip64(tmp_path, mocker):
+def test_transmute_conditional_zip64(tmp_path, monkeypatch):
"""
Test that zip64 is used in transmute after a threshold.
"""
@@ -149,8 +153,8 @@ def test_transmute_conditional_zip64(tmp_path, mocker):
LIMIT = 16384
for test_size, extra_expected in (LIMIT // 2, False), (LIMIT * 2, True):
- mocker.patch("conda_package_streaming.create.CONDA_ZIP64_LIMIT", new=LIMIT)
- mocker.patch("zipfile.ZIP64_LIMIT", new=LIMIT)
+ monkeypatch.setattr("conda_package_streaming.create.CONDA_ZIP64_LIMIT", LIMIT)
+ monkeypatch.setattr("zipfile.ZIP64_LIMIT", LIMIT)
tmp_tar = tmp_path / f"{test_size}.tar.bz2"
with tarfile.open(tmp_tar, "w:bz2") as tar:
@@ -184,18 +188,28 @@ def test_transmute_stream(tmpdir, conda_paths):
conda_packages.append(path)
for package in conda_packages[:3]:
- file_id = package.name
+ file_id = package.stem
+
+ # also testing usage of LegacyCompressor here
+ class LegacyCompressor:
+ def stream_writer(self, writer, *, size, closefd):
+ return zstd.open(writer, mode="wb")
transmute_stream(
file_id,
tmpdir,
- compressor=lambda: zstandard.ZstdCompressor(),
+ compressor=LegacyCompressor(),
package_stream=itertools.chain(
stream_conda_component(package, component=CondaComponent.pkg),
stream_conda_component(package, component=CondaComponent.info),
),
)
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ tmpdir / package.name, package, strict=True
+ )
+ assert missing == mismatched == []
+
def test_anonymize_helper():
ti = tarfile.TarInfo(name="info")
@@ -205,3 +219,140 @@ def test_anonymize_helper():
assert anon.name == ti.name # they are also the same object
assert anon.uid == anon.gid == 0
assert anon.uname == anon.gname == ""
+
+
+def test_compression_params_mutual_exclusivity_level(tmpdir, testtar_bytes):
+ """Test for error if compressor= and level options are passed."""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ class LegacyCompressor:
+ def stream_writer(self, writer, *, size, closefd):
+ return zstd.open(writer, mode="wb")
+
+ with pytest.raises(
+ ValueError,
+ match="`compressor` overrides `compression_level` and `compression_threads`",
+ ):
+ transmute_stream(
+ "test",
+ tmpdir,
+ compressor=LegacyCompressor(),
+ compression_level=19,
+ package_stream=stream_conda_component(testtar),
+ )
+
+
+def test_compression_params_mutual_exclusivity_threads(tmpdir, testtar_bytes):
+ """Test for error if compressor= and threads options are passed."""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ class LegacyCompressor:
+ def stream_writer(self, writer, *, size, closefd):
+ return zstd.open(writer, mode="wb")
+
+ with pytest.raises(
+ ValueError,
+ match="`compressor`",
+ ):
+ transmute_stream(
+ "test",
+ tmpdir,
+ compressor=LegacyCompressor(),
+ compression_threads=4,
+ package_stream=stream_conda_component(testtar),
+ )
+
+
+def test_compression_params_mutual_exclusivity_both(tmpdir, testtar_bytes):
+ """Test that compressor is mutually exclusive with both compression parameters"""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ class LegacyCompressor:
+ def stream_writer(self, writer, *, size, closefd):
+ return zstd.open(writer, mode="wb")
+
+ with pytest.raises(
+ ValueError,
+ match="`compressor`",
+ ):
+ transmute_stream(
+ "test",
+ tmpdir,
+ compressor=LegacyCompressor(),
+ compression_level=19,
+ compression_threads=4,
+ package_stream=stream_conda_component(testtar),
+ )
+
+
+def test_compressor_missing_stream_writer(tmpdir, testtar_bytes):
+ """Test that compressor must have stream_writer method"""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ class BadCompressor:
+ """Compressor without stream_writer method"""
+
+ pass
+
+ with pytest.raises(
+ TypeError,
+ match="compressor must provide stream_writer",
+ ):
+ transmute_stream(
+ "test",
+ tmpdir,
+ compressor=BadCompressor(),
+ package_stream=stream_conda_component(testtar),
+ )
+
+
+def test_compression_level_thread_defaults(tmpdir, testtar_bytes):
+ """Test that compression_level and compression_threads use defaults when None"""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ # Transmute with None values should use defaults (ZSTD_COMPRESS_LEVEL=19,
+ # ZSTD_COMPRESS_THREADS=1)
+ #
+ # This test verifies that the defaults are applied and the .conda file is
+ # created successfully
+ out = transmute_stream(
+ "test",
+ tmpdir,
+ compression_level=None,
+ compression_threads=None,
+ package_stream=stream_conda_component(testtar),
+ )
+
+ assert out.exists()
+ assert out.name == "test.conda"
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
+
+
+def test_compression_explicit_values(tmpdir, testtar_bytes):
+ """Test that explicit compression parameters are used"""
+ testtar = Path(tmpdir, "test.tar.bz2")
+ testtar.write_bytes(testtar_bytes)
+
+ # Use explicit compression values
+ out = transmute_stream(
+ "test",
+ tmpdir,
+ compression_level=10,
+ compression_threads=2,
+ package_stream=stream_conda_component(testtar),
+ )
+
+ assert out.exists()
+ assert out.name == "test.conda"
+ _, missing, mismatched = validate_converted_files_match_streaming(
+ out, testtar, strict=True
+ )
+ assert missing == mismatched == []
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/commit/daa11965cb848330c6cfc7f62ded98e283efcac3
--
View it on GitLab: https://salsa.debian.org/med-team/conda-package-streaming/-/commit/daa11965cb848330c6cfc7f62ded98e283efcac3
You're receiving this email because of your account on salsa.debian.org. Manage all notifications: https://salsa.debian.org/-/profile/notifications | Help: https://salsa.debian.org/help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20260911/7d0031c3/attachment-0001.htm>
More information about the debian-med-commit
mailing list