[med-svn] [Git][med-team/qiime][master] 5 commits: New upstream version 2022.8.3
Étienne Mollier (@emollier)
gitlab at salsa.debian.org
Sat Oct 22 13:45:41 BST 2022
Étienne Mollier pushed to branch master at Debian Med / qiime
Commits:
329a4079 by Étienne Mollier at 2022-10-22T14:16:47+02:00
New upstream version 2022.8.3
- - - - -
a1f267b4 by Étienne Mollier at 2022-10-22T14:16:47+02:00
routine-update: New upstream version
- - - - -
a4c0cfcd by Étienne Mollier at 2022-10-22T14:16:49+02:00
Update upstream source from tag 'upstream/2022.8.3'
Update to upstream version '2022.8.3'
with Debian dir e9f54fc223ec8e77bbfe384fbe84772dee5494ca
- - - - -
e7c4ec94 by Étienne Mollier at 2022-10-22T14:16:52+02:00
routine-update: Remove trailing whitespace in debian/changelog
- - - - -
6f81e477 by Étienne Mollier at 2022-10-22T14:38:39+02:00
ready to upload to experimental
- - - - -
19 changed files:
- + .github/CONTRIBUTING.md
- + .github/ISSUE_TEMPLATE/1-user-need-help.md
- + .github/ISSUE_TEMPLATE/2-dev-need-help.md
- + .github/ISSUE_TEMPLATE/3-found-bug.md
- + .github/ISSUE_TEMPLATE/4-make-better.md
- + .github/ISSUE_TEMPLATE/5-make-new.md
- + .github/ISSUE_TEMPLATE/6-where-to-go.md
- + .github/SUPPORT.md
- + .github/pull_request_template.md
- + .github/rubric.png
- + .github/workflows/ci.yml
- debian/changelog
- qiime2/_version.py
- qiime2/core/archive/archiver.py
- qiime2/core/archive/tests/test_archiver.py
- qiime2/core/cache.py
- qiime2/core/tests/test_cache.py
- qiime2/core/util.py
- qiime2/jupyter/handlers.py
Changes:
=====================================
.github/CONTRIBUTING.md
=====================================
@@ -0,0 +1,23 @@
+# Contributing to this project
+
+Thanks for thinking of us :heart: :tada: - we would love a helping hand!
+
+## I just have a question
+
+> Note: Please don't file an issue to ask a question. You'll get faster results
+> by using the resources below.
+
+### QIIME 2 Users
+
+Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials,
+walkthroughs, and guides available. If you still need help, please visit us at
+the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support).
+
+### QIIME 2 Developers
+
+Check out the [Developer Docs](https://dev.qiime2.org) - there are many
+tutorials, walkthroughs, and guides available. If you still need help, please
+visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion).
+
+This document is based heavily on the following:
+https://github.com/atom/atom/blob/master/CONTRIBUTING.md
=====================================
.github/ISSUE_TEMPLATE/1-user-need-help.md
=====================================
@@ -0,0 +1,14 @@
+---
+name: I am a user and I need help with QIIME 2...
+about: I am using QIIME 2 and have a question or am experiencing a problem
+
+---
+
+Have you had a chance to check out the docs?
+https://docs.qiime2.org
+There are many tutorials, walkthroughs, and guides available.
+
+If you still need help, please visit:
+https://forum.qiime2.org/c/user-support
+
+Help requests filed here will not be answered.
=====================================
.github/ISSUE_TEMPLATE/2-dev-need-help.md
=====================================
@@ -0,0 +1,12 @@
+---
+name: I am a developer and I need help with QIIME 2...
+about: I am developing a QIIME 2 plugin or interface and have a question or a problem
+
+---
+
+Have you had a chance to check out the developer docs?
+https://dev.qiime2.org
+There are many tutorials, walkthroughs, and guides available.
+
+If you still need help, please visit:
+https://forum.qiime2.org/c/dev-discussion
=====================================
.github/ISSUE_TEMPLATE/3-found-bug.md
=====================================
@@ -0,0 +1,36 @@
+---
+name: I am a developer and I found a bug...
+about: I am a developer and I found a bug that I can describe
+
+---
+
+**Bug Description**
+A clear and concise description of what the bug is.
+
+**Steps to reproduce the behavior**
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Computation Environment**
+- OS: [e.g. macOS High Sierra]
+- QIIME 2 Release [e.g. 2018.6]
+
+**Questions**
+1. An enumerated list with any questions about the problem here.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list with any other context or comments about the problem here.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/4-make-better.md
=====================================
@@ -0,0 +1,26 @@
+---
+name: I am a developer and I have an idea for an improvement...
+about: I am a developer and I have an idea for an improvement to existing functionality
+
+---
+
+**Improvement Description**
+A clear and concise description of what the improvement is.
+
+**Current Behavior**
+Please provide a brief description of the current behavior.
+
+**Proposed Behavior**
+Please provide a brief description of the proposed behavior.
+
+**Questions**
+1. An enumerated list of questions related to the proposal.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list of comments related to the proposal that don't fit anywhere else.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/5-make-new.md
=====================================
@@ -0,0 +1,26 @@
+---
+name: I am a developer and I have an idea for a new feature...
+about: I am a developer and I have an idea for new functionality
+
+---
+
+**Addition Description**
+A clear and concise description of what the addition is.
+
+**Current Behavior**
+Please provide a brief description of the current behavior, if applicable.
+
+**Proposed Behavior**
+Please provide a brief description of the proposed behavior.
+
+**Questions**
+1. An enumerated list of questions related to the proposal.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list of comments related to the proposal that don't fit anywhere else.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/6-where-to-go.md
=====================================
@@ -0,0 +1,147 @@
+---
+name: I don't know where to file my issue...
+about: I am a developer and I don't know which repo to file this in
+
+---
+
+The repos within the QIIME 2 GitHub Organization are listed below, with a brief description about the repo.
+
+Sorted alphabetically by repo name.
+
+- The CI automation engine that builds and distributes QIIME 2
+ https://github.com/qiime2/busywork/issues
+
+- A Concourse resource for working with conda
+ https://github.com/qiime2/conda-channel-resource/issues
+
+- Web app for vanity URLs for QIIME 2 data assets
+ https://github.com/qiime2/data.qiime2.org/issues
+
+- The Developer Documentation
+ https://github.com/qiime2/dev-docs/issues
+
+- A discourse plugin for handling queued/unqueued topics
+ https://github.com/qiime2/discourse-unhandled-tagger/issues
+
+- The User Documentation
+ https://github.com/qiime2/docs/issues
+
+- Rendered QIIME 2 environment files for conda
+ https://github.com/qiime2/environment-files/issues
+
+- Google Sheets Add-On for validating tabular data
+ https://github.com/qiime2/Keemei/issues
+
+- A docker image for linux-based busywork workers
+ https://github.com/qiime2/linux-worker-docker/issues
+
+- Official project logos
+ https://github.com/qiime2/logos/issues
+
+- The q2-alignment plugin
+ https://github.com/qiime2/q2-alignment/issues
+
+- The q2-composition plugin
+ https://github.com/qiime2/q2-composition/issues
+
+- The q2-cutadapt plugin
+ https://github.com/qiime2/q2-cutadapt/issues
+
+- The q2-dada2 plugin
+ https://github.com/qiime2/q2-dada2/issues
+
+- The q2-deblur plugin
+ https://github.com/qiime2/q2-deblur/issues
+
+- The q2-demux plugin
+ https://github.com/qiime2/q2-demux/issues
+
+- The q2-diversity plugin
+ https://github.com/qiime2/q2-diversity/issues
+
+- The q2-diversity-lib plugin
+ https://github.com/qiime2/q2-diversity-lib/issues
+
+- The q2-emperor plugin
+ https://github.com/qiime2/q2-emperor/issues
+
+- The q2-feature-classifier plugin
+ https://github.com/qiime2/q2-feature-classifier/issues
+
+- The q2-feature-table plugin
+ https://github.com/qiime2/q2-feature-table/issues
+
+- The q2-fragment-insertion plugin
+ https://github.com/qiime2/q2-fragment-insertion/issues
+
+- The q2-gneiss plugin
+ https://github.com/qiime2/q2-gneiss/issues
+
+- The q2-longitudinal plugin
+ https://github.com/qiime2/q2-longitudinal/issues
+
+- The q2-metadata plugin
+ https://github.com/qiime2/q2-metadata/issues
+
+- The q2-phylogeny plugin
+ https://github.com/qiime2/q2-phylogeny/issues
+
+- The q2-quality-control plugin
+ https://github.com/qiime2/q2-quality-control/issues
+
+- The q2-quality-filter plugin
+ https://github.com/qiime2/q2-quality-filter/issues
+
+- The q2-sample-classifier plugin
+ https://github.com/qiime2/q2-sample-classifier/issues
+
+- The q2-shogun plugin
+ https://github.com/qiime2/q2-shogun/issues
+
+- The q2-taxa plugin
+ https://github.com/qiime2/q2-taxa/issues
+
+- The q2-types plugin
+ https://github.com/qiime2/q2-types/issues
+
+- The q2-vsearch plugin
+ https://github.com/qiime2/q2-vsearch/issues
+
+- The CLI interface
+ https://github.com/qiime2/q2cli/issues
+
+- The prototype CWL interface
+ https://github.com/qiime2/q2cwl/issues
+
+- The prototype Galaxy interface
+ https://github.com/qiime2/q2galaxy/issues
+
+- An internal tool for ensuring header text and copyrights are present
+ https://github.com/qiime2/q2lint/issues
+
+- The prototype GUI interface
+ https://github.com/qiime2/q2studio/issues
+
+- A base template for use in official QIIME 2 plugins
+ https://github.com/qiime2/q2templates/issues
+
+- The read-only web interface at view.qiime2.org
+ https://github.com/qiime2/q2view/issues
+
+- The QIIME 2 homepage at qiime2.org
+ https://github.com/qiime2/qiime2.github.io/issues
+
+- The QIIME 2 framework
+ https://github.com/qiime2/qiime2/issues
+
+- Centralized templates for repo assets
+ https://github.com/qiime2/template-repo/issues
+
+- Scripts for building QIIME 2 VMs
+ https://github.com/qiime2/vm-playbooks/issues
+
+- Scripts for building QIIME 2 workshop clusters
+ https://github.com/qiime2/workshop-playbooks/issues
+
+- The web app that runs workshops.qiime2.org
+ https://github.com/qiime2/workshops.qiime2.org/issues
=====================================
.github/SUPPORT.md
=====================================
@@ -0,0 +1,112 @@
+# QIIME 2 Users
+
+Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials,
+walkthroughs, and guides available. If you still need help, please visit us at
+the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support).
+
+# QIIME 2 Developers
+
+Check out the [Developer Docs](https://dev.qiime2.org) - there are many
+tutorials, walkthroughs, and guides available. If you still need help, please
+visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion).
+
+# General Bug/Issue Triage Discussion
+
+
+
+# Projects/Repositories in the QIIME 2 GitHub Organization
+
+Sorted alphabetically by repo name.
+
+- [busywork](https://github.com/qiime2/busywork/issues)
+ | The CI automation engine that builds and distributes QIIME 2
+- [conda-channel-resource](https://github.com/qiime2/conda-channel-resource/issues)
+ | A Concourse resource for working with conda
+- [data.qiime2.org](https://github.com/qiime2/data.qiime2.org/issues)
+ | Web app for vanity URLs for QIIME 2 data assets
+- [dev-docs](https://github.com/qiime2/dev-docs/issues)
+ | The Developer Documentation
+- [discourse-unhandled-tagger](https://github.com/qiime2/discourse-unhandled-tagger/issues)
+ | A discourse plugin for handling queued/unqueued topics
+- [docs](https://github.com/qiime2/docs/issues)
+ | The User Documentation
+- [environment-files](https://github.com/qiime2/environment-files/issues)
+ | Rendered QIIME 2 environment files for conda
+- [Keemei](https://github.com/qiime2/Keemei/issues)
+ | Google Sheets Add-On for validating tabular data
+- [linux-worker-docker](https://github.com/qiime2/linux-worker-docker/issues)
+ | A docker image for linux-based busywork workers
+- [logos](https://github.com/qiime2/logos/issues)
+ | Official project logos
+- [q2-alignment](https://github.com/qiime2/q2-alignment/issues)
+ | The q2-alignment plugin
+- [q2-composition](https://github.com/qiime2/q2-composition/issues)
+ | The q2-composition plugin
+- [q2-cutadapt](https://github.com/qiime2/q2-cutadapt/issues)
+ | The q2-cutadapt plugin
+- [q2-dada2](https://github.com/qiime2/q2-dada2/issues)
+ | The q2-dada2 plugin
+- [q2-deblur](https://github.com/qiime2/q2-deblur/issues)
+ | The q2-deblur plugin
+- [q2-demux](https://github.com/qiime2/q2-demux/issues)
+ | The q2-demux plugin
+- [q2-diversity](https://github.com/qiime2/q2-diversity/issues)
+ | The q2-diversity plugin
+- [q2-diversity-lib](https://github.com/qiime2/q2-diversity-lib/issues)
+ | The q2-diversity-lib plugin
+- [q2-emperor](https://github.com/qiime2/q2-emperor/issues)
+ | The q2-emperor plugin
+- [q2-feature-classifier](https://github.com/qiime2/q2-feature-classifier/issues)
+ | The q2-feature-classifier plugin
+- [q2-feature-table](https://github.com/qiime2/q2-feature-table/issues)
+ | The q2-feature-table plugin
+- [q2-fragment-insertion](https://github.com/qiime2/q2-fragment-insertion/issues)
+ | The q2-fragment-insertion plugin
+- [q2-gneiss](https://github.com/qiime2/q2-gneiss/issues)
+ | The q2-gneiss plugin
+- [q2-longitudinal](https://github.com/qiime2/q2-longitudinal/issues)
+ | The q2-longitudinal plugin
+- [q2-metadata](https://github.com/qiime2/q2-metadata/issues)
+ | The q2-metadata plugin
+- [q2-phylogeny](https://github.com/qiime2/q2-phylogeny/issues)
+ | The q2-phylogeny plugin
+- [q2-quality-control](https://github.com/qiime2/q2-quality-control/issues)
+ | The q2-quality-control plugin
+- [q2-quality-filter](https://github.com/qiime2/q2-quality-filter/issues)
+ | The q2-quality-filter plugin
+- [q2-sample-classifier](https://github.com/qiime2/q2-sample-classifier/issues)
+ | The q2-sample-classifier plugin
+- [q2-shogun](https://github.com/qiime2/q2-shogun/issues)
+ | The q2-shogun plugin
+- [q2-taxa](https://github.com/qiime2/q2-taxa/issues)
+ | The q2-taxa plugin
+- [q2-types](https://github.com/qiime2/q2-types/issues)
+ | The q2-types plugin
+- [q2-vsearch](https://github.com/qiime2/q2-vsearch/issues)
+ | The q2-vsearch plugin
+- [q2cli](https://github.com/qiime2/q2cli/issues)
+ | The CLI interface
+- [q2cwl](https://github.com/qiime2/q2cwl/issues)
+ | The prototype CWL interface
+- [q2galaxy](https://github.com/qiime2/q2galaxy/issues)
+ | The prototype Galaxy interface
+- [q2lint](https://github.com/qiime2/q2lint/issues)
+ | An internal tool for ensuring header text and copyrights are present
+- [q2studio](https://github.com/qiime2/q2studio/issues)
+ | The prototype GUI interface
+- [q2templates](https://github.com/qiime2/q2templates/issues)
+ | A base template for use in official QIIME 2 plugins
+- [q2view](https://github.com/qiime2/q2view/issues)
+ | The read-only web interface at view.qiime2.org
+- [qiime2.github.io](https://github.com/qiime2/qiime2.github.io/issues)
+ | The QIIME 2 homepage at qiime2.org
+- [qiime2](https://github.com/qiime2/qiime2/issues)
+ | The QIIME 2 framework
+- [template-repo](https://github.com/qiime2/template-repo/issues)
+ | Centralized templates for repo assets
+- [vm-playbooks](https://github.com/qiime2/vm-playbooks/issues)
+ | Scripts for building QIIME 2 VMs
+- [workshop-playbooks](https://github.com/qiime2/workshop-playbooks/issues)
+ | Scripts for building QIIME 2 workshop clusters
+- [workshops.qiime2.org](https://github.com/qiime2/workshops.qiime2.org/issues)
+ | The web app that runs workshops.qiime2.org
=====================================
.github/pull_request_template.md
=====================================
@@ -0,0 +1,11 @@
+Brief summary of the Pull Request, including any issues it may fix using the GitHub closing syntax:
+
+https://help.github.com/articles/closing-issues-using-keywords/
+
+Also, include any co-authors or contributors using the GitHub coauthor tag:
+
+https://help.github.com/articles/creating-a-commit-with-multiple-authors/
+
+---
+
+Include any questions for reviewers, screenshots, sample outputs, etc.
=====================================
.github/rubric.png
=====================================
Binary files /dev/null and b/.github/rubric.png differ
=====================================
.github/workflows/ci.yml
=====================================
@@ -0,0 +1,55 @@
+# This file is automatically generated by busywork.qiime2.org and
+# template-repos - any manual edits made to this file will be erased when
+# busywork performs maintenance updates.
+
+name: ci
+
+on:
+ pull_request:
+ push:
+ branches:
+ - 2022.8-patches
+
+jobs:
+ lint:
+ runs-on: ubuntu-latest
+ steps:
+ - name: checkout source
+ uses: actions/checkout at v2
+
+ - name: set up python 3.8
+ uses: actions/setup-python at v1
+ with:
+ python-version: 3.8
+
+ - name: install dependencies
+ run: python -m pip install --upgrade pip
+
+ - name: lint
+ run: |
+ pip install -q https://github.com/qiime2/q2lint/archive/master.zip
+ q2lint
+ pip install -q flake8
+ flake8
+
+ build-and-test:
+ needs: lint
+ strategy:
+ matrix:
+ os: [ubuntu-latest, macos-latest]
+ runs-on: ${{ matrix.os }}
+ steps:
+ - name: checkout source
+ uses: actions/checkout at v2
+ with:
+ fetch-depth: 0
+
+ - name: set up git repo for versioneer
+ run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+
+ - uses: qiime2/action-library-packaging at alpha1
+ with:
+ package-name: qiime2
+ build-target: release
+ additional-tests: QIIMETEST= py.test --pyargs qiime2
+ library-token: ${{ secrets.LIBRARY_TOKEN }}
=====================================
debian/changelog
=====================================
@@ -1,3 +1,10 @@
+qiime (2022.8.3-1) experimental; urgency=medium
+
+ * New upstream version
+ * Remove trailing whitespace in debian/changelog (routine-update)
+
+ -- Étienne Mollier <emollier at debian.org> Sat, 22 Oct 2022 14:16:59 +0200
+
qiime (2022.8.1-3) unstable; urgency=medium
* Team Upload.
@@ -27,7 +34,7 @@ qiime (2022.8.1-1) unstable; urgency=medium
qiime (2022.2.1-2) unstable; urgency=medium
- * d/control: Extended package description
+ * d/control: Extended package description
-- Steffen Moeller <moeller at debian.org> Sun, 10 Jul 2022 13:31:57 +0200
=====================================
qiime2/_version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (HEAD -> master, tag: 2022.8.1)"
- git_full = "047b6a46ac714c56e2571a14904852917847d91c"
- git_date = "2022-08-24 17:43:54 -0700"
+ git_refnames = " (tag: 2022.8.3, 2022.8-patches)"
+ git_full = "d0845af30335929f51f59860eb4a7cd931c4c45d"
+ git_date = "2022-09-08 18:48:41 -0700"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
=====================================
qiime2/core/archive/archiver.py
=====================================
@@ -255,6 +255,22 @@ class _NoOpArchive(_Archive):
self.uuid, self.version, self.framework_version)
+class ArchiveCheck(_Archive):
+ """Used by the Jupyter handlers"""
+
+ # TODO: make this part of the archiver API at some point
+ def open(self, relpath):
+ abspath = os.path.join(str(self.path), relpath)
+ return open(abspath, 'r')
+
+ def relative_iterdir(self, relpath='.'):
+ for p in pathlib.Path(self.path).iterdir():
+ yield str(p.relative_to(self.path))
+
+ def _get_uuid(self):
+ return os.path.basename(self.path)
+
+
class Archiver:
CURRENT_FORMAT_VERSION = '5'
_FORMAT_REGISTRY = {
=====================================
qiime2/core/archive/tests/test_archiver.py
=====================================
@@ -15,7 +15,7 @@ import pathlib
from qiime2.core.archive import Archiver
from qiime2.core.archive import ImportProvenanceCapture
-from qiime2.core.archive.archiver import _ZipArchive
+from qiime2.core.archive.archiver import _ZipArchive, ArchiveCheck
from qiime2.core.archive.format.util import artifact_version
from qiime2.core.testing.format import IntSequenceDirectoryFormat
from qiime2.core.testing.type import IntSequence1
@@ -324,6 +324,26 @@ class TestArchiver(unittest.TestCase, ArchiveTestingMixin):
self.assertEqual(diff.removed, {})
self.assertEqual(diff.changed, {})
+ def test_archive_check(self):
+ """Rough test of our machinery to support showing visualizations in
+ Jupyter notebooks without actually spoofing the notebook
+ """
+ archive = ArchiveCheck(self.archiver.path)
+
+ # Make sure this _get_uuid actually works
+ self.assertEqual(archive._get_uuid(), archive.uuid)
+
+ expected = set([
+ 'metadata.yaml',
+ 'data',
+ 'checksums.md5',
+ 'provenance',
+ 'VERSION'
+ ])
+
+ observed = set(file for file in archive.relative_iterdir())
+ self.assertEqual(observed, expected)
+
if __name__ == '__main__':
unittest.main()
=====================================
qiime2/core/cache.py
=====================================
@@ -16,20 +16,21 @@ import psutil
import shutil
import getpass
import pathlib
+import weakref
import tempfile
+import warnings
import threading
from sys import maxsize
from random import randint
from datetime import timedelta
-
from flufl.lock import Lock
import qiime2
from .path import ArchivePath
from qiime2.sdk.result import Result
-from qiime2.core.util import (is_uuid4, set_permissions, READ_ONLY_FILE,
- READ_ONLY_DIR, ALL_PERMISSIONS)
+from qiime2.core.util import (is_uuid4, set_permissions, touch_under_path,
+ READ_ONLY_FILE, READ_ONLY_DIR, ALL_PERMISSIONS)
from qiime2.core.archive.archiver import Archiver
_VERSION_TEMPLATE = """\
@@ -56,6 +57,9 @@ _CACHE.temp_cache = None
# Keep track of every cache used by this process for cleanup later
USED_CACHES = set()
+# These permissions are directory with sticky bit and rwx for all set
+EXPECTED_PERMISSIONS = 0o41777
+
def get_cache():
""" Gets our cache if we have one and creates one in temp if we don't
@@ -127,6 +131,15 @@ def _exit_cleanup():
cache.garbage_collection()
+def monitor_thread(cache_dir, is_done):
+ """This function will be running in a seperate thread and making sure Mac
+ doesn't cull our stuff by updating its access and modified times
+ """
+ while not is_done.is_set():
+ touch_under_path(cache_dir)
+ time.sleep(60 * 60 * 6)
+
+
class Cache:
"""General structure of the cache (tmp optional)
artifact_cache/
@@ -173,8 +186,22 @@ class Cache:
if not os.path.exists(self.path):
self._create_cache()
elif not self.is_cache(self.path):
- raise ValueError(f"Path: \'{path}\' already exists and is not a"
- " cache")
+ # MacOS culls files in the temp dir that haven't been used for a
+ # few days. This can lead to the VERSION file being deleted while
+ # we still have a cache dir, so we see the directory but don't
+ # think it's a cache. Our solution is to just kill this directory
+ # and recreate it. We only do this on the temp cache which is not
+ # storing anything long term anyway
+ if path is None:
+ warnings.warn("Your temporary cache was found to be in an "
+ "inconsistent state. It has been recreated.")
+ set_permissions(self.path, ALL_PERMISSIONS, ALL_PERMISSIONS)
+ shutil.rmtree(self.path)
+ self._create_cache()
+ else:
+ raise ValueError(
+ f"Path: \'{self.path}\' already exists and is not a "
+ "cache.")
self.lock = Lock(str(self.lockfile), lifetime=timedelta(minutes=10))
# Make our process pool.
@@ -188,6 +215,19 @@ class Cache:
# We were used by this process
USED_CACHES.add(self)
+ # Start thread that pokes things in the cache to ensure they aren't
+ # culled for being too old (only if we are in a temp cache)
+ if path is None:
+ self._thread_is_done = threading.Event()
+ self._thread_destructor = \
+ weakref.finalize(self, self._thread_is_done.set)
+
+ self._thread = threading.Thread(
+ target=monitor_thread, args=(self.path, self._thread_is_done),
+ daemon=True)
+
+ self._thread.start()
+
def __enter__(self):
"""Set this cache on the thread local
"""
@@ -204,6 +244,21 @@ class Cache:
"""
_CACHE.cache = None
+ def __getstate__(self):
+ """We don't want to pickle any of the thread stuff because it won't
+ rehydrate, and we don't care about it
+ """
+ threadless_dict = self.__dict__.copy()
+
+ # This will only even exist if we are a temp cache not a named cache.
+ # If _thread exists the others should as well
+ if '_thread' in threadless_dict:
+ del threadless_dict['_thread_is_done']
+ del threadless_dict['_thread_destructor']
+ del threadless_dict['_thread']
+
+ return threadless_dict
+
@classmethod
def is_cache(cls, path):
"""Tells us if the path we were given is a cache
@@ -250,18 +305,29 @@ class Cache:
tmpdir = tempfile.gettempdir()
cache_dir = os.path.join(tmpdir, 'qiime2')
- if not os.path.exists(cache_dir):
- os.mkdir(cache_dir)
# Make sure the sticky bit is set on the cache directory. Documentation
# on what a sitcky bit is found here
# https://docs.python.org/3/library/stat.html#stat.S_ISVTX
# We also set read/write/execute permissions for everyone on this
- # directory
- permissions = os.stat(cache_dir).st_mode
- sticky_permissions = permissions | stat.S_ISVTX | stat.S_IRWXU | \
- stat.S_IRGRP | stat.S_IRWXO
- os.chmod(cache_dir, sticky_permissions)
+ # directory. We only do this if we are the owner of the /tmp/qiime2
+ # directory or in other words the first person to run QIIME 2 with this
+ # /tmp since the /tmp was wiped
+ if not os.path.exists(cache_dir):
+ os.mkdir(cache_dir)
+ sticky_permissions = stat.S_ISVTX | stat.S_IRWXU | stat.S_IRWXG \
+ | stat.S_IRWXO
+ os.chmod(cache_dir, sticky_permissions)
+ elif os.stat(cache_dir).st_mode != EXPECTED_PERMISSIONS:
+ raise ValueError(f"Directory '{cache_dir}' already exists without "
+ f"proper permissions "
+ f"'{oct(EXPECTED_PERMISSIONS)}' set. Current "
+ "permissions are "
+ f"'{oct(os.stat(cache_dir).st_mode)}.' This most "
+ "likely means something other than QIIME 2 "
+ f"created the directory '{cache_dir}' or QIIME 2 "
+ f"failed between creating '{cache_dir}' and "
+ "setting permissions on it.")
user = _get_user()
user_dir = os.path.join(cache_dir, user)
=====================================
qiime2/core/tests/test_cache.py
=====================================
@@ -8,15 +8,23 @@
import os
import gc
+import pwd
+import crypt
+import shutil
+import string
import atexit
+import psutil
+import random
+import platform
import tempfile
import unittest
+from contextlib import contextmanager
import pytest
from flufl.lock import LockState
import qiime2
-from qiime2.core.cache import Cache, _exit_cleanup, get_cache
+from qiime2.core.cache import Cache, _exit_cleanup, get_cache, _get_user
from qiime2.core.testing.type import IntSequence1, IntSequence2
from qiime2.core.testing.util import get_dummy_plugin
from qiime2.sdk.result import Artifact
@@ -65,6 +73,49 @@ def _on_exit_validate(cache, expected):
assert expected.issubset(observed)
+ at contextmanager
+def _fake_user_for_cache(cache_prefix, i_acknowledge_this_is_dangerous=False):
+ """Creates a fake user with a uname that is 8 random alphanumeric
+ characters that we ensure does not collide with an existing uname and
+ create a cache for said user under cache_prefix
+ """
+ if not i_acknowledge_this_is_dangerous:
+ raise ValueError('YOU MUST ACCEPT THE DANGER OF LETTING THIS SCRIPT '
+ 'MAKE AND REMOVE A USER')
+
+ if not os.getegid() == 0:
+ raise ValueError('This action requires super user permissions which '
+ 'you do not have')
+
+ user_list = psutil.users()
+ uname = ''.join(random.choices(string.ascii_letters + string.digits, k=8))
+
+ # Highly unlikely this will ever happen, but we really don't want to
+ # have collisions here
+ while uname in user_list:
+ uname = ''.join(
+ random.choices(string.ascii_letters + string.digits, k=8))
+
+ password = crypt.crypt('test', '22')
+ os.system(f'useradd -p {password} {uname}')
+
+ os.seteuid(pwd.getpwnam(uname).pw_uid)
+ # seteuid does not convice getpass.getuser we are not root because it uses
+ # getuid not geteuid. I cannot use setuid because then I would not be able
+ # to get root permissions back, so I give it the cache path manually under
+ # tmp. This should be functionally no different as far as permissions on
+ # /tmp/qiime2 are concerned. It still thinks we are not root as far as
+ # file system operations go
+ user_cache = Cache(os.path.join(cache_prefix, uname))
+
+ try:
+ yield (uname, user_cache)
+ finally:
+ os.seteuid(0)
+ os.system(f'userdel {uname}')
+ shutil.rmtree(user_cache.path)
+
+
class TestCache(unittest.TestCase):
def setUp(self):
# Create temp test dir
@@ -347,7 +398,9 @@ class TestCache(unittest.TestCase):
self.assertIn(uuid, os.listdir(self.cache.data))
self.assertIn(uuid, os.listdir(self.cache.pools / 'pool'))
- def test_asynchronous_pool_post_exit(self):
+ # This test has zzz in front of it because unittest.Testcase runs the tests
+ # in alphabetical order, and we want this test to run last
+ def test_zzz_asynchronous_pool_post_exit(self):
"""This test determines if all of the data is still in the cache when
we are getting ready to exit. This was put here when ensuring we do not
destroy our data when running asynchronous actions, and it can probably
@@ -396,3 +449,65 @@ class TestCache(unittest.TestCase):
f"Permission denied: '{target}'"):
with open(target, mode='w') as fh:
fh.write('extra file')
+
+ @pytest.mark.skipif(
+ os.geteuid() != 0, reason="only sudo can mess with users")
+ @pytest.mark.skipif(
+ platform.system() == "Darwin",
+ reason="Mac clusters not really a thing")
+ def test_multi_user(self):
+ """This test determines if we can have multiple users successfully
+ accessing the cache under the /tmp/qiime2 directory. This test came
+ from this issue https://github.com/qiime2/qiime2/issues/639. It should
+ only run as root because only root can create and delete users, and for
+ now at least it won't run on Mac
+ """
+ plugin = get_dummy_plugin()
+ concatenate_ints = plugin.methods['concatenate_ints']
+
+ root_cache = get_cache()
+ root_user = _get_user()
+
+ # This should ensure that the /tmp/qiime2/root cache exists and has
+ # things in it
+ with root_cache:
+ root_result = \
+ concatenate_ints(self.art1, self.art2, self.art4, 4, 5)[0]
+
+ root_expected = set((
+ './VERSION', f'data/{root_result._archiver.uuid}'
+ ))
+
+ # The location we put the root cache in is also where we want the fake
+ # user cache
+ cache_prefix = os.path.split(root_cache.path)[0]
+ # Temporarily create a new user and user cache for multi-user testing
+ # purposes
+ with _fake_user_for_cache(
+ cache_prefix,
+ i_acknowledge_this_is_dangerous=True) as (uname, user_cache):
+ with user_cache:
+ user_result = concatenate_ints(
+ self.art1, self.art2, self.art4, 4, 5)[0]
+
+ user_expected = set((
+ './VERSION', f'data/{user_result._archiver.uuid}',
+ ))
+
+ self.assertEqual(os.path.basename(root_cache.path), root_user)
+ self.assertEqual(os.path.basename(user_cache.path), uname)
+
+ root_observed = _get_cache_contents(root_cache)
+ user_observed = _get_cache_contents(user_cache)
+
+ self.assertTrue(root_expected.issubset(root_observed))
+ self.assertTrue(user_expected.issubset(user_observed))
+
+ def test_inconsistent_cache(self):
+ cache = Cache()
+ (cache.path / 'VERSION').unlink()
+
+ del cache
+
+ with self.assertWarnsRegex(UserWarning, "in an inconsistent state"):
+ Cache()
=====================================
qiime2/core/util.py
=====================================
@@ -285,8 +285,32 @@ def set_permissions(path, file_permissions=None, dir_permissions=None):
"""
for directory, _, files in os.walk(path):
if dir_permissions:
- os.chmod(directory, dir_permissions)
+ try:
+ os.chmod(directory, dir_permissions)
+ except FileNotFoundError:
+ pass
for file in files:
if file_permissions:
- os.chmod(os.path.join(directory, file), file_permissions)
+ try:
+ os.chmod(os.path.join(directory, file), file_permissions)
+ except FileNotFoundError:
+ pass
+
+
+def touch_under_path(path):
+ """Touches everything under a given path to ensure they don't get culled by
+ Mac
+ """
+ for directory, _, files in os.walk(path):
+ try:
+ os.utime(directory, None, follow_symlinks=False)
+ except FileNotFoundError:
+ pass
+
+ for file in files:
+ try:
+ os.utime(
+ os.path.join(directory, file), None, follow_symlinks=False)
+ except FileNotFoundError:
+ pass
=====================================
qiime2/jupyter/handlers.py
=====================================
@@ -12,19 +12,7 @@ import pathlib
import tornado.web as web
from notebook.base.handlers import IPythonHandler
-import qiime2.core.archive.archiver as archiver
-
-
-class _ArchiveCheck(archiver._Archive):
- """This is only what is needed to verify a path is an archive"""
- # TODO: make this part of the archiver API at some point
- def open(self, relpath):
- abspath = os.path.join(str(self.path), str(self.uuid), relpath)
- return open(abspath, 'r')
-
- def relative_iterdir(self, relpath='.'):
- for p in pathlib.Path(self.path).iterdir():
- yield str(p.relative_to(self.path))
+from qiime2.core.archive.archiver import ArchiveCheck
class QIIME2RedirectHandler(IPythonHandler):
@@ -39,9 +27,8 @@ class QIIME2RedirectHandler(IPythonHandler):
self.send_error(409) # Conflict
return
# is it actually a QIIME 2 result, or a random part of the filesystem
- archive = _ArchiveCheck(pathlib.Path(location))
- self.result_store[archive.uuid] = os.path.join(
- location, str(archive.uuid), 'data')
+ archive = ArchiveCheck(pathlib.Path(location))
+ self.result_store[archive.uuid] = os.path.join(location, 'data')
self.redirect('view/%s/' % archive.uuid)
View it on GitLab: https://salsa.debian.org/med-team/qiime/-/compare/662f5c315d2b1c2e7f5082c91d8567bb67f4e224...6f81e4770f565491943c431690b9a4239ffb6d46
--
View it on GitLab: https://salsa.debian.org/med-team/qiime/-/compare/662f5c315d2b1c2e7f5082c91d8567bb67f4e224...6f81e4770f565491943c431690b9a4239ffb6d46
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20221022/2a4b82ba/attachment-0001.htm>
More information about the debian-med-commit
mailing list