[med-svn] [Git][med-team/q2cli][master] 8 commits: New upstream version 2022.8.0
Andreas Tille (@tille)
gitlab at salsa.debian.org
Thu Jan 12 19:19:19 GMT 2023
Andreas Tille pushed to branch master at Debian Med / q2cli
Commits:
7f1d1d25 by Nilesh Patra at 2022-09-05T19:10:39+05:30
New upstream version 2022.8.0
- - - - -
c68797f8 by Andreas Tille at 2023-01-12T18:43:13+01:00
Bump versioned Depends
- - - - -
09a6e214 by Andreas Tille at 2023-01-12T18:43:24+01:00
New upstream version 2022.11.1
- - - - -
b75e2c62 by Andreas Tille at 2023-01-12T18:43:24+01:00
routine-update: New upstream version
- - - - -
bf27048e by Andreas Tille at 2023-01-12T18:43:25+01:00
Update upstream source from tag 'upstream/2022.11.1'
Update to upstream version '2022.11.1'
with Debian dir ab30ccbd7e5889ca5006bb38811f204468f8f0b8
- - - - -
32202c40 by Andreas Tille at 2023-01-12T18:43:25+01:00
routine-update: Standards-Version: 4.6.2
- - - - -
4c45736e by Andreas Tille at 2023-01-12T18:52:33+01:00
Test-Depends: python3-sklearn
- - - - -
72b86e92 by Andreas Tille at 2023-01-12T20:17:48+01:00
Upload to experimental
- - - - -
23 changed files:
- + .github/CONTRIBUTING.md
- + .github/ISSUE_TEMPLATE/1-user-need-help.md
- + .github/ISSUE_TEMPLATE/2-dev-need-help.md
- + .github/ISSUE_TEMPLATE/3-found-bug.md
- + .github/ISSUE_TEMPLATE/4-make-better.md
- + .github/ISSUE_TEMPLATE/5-make-new.md
- + .github/ISSUE_TEMPLATE/6-where-to-go.md
- + .github/SUPPORT.md
- + .github/pull_request_template.md
- + .github/rubric.png
- + .github/workflows/add-to-project-ci.yml
- + .github/workflows/ci.yml
- debian/changelog
- debian/control
- debian/tests/control
- q2cli/_version.py
- q2cli/builtin/tools.py
- q2cli/click/type.py
- q2cli/commands.py
- + q2cli/tests/test_cache_cli.py
- q2cli/tests/test_cli.py
- q2cli/tests/test_tools.py
- q2cli/util.py
Changes:
=====================================
.github/CONTRIBUTING.md
=====================================
@@ -0,0 +1,23 @@
+# Contributing to this project
+
+Thanks for thinking of us :heart: :tada: - we would love a helping hand!
+
+## I just have a question
+
+> Note: Please don't file an issue to ask a question. You'll get faster results
+> by using the resources below.
+
+### QIIME 2 Users
+
+Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials,
+walkthroughs, and guides available. If you still need help, please visit us at
+the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support).
+
+### QIIME 2 Developers
+
+Check out the [Developer Docs](https://dev.qiime2.org) - there are many
+tutorials, walkthroughs, and guides available. If you still need help, please
+visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion).
+
+This document is based heavily on the following:
+https://github.com/atom/atom/blob/master/CONTRIBUTING.md
=====================================
.github/ISSUE_TEMPLATE/1-user-need-help.md
=====================================
@@ -0,0 +1,14 @@
+---
+name: I am a user and I need help with QIIME 2...
+about: I am using QIIME 2 and have a question or am experiencing a problem
+
+---
+
+Have you had a chance to check out the docs?
+https://docs.qiime2.org
+There are many tutorials, walkthroughs, and guides available.
+
+If you still need help, please visit:
+https://forum.qiime2.org/c/user-support
+
+Help requests filed here will not be answered.
=====================================
.github/ISSUE_TEMPLATE/2-dev-need-help.md
=====================================
@@ -0,0 +1,12 @@
+---
+name: I am a developer and I need help with QIIME 2...
+about: I am developing a QIIME 2 plugin or interface and have a question or a problem
+
+---
+
+Have you had a chance to check out the developer docs?
+https://dev.qiime2.org
+There are many tutorials, walkthroughs, and guides available.
+
+If you still need help, please visit:
+https://forum.qiime2.org/c/dev-discussion
=====================================
.github/ISSUE_TEMPLATE/3-found-bug.md
=====================================
@@ -0,0 +1,36 @@
+---
+name: I am a developer and I found a bug...
+about: I am a developer and I found a bug that I can describe
+
+---
+
+**Bug Description**
+A clear and concise description of what the bug is.
+
+**Steps to reproduce the behavior**
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Computation Environment**
+- OS: [e.g. macOS High Sierra]
+- QIIME 2 Release [e.g. 2018.6]
+
+**Questions**
+1. An enumerated list with any questions about the problem here.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list with any other context or comments about the problem here.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/4-make-better.md
=====================================
@@ -0,0 +1,26 @@
+---
+name: I am a developer and I have an idea for an improvement...
+about: I am a developer and I have an idea for an improvement to existing functionality
+
+---
+
+**Improvement Description**
+A clear and concise description of what the improvement is.
+
+**Current Behavior**
+Please provide a brief description of the current behavior.
+
+**Proposed Behavior**
+Please provide a brief description of the proposed behavior.
+
+**Questions**
+1. An enumerated list of questions related to the proposal.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list of comments related to the proposal that don't fit anywhere else.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/5-make-new.md
=====================================
@@ -0,0 +1,26 @@
+---
+name: I am a developer and I have an idea for a new feature...
+about: I am a developer and I have an idea for new functionality
+
+---
+
+**Addition Description**
+A clear and concise description of what the addition is.
+
+**Current Behavior**
+Please provide a brief description of the current behavior, if applicable.
+
+**Proposed Behavior**
+Please provide a brief description of the proposed behavior.
+
+**Questions**
+1. An enumerated list of questions related to the proposal.
+2. If not applicable, please delete this section.
+
+**Comments**
+1. An enumerated list of comments related to the proposal that don't fit anywhere else.
+2. If not applicable, please delete this section.
+
+**References**
+1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
+2. If not applicable, please delete this section.
=====================================
.github/ISSUE_TEMPLATE/6-where-to-go.md
=====================================
@@ -0,0 +1,147 @@
+---
+name: I don't know where to file my issue...
+about: I am a developer and I don't know which repo to file this in
+
+---
+
+The repos within the QIIME 2 GitHub Organization are listed below, with a brief description about the repo.
+
+Sorted alphabetically by repo name.
+
+- The CI automation engine that builds and distributes QIIME 2
+ https://github.com/qiime2/busywork/issues
+
+- A Concourse resource for working with conda
+ https://github.com/qiime2/conda-channel-resource/issues
+
+- Web app for vanity URLs for QIIME 2 data assets
+ https://github.com/qiime2/data.qiime2.org/issues
+
+- The Developer Documentation
+ https://github.com/qiime2/dev-docs/issues
+
+- A discourse plugin for handling queued/unqueued topics
+ https://github.com/qiime2/discourse-unhandled-tagger/issues
+
+- The User Documentation
+ https://github.com/qiime2/docs/issues
+
+- Rendered QIIME 2 environment files for conda
+ https://github.com/qiime2/environment-files/issues
+
+- Google Sheets Add-On for validating tabular data
+ https://github.com/qiime2/Keemei/issues
+
+- A docker image for linux-based busywork workers
+ https://github.com/qiime2/linux-worker-docker/issues
+
+- Official project logos
+ https://github.com/qiime2/logos/issues
+
+- The q2-alignment plugin
+ https://github.com/qiime2/q2-alignment/issues
+
+- The q2-composition plugin
+ https://github.com/qiime2/q2-composition/issues
+
+- The q2-cutadapt plugin
+ https://github.com/qiime2/q2-cutadapt/issues
+
+- The q2-dada2 plugin
+ https://github.com/qiime2/q2-dada2/issues
+
+- The q2-deblur plugin
+ https://github.com/qiime2/q2-deblur/issues
+
+- The q2-demux plugin
+ https://github.com/qiime2/q2-demux/issues
+
+- The q2-diversity plugin
+ https://github.com/qiime2/q2-diversity/issues
+
+- The q2-diversity-lib plugin
+ https://github.com/qiime2/q2-diversity-lib/issues
+
+- The q2-emperor plugin
+ https://github.com/qiime2/q2-emperor/issues
+
+- The q2-feature-classifier plugin
+ https://github.com/qiime2/q2-feature-classifier/issues
+
+- The q2-feature-table plugin
+ https://github.com/qiime2/q2-feature-table/issues
+
+- The q2-fragment-insertion plugin
+ https://github.com/qiime2/q2-fragment-insertion/issues
+
+- The q2-gneiss plugin
+ https://github.com/qiime2/q2-gneiss/issues
+
+- The q2-longitudinal plugin
+ https://github.com/qiime2/q2-longitudinal/issues
+
+- The q2-metadata plugin
+ https://github.com/qiime2/q2-metadata/issues
+
+- The q2-phylogeny plugin
+ https://github.com/qiime2/q2-phylogeny/issues
+
+- The q2-quality-control plugin
+ https://github.com/qiime2/q2-quality-control/issues
+
+- The q2-quality-filter plugin
+ https://github.com/qiime2/q2-quality-filter/issues
+
+- The q2-sample-classifier plugin
+ https://github.com/qiime2/q2-sample-classifier/issues
+
+- The q2-shogun plugin
+ https://github.com/qiime2/q2-shogun/issues
+
+- The q2-taxa plugin
+ https://github.com/qiime2/q2-taxa/issues
+
+- The q2-types plugin
+ https://github.com/qiime2/q2-types/issues
+
+- The q2-vsearch plugin
+ https://github.com/qiime2/q2-vsearch/issues
+
+- The CLI interface
+ https://github.com/qiime2/q2cli/issues
+
+- The prototype CWL interface
+ https://github.com/qiime2/q2cwl/issues
+
+- The prototype Galaxy interface
+ https://github.com/qiime2/q2galaxy/issues
+
+- An internal tool for ensuring header text and copyrights are present
+ https://github.com/qiime2/q2lint/issues
+
+- The prototype GUI interface
+ https://github.com/qiime2/q2studio/issues
+
+- A base template for use in official QIIME 2 plugins
+ https://github.com/qiime2/q2templates/issues
+
+- The read-only web interface at view.qiime2.org
+ https://github.com/qiime2/q2view/issues
+
+- The QIIME 2 homepage at qiime2.org
+ https://github.com/qiime2/qiime2.github.io/issues
+
+- The QIIME 2 framework
+ https://github.com/qiime2/qiime2/issues
+
+- Centralized templates for repo assets
+ https://github.com/qiime2/template-repo/issues
+
+- Scripts for building QIIME 2 VMs
+ https://github.com/qiime2/vm-playbooks/issues
+
+- Scripts for building QIIME 2 workshop clusters
+ https://github.com/qiime2/workshop-playbooks/issues
+
+- The web app that runs workshops.qiime2.org
+ https://github.com/qiime2/workshops.qiime2.org/issues
=====================================
.github/SUPPORT.md
=====================================
@@ -0,0 +1,112 @@
+# QIIME 2 Users
+
+Check out the [User Docs](https://docs.qiime2.org) - there are many tutorials,
+walkthroughs, and guides available. If you still need help, please visit us at
+the [QIIME 2 Forum](https://forum.qiime2.org/c/user-support).
+
+# QIIME 2 Developers
+
+Check out the [Developer Docs](https://dev.qiime2.org) - there are many
+tutorials, walkthroughs, and guides available. If you still need help, please
+visit us at the [QIIME 2 Forum](https://forum.qiime2.org/c/dev-discussion).
+
+# General Bug/Issue Triage Discussion
+
+
+
+# Projects/Repositories in the QIIME 2 GitHub Organization
+
+Sorted alphabetically by repo name.
+
+- [busywork](https://github.com/qiime2/busywork/issues)
+ | The CI automation engine that builds and distributes QIIME 2
+- [conda-channel-resource](https://github.com/qiime2/conda-channel-resource/issues)
+ | A Concourse resource for working with conda
+- [data.qiime2.org](https://github.com/qiime2/data.qiime2.org/issues)
+ | Web app for vanity URLs for QIIME 2 data assets
+- [dev-docs](https://github.com/qiime2/dev-docs/issues)
+ | The Developer Documentation
+- [discourse-unhandled-tagger](https://github.com/qiime2/discourse-unhandled-tagger/issues)
+ | A discourse plugin for handling queued/unqueued topics
+- [docs](https://github.com/qiime2/docs/issues)
+ | The User Documentation
+- [environment-files](https://github.com/qiime2/environment-files/issues)
+ | Rendered QIIME 2 environment files for conda
+- [Keemei](https://github.com/qiime2/Keemei/issues)
+ | Google Sheets Add-On for validating tabular data
+- [linux-worker-docker](https://github.com/qiime2/linux-worker-docker/issues)
+ | A docker image for linux-based busywork workers
+- [logos](https://github.com/qiime2/logos/issues)
+ | Official project logos
+- [q2-alignment](https://github.com/qiime2/q2-alignment/issues)
+ | The q2-alignment plugin
+- [q2-composition](https://github.com/qiime2/q2-composition/issues)
+ | The q2-composition plugin
+- [q2-cutadapt](https://github.com/qiime2/q2-cutadapt/issues)
+ | The q2-cutadapt plugin
+- [q2-dada2](https://github.com/qiime2/q2-dada2/issues)
+ | The q2-dada2 plugin
+- [q2-deblur](https://github.com/qiime2/q2-deblur/issues)
+ | The q2-deblur plugin
+- [q2-demux](https://github.com/qiime2/q2-demux/issues)
+ | The q2-demux plugin
+- [q2-diversity](https://github.com/qiime2/q2-diversity/issues)
+ | The q2-diversity plugin
+- [q2-diversity-lib](https://github.com/qiime2/q2-diversity-lib/issues)
+ | The q2-diversity-lib plugin
+- [q2-emperor](https://github.com/qiime2/q2-emperor/issues)
+ | The q2-emperor plugin
+- [q2-feature-classifier](https://github.com/qiime2/q2-feature-classifier/issues)
+ | The q2-feature-classifier plugin
+- [q2-feature-table](https://github.com/qiime2/q2-feature-table/issues)
+ | The q2-feature-table plugin
+- [q2-fragment-insertion](https://github.com/qiime2/q2-fragment-insertion/issues)
+ | The q2-fragment-insertion plugin
+- [q2-gneiss](https://github.com/qiime2/q2-gneiss/issues)
+ | The q2-gneiss plugin
+- [q2-longitudinal](https://github.com/qiime2/q2-longitudinal/issues)
+ | The q2-longitudinal plugin
+- [q2-metadata](https://github.com/qiime2/q2-metadata/issues)
+ | The q2-metadata plugin
+- [q2-phylogeny](https://github.com/qiime2/q2-phylogeny/issues)
+ | The q2-phylogeny plugin
+- [q2-quality-control](https://github.com/qiime2/q2-quality-control/issues)
+ | The q2-quality-control plugin
+- [q2-quality-filter](https://github.com/qiime2/q2-quality-filter/issues)
+ | The q2-quality-filter plugin
+- [q2-sample-classifier](https://github.com/qiime2/q2-sample-classifier/issues)
+ | The q2-sample-classifier plugin
+- [q2-shogun](https://github.com/qiime2/q2-shogun/issues)
+ | The q2-shogun plugin
+- [q2-taxa](https://github.com/qiime2/q2-taxa/issues)
+ | The q2-taxa plugin
+- [q2-types](https://github.com/qiime2/q2-types/issues)
+ | The q2-types plugin
+- [q2-vsearch](https://github.com/qiime2/q2-vsearch/issues)
+ | The q2-vsearch plugin
+- [q2cli](https://github.com/qiime2/q2cli/issues)
+ | The CLI interface
+- [q2cwl](https://github.com/qiime2/q2cwl/issues)
+ | The prototype CWL interface
+- [q2galaxy](https://github.com/qiime2/q2galaxy/issues)
+ | The prototype Galaxy interface
+- [q2lint](https://github.com/qiime2/q2lint/issues)
+ | An internal tool for ensuring header text and copyrights are present
+- [q2studio](https://github.com/qiime2/q2studio/issues)
+ | The prototype GUI interface
+- [q2templates](https://github.com/qiime2/q2templates/issues)
+ | A base template for use in official QIIME 2 plugins
+- [q2view](https://github.com/qiime2/q2view/issues)
+ | The read-only web interface at view.qiime2.org
+- [qiime2.github.io](https://github.com/qiime2/qiime2.github.io/issues)
+ | The QIIME 2 homepage at qiime2.org
+- [qiime2](https://github.com/qiime2/qiime2/issues)
+ | The QIIME 2 framework
+- [template-repo](https://github.com/qiime2/template-repo/issues)
+ | Centralized templates for repo assets
+- [vm-playbooks](https://github.com/qiime2/vm-playbooks/issues)
+ | Scripts for building QIIME 2 VMs
+- [workshop-playbooks](https://github.com/qiime2/workshop-playbooks/issues)
+ | Scripts for building QIIME 2 workshop clusters
+- [workshops.qiime2.org](https://github.com/qiime2/workshops.qiime2.org/issues)
+ | The web app that runs workshops.qiime2.org
=====================================
.github/pull_request_template.md
=====================================
@@ -0,0 +1,11 @@
+Brief summary of the Pull Request, including any issues it may fix using the GitHub closing syntax:
+
+https://help.github.com/articles/closing-issues-using-keywords/
+
+Also, include any co-authors or contributors using the GitHub coauthor tag:
+
+https://help.github.com/articles/creating-a-commit-with-multiple-authors/
+
+---
+
+Include any questions for reviewers, screenshots, sample outputs, etc.
=====================================
.github/rubric.png
=====================================
Binary files /dev/null and b/.github/rubric.png differ
=====================================
.github/workflows/add-to-project-ci.yml
=====================================
@@ -0,0 +1,21 @@
+name: Add new issues and PRs to triage project board
+
+on:
+ issues:
+ types:
+ - opened
+ pull_request_target:
+ types:
+ - opened
+
+jobs:
+ add-to-project:
+ name: Add issue to project
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/add-to-project at v0.3.0
+ with:
+ project-url: https://github.com/orgs/qiime2/projects/36
+ github-token: ${{ secrets.ADD_TO_PROJECT_PAT }}
+ labeled: skip-triage
+ label-operator: NOT
=====================================
.github/workflows/ci.yml
=====================================
@@ -0,0 +1,55 @@
+# This file is automatically generated by busywork.qiime2.org and
+# template-repos - any manual edits made to this file will be erased when
+# busywork performs maintenance updates.
+
+name: ci
+
+on:
+ pull_request:
+ push:
+ branches:
+ - master
+
+jobs:
+ lint:
+ runs-on: ubuntu-latest
+ steps:
+ - name: checkout source
+ uses: actions/checkout at v2
+
+ - name: set up python 3.8
+ uses: actions/setup-python at v1
+ with:
+ python-version: 3.8
+
+ - name: install dependencies
+ run: python -m pip install --upgrade pip
+
+ - name: lint
+ run: |
+ pip install -q https://github.com/qiime2/q2lint/archive/master.zip
+ q2lint
+ pip install -q flake8
+ flake8
+
+ build-and-test:
+ needs: lint
+ strategy:
+ matrix:
+ os: [ubuntu-latest, macos-latest]
+ runs-on: ${{ matrix.os }}
+ steps:
+ - name: checkout source
+ uses: actions/checkout at v2
+ with:
+ fetch-depth: 0
+
+ - name: set up git repo for versioneer
+ run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+
+ - uses: qiime2/action-library-packaging at alpha1
+ with:
+ package-name: q2cli
+ build-target: dev
+ additional-tests: QIIMETEST= py.test --pyargs q2cli
+ library-token: ${{ secrets.LIBRARY_TOKEN }}
=====================================
debian/changelog
=====================================
@@ -1,3 +1,13 @@
+q2cli (2022.11.1-1) experimental; urgency=medium
+
+ * Team upload.
+ * New upstream version
+ * Bump versioned Depends
+ * Standards-Version: 4.6.2 (routine-update)
+ * Test-Depends: python3-sklearn
+
+ -- Andreas Tille <tille at debian.org> Thu, 12 Jan 2023 18:43:42 +0100
+
q2cli (2022.8.0-1) unstable; urgency=medium
* Team upload.
=====================================
debian/control
=====================================
@@ -9,9 +9,9 @@ Build-Depends: debhelper-compat (= 13),
python3,
python3-pytest <!nocheck>,
python3-setuptools,
- qiime (>= 2022.8.0),
+ qiime (>= 2022.11.1),
python3-click
-Standards-Version: 4.6.1
+Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/med-team/q2cli
Vcs-Git: https://salsa.debian.org/med-team/q2cli.git
Homepage: https://qiime2.org/
@@ -22,8 +22,8 @@ Architecture: all
Depends: ${shlibs:Depends},
${misc:Depends},
${python3:Depends},
- q2-feature-table (>= 2021.8.0),
- qiime (>= 2021.8.0),
+ q2-feature-table (>= 2021.11.1),
+ qiime (>= 2021.11.1),
python3-setuptools,
python3-click
Description: Click-based command line interface for QIIME 2
=====================================
debian/tests/control
=====================================
@@ -1,3 +1,3 @@
Tests: run-unit-test
-Depends: @, q2-sample-classifier, file
+Depends: @, q2-sample-classifier, file, python3-sklearn
Restrictions: allow-stderr
=====================================
q2cli/_version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (tag: 2022.8.0)"
- git_full = "469e5eb69d7fe6b213f628d299fff1545cdec9f7"
- git_date = "2022-08-22 21:21:57 +0000"
+ git_refnames = " (HEAD -> master, tag: 2022.11.1)"
+ git_full = "da2a37b63494b67a5bc7a8b9f5b226b5cfd1f921"
+ git_date = "2022-12-21 21:36:54 +0000"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
=====================================
q2cli/builtin/tools.py
=====================================
@@ -177,22 +177,69 @@ def import_data(type, input_path, output_path, input_format):
help="Display basic information about a QIIME 2 Artifact or "
"Visualization, including its UUID and type.",
cls=ToolCommand)
- at click.argument('path', type=click.Path(exists=True, file_okay=True,
- dir_okay=False, readable=True),
- metavar=_COMBO_METAVAR)
-def peek(path):
+ at click.argument('paths', nargs=-1, required=True,
+ type=click.Path(exists=True, file_okay=True, dir_okay=False,
+ readable=True), metavar=_COMBO_METAVAR)
+ at click.option('--tsv/--no-tsv', default=False,
+ help='Print as machine-readable tab separated values.')
+def peek(paths, tsv):
import qiime2.sdk
from q2cli.core.config import CONFIG
- metadata = qiime2.sdk.Result.peek(path)
+ metadatas = {os.path.basename(path):
+ qiime2.sdk.Result.peek(path) for path in paths}
- click.echo(CONFIG.cfg_style('type', "UUID")+": ", nl=False)
- click.echo(metadata.uuid)
- click.echo(CONFIG.cfg_style('type', "Type")+": ", nl=False)
- click.echo(metadata.type)
- if metadata.format is not None:
- click.echo(CONFIG.cfg_style('type', "Data format")+": ", nl=False)
- click.echo(metadata.format)
+ if tsv:
+ click.echo("Filename\tType\tUUID\tData Format")
+ for path, m in metadatas.items():
+ click.echo(f"{path}\t{m.type}\t{m.uuid}\t{m.format}")
+
+ elif len(metadatas) == 1:
+ metadata = metadatas[os.path.basename(paths[0])]
+ click.echo(CONFIG.cfg_style('type', "UUID")+": ", nl=False)
+ click.echo(metadata.uuid)
+ click.echo(CONFIG.cfg_style('type', "Type")+": ", nl=False)
+ click.echo(metadata.type)
+ if metadata.format is not None:
+ click.echo(CONFIG.cfg_style('type', "Data format")+": ", nl=False)
+ click.echo(metadata.format)
+
+ else:
+ COLUMN_FILENAME = "Filename"
+ COLUMN_TYPE = "Type"
+ COLUMN_UUID = "UUID"
+ COLUMN_DATA_FORMAT = "Data Format"
+
+ filename_width = max([len(p) for p in paths]
+ + [len(COLUMN_FILENAME)])
+ type_width = max([len(i.type) for i in metadatas.values()]
+ + [len(COLUMN_TYPE)])
+ uuid_width = max([len(i.uuid) for i in metadatas.values()]
+ + [len(COLUMN_UUID)])
+ data_format_width = \
+ max([len(i.format) if i.format is not None else 0
+ for i in metadatas.values()] + [len(COLUMN_DATA_FORMAT)])
+
+ padding = 2
+ format_string = f"{{f:<{filename_width + padding}}} " + \
+ f"{{t:<{type_width + padding}}} " + \
+ f"{{u:<{uuid_width + padding}}} " + \
+ f"{{d:<{data_format_width + padding}}}"
+
+ click.secho(
+ format_string.format(
+ f=COLUMN_FILENAME,
+ t=COLUMN_TYPE,
+ u=COLUMN_UUID,
+ d=COLUMN_DATA_FORMAT),
+ bold=True, fg="green")
+ for path, m in metadatas.items():
+ click.echo(
+ format_string.format(
+ f=path,
+ t=m.type,
+ u=m.uuid,
+ d=(m.format if m.format is not None else 'N/A')))
_COLUMN_TYPES = ['categorical', 'numeric']
@@ -324,8 +371,7 @@ def cast_metadata(paths, cast, output_file, ignore_extra,
@click.option('--tsv/--no-tsv', default=False,
help='Print as machine-readable TSV instead of text.')
@click.argument('paths', nargs=-1, required=True, metavar='METADATA...',
- type=click.Path(exists=True, file_okay=True, dir_okay=False,
- readable=True))
+ type=click.Path(file_okay=True, dir_okay=False, readable=True))
@q2cli.util.pretty_failure(traceback=None)
def inspect_metadata(paths, tsv, failure):
metadata = _merge_metadata(paths)
@@ -373,30 +419,8 @@ def inspect_metadata(paths, tsv, failure):
click.echo(metadata.column_count)
-def _load_metadata(path):
- import qiime2
- import qiime2.sdk
-
- # TODO: clean up duplication between this and the metadata handlers.
- try:
- artifact = qiime2.sdk.Result.load(path)
- except Exception:
- metadata = qiime2.Metadata.load(path)
- else:
- if isinstance(artifact, qiime2.Visualization):
- raise Exception("Visualizations cannot be viewed as QIIME 2"
- " metadata:\n%r" % path)
- elif artifact.has_metadata():
- metadata = artifact.view(qiime2.Metadata)
- else:
- raise Exception("Artifacts with type %r cannot be viewed as"
- " QIIME 2 metadata:\n%r" % (artifact.type, path))
-
- return metadata
-
-
def _merge_metadata(paths):
- m = [_load_metadata(p) for p in paths]
+ m = [q2cli.util.load_metadata(p) for p in paths]
metadata = m[0]
if m[1:]:
metadata = metadata.merge(*m[1:])
@@ -410,14 +434,15 @@ def _merge_metadata(paths):
"used after the command exits, use 'qiime tools extract'.",
cls=ToolCommand)
@click.argument('visualization-path', metavar='VISUALIZATION',
- type=click.Path(exists=True, file_okay=True, dir_okay=False,
- readable=True))
+ type=click.Path(file_okay=True, dir_okay=False, readable=True))
@click.option('--index-extension', required=False, default='html',
help='The extension of the index file that should be opened. '
'[default: html]')
def view(visualization_path, index_extension):
# Guard headless envs from having to import anything large
import sys
+ from qiime2 import Visualization
+ from q2cli.util import _load_input
from q2cli.core.config import CONFIG
if not os.getenv("DISPLAY") and sys.platform != "darwin":
raise click.UsageError(
@@ -426,16 +451,11 @@ def view(visualization_path, index_extension):
'https://view.qiime2.org, or move the Visualization to an '
'environment with a display and view it with `qiime tools view`.')
- import zipfile
- import qiime2.sdk
-
if index_extension.startswith('.'):
index_extension = index_extension[1:]
- try:
- visualization = qiime2.sdk.Visualization.load(visualization_path)
- # TODO: currently a KeyError is raised if a zipped file that is not a
- # QIIME 2 result is passed. This should be handled better by the framework.
- except (zipfile.BadZipFile, KeyError, TypeError):
+
+ visualization = _load_input(visualization_path, view=True)[0]
+ if not isinstance(visualization, Visualization):
raise click.BadParameter(
'%s is not a QIIME 2 Visualization. Only QIIME 2 Visualizations '
'can be viewed.' % visualization_path)
@@ -580,3 +600,199 @@ def citations(path):
click.echo(CONFIG.cfg_style('problem', 'No citations found.'),
err=True)
ctx.exit(1)
+
+
+ at tools.command(name='cache-create',
+ short_help='Create an empty cache at the given location.',
+ help='Create an empty cache at the given location.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=False, readable=True),
+ help='Path to a nonexistent directory to be created as a cache.')
+def cache_create(cache):
+ from qiime2.core.cache import Cache
+ from q2cli.core.config import CONFIG
+
+ try:
+ Cache(cache)
+ except Exception as e:
+ header = "There was a problem creating a cache at '%s':" % cache
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ success = "Created cache at '%s'" % cache
+ click.echo(CONFIG.cfg_style('success', success))
+
+
+ at tools.command(name='cache-remove',
+ short_help='Removes a given key from a cache.',
+ help='Removes a given key from a cache then runs garbage '
+ 'collection on the cache.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=True, file_okay=False, dir_okay=True,
+ readable=True),
+ help='Path to an existing cache to remove the key from.')
+ at click.option('--key', required=True,
+ help='The key to remove from the cache.')
+def cache_remove(cache, key):
+ from qiime2.core.cache import Cache
+ from q2cli.core.config import CONFIG
+
+ try:
+ _cache = Cache(cache)
+ _cache.remove(key)
+ except Exception as e:
+ header = "There was a problem removing the key '%s' from the " \
+ "cache '%s':" % (key, cache)
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ success = "Removed key '%s' from cache '%s'" % (key, cache)
+ click.echo(CONFIG.cfg_style('success', success))
+
+
+ at tools.command(name='cache-garbage-collection',
+ short_help='Runs garbage collection on the cache at the '
+ 'specified location.',
+ help='Runs garbage collection on the cache at the specified '
+ 'location if the specified location is a cache.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=True, file_okay=False, dir_okay=True,
+ readable=True),
+ help='Path to an existing cache to run garbage collection on.')
+def cache_garbage_collection(cache):
+ from qiime2.core.cache import Cache
+ from q2cli.core.config import CONFIG
+
+ try:
+ _cache = Cache(cache)
+ _cache.garbage_collection()
+ except Exception as e:
+ header = "There was a problem running garbage collection on the " \
+ "cache at '%s':" % cache
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ success = "Ran garbage collection on cache at '%s'" % cache
+ click.echo(CONFIG.cfg_style('success', success))
+
+
+ at tools.command(name='cache-store',
+ short_help='Stores a .qza in the cache under a key.',
+ help='Stores a .qza in the cache under a key.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=True, file_okay=False, dir_okay=True,
+ readable=True),
+ help='Path to an existing cache to save into.')
+ at click.option('--artifact-path', required=True,
+ type=click.Path(exists=True, file_okay=True, dir_okay=False,
+ readable=True),
+ help='Path to a .qza to save into the cache.')
+ at click.option('--key', required=True,
+ help='The key to save the artifact under (must be a valid '
+ 'Python identifier).')
+def cache_store(cache, artifact_path, key):
+ from qiime2.sdk.result import Result
+ from qiime2.core.cache import Cache
+ from q2cli.core.config import CONFIG
+
+ try:
+ artifact = Result.load(artifact_path)
+ _cache = Cache(cache)
+ _cache.save(artifact, key)
+ except Exception as e:
+ header = "There was a problem saving the artifact '%s' to the cache " \
+ "'%s' under the key '%s':" % (artifact_path, cache, key)
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ success = "Saved the artifact '%s' to the cache '%s' under the key " \
+ "'%s'" % (artifact_path, cache, key)
+ click.echo(CONFIG.cfg_style('success', success))
+
+
+ at tools.command(name='cache-fetch',
+ short_help='Fetches an artifact out of a cache into a .qza.',
+ help='Fetches the artifact saved to the specified cache under '
+ 'the specified key into a .qza at the specified location.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=True, file_okay=False, dir_okay=True,
+ readable=True),
+ help='Path to an existing cache to load from.')
+ at click.option('--key', required=True,
+ help='The key to the artifact being loaded.')
+ at click.option('--output-path', required=True,
+ type=click.Path(exists=False, readable=True),
+ help='Path to put the .qza we are loading the artifact into.')
+def cache_fetch(cache, key, output_path):
+ from qiime2.core.cache import Cache
+ from q2cli.core.config import CONFIG
+
+ try:
+ _cache = Cache(cache)
+ artifact = _cache.load(key)
+ artifact.save(output_path)
+ except Exception as e:
+ header = "There was a problem loading the artifact with the key " \
+ "'%s' from the cache '%s' and saving it to the file '%s':" % \
+ key, cache, output_path
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ success = "Loaded artifact with the key '%s' from the cache '%s' and " \
+ "saved it to the file '%s'" % (key, cache, output_path)
+ click.echo(CONFIG.cfg_style('success', success))
+
+
+ at tools.command(name='cache-status',
+ short_help='Checks the status of the cache.',
+ help='Lists all keys in the given cache. Peeks artifacts '
+ 'pointed to by keys to data and lists the number of '
+ 'artifacts in the pool for keys to pools.',
+ cls=ToolCommand)
+ at click.option('--cache', required=True,
+ type=click.Path(exists=True, file_okay=False, dir_okay=True,
+ readable=True),
+ help='Path to an existing cache to check the status of.')
+def cache_status(cache):
+ from qiime2.core.cache import Cache
+ from qiime2.sdk.result import Result
+
+ from q2cli.core.config import CONFIG
+
+ data_output = []
+ pool_output = []
+ try:
+ _cache = Cache(cache)
+ with _cache.lock:
+ for key in _cache.get_keys():
+ key_values = _cache.read_key(key)
+
+ if (data := key_values['data']) is not None:
+ data_output.append(
+ 'data: %s -> %s' %
+ (key, str(Result.peek(_cache.data / data))))
+ elif (pool := key_values['pool']) is not None:
+ pool_output.append(
+ 'pool: %s -> size = %s' %
+ (key, str(len(os.listdir(_cache.pools / pool)))))
+ except Exception as e:
+ header = "There was a problem getting the status of the cache at " \
+ "path '%s':" % cache
+ q2cli.util.exit_with_error(e, header=header, traceback=None)
+
+ if not data_output:
+ data_output = 'No data keys in cache'
+ else:
+ data_output = '\n'.join(data_output)
+ data_output = 'Data keys in cache:\n' + data_output
+
+ if not pool_output:
+ pool_output = 'No pool keys in cache'
+ else:
+ pool_output = '\n'.join(pool_output)
+ pool_output = 'Pool keys in cache:\n' + pool_output
+
+ output = data_output + '\n\n' + pool_output
+ success = "Status of the cache at the path '%s':\n\n%s" % \
+ (cache, output)
+ click.echo(CONFIG.cfg_style('success', success))
=====================================
q2cli/click/type.py
=====================================
@@ -46,10 +46,6 @@ class OutDirType(click.Path):
return value
-class ControlFlowException(Exception):
- pass
-
-
class QIIME2Type(click.ParamType):
def __init__(self, type_ast, type_repr, is_output=False):
self.type_repr = type_repr
@@ -84,8 +80,14 @@ class QIIME2Type(click.ParamType):
def _convert_output(self, value, param, ctx):
import os
+ from q2cli.util import output_in_cache
# Click path fails to validate writability on new paths
+ # Check if our output path is actually in a cache and if it is skip our
+ # other checks
+ if output_in_cache(value):
+ return value
+
if os.path.exists(value):
if os.path.isdir(value):
self.fail('%r is already a directory.' % (value,), param, ctx)
@@ -102,40 +104,21 @@ class QIIME2Type(click.ParamType):
def _convert_input(self, value, param, ctx):
import os
- import tempfile
import qiime2.sdk
import qiime2.sdk.util
import q2cli.util
try:
- try:
- q2cli.util.get_plugin_manager()
- result = qiime2.sdk.Result.load(value)
- except OSError as e:
- if e.errno == 28:
- temp = tempfile.tempdir
- self.fail(f'There was not enough space left on {temp!r} '
- f'to extract the artifact {value!r}. '
- '(Try setting $TMPDIR to a directory with '
- 'more space, or increasing the size of '
- f'{temp!r})', param, ctx)
- else:
- raise ControlFlowException
- except ValueError as e:
- if 'does not exist' in str(e):
- self.fail(f'{value!r} is not a valid filepath', param, ctx)
- else:
- raise ControlFlowException
- except Exception as e:
- # If we made it here, QIIME 2 was confident that the thing we
- # are trying to load is a QIIME 2 Result, however, we have run
- # into some kind of catastrophic error.
- header = ('There was a problem loading %s as a '
- 'QIIME 2 Result:' % value)
- q2cli.util.exit_with_error(e, header=header)
- except ControlFlowException:
- self.fail('%r is not a QIIME 2 Artifact (.qza)' % value, param,
- ctx)
+ result, error = q2cli.util._load_input(value)
+ except Exception as e:
+ header = f'There was a problem loading {value!r} as an artifact:'
+ q2cli.util.exit_with_error(
+ e, header=header, traceback='stderr')
+
+ if error:
+ self.fail(str(error), param, ctx)
+ # We want to use click's fail to pretty print whatever error we got
+ # from get_input
if isinstance(result, qiime2.sdk.Visualization):
maybe = value[:-1] + 'a'
@@ -158,33 +141,12 @@ class QIIME2Type(click.ParamType):
return result
def _convert_metadata(self, value, param, ctx):
- import sys
- import qiime2
import q2cli.util
if self.type_expr.name == 'MetadataColumn':
value, column = value
- fp = value
- try:
- q2cli.util.get_plugin_manager()
- artifact = qiime2.Artifact.load(fp)
- except Exception:
- try:
- metadata = qiime2.Metadata.load(fp)
- except Exception as e:
- header = ("There was an issue with loading the file %s as "
- "metadata:" % fp)
- tb = 'stderr' if '--verbose' in sys.argv else None
- q2cli.util.exit_with_error(e, header=header, traceback=tb)
- else:
- try:
- metadata = artifact.view(qiime2.Metadata)
- except Exception as e:
- header = ("There was an issue with viewing the artifact "
- "%s as QIIME 2 Metadata:" % fp)
- tb = 'stderr' if '--verbose' in sys.argv else None
- q2cli.util.exit_with_error(e, header=header, traceback=tb)
+ metadata = q2cli.util.load_metadata(value)
if self.type_expr.name != 'MetadataColumn':
return metadata
=====================================
q2cli/commands.py
=====================================
@@ -286,8 +286,21 @@ class ActionCommand(BaseCommandMixin, click.Command):
"""Called when user hits return, **kwargs are Dict[click_names, Obj]"""
import os
import qiime2.util
+ from q2cli.util import output_in_cache
+ from qiime2.core.cache import Cache
output_dir = kwargs.pop('output_dir')
+ # If they gave us a cache and key combo as an output dir, we want to
+ # error out, so we check if their output dir contains a : and the part
+ # before it is a cache
+ if output_dir:
+ potential_cache = output_dir.rsplit(':', 1)[0]
+ if potential_cache and os.path.exists(potential_cache) and \
+ Cache.is_cache(potential_cache):
+ raise ValueError(f"The given output dir '{output_dir}' "
+ "appears to be a cache:key combo. Cache keys "
+ "cannot be used as output dirs.")
+
verbose = kwargs.pop('verbose')
if verbose is None:
verbose = False
@@ -359,7 +372,14 @@ class ActionCommand(BaseCommandMixin, click.Command):
os.makedirs(output_dir)
for result, output in zip(results, outputs):
- path = result.save(output)
+ if output_in_cache(output) and output_dir is None:
+ cache_path, key = output.split(':')
+ cache = Cache(cache_path)
+ cache.save(result, key)
+ path = output
+ else:
+ path = result.save(output)
+
if not quiet:
click.echo(
CONFIG.cfg_style('success', 'Saved %s to: %s' %
=====================================
q2cli/tests/test_cache_cli.py
=====================================
@@ -0,0 +1,312 @@
+# ----------------------------------------------------------------------------
+# Copyright (c) 2016-2022, QIIME 2 development team.
+#
+# Distributed under the terms of the Modified BSD License.
+#
+# The full license is in the file LICENSE, distributed with this software.
+# ----------------------------------------------------------------------------
+
+import os.path
+import unittest
+import unittest.mock
+import tempfile
+
+from click.testing import CliRunner
+from qiime2 import Artifact
+from qiime2.core.testing.type import IntSequence1, IntSequence2, Mapping
+from qiime2.core.testing.util import get_dummy_plugin
+from qiime2.core.cache import Cache
+
+from q2cli.commands import RootCommand
+from q2cli.builtin.tools import tools
+
+
+class TestCacheCli(unittest.TestCase):
+ def setUp(self):
+ get_dummy_plugin()
+
+ self.runner = CliRunner()
+ self.plugin_command = RootCommand().get_command(
+ ctx=None, name='dummy-plugin')
+ self.tempdir = \
+ tempfile.TemporaryDirectory(prefix='qiime2-q2cli-test-temp-')
+ self.cache = Cache(os.path.join(self.tempdir.name, 'new_cache'))
+
+ self.art1 = Artifact.import_data(IntSequence1, [0, 1, 2])
+ self.art2 = Artifact.import_data(IntSequence1, [3, 4, 5])
+ self.art3 = Artifact.import_data(IntSequence2, [6, 7, 8])
+ self.mapping = Artifact.import_data(Mapping, {'a': '1', 'b': '2'})
+
+ self.non_cache_output = os.path.join(self.tempdir.name, 'output.qza')
+ self.art3_non_cache = os.path.join(self.tempdir.name, 'art3.qza')
+
+ def tearDown(self):
+ self.tempdir.cleanup()
+
+ def _run_command(self, *args):
+ return self.runner.invoke(self.plugin_command, args)
+
+ def test_inputs_from_cache(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.cache.save(self.art3, 'art3')
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+ art3_path = str(self.cache.path) + ':art3'
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', art3_path, '--p-int1', '9', '--p-int2', '10',
+ '--o-concatenated-ints', self.non_cache_output, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(Artifact.load(self.non_cache_output).view(list),
+ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
+
+ def test_inputs_split(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.art3.save(self.art3_non_cache)
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', self.art3_non_cache, '--p-int1', '9', '--p-int2',
+ '10', '--o-concatenated-ints', self.non_cache_output, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(Artifact.load(self.non_cache_output).view(list),
+ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
+
+ def test_colon_in_input_path_not_cache(self):
+ art_path = os.path.join(self.tempdir.name, 'art:1.qza')
+ self.art1.save(art_path)
+
+ left_path = os.path.join(self.tempdir.name, 'left.qza')
+ right_path = os.path.join(self.tempdir.name, 'right.qza')
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art_path, '--o-left', left_path,
+ '--o-right', right_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(Artifact.load(left_path).view(list), [0])
+ self.assertEqual(Artifact.load(right_path).view(list), [1, 2])
+
+ def test_colon_in_cache_path(self):
+ cache = Cache(os.path.join(self.tempdir.name, 'new:cache'))
+ cache.save(self.art1, 'art')
+
+ art_path = str(cache.path) + ':art'
+
+ left_path = os.path.join(self.tempdir.name, 'left.qza')
+ right_path = os.path.join(self.tempdir.name, 'right.qza')
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art_path, '--o-left', left_path,
+ '--o-right', right_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(Artifact.load(left_path).view(list), [0])
+ self.assertEqual(Artifact.load(right_path).view(list), [1, 2])
+
+ def test_output_to_cache(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.cache.save(self.art3, 'art3')
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+ art3_path = str(self.cache.path) + ':art3'
+
+ out_path = str(self.cache.path) + ':out'
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', art3_path, '--p-int1', '9', '--p-int2', '10',
+ '--o-concatenated-ints', out_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(self.cache.load('out').view(list),
+ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
+
+ def test_outputs_to_cache(self):
+ self.cache.save(self.art1, 'art1')
+ art1_path = str(self.cache.path) + ':art1'
+
+ left_path = str(self.cache.path) + ':left'
+ right_path = str(self.cache.path) + ':right'
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art1_path, '--o-left', left_path,
+ '--o-right', right_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(self.cache.load('left').view(list), [0])
+ self.assertEqual(self.cache.load('right').view(list), [1, 2])
+
+ def test_outputs_split(self):
+ self.cache.save(self.art1, 'art1')
+ art1_path = str(self.cache.path) + ':art1'
+
+ left_path = str(self.cache.path) + ':left'
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art1_path, '--o-left', left_path,
+ '--o-right', self.non_cache_output, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(self.cache.load('left').view(list), [0])
+ self.assertEqual(Artifact.load(self.non_cache_output).view(list),
+ [1, 2])
+
+ def test_invalid_cache_path_input(self):
+ art1_path = 'not_a_cache:art1'
+
+ left_path = str(self.cache.path) + ':left'
+ right_path = str(self.cache.path) + ':right'
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art1_path, '--o-left', left_path,
+ '--o-right', right_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertRegex(result.output, r"cache")
+
+ def test_invalid_cache_path_output(self):
+ self.cache.save(self.art1, 'art1')
+ art1_path = str(self.cache.path) + ':art1'
+
+ left_path = '/this/is/not_a_cache:left'
+ right_path = str(self.cache.path) + ':right'
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art1_path, '--o-left', left_path,
+ '--o-right', right_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn('does not exist', result.output)
+
+ def test_colon_in_out_path_not_cache(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.cache.save(self.art3, 'art3')
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+ art3_path = str(self.cache.path) + ':art3'
+
+ out_path = os.path.join(self.tempdir.name, 'out:put.qza')
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', art3_path, '--p-int1', '9', '--p-int2', '10',
+ '--o-concatenated-ints', out_path, '--verbose'
+ )
+
+ if result.exception:
+ raise result.exception
+ self.assertEqual(result.exit_code, 0)
+ self.assertEqual(Artifact.load(out_path).view(list),
+ [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
+
+ def test_nonexistent_input_key(self):
+ art1_path = str(self.cache.path) + ':art1'
+
+ left_path = str(self.cache.path) + ':left'
+
+ result = self._run_command(
+ 'split-ints', '--i-ints', art1_path, '--o-left', left_path,
+ '--o-right', self.non_cache_output, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn("does not contain the key 'art1'",
+ str(result.output))
+
+ def test_output_key_invalid(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.cache.save(self.art3, 'art3')
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+ art3_path = str(self.cache.path) + ':art3'
+
+ out_path = str(self.cache.path) + ':not_valid_identifier$&;'
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', art3_path, '--p-int1', '9', '--p-int2', '10',
+ '--o-concatenated-ints', out_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn('Keys must be valid Python identifiers',
+ str(result.exception))
+
+ def test_artifact_as_metadata_cache(self):
+ self.cache.save(self.mapping, 'mapping')
+ mapping_path = str(self.cache.path) + ':mapping'
+
+ result = self.runner.invoke(tools, ['inspect-metadata', mapping_path])
+
+ self.assertEqual(result.exit_code, 0)
+ self.assertIn('COLUMN NAME TYPE', result.output)
+ self.assertIn("=========== ===========", result.output)
+ self.assertIn("a categorical", result.output)
+ self.assertIn("b categorical", result.output)
+ self.assertIn("IDS: 1", result.output)
+ self.assertIn("COLUMNS: 2", result.output)
+
+ def test_artifact_as_metadata_cache_bad_key(self):
+ mapping_path = str(self.cache.path) + ':mapping'
+
+ result = self.runner.invoke(tools, ['inspect-metadata', mapping_path])
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn("does not contain the key 'mapping'", result.output)
+
+ def test_artifact_as_metadata_cache_bad_cache(self):
+ result = self.runner.invoke(
+ tools, ['inspect-metadata', 'not_a_cache:key'])
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn('is not a valid cache', result.output)
+
+ def test_output_dir_as_cache(self):
+ self.cache.save(self.art1, 'art1')
+ self.cache.save(self.art2, 'art2')
+ self.cache.save(self.art3, 'art3')
+
+ art1_path = str(self.cache.path) + ':art1'
+ art2_path = str(self.cache.path) + ':art2'
+ art3_path = str(self.cache.path) + ':art3'
+
+ out_path = str(self.cache.path) + ':out'
+
+ result = self._run_command(
+ 'concatenate-ints', '--i-ints1', art1_path, '--i-ints2', art2_path,
+ '--i-ints3', art3_path, '--p-int1', '9', '--p-int2', '10',
+ '--output-dir', out_path, '--verbose'
+ )
+
+ self.assertEqual(result.exit_code, 1)
+ self.assertIn(
+ 'Cache keys cannot be used as output dirs.', str(result.exception))
+
+
+if __name__ == "__main__":
+ unittest.main()
=====================================
q2cli/tests/test_cli.py
=====================================
@@ -16,6 +16,7 @@ import errno
from click.testing import CliRunner
from qiime2 import Artifact, Visualization
+from qiime2.core.cache import get_cache
from qiime2.core.testing.type import IntSequence1, IntSequence2
from qiime2.core.testing.util import get_dummy_plugin
@@ -56,17 +57,17 @@ class CliTests(unittest.TestCase):
# top level commands, including a plugin, are present
qiime_cli = RootCommand()
commands = qiime_cli.list_commands(ctx=None)
- self.assertTrue('info' in commands)
- self.assertTrue('tools' in commands)
- self.assertTrue('dummy-plugin' in commands)
+ self.assertIn('info', commands)
+ self.assertIn('tools', commands)
+ self.assertIn('dummy-plugin', commands)
def test_plugin_list_commands(self):
# plugin commands are present including a method and visualizer
qiime_cli = RootCommand()
command = qiime_cli.get_command(ctx=None, name='dummy-plugin')
commands = command.list_commands(ctx=None)
- self.assertTrue('split-ints' in commands)
- self.assertTrue('mapping-viz' in commands)
+ self.assertIn('split-ints', commands)
+ self.assertIn('mapping-viz', commands)
self.assertFalse('split_ints' in commands)
self.assertFalse('mapping_viz' in commands)
@@ -96,25 +97,25 @@ class CliTests(unittest.TestCase):
result = self.runner.invoke(
tools, ['import', '--show-importable-types'])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('FourInts' in result.output)
- self.assertTrue('IntSequence1' in result.output)
- self.assertTrue('IntSequence2' in result.output)
- self.assertTrue('Kennel[Cat]' in result.output)
- self.assertTrue('Kennel[Dog]' in result.output)
- self.assertTrue('Mapping' in result.output)
+ self.assertIn('FourInts', result.output)
+ self.assertIn('IntSequence1', result.output)
+ self.assertIn('IntSequence2', result.output)
+ self.assertIn('Kennel[Cat]', result.output)
+ self.assertIn('Kennel[Dog]', result.output)
+ self.assertIn('Mapping', result.output)
def test_show_importable_formats(self):
result = self.runner.invoke(
tools, ['import', '--show-importable-formats'])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('FourIntsDirectoryFormat' in result.output)
- self.assertTrue('IntSequenceDirectoryFormat' in result.output)
- self.assertFalse('UnimportableFormat' in result.output)
- self.assertFalse('UnimportableDirectoryFormat' in result.output)
- self.assertTrue('MappingDirectoryFormat' in result.output)
- self.assertTrue('IntSequenceFormat' in result.output)
- self.assertTrue('IntSequenceFormatV2' in result.output)
- self.assertTrue('IntSequenceV2DirectoryFormat' in result.output)
+ self.assertIn('FourIntsDirectoryFormat', result.output)
+ self.assertIn('IntSequenceDirectoryFormat', result.output)
+ self.assertNotIn('UnimportableFormat', result.output)
+ self.assertNotIn('UnimportableDirectoryFormat', result.output)
+ self.assertIn('MappingDirectoryFormat', result.output)
+ self.assertIn('IntSequenceFormat', result.output)
+ self.assertIn('IntSequenceFormatV2', result.output)
+ self.assertIn('IntSequenceV2DirectoryFormat', result.output)
def test_extract(self):
result = self.runner.invoke(
@@ -134,17 +135,17 @@ class CliTests(unittest.TestCase):
result = self.runner.invoke(
tools, ['validate', self.artifact1_path, '--level', 'min'])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('appears to be valid at level=min' in result.output)
+ self.assertIn('appears to be valid at level=min', result.output)
def test_validate_max(self):
result = self.runner.invoke(
tools, ['validate', self.artifact1_path, '--level', 'max'])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('appears to be valid at level=max' in result.output)
+ self.assertIn('appears to be valid at level=max', result.output)
result = self.runner.invoke(tools, ['validate', self.artifact1_path])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('appears to be valid at level=max' in result.output)
+ self.assertIn('appears to be valid at level=max', result.output)
def test_split_ints(self):
qiime_cli = RootCommand()
@@ -290,18 +291,18 @@ class CliTests(unittest.TestCase):
obj = QIIME2Type(IntSequence1.to_ast(), repr(IntSequence1))
with self.assertRaisesRegex(click.exceptions.BadParameter,
- f'{self.tempdir!r} is not a QIIME 2 '
- 'Artifact'):
+ f'{self.tempdir!r} is a directory,'
+ ' not a QIIME 2 Artifact'):
obj._convert_input(self.tempdir, None, None)
with self.assertRaisesRegex(click.exceptions.BadParameter,
- "'x' is not a valid filepath"):
+ "x does not exist"):
obj._convert_input('x', None, None)
# This is to ensure the temp in the regex matches the temp used in the
# method under test in type.py
- temp = tempfile.tempdir
- with unittest.mock.patch('qiime2.sdk.Result.load',
+ temp = str(get_cache().path)
+ with unittest.mock.patch('qiime2.sdk.Result.peek',
side_effect=OSError(errno.ENOSPC,
'No space left on '
'device')):
@@ -317,15 +318,15 @@ class CliTests(unittest.TestCase):
viz_path = os.path.join(self.tempdir, 'viz')
- with unittest.mock.patch('qiime2.sdk.Result.load',
+ with unittest.mock.patch('qiime2.sdk.Result.peek',
side_effect=SyntaxError):
result = self.runner.invoke(
command, ['most-common-viz', '--i-ints', self.artifact1_path,
'--o-visualization', viz_path, '--verbose'])
self.assertEqual(result.exit_code, 1)
- self.assertTrue('problem loading' in result.output)
- self.assertTrue(self.artifact1_path in result.output)
+ self.assertIn('problem loading', result.output)
+ self.assertIn(self.artifact1_path, result.output)
def test_deprecated_help_text(self):
qiime_cli = RootCommand()
@@ -334,8 +335,8 @@ class CliTests(unittest.TestCase):
result = self.runner.invoke(command, ['deprecated-method', '--help'])
self.assertEqual(result.exit_code, 0)
- self.assertTrue('WARNING' in result.output)
- self.assertTrue('deprecated' in result.output)
+ self.assertIn('WARNING', result.output)
+ self.assertIn('deprecated', result.output)
def test_run_deprecated_gets_warning_msg(self):
qiime_cli = RootCommand()
@@ -354,7 +355,7 @@ class CliTests(unittest.TestCase):
# Just make sure that the command ran as expected
self.assertEqual(artifact.view(dict), {'foo': '43'})
- self.assertTrue('deprecated' in result.output)
+ self.assertIn('deprecated', result.output)
class TestOptionalArtifactSupport(unittest.TestCase):
@@ -612,6 +613,8 @@ class TestMetadataColumnSupport(MetadataTestsBase):
'--m-metadata-column', 'col1', '--verbose')
exp_tsv = 'id\tcol1\n#q2:types\tcategorical\n0\tfoo\nid1\tbar\n'
+ if result.exit_code != 0:
+ raise ValueError(result.exception)
self._assertMetadataOutput(
result, exp_tsv=exp_tsv,
exp_yaml="metadata: !metadata 'metadata.tsv'")
=====================================
q2cli/tests/test_tools.py
=====================================
@@ -7,6 +7,7 @@
# ----------------------------------------------------------------------------
import os
+import gc
import shutil
import unittest
import tempfile
@@ -15,8 +16,11 @@ from click.testing import CliRunner
from qiime2 import Artifact
from qiime2.core.testing.util import get_dummy_plugin
from qiime2.metadata.base import SUPPORTED_COLUMN_TYPES
+from qiime2.core.cache import Cache
+from qiime2.sdk.result import Result
-from q2cli.builtin.tools import tools, _load_metadata
+from q2cli.util import load_metadata
+from q2cli.builtin.tools import tools
from q2cli.commands import RootCommand
@@ -102,7 +106,7 @@ class TestCastMetadata(unittest.TestCase):
'--ignore-extra', '--output-file', self.output_file])
self.assertEqual(result.exit_code, 0)
- casted_metadata = _load_metadata(self.output_file)
+ casted_metadata = load_metadata(self.output_file)
self.assertNotIn('extra', casted_metadata.columns.keys())
def test_complete_successful_run(self):
@@ -111,10 +115,10 @@ class TestCastMetadata(unittest.TestCase):
'numbers:categorical', '--output-file', self.output_file])
self.assertEqual(result.exit_code, 0)
- input_metadata = _load_metadata(self.metadata_file)
+ input_metadata = load_metadata(self.metadata_file)
self.assertEqual('numeric', input_metadata.columns['numbers'].type)
- casted_metadata = _load_metadata(self.output_file)
+ casted_metadata = load_metadata(self.output_file)
self.assertEqual('categorical',
casted_metadata.columns['numbers'].type)
@@ -445,5 +449,280 @@ class TestExportToFileFormat(TestInspectMetadata):
self.assertEqual(success, result.output)
+class TestCacheTools(unittest.TestCase):
+ def setUp(self):
+ get_dummy_plugin()
+
+ self.runner = CliRunner()
+ self.plugin_command = RootCommand().get_command(
+ ctx=None, name='dummy-plugin')
+ self.tempdir = \
+ tempfile.TemporaryDirectory(prefix='qiime2-q2cli-test-temp-')
+
+ self.art1 = Artifact.import_data('IntSequence1', [0, 1, 2])
+ self.art2 = Artifact.import_data('IntSequence1', [3, 4, 5])
+ self.art3 = Artifact.import_data('IntSequence1', [6, 7, 8])
+ self.art4 = Artifact.import_data('IntSequence2', [9, 10, 11])
+ self.cache = Cache(os.path.join(self.tempdir.name, 'new_cache'))
+
+ def tearDown(self):
+ self.tempdir.cleanup()
+
+ def test_cache_create(self):
+ cache_path = os.path.join(self.tempdir.name, 'created_cache')
+
+ result = self.runner.invoke(
+ tools, ['cache-create', '--cache', cache_path])
+
+ success = "Created cache at '%s'\n" % cache_path
+ self.assertEqual(success, result.output)
+ self.assertTrue(Cache.is_cache(cache_path))
+
+ def test_cache_remove(self):
+ self.cache.save(self.art1, 'key')
+ self.assertTrue('key' in self.cache.get_keys())
+
+ result = self.runner.invoke(
+ tools,
+ ['cache-remove', '--cache', str(self.cache.path), '--key', 'key'])
+
+ success = "Removed key 'key' from cache '%s'\n" % self.cache.path
+ self.assertEqual(success, result.output)
+ self.assertFalse('key' in self.cache.get_keys())
+
+ def test_cache_garbage_collection(self):
+ # Data referenced directly by key
+ self.cache.save(self.art1, 'foo')
+ # Data referenced by pool that is referenced by key
+ pool = self.cache.create_pool(['bar'])
+ pool.save(self.art2)
+ # We will be manually deleting the keys that back these two
+ self.cache.save(self.art3, 'baz')
+ pool = self.cache.create_pool(['qux'])
+ pool.save(self.art4)
+
+ # What we expect to see before and after gc
+ expected_pre_gc_contents = \
+ set(('./VERSION', 'keys/foo', 'keys/bar',
+ 'keys/baz', 'keys/qux',
+ f'pools/bar/{self.art2.uuid}',
+ f'pools/qux/{self.art4.uuid}',
+ f'data/{self.art1.uuid}', f'data/{self.art2.uuid}',
+ f'data/{self.art3.uuid}', f'data/{self.art4.uuid}'))
+
+ expected_post_gc_contents = \
+ set(('./VERSION', 'keys/foo', 'keys/bar',
+ f'pools/bar/{self.art2.uuid}',
+ f'data/{self.art1.uuid}', f'data/{self.art2.uuid}'))
+
+ # Assert cache looks how we want pre gc
+ pre_gc_contents = _get_cache_contents(self.cache)
+ self.assertEqual(expected_pre_gc_contents, pre_gc_contents)
+
+ # Delete keys
+ self.cache.remove(self.cache.keys / 'baz')
+ self.cache.remove(self.cache.keys / 'qux')
+
+ # Make sure Python's garbage collector gets the process pool symlinks
+ # to the artifact that was keyed on baz and the one in the qux pool
+ gc.collect()
+ result = self.runner.invoke(
+ tools,
+ ['cache-garbage-collection', '--cache', str(self.cache.path)])
+
+ success = "Ran garbage collection on cache at '%s'\n" % self.cache.path
+ self.assertEqual(success, result.output)
+
+ # Assert cache looks how we want post gc
+ post_gc_contents = _get_cache_contents(self.cache)
+ self.assertEqual(expected_post_gc_contents, post_gc_contents)
+
+ def test_cache_store(self):
+ artifact = os.path.join(self.tempdir.name, 'artifact.qza')
+ self.art1.save(artifact)
+
+ result = self.runner.invoke(
+ tools, ['cache-store', '--cache', str(self.cache.path),
+ '--artifact-path', artifact, '--key', 'key'])
+
+ success = "Saved the artifact '%s' to the cache '%s' under the key " \
+ "'key'\n" % (artifact, self.cache.path)
+ self.assertEqual(success, result.output)
+
+ def test_cache_fetch(self):
+ artifact = os.path.join(self.tempdir.name, 'artifact.qza')
+ self.cache.save(self.art1, 'key')
+
+ result = self.runner.invoke(
+ tools, ['cache-fetch', '--cache', str(self.cache.path),
+ '--key', 'key', '--output-path', artifact])
+
+ success = "Loaded artifact with the key 'key' from the cache '%s' " \
+ "and saved it to the file '%s'\n" % (self.cache.path, artifact)
+ self.assertEqual(success, result.output)
+
+ def test_cache_roundtrip(self):
+ in_artifact = os.path.join(self.tempdir.name, 'in_artifact.qza')
+ out_artifact = os.path.join(self.tempdir.name, 'out_artifact.qza')
+
+ self.art1.save(in_artifact)
+
+ result = self.runner.invoke(
+ tools, ['cache-store', '--cache', str(self.cache.path),
+ '--artifact-path', in_artifact, '--key', 'key'])
+
+ success = "Saved the artifact '%s' to the cache '%s' under the key " \
+ "'key'\n" % (in_artifact, self.cache.path)
+ self.assertEqual(success, result.output)
+
+ result = self.runner.invoke(
+ tools, ['cache-fetch', '--cache', str(self.cache.path),
+ '--key', 'key', '--output-path', out_artifact])
+
+ success = "Loaded artifact with the key 'key' from the cache '%s' " \
+ "and saved it to the file '%s'\n" % (self.cache.path, out_artifact)
+ self.assertEqual(success, result.output)
+
+ artifact = Artifact.load(out_artifact)
+ self.assertEqual([0, 1, 2], artifact.view(list))
+
+ def test_cache_status(self):
+ success_template = \
+ "Status of the cache at the path '%s':\n\n%s\n\n%s\n"
+
+ # Empty cache
+ result = self.runner.invoke(
+ tools, ['cache-status', '--cache', str(self.cache.path)])
+ success = \
+ success_template % (str(self.cache.path), 'No data keys in cache',
+ 'No pool keys in cache')
+ self.assertEqual(success, result.output)
+
+ # Cache with only data
+ in_artifact = os.path.join(self.tempdir.name, 'in_artifact.qza')
+ self.art1.save(in_artifact)
+ self.runner.invoke(
+ tools, ['cache-store', '--cache', str(self.cache.path),
+ '--artifact-path', in_artifact, '--key', 'key'])
+
+ result = self.runner.invoke(
+ tools, ['cache-status', '--cache', str(self.cache.path)])
+ data_output = 'Data keys in cache:\ndata: key -> %s' % \
+ str(Result.peek(self.cache.data / str(self.art1.uuid)))
+ success = \
+ success_template % (str(self.cache.path), data_output,
+ 'No pool keys in cache')
+ self.assertEqual(success, result.output)
+
+ # Cache with data and pool
+ pool = self.cache.create_pool(keys=['pool_key'])
+ pool.save(self.art2)
+
+ result = self.runner.invoke(
+ tools, ['cache-status', '--cache', str(self.cache.path)])
+ pool_output = 'Pool keys in cache:\npool: pool_key -> size = 1'
+ success = \
+ success_template % (str(self.cache.path), data_output,
+ pool_output)
+ self.assertEqual(success, result.output)
+
+
+def _get_cache_contents(cache):
+ """Gets contents of cache not including contents of the artifacts
+ themselves relative to the root of the cache
+ """
+ cache_contents = set()
+
+ rel_keys = os.path.relpath(cache.keys, cache.path)
+ rel_data = os.path.relpath(cache.data, cache.path)
+ rel_pools = os.path.relpath(cache.pools, cache.path)
+ rel_cache = os.path.relpath(cache.path, cache.path)
+
+ for key in os.listdir(cache.keys):
+ cache_contents.add(os.path.join(rel_keys, key))
+
+ for art in os.listdir(cache.data):
+ cache_contents.add(os.path.join(rel_data, art))
+
+ for pool in os.listdir(cache.pools):
+ for link in os.listdir(os.path.join(cache.pools, pool)):
+ cache_contents.add(os.path.join(rel_pools, pool, link))
+
+ for elem in os.listdir(cache.path):
+ if os.path.isfile(os.path.join(cache.path, elem)):
+ cache_contents.add(os.path.join(rel_cache, elem))
+
+ return cache_contents
+
+
+class TestPeek(unittest.TestCase):
+ def setUp(self):
+ self.runner = CliRunner()
+ self.tempdir = tempfile.mkdtemp(prefix='qiime2-q2cli-test-temp-')
+
+ # create artifact
+ self.artifact = os.path.join(self.tempdir, 'artifact.qza')
+ Artifact.import_data(
+ 'Mapping', {'foo': 'bar'}).save(self.artifact)
+
+ # create visualization
+ qiime_cli = RootCommand()
+ command = qiime_cli.get_command(ctx=None, name='dummy-plugin')
+ self.viz = os.path.join(self.tempdir, 'viz.qzv')
+
+ self.ints = os.path.join(self.tempdir, 'ints.qza')
+ ints = Artifact.import_data(
+ 'IntSequence1', [0, 42, 43], list)
+ ints.save(self.ints)
+
+ self.runner.invoke(
+ command, ['most-common-viz', '--i-ints', self.ints,
+ '--o-visualization', self.viz, '--verbose'])
+
+ def tearDown(self):
+ shutil.rmtree(self.tempdir)
+
+ def test_single_artifact(self):
+ result = self.runner.invoke(tools, ['peek', self.artifact])
+ self.assertEqual(result.exit_code, 0)
+ self.assertIn("UUID:", result.output)
+ self.assertIn("Type:", result.output)
+ self.assertIn("Data format:", result.output)
+ self.assertEqual(result.output.count('\n'), 3)
+
+ def test_single_visualization(self):
+ result = self.runner.invoke(tools, ['peek', self.viz])
+ self.assertEqual(result.exit_code, 0)
+ self.assertIn("UUID:", result.output)
+ self.assertIn("Type:", result.output)
+ self.assertNotIn("Data format:", result.output)
+ self.assertEqual(result.output.count('\n'), 2)
+
+ def test_artifact_and_visualization(self):
+ result = self.runner.invoke(tools, ['peek', self.artifact, self.viz])
+ self.assertEqual(result.exit_code, 0)
+ self.assertIn("UUID", result.output)
+ self.assertIn("Type", result.output)
+ self.assertIn("Data Format", result.output)
+ self.assertIn("N/A", result.output)
+ self.assertEqual(result.output.count('\n'), 3)
+
+ def test_single_file_tsv(self):
+ result = self.runner.invoke(tools, ['peek', '--tsv', self.artifact])
+ self.assertIn("Filename\tType\tUUID\tData Format\n", result.output)
+ self.assertIn("artifact.qza", result.output)
+ self.assertEqual(result.output.count('\t'), 6)
+ self.assertEqual(result.output.count('\n'), 2)
+
+ def test_multiple_file_tsv(self):
+ result = self.runner.invoke(tools, ['peek', '--tsv', self.artifact,
+ self.viz])
+ self.assertIn("Filename\tType\tUUID\tData Format\n", result.output)
+ self.assertIn("artifact.qza", result.output)
+ self.assertIn("viz.qzv", result.output)
+ self.assertEqual(result.output.count('\t'), 9)
+ self.assertEqual(result.output.count('\n'), 3)
+
+
if __name__ == "__main__":
unittest.main()
=====================================
q2cli/util.py
=====================================
@@ -7,6 +7,10 @@
# ----------------------------------------------------------------------------
+class OutOfDisk(Exception):
+ pass
+
+
def get_app_dir():
import os
conda_prefix = os.environ.get('CONDA_PREFIX')
@@ -70,7 +74,43 @@ def exit_with_error(e, header='An error has been encountered:',
if not footer:
click.echo(err=True) # extra newline to look normal
- click.get_current_context().exit(status)
+ try:
+ click.get_current_context().exit(status)
+ except RuntimeError:
+ sys.exit(status)
+
+
+def output_in_cache(fp):
+ """Determines if an output path follows the format
+ /path_to_extant_cache:key
+ """
+ from qiime2.core.cache import Cache
+
+ # Tells us right away this isn't in a cache
+ if ':' not in fp:
+ return False
+
+ cache_path, key = _get_cache_path_and_key(fp)
+
+ try:
+ if Cache.is_cache(cache_path):
+ if not key.isidentifier():
+ raise ValueError(
+ f"Key '{key}' is not a valid Python identifier. Keys must "
+ "be valid Python identifiers. Python identifier rules may "
+ "be found here https://www.askpython.com/python/"
+ "python-identifiers-rules-best-practices")
+ else:
+ return True
+ except FileNotFoundError as e:
+ # If cache_path doesn't exist, don't treat this as a cache output
+ if 'No such file or directory' in str(e):
+ pass
+ else:
+ raise e
+
+ # We don't have a cache at all
+ return False
def get_close_matches(name, possibilities):
@@ -112,10 +152,12 @@ class pretty_failure:
return self
def __exit__(self, exc_type, exc_val, exc_tb):
- if exc_val is not None:
+ # if exit_with_error is called twice, then click.exit(1) or sys.exit(1)
+ # will happen, no need to exit_with_error again in that case.
+ if exc_val is not None and str(exc_val) != '1':
exit_with_error(exc_val, self.header, self.traceback, self.status)
- return True
+ return False
def convert_primitive(ast):
@@ -234,3 +276,157 @@ def get_plugin_manager():
return pm
return qiime2.sdk.PluginManager()
+
+
+def load_metadata(fp):
+ import qiime2
+ import sys
+
+ metadata, error = _load_metadata_artifact(fp)
+ if metadata is None:
+ try:
+ metadata = qiime2.Metadata.load(fp)
+ except Exception as e:
+ if error and ':' in fp:
+ e = error
+ header = ("There was an issue with loading the file %s as "
+ "metadata:" % fp)
+ tb = 'stderr' if '--verbose' in sys.argv else None
+ exit_with_error(e, header=header, traceback=tb)
+
+ return metadata
+
+
+def _load_metadata_artifact(fp):
+ import qiime2
+ import sys
+
+ artifact, error = _load_input(fp)
+ if isinstance(error, OutOfDisk):
+ raise error
+
+ default_tb = 'stderr'
+ # if that worked, we have an artifact or we've
+ # already raised a critical error
+ # otherwise, any normal errors can be ignored as its
+ # most likely actually metadata not a qza
+ if artifact:
+ try:
+ default_tb = None
+ if isinstance(artifact, qiime2.Visualization):
+ raise Exception(
+ 'Visualizations cannot be viewed as QIIME 2 metadata.')
+ if not artifact.has_metadata():
+ raise Exception(
+ f"Artifacts with type {artifact.type!r} cannot be viewed"
+ " as QIIME 2 metadata.")
+
+ default_tb = 'stderr'
+ return artifact.view(qiime2.Metadata), None
+
+ except Exception as e:
+ header = ("There was an issue with viewing the artifact "
+ f"{fp!r} as QIIME 2 Metadata:")
+ tb = 'stderr' if '--verbose' in sys.argv else default_tb
+ exit_with_error(e, header=header, traceback=tb)
+
+ else:
+ return None, error
+
+
+def _load_input(fp, view=False):
+ # Just initialize the plugin manager. This is slow and not necessary if we
+ # called this from qiime tools view.
+ if not view:
+ _ = get_plugin_manager()
+
+ if ':' in fp:
+ artifact, error = _load_input_cache(fp)
+ if error:
+ artifact, _ = _load_input_file(fp)
+ if artifact is not None:
+ error = None
+ # ignore this error (`_`), it was more likely
+ # a bad cache than an really weird filepath
+ else:
+ artifact, error = _load_input_file(fp)
+
+ if isinstance(error, OSError) and error.errno == 28:
+ # abort as there's nothing anyone can do about this
+ from qiime2.core.cache import get_cache
+
+ path = str(get_cache().path)
+ return None, OutOfDisk(f'There was not enough space left on {path!r} '
+ f'to use the artifact {fp!r}. (Try '
+ f'setting $TMPDIR to a directory with more '
+ f'space, or increasing the size of {path!r})')
+
+ return artifact, error
+
+
+def _load_input_cache(fp):
+ artifact = error = None
+ try:
+ artifact = try_as_cache_input(fp)
+ except Exception as e:
+ error = e
+
+ return artifact, error
+
+
+def _load_input_file(fp):
+ import qiime2.sdk
+ import os
+
+ if os.path.exists(fp) and os.path.isdir(fp):
+ return None, ValueError(
+ f"{fp!r} is a directory, not a QIIME 2 Artifact.")
+
+ # test if valid
+ peek = None
+ try:
+ peek = qiime2.sdk.Result.peek(fp)
+ except Exception as error:
+ if isinstance(error, SyntaxError):
+ raise error
+ # ideally ValueError: X is not a QIIME archive.
+ # but sometimes SyntaxError or worse
+ return None, error
+
+ # try to actually load
+ try:
+ artifact = qiime2.sdk.Result.load(fp)
+ return artifact, None
+
+ except Exception as e:
+ if peek:
+ # abort early as there's nothing else to do
+ raise ValueError(
+ "It looks like you have an Artifact but are missing the"
+ " plugin(s) necessary to load it. Artifact has type"
+ f" {peek.type!r} and format {peek.format!r}") from e
+ else:
+ error = e
+
+ return None, error
+
+
+def try_as_cache_input(fp):
+ """ Determine if an input is in a cache and load it from the cache if it is
+ """
+ import os
+ from qiime2 import Cache
+
+ cache_path, key = _get_cache_path_and_key(fp)
+
+ # We don't want to invent a new cache on disk here because if their input
+ # exists their cache must also already exist
+ if not os.path.exists(cache_path) or not Cache.is_cache(cache_path):
+ raise ValueError(f"The path {cache_path!r} is not a valid cache.")
+
+ cache = Cache(cache_path)
+ return cache.load(key)
+
+
+def _get_cache_path_and_key(fp):
+ return fp.rsplit(':', 1)
View it on GitLab: https://salsa.debian.org/med-team/q2cli/-/compare/f7facc8e6c0d9d19f48c1c03fd1a7c52cb080b27...72b86e920938b5f3246e09342ad312a86360c743
--
View it on GitLab: https://salsa.debian.org/med-team/q2cli/-/compare/f7facc8e6c0d9d19f48c1c03fd1a7c52cb080b27...72b86e920938b5f3246e09342ad312a86360c743
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20230112/6ad753b3/attachment-0001.htm>
More information about the debian-med-commit
mailing list