[med-svn] [Git][med-team/qiime][upstream] New upstream version 2019.10.0
Liubov Chuprikova
gitlab at salsa.debian.org
Sat Dec 14 21:22:32 GMT 2019
Liubov Chuprikova pushed to branch upstream at Debian Med / qiime
Commits:
970d3394 by Liubov Chuprikova at 2019-12-14T21:19:22Z
New upstream version 2019.10.0
- - - - -
12 changed files:
- README.md
- ci/recipe/meta.yaml
- qiime2/_version.py
- qiime2/citations.bib
- qiime2/core/type/primitive.py
- qiime2/core/type/tests/test_primitive.py
- qiime2/metadata/metadata.py
- qiime2/metadata/tests/test_metadata.py
- qiime2/metadata/tests/test_metadata_column.py
- qiime2/plugin/model/file_format.py
- qiime2/plugin/model/tests/test_file_format.py
- qiime2/sdk/actiongraph.py
Changes:
=====================================
README.md
=====================================
@@ -23,4 +23,4 @@ Please visit the [contributing page](https://github.com/qiime2/qiime2/blob/maste
## Citing QIIME 2
If you use QIIME 2 for any published research, please include the following citation:
-> Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology. https://doi.org/10.1038/s41587-019-0209-9
+> Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology 37:852–857. https://doi.org/10.1038/s41587-019-0209-9
=====================================
ci/recipe/meta.yaml
=====================================
@@ -20,11 +20,9 @@ requirements:
- python {{ python }}
- pyyaml
- decorator
- # TODO: unset this pin once pandas 0.25.x is sorted out
- - pandas 0.24.2
+ - pandas
- tzlocal
- python-dateutil
- - pyparsing ==2.3.0
- bibtexparser
- networkx
=====================================
qiime2/_version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (tag: 2019.7.0)"
- git_full = "a7d3da5a733aa14749c21ddda969e7ccd403c311"
- git_date = "2019-07-30 18:15:55 +0000"
+ git_refnames = " (tag: 2019.10.0)"
+ git_full = "1ef06fd6c0c78b41777afc7688d45e69d22a693b"
+ git_date = "2019-11-01 01:04:24 +0000"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
=====================================
qiime2/citations.bib
=====================================
@@ -3,6 +3,9 @@
title={Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2},
journal={Nature Biotechnology},
year={2019},
+ volume={37},
+ number={8},
+ pages={852-857},
issn={1546-1696},
doi={10.1038/s41587-019-0209-9},
url={https://doi.org/10.1038/s41587-019-0209-9}
=====================================
qiime2/core/type/primitive.py
=====================================
@@ -419,12 +419,12 @@ class _MetadataColumn(_PrimitiveTemplateBase):
raise TypeError("Unsupported type in field: %r"
% (field.get_name(),))
- def decode(self, metadata):
+ def decode(self, value):
# This interface should have already retrieved this object.
- if not self.is_element(metadata):
+ if not isinstance(value, metadata.MetadataColumn):
raise TypeError("`Metadata` must be provided by the interface"
" directly.")
- return metadata
+ return value
def encode(self, value):
# TODO: Should this be the provenance representation? Does that affect
=====================================
qiime2/core/type/tests/test_primitive.py
=====================================
@@ -8,6 +8,9 @@
import unittest
+import pandas as pd
+
+import qiime2.metadata as metadata
import qiime2.core.type.primitive as primitive
import qiime2.core.type.grammar as grammar
@@ -47,5 +50,29 @@ class TestIntersectTwoRanges(unittest.TestCase):
self.assertIntersectEqual(a, b, grammar.UnionExp())
+class TestMetadataColumn(unittest.TestCase):
+
+ def test_decode_categorical_value(self):
+ value = pd.Series({'a': 'a', 'b': 'b', 'c': 'c'}, name='foo')
+ value.index.name = 'id'
+ cat_md = metadata.CategoricalMetadataColumn(value)
+
+ res = primitive.MetadataColumn[primitive.Categorical].decode(cat_md)
+ self.assertIs(res, cat_md)
+
+ def test_decode_numeric_value(self):
+ value = pd.Series({'a': 1, 'b': 2, 'c': 3}, name='foo')
+ value.index.name = 'id'
+ num_md = metadata.NumericMetadataColumn(value)
+
+ res = primitive.MetadataColumn[primitive.Categorical].decode(num_md)
+ self.assertIs(res, num_md)
+
+ def test_decode_other(self):
+ with self.assertRaisesRegex(TypeError, 'provided.*directly'):
+ primitive.MetadataColumn[primitive.Categorical].decode(
+ "<metadata>")
+
+
if __name__ == '__main__':
unittest.main()
=====================================
qiime2/metadata/metadata.py
=====================================
@@ -156,11 +156,6 @@ class _MetadataBase:
"Detected empty metadata %s. %ss must consist of at least "
"one character." % (label, label))
- if value != value.strip():
- raise ValueError(
- "Detected metadata %s with leading or trailing "
- "whitespace characters: %r" % (label, value))
-
if axis == 'id' and value.startswith('#'):
raise ValueError(
"Detected metadata %s that begins with a pound sign "
@@ -362,13 +357,19 @@ class Metadata(_MetadataBase):
super().__init__(dataframe.index)
self._dataframe, self._columns = self._normalize_dataframe(dataframe)
+ self._validate_index(self._dataframe.columns, axis='column')
def _normalize_dataframe(self, dataframe):
- self._validate_index(dataframe.columns, axis='column')
-
norm_df = dataframe.copy()
+
+ # Do not attempt to strip empty metadata
+ if not norm_df.columns.empty:
+ norm_df.columns = norm_df.columns.str.strip()
+
+ norm_df.index = norm_df.index.str.strip()
+
columns = collections.OrderedDict()
- for column_name, series in dataframe.items():
+ for column_name, series in norm_df.items():
metadata_column = self._metadata_column_factory(series)
norm_df[column_name] = metadata_column.to_series()
properties = ColumnProperties(type=metadata_column.type)
@@ -870,8 +871,6 @@ class MetadataColumn(_MetadataBase, metaclass=abc.ABCMeta):
super().__init__(series.index)
- self._validate_index([series.name], axis='column')
-
if not self._is_supported_dtype(series.dtype):
raise TypeError(
"%s %r does not support a pandas.Series object with dtype %s" %
@@ -879,6 +878,8 @@ class MetadataColumn(_MetadataBase, metaclass=abc.ABCMeta):
self._series = self._normalize_(series)
+ self._validate_index([self._series.name], axis='column')
+
def __repr__(self):
"""String summary of the metadata column."""
return '<%s name=%r id_count=%d>' % (self.__class__.__name__,
@@ -1125,6 +1126,7 @@ class CategoricalMetadataColumn(MetadataColumn):
def _normalize_(cls, series):
def normalize(value):
if isinstance(value, str):
+ value = value.strip()
if value == '':
raise ValueError(
"%s does not support empty strings as values. Use an "
@@ -1132,11 +1134,6 @@ class CategoricalMetadataColumn(MetadataColumn):
"(e.g. `numpy.nan`) or supply a non-empty string as "
"the value in column %r." %
(cls.__name__, series.name))
- elif value != value.strip():
- raise ValueError(
- "%s does not support values with leading or trailing "
- "whitespace characters. Column %r has the following "
- "value: %r" % (cls.__name__, series.name, value))
else:
return value
elif pd.isnull(value): # permits np.nan, Python float nan, None
@@ -1147,7 +1144,10 @@ class CategoricalMetadataColumn(MetadataColumn):
"%r of type %r in column %r." %
(cls.__name__, value, type(value), series.name))
- return series.apply(normalize, convert_dtype=False)
+ norm_series = series.apply(normalize, convert_dtype=False)
+ norm_series.index = norm_series.index.str.strip()
+ norm_series.name = norm_series.name.strip()
+ return norm_series
class NumericMetadataColumn(MetadataColumn):
=====================================
qiime2/metadata/tests/test_metadata.py
=====================================
@@ -76,22 +76,6 @@ class TestInvalidMetadataConstruction(unittest.TestCase):
{'col': [1, 2, 3],
'': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id')))
- def test_leading_trailing_whitespace_id(self):
- with self.assertRaisesRegex(ValueError, "metadata ID.*leading or "
- "trailing whitespace.*' b '"):
- Metadata(pd.DataFrame(
- {'col': [1, 2, 3]},
- index=pd.Index(['a', ' b ', 'c'], name='id')))
-
- def test_leading_trailing_whitespace_column_name(self):
- with self.assertRaisesRegex(
- ValueError, "metadata column name.*leading or trailing "
- "whitespace.*' col2 '"):
- Metadata(pd.DataFrame(
- {'col1': [1, 2, 3],
- ' col2 ': [4, 5, 6]},
- index=pd.Index(['a', 'b', 'c'], name='id')))
-
def test_pound_sign_id(self):
with self.assertRaisesRegex(
ValueError, "metadata ID.*begins with a pound sign.*'#b'"):
@@ -157,15 +141,6 @@ class TestInvalidMetadataConstruction(unittest.TestCase):
'col2': ['foo', '', 'bar']},
index=pd.Index(['a', 'b', 'c'], name='id')))
- def test_categorical_column_leading_trailing_whitespace_value(self):
- with self.assertRaisesRegex(
- ValueError, "CategoricalMetadataColumn.*leading or trailing "
- "whitespace characters.*Column 'col2'.*' bar '"):
- Metadata(pd.DataFrame(
- {'col1': [1, 2, 3],
- 'col2': ['foo', ' bar ', 'baz']},
- index=pd.Index(['a', 'b', 'c'], name='id')))
-
def test_numeric_column_infinity(self):
with self.assertRaisesRegex(
ValueError, "NumericMetadataColumn.*positive or negative "
@@ -402,6 +377,38 @@ class TestMetadataConstructionAndProperties(unittest.TestCase):
self.assertEqual(set(metadata.columns), {'column', 'Column'})
+ def test_categorical_column_leading_trailing_whitespace_value(self):
+ md1 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3],
+ 'col2': ['foo', ' bar ', 'baz']},
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+ md2 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3],
+ 'col2': ['foo', 'bar', 'baz']},
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(md1, md2)
+
+ def test_leading_trailing_whitespace_id(self):
+ md1 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+ index=pd.Index(['a', ' b ', 'c'], name='id')))
+ md2 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(md1, md2)
+
+ def test_leading_trailing_whitespace_column_name(self):
+ md1 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3], ' col2 ': [4, 5, 6]},
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+ md2 = Metadata(pd.DataFrame(
+ {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(md1, md2)
+
class TestSourceArtifacts(unittest.TestCase):
def setUp(self):
=====================================
qiime2/metadata/tests/test_metadata_column.py
=====================================
@@ -96,21 +96,6 @@ class TestInvalidMetadataColumnConstruction(unittest.TestCase):
[1, 2, 3], name='',
index=pd.Index(['a', 'b', 'c'], name='id')))
- def test_leading_trailing_whitespace_id(self):
- with self.assertRaisesRegex(ValueError, "metadata ID.*leading or "
- "trailing whitespace.*' b '"):
- DummyMetadataColumn(pd.Series(
- [1, 2, 3], name='col',
- index=pd.Index(['a', ' b ', 'c'], name='id')))
-
- def test_leading_trailing_whitespace_column_name(self):
- with self.assertRaisesRegex(
- ValueError, "metadata column name.*leading or trailing "
- "whitespace.*' col2 '"):
- DummyMetadataColumn(pd.Series(
- [1, 2, 3], name=' col2 ',
- index=pd.Index(['a', 'b', 'c'], name='id')))
-
def test_pound_sign_id(self):
with self.assertRaisesRegex(
ValueError, "metadata ID.*begins with a pound sign.*'#b'"):
@@ -826,14 +811,6 @@ class TestCategoricalMetadataColumn(unittest.TestCase):
['foo', '', 'bar'], name='col1',
index=pd.Index(['a', 'b', 'c'], name='id')))
- def test_leading_trailing_whitespace_value(self):
- with self.assertRaisesRegex(
- ValueError, "CategoricalMetadataColumn.*leading or trailing "
- "whitespace characters.*Column 'col1'.*' bar '"):
- CategoricalMetadataColumn(pd.Series(
- ['foo', ' bar ', 'baz'], name='col1',
- index=pd.Index(['a', 'b', 'c'], name='id')))
-
def test_type_property(self):
self.assertEqual(CategoricalMetadataColumn.type, 'categorical')
@@ -899,6 +876,36 @@ class TestCategoricalMetadataColumn(unittest.TestCase):
pdt.assert_series_equal(obs, exp)
self.assertEqual(obs.dtype, object)
+ def test_leading_trailing_whitespace_value(self):
+ col1 = CategoricalMetadataColumn(pd.Series(
+ ['foo', ' bar ', 'baz'], name='col1',
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+ col2 = CategoricalMetadataColumn(pd.Series(
+ ['foo', 'bar', 'baz'], name='col1',
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(col1, col2)
+
+ def test_leading_trailing_whitespace_id(self):
+ col1 = CategoricalMetadataColumn(pd.Series(
+ ['foo', ' bar ', 'baz'], name='col',
+ index=pd.Index(['a', ' b ', 'c'], name='id')))
+ col2 = CategoricalMetadataColumn(pd.Series(
+ ['foo', ' bar ', 'baz'], name='col',
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(col1, col2)
+
+ def test_leading_trailing_whitespace_column_name(self):
+ col1 = CategoricalMetadataColumn(pd.Series(
+ ['foo', ' bar ', 'baz'], name=' col ',
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+ col2 = CategoricalMetadataColumn(pd.Series(
+ ['foo', ' bar ', 'baz'], name='col',
+ index=pd.Index(['a', 'b', 'c'], name='id')))
+
+ self.assertEqual(col1, col2)
+
class TestNumericMetadataColumn(unittest.TestCase):
def test_unsupported_dtype(self):
=====================================
qiime2/plugin/model/file_format.py
=====================================
@@ -8,6 +8,7 @@
import abc
+from qiime2.core import transform
from .base import FormatBase, ValidationError, _check_validation_level
@@ -39,6 +40,13 @@ class _FileFormat(FormatBase, metaclass=abc.ABCMeta):
raise NotImplementedError("%r does not implement validate."
% type(self))
+ def view(self, view_type):
+ from_type = transform.ModelType.from_view_type(self.__class__)
+ to_type = transform.ModelType.from_view_type(view_type)
+
+ transformation = from_type.make_transformation(to_type)
+ return transformation(self)
+
class TextFileFormat(_FileFormat):
def open(self):
=====================================
qiime2/plugin/model/tests/test_file_format.py
=====================================
@@ -11,6 +11,7 @@ import unittest
import tempfile
import qiime2.plugin.model as model
+from qiime2.core.testing.plugin import SingleIntFormat
class TestTextFileFormat(unittest.TestCase):
@@ -59,5 +60,28 @@ class TestTextFileFormat(unittest.TestCase):
self.assertEqual(b'S', fh.read(1))
+class TestFileFormat(unittest.TestCase):
+ def setUp(self):
+ self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-')
+
+ path = os.path.join(self.test_dir.name, 'int')
+ with open(path, 'w') as fh:
+ fh.write('1')
+
+ self.format = SingleIntFormat(path, mode='r')
+
+ def tearDown(self):
+ self.test_dir.cleanup()
+
+ def test_view_expected(self):
+ number = self.format.view(int)
+ self.assertEqual(1, number)
+
+ def test_view_invalid_type(self):
+ with self.assertRaisesRegex(
+ Exception, "No transformation.*SingleIntFormat.*float"):
+ self.format.view(float)
+
+
if __name__ == '__main__':
unittest.main()
=====================================
qiime2/sdk/actiongraph.py
=====================================
@@ -187,15 +187,15 @@ def build_graph(action_list=[], opt=False):
name = "opt_"+str(in_v)
G.add_edge(name, str(dict_))
G[name][str(dict_)]['name'] = in_k[1:]
- G.node[name]['type'] = in_v
- G.node[name]['optional'] = True
- G.node[name]['node'] = 'type'
+ G.nodes[name]['type'] = in_v
+ G.nodes[name]['optional'] = True
+ G.nodes[name]['node'] = 'type'
else:
G.add_edge(in_v, str(dict_))
G[in_v][str(dict_)]['name'] = in_k
- G.node[in_v]['type'] = in_v
- G.node[in_v]['optional'] = False
- G.node[in_v]['node'] = 'type'
+ G.nodes[in_v]['type'] = in_v
+ G.nodes[in_v]['optional'] = False
+ G.nodes[in_v]['node'] = 'type'
else:
for out_k, out_v in v.items():
if not out_v:
@@ -204,13 +204,13 @@ def build_graph(action_list=[], opt=False):
name = "opt_"+str(out_v)
G.add_edge("opt_"+str(out_v), str(dict_))
G[str(dict_)][name]['name'] = out_k[1:]
- G.node[name]['type'] = in_v
- G.node[name]['optional'] = True
- G.node[name]['node'] = 'type'
+ G.nodes[name]['type'] = in_v
+ G.nodes[name]['optional'] = True
+ G.nodes[name]['node'] = 'type'
else:
G.add_edge(str(dict_), out_v)
G[str(dict_)][out_v]['name'] = out_k
- G.node[out_v]['type'] = out_v
- G.node[out_v]['optional'] = False
- G.node[out_v]['node'] = 'type'
+ G.nodes[out_v]['type'] = out_v
+ G.nodes[out_v]['optional'] = False
+ G.nodes[out_v]['node'] = 'type'
return G
View it on GitLab: https://salsa.debian.org/med-team/qiime/commit/970d3394afddcc67d1ab018c785d84059282f675
--
View it on GitLab: https://salsa.debian.org/med-team/qiime/commit/970d3394afddcc67d1ab018c785d84059282f675
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20191214/140988ce/attachment-0001.html>
More information about the debian-med-commit
mailing list