[med-svn] [Git][med-team/hdmf][master] 7 commits: h5py is fixed
Andreas Tille
gitlab at salsa.debian.org
Thu May 14 09:55:45 BST 2020
Andreas Tille pushed to branch master at Debian Med / hdmf
Commits:
43b36797 by Andreas Tille at 2020-05-14T10:51:19+02:00
h5py is fixed
- - - - -
16f5a5de by Andreas Tille at 2020-05-14T10:51:54+02:00
New upstream version 1.6.1
- - - - -
e3b4854d by Andreas Tille at 2020-05-14T10:51:54+02:00
routine-update: New upstream version
- - - - -
5283613c by Andreas Tille at 2020-05-14T10:51:55+02:00
Update upstream source from tag 'upstream/1.6.1'
Update to upstream version '1.6.1'
with Debian dir cf819c97a83813e618f76f795db3b5f6e9f8fed9
- - - - -
91ba84a4 by Andreas Tille at 2020-05-14T10:51:55+02:00
routine-update: debhelper-compat 12
- - - - -
b7a9b4b5 by Andreas Tille at 2020-05-14T10:53:45+02:00
routine-update: Ready to upload to unstable
- - - - -
cbc90802 by Andreas Tille at 2020-05-14T10:54:37+02:00
Slight copyright fix
- - - - -
25 changed files:
- PKG-INFO
- debian/changelog
- debian/control
- debian/copyright
- setup.cfg
- setup.py
- src/hdmf.egg-info/PKG-INFO
- src/hdmf/_version.py
- src/hdmf/backends/hdf5/h5tools.py
- src/hdmf/build/manager.py
- src/hdmf/build/objectmapper.py
- src/hdmf/common/io/table.py
- src/hdmf/common/table.py
- src/hdmf/spec/namespace.py
- src/hdmf/spec/write.py
- src/hdmf/utils.py
- src/hdmf/validate/validator.py
- tests/coloredtestrunner.py
- tests/unit/build_tests/test_io_map.py
- tests/unit/common/test_table.py
- tests/unit/spec_tests/test_load_namespace.py
- tests/unit/spec_tests/test_spec_write.py
- tests/unit/test_io_hdf5_h5tools.py
- tests/unit/utils_test/test_docval.py
- tests/unit/validator_tests/test_validate.py
Changes:
=====================================
PKG-INFO
=====================================
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: hdmf
-Version: 1.5.4
+Version: 1.6.1
Summary: A package for standardizing hierarchical object data
Home-page: https://github.com/hdmf-dev/hdmf
Author: Andrew Tritt
@@ -106,7 +106,7 @@ Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: BSD License
-Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Microsoft :: Windows
=====================================
debian/changelog
=====================================
@@ -1,4 +1,4 @@
-hdmf (1.5.4-2) UNRELEASED; urgency=medium
+hdmf (1.6.1-1) unstable; urgency=medium
* Team upload.
* Standards-Version: 4.5.0 (routine-update)
@@ -10,10 +10,9 @@ hdmf (1.5.4-2) UNRELEASED; urgency=medium
* Use secure URI in debian/watch.
* Set upstream metadata fields: Bug-Database, Bug-Submit, Repository,
Repository-Browse.
- TODO: Build issue connected to hdf5 / Python 3.8?
- Same is true for new upstream version 1.6
+ * debhelper-compat 12 (routine-update)
- -- Andreas Tille <tille at debian.org> Sat, 18 Apr 2020 07:57:30 +0200
+ -- Andreas Tille <tille at debian.org> Thu, 14 May 2020 10:52:02 +0200
hdmf (1.5.4-1) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -7,7 +7,7 @@ Priority: optional
Build-Depends: dh-python,
python3-setuptools,
python3-all,
- debhelper-compat (= 10),
+ debhelper-compat (= 12),
python3-certifi,
python3-chardet,
python3-h5py (>= 2.9~),
=====================================
debian/copyright
=====================================
@@ -15,7 +15,7 @@ Files: versioneer.py
Copyright: 2016, Brian Warner
License: CC0-1.0
-Files: debian
+Files: debian/*
Copyright: 2019, Yaroslav Halchenko <debian at onerussian.com>
License: BSD-3-custom
=====================================
setup.cfg
=====================================
@@ -17,6 +17,7 @@ exclude =
__pycache__,
build/,
dist/,
+ src/hdmf/common/hdmf-common-schema,
docs/source/conf.py
versioneer.py
per-file-ignores =
=====================================
setup.py
=====================================
@@ -41,7 +41,7 @@ setup_args = {
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"License :: OSI Approved :: BSD License",
- "Development Status :: 2 - Pre-Alpha",
+ "Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Operating System :: Microsoft :: Windows",
=====================================
src/hdmf.egg-info/PKG-INFO
=====================================
@@ -1,6 +1,6 @@
Metadata-Version: 2.1
Name: hdmf
-Version: 1.5.4
+Version: 1.6.1
Summary: A package for standardizing hierarchical object data
Home-page: https://github.com/hdmf-dev/hdmf
Author: Andrew Tritt
@@ -106,7 +106,7 @@ Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: BSD License
-Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Microsoft :: Windows
=====================================
src/hdmf/_version.py
=====================================
@@ -8,11 +8,11 @@ import json
version_json = '''
{
- "date": "2020-01-21T19:23:33-0500",
+ "date": "2020-03-02T17:50:36-0800",
"dirty": false,
"error": null,
- "full-revisionid": "594d45ad3094b9f6545a208943b02c22c580c91d",
- "version": "1.5.4"
+ "full-revisionid": "cc6fba3d7441599423dc5aa80dc1cc431590620b",
+ "version": "1.6.1"
}
''' # END VERSION_JSON
=====================================
src/hdmf/backends/hdf5/h5tools.py
=====================================
@@ -10,8 +10,7 @@ from ...utils import docval, getargs, popargs, call_docval_func, get_data_shape
from ...data_utils import AbstractDataChunkIterator
from ...build import Builder, GroupBuilder, DatasetBuilder, LinkBuilder, BuildManager,\
RegionBuilder, ReferenceBuilder, TypeMap, ObjectMapper
-from ...spec import RefSpec, DtypeSpec, NamespaceCatalog, GroupSpec
-from ...spec import NamespaceBuilder
+from ...spec import RefSpec, DtypeSpec, NamespaceCatalog, GroupSpec, NamespaceBuilder
from .h5_utils import BuilderH5ReferenceDataset, BuilderH5RegionDataset, BuilderH5TableDataset,\
H5DataIO, H5SpecReader, H5SpecWriter
@@ -109,7 +108,19 @@ class HDF5IO(HDMFIO):
deps = dict()
for ns in namespaces:
ns_group = spec_group[ns]
- latest_version = list(ns_group.keys())[-1]
+ # NOTE: by default, objects within groups are iterated in alphanumeric order
+ version_names = list(ns_group.keys())
+ if len(version_names) > 1:
+ # prior to HDMF 1.6.1, extensions without a version were written under the group name "unversioned"
+ # make sure that if there is another group representing a newer version, that is read instead
+ if 'unversioned' in version_names:
+ version_names.remove('unversioned')
+ if len(version_names) > 1:
+ # as of HDMF 1.6.1, extensions without a version are written under the group name "None"
+ # make sure that if there is another group representing a newer version, that is read instead
+ if 'None' in version_names:
+ version_names.remove('None')
+ latest_version = version_names[-1]
ns_group = ns_group[latest_version]
reader = H5SpecReader(ns_group)
readers[ns] = reader
@@ -280,10 +291,7 @@ class HDF5IO(HDMFIO):
for ns_name in ns_catalog.namespaces:
ns_builder = self.__convert_namespace(ns_catalog, ns_name)
namespace = ns_catalog.get_namespace(ns_name)
- if namespace.version is None:
- group_name = '%s/unversioned' % ns_name
- else:
- group_name = '%s/%s' % (ns_name, namespace.version)
+ group_name = '%s/%s' % (ns_name, namespace.version)
ns_group = spec_group.require_group(group_name)
writer = H5SpecWriter(ns_group)
ns_builder.export('namespace', writer=writer)
@@ -910,18 +918,6 @@ class HDF5IO(HDMFIO):
self.__exhaust_dcis()
return
- @classmethod
- def __selection_max_bounds__(cls, selection):
- """Determine the bounds of a numpy selection index tuple"""
- if isinstance(selection, int):
- return selection+1
- elif isinstance(selection, slice):
- return selection.stop
- elif isinstance(selection, list) or isinstance(selection, np.ndarray):
- return np.nonzero(selection)[0][-1]+1
- elif isinstance(selection, tuple):
- return tuple([cls.__selection_max_bounds__(i) for i in selection])
-
@classmethod
def __scalar_fill__(cls, parent, name, data, options=None):
dtype = None
@@ -997,21 +993,25 @@ class HDF5IO(HDMFIO):
"""
try:
chunk_i = next(data)
- # Determine the minimum array dimensions to fit the chunk selection
- max_bounds = cls.__selection_max_bounds__(chunk_i.selection)
- if not hasattr(max_bounds, '__len__'):
- max_bounds = (max_bounds,)
- # Determine if we need to expand any of the data dimensions
- expand_dims = [i for i, v in enumerate(max_bounds) if v is not None and v > dset.shape[i]]
- # Expand the dataset if needed
- if len(expand_dims) > 0:
- new_shape = np.asarray(dset.shape)
- new_shape[expand_dims] = np.asarray(max_bounds)[expand_dims]
- dset.resize(new_shape)
- # Process and write the data
- dset[chunk_i.selection] = chunk_i.data
except StopIteration:
return False
+ if isinstance(chunk_i.selection, tuple):
+ # Determine the minimum array dimensions to fit the chunk selection
+ max_bounds = tuple([x.stop or 0 if isinstance(x, slice) else x+1 for x in chunk_i.selection])
+ elif isinstance(chunk_i.selection, int):
+ max_bounds = (chunk_i.selection+1, )
+ elif isinstance(chunk_i.selection, slice):
+ max_bounds = (chunk_i.selection.stop or 0, )
+ else:
+ msg = ("Chunk selection %s must be a single int, single slice, or tuple of slices "
+ "and/or integers") % str(chunk_i.selection)
+ raise TypeError(msg)
+
+ # Expand the dataset if needed
+ dset.id.extend(max_bounds)
+ # Write the data
+ dset[chunk_i.selection] = chunk_i.data
+
return True
@classmethod
@@ -1140,3 +1140,10 @@ class HDF5IO(HDMFIO):
else:
ret.append(elem)
return ret
+
+ @property
+ def mode(self):
+ """
+ Return the HDF5 file mode. One of ("w", "r", "r+", "a", "w-", "x").
+ """
+ return self.__mode
=====================================
src/hdmf/build/manager.py
=====================================
@@ -136,10 +136,10 @@ class BuildManager:
@docval({"name": "container", "type": AbstractContainer, "doc": "the container to convert to a Builder"},
{"name": "source", "type": str,
"doc": "the source of container being built i.e. file path", 'default': None},
- {"name": "spec_ext", "type": BaseStorageSpec, "doc": "a spec that further refines the base specificatoin",
+ {"name": "spec_ext", "type": BaseStorageSpec, "doc": "a spec that further refines the base specification",
'default': None})
def build(self, **kwargs):
- """ Build the GroupBuilder for the given AbstractContainer"""
+ """ Build the GroupBuilder/DatasetBuilder for the given AbstractContainer"""
container = getargs('container', kwargs)
container_id = self.__conthash__(container)
result = self.__builders.get(container_id)
@@ -152,13 +152,13 @@ class BuildManager:
source = container.container_source
else:
if container.container_source != source:
- raise ValueError("Can't change container_source once set")
+ raise ValueError("Cannot change container_source once set: '%s' %s.%s"
+ % (container.name, container.__class__.__module__,
+ container.__class__.__name__))
result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext)
self.prebuilt(container, result)
- elif container.modified:
- if isinstance(result, GroupBuilder):
- # TODO: if Datasets attributes are allowed to be modified, we need to
- # figure out how to handle that starting here.
+ elif container.modified or spec_ext is not None:
+ if isinstance(result, BaseBuilder):
result = self.__type_map.build(container, self, builder=result, source=source, spec_ext=spec_ext)
return result
@@ -727,10 +727,10 @@ class TypeMap:
"doc": "the BuildManager to use for managing this build", 'default': None},
{"name": "source", "type": str,
"doc": "the source of container being built i.e. file path", 'default': None},
- {"name": "builder", "type": GroupBuilder, "doc": "the Builder to build on", 'default': None},
+ {"name": "builder", "type": BaseBuilder, "doc": "the Builder to build on", 'default': None},
{"name": "spec_ext", "type": BaseStorageSpec, "doc": "a spec extension", 'default': None})
def build(self, **kwargs):
- """ Build the GroupBuilder for the given AbstractContainer"""
+ """Build the GroupBuilder/DatasetBuilder for the given AbstractContainer"""
container, manager, builder = getargs('container', 'manager', 'builder', kwargs)
source, spec_ext = getargs('source', 'spec_ext', kwargs)
=====================================
src/hdmf/build/objectmapper.py
=====================================
@@ -11,7 +11,7 @@ from ..spec import Spec, AttributeSpec, DatasetSpec, GroupSpec, LinkSpec, NAME_W
from ..data_utils import DataIO, AbstractDataChunkIterator
from ..query import ReferenceResolver
from ..spec.spec import BaseStorageSpec
-from .builders import DatasetBuilder, GroupBuilder, LinkBuilder, Builder, ReferenceBuilder, RegionBuilder
+from .builders import DatasetBuilder, GroupBuilder, LinkBuilder, Builder, ReferenceBuilder, RegionBuilder, BaseBuilder
from .manager import Proxy, BuildManager
from .warnings import OrphanContainerWarning, MissingRequiredWarning
@@ -122,21 +122,46 @@ class ObjectMapper(metaclass=ExtenderMeta):
"""
Determine the dtype to use from the dtype of the given value and the specified dtype.
This amounts to determining the greater precision of the two arguments, but also
- checks to make sure the same base dtype is being used.
+ checks to make sure the same base dtype is being used. A warning is raised if the
+ base type of the specified dtype differs from the base type of the given dtype and
+ a conversion will result (e.g., float32 -> uint32).
"""
g = np.dtype(given)
s = np.dtype(specified)
- if g.itemsize <= s.itemsize:
+ if g is s:
return s.type
+ if g.itemsize <= s.itemsize: # given type has precision < precision of specified type
+ # note: this allows float32 -> int32, bool -> int8, int16 -> uint16 which may involve buffer overflows,
+ # truncated values, and other unexpected consequences.
+ warnings.warn('Value with data type %s is being converted to data type %s as specified.'
+ % (g.name, s.name))
+ return s.type
+ elif g.name[:3] == s.name[:3]:
+ return g.type # same base type, use higher-precision given type
else:
- if g.name[:3] != s.name[:3]: # different types
- if s.itemsize < 8:
- msg = "expected %s, received %s - must supply %s or higher precision" % (s.name, g.name, s.name)
- else:
- msg = "expected %s, received %s - must supply %s" % (s.name, g.name, s.name)
+ if np.issubdtype(s, np.unsignedinteger):
+ # e.g.: given int64 and spec uint32, return uint64. given float32 and spec uint8, return uint32.
+ ret_type = np.dtype('uint' + str(int(g.itemsize*8)))
+ warnings.warn('Value with data type %s is being converted to data type %s (min specification: %s).'
+ % (g.name, ret_type.name, s.name))
+ return ret_type.type
+ if np.issubdtype(s, np.floating):
+ # e.g.: given int64 and spec float32, return float64. given uint64 and spec float32, return float32.
+ ret_type = np.dtype('float' + str(max(int(g.itemsize*8), 32)))
+ warnings.warn('Value with data type %s is being converted to data type %s (min specification: %s).'
+ % (g.name, ret_type.name, s.name))
+ return ret_type.type
+ if np.issubdtype(s, np.integer):
+ # e.g.: given float64 and spec int8, return int64. given uint32 and spec int8, return int32.
+ ret_type = np.dtype('int' + str(int(g.itemsize*8)))
+ warnings.warn('Value with data type %s is being converted to data type %s (min specification: %s).'
+ % (g.name, ret_type.name, s.name))
+ return ret_type.type
+ if s.type is np.bool_:
+ msg = "expected %s, received %s - must supply %s" % (s.name, g.name, s.name)
raise ValueError(msg)
- else:
- return g.type
+ # all numeric types in __dtypes should be caught by the above
+ raise ValueError('Unsupported conversion to specification data type: %s' % s.name)
@classmethod
def no_convert(cls, obj_type):
@@ -520,10 +545,10 @@ class ObjectMapper(metaclass=ExtenderMeta):
@docval({"name": "container", "type": AbstractContainer, "doc": "the container to convert to a Builder"},
{"name": "manager", "type": BuildManager, "doc": "the BuildManager to use for managing this build"},
- {"name": "parent", "type": Builder, "doc": "the parent of the resulting Builder", 'default': None},
+ {"name": "parent", "type": GroupBuilder, "doc": "the parent of the resulting Builder", 'default': None},
{"name": "source", "type": str,
"doc": "the source of container being built i.e. file path", 'default': None},
- {"name": "builder", "type": GroupBuilder, "doc": "the Builder to build on", 'default': None},
+ {"name": "builder", "type": BaseBuilder, "doc": "the Builder to build on", 'default': None},
{"name": "spec_ext", "type": BaseStorageSpec, "doc": "a spec extension", 'default': None},
returns="the Builder representing the given AbstractContainer", rtype=Builder)
def build(self, **kwargs):
@@ -539,55 +564,65 @@ class ObjectMapper(metaclass=ExtenderMeta):
self.__add_groups(builder, self.__spec.groups, container, manager, source)
self.__add_links(builder, self.__spec.links, container, manager, source)
else:
- if not isinstance(container, Data):
- msg = "'container' must be of type Data with DatasetSpec"
- raise ValueError(msg)
- spec_dtype, spec_shape, spec = self.__check_dset_spec(self.spec, spec_ext)
- if isinstance(spec_dtype, RefSpec):
- # a dataset of references
- bldr_data = self.__get_ref_builder(spec_dtype, spec_shape, container, manager, source=source)
- builder = DatasetBuilder(name, bldr_data, parent=parent, source=source, dtype=spec_dtype.reftype)
- elif isinstance(spec_dtype, list):
- # a compound dataset
- #
- # check for any references in the compound dtype, and
- # convert them if necessary
- refs = [(i, subt) for i, subt in enumerate(spec_dtype) if isinstance(subt.dtype, RefSpec)]
- bldr_data = copy(container.data)
- bldr_data = list()
- for i, row in enumerate(container.data):
- tmp = list(row)
- for j, subt in refs:
- tmp[j] = self.__get_ref_builder(subt.dtype, None, row[j], manager, source=source)
- bldr_data.append(tuple(tmp))
- try:
- bldr_data, dtype = self.convert_dtype(spec, bldr_data)
- except Exception as ex:
- msg = 'could not resolve dtype for %s \'%s\'' % (type(container).__name__, container.name)
- raise Exception(msg) from ex
- builder = DatasetBuilder(name, bldr_data, parent=parent, source=source, dtype=dtype)
- else:
- # a regular dtype
- if spec_dtype is None and self.__is_reftype(container.data):
- # an unspecified dtype and we were given references
+ if builder is None:
+ if not isinstance(container, Data):
+ msg = "'container' must be of type Data with DatasetSpec"
+ raise ValueError(msg)
+ spec_dtype, spec_shape, spec = self.__check_dset_spec(self.spec, spec_ext)
+ if isinstance(spec_dtype, RefSpec):
+ # a dataset of references
+ bldr_data = self.__get_ref_builder(spec_dtype, spec_shape, container, manager, source=source)
+ builder = DatasetBuilder(name, bldr_data, parent=parent, source=source, dtype=spec_dtype.reftype)
+ elif isinstance(spec_dtype, list):
+ # a compound dataset
+ #
+ # check for any references in the compound dtype, and
+ # convert them if necessary
+ refs = [(i, subt) for i, subt in enumerate(spec_dtype) if isinstance(subt.dtype, RefSpec)]
+ bldr_data = copy(container.data)
bldr_data = list()
- for d in container.data:
- if d is None:
- bldr_data.append(None)
- else:
- bldr_data.append(ReferenceBuilder(manager.build(d, source=source)))
- builder = DatasetBuilder(name, bldr_data, parent=parent, source=source,
- dtype='object')
- else:
- # a dataset that has no references, pass the donversion off to
- # the convert_dtype method
+ for i, row in enumerate(container.data):
+ tmp = list(row)
+ for j, subt in refs:
+ tmp[j] = self.__get_ref_builder(subt.dtype, None, row[j], manager, source=source)
+ bldr_data.append(tuple(tmp))
try:
- bldr_data, dtype = self.convert_dtype(spec, container.data)
+ bldr_data, dtype = self.convert_dtype(spec, bldr_data)
except Exception as ex:
msg = 'could not resolve dtype for %s \'%s\'' % (type(container).__name__, container.name)
raise Exception(msg) from ex
builder = DatasetBuilder(name, bldr_data, parent=parent, source=source, dtype=dtype)
- self.__add_attributes(builder, self.__spec.attributes, container, manager, source)
+ else:
+ # a regular dtype
+ if spec_dtype is None and self.__is_reftype(container.data):
+ # an unspecified dtype and we were given references
+ bldr_data = list()
+ for d in container.data:
+ if d is None:
+ bldr_data.append(None)
+ else:
+ bldr_data.append(ReferenceBuilder(manager.build(d, source=source)))
+ builder = DatasetBuilder(name, bldr_data, parent=parent, source=source,
+ dtype='object')
+ else:
+ # a dataset that has no references, pass the donversion off to
+ # the convert_dtype method
+ try:
+ bldr_data, dtype = self.convert_dtype(spec, container.data)
+ except Exception as ex:
+ msg = 'could not resolve dtype for %s \'%s\'' % (type(container).__name__, container.name)
+ raise Exception(msg) from ex
+ builder = DatasetBuilder(name, bldr_data, parent=parent, source=source, dtype=dtype)
+
+ # Add attributes from the specification extension to the list of attributes
+ all_attrs = self.__spec.attributes + getattr(spec_ext, 'attributes', tuple())
+ # If the spec_ext refines an existing attribute it will now appear twice in the list. The
+ # refinement should only be relevant for validation (not for write). To avoid problems with the
+ # write we here remove duplicates and keep the original spec of the two to make write work.
+ # TODO: We should add validation in the AttributeSpec to make sure refinements are valid
+ # TODO: Check the BuildManager as refinements should probably be resolved rather than be passed in via spec_ext
+ all_attrs = list({a.name: a for a in all_attrs[::-1]}.values())
+ self.__add_attributes(builder, all_attrs, container, manager, source)
return builder
def __check_dset_spec(self, orig, ext):
@@ -965,7 +1000,7 @@ class ObjectMapper(metaclass=ExtenderMeta):
object_id=builder.attributes.get(self.__spec.id_key()))
obj.__init__(**kwargs)
except Exception as ex:
- msg = 'Could not construct %s object due to %s' % (cls.__name__, ex)
+ msg = 'Could not construct %s object due to: %s' % (cls.__name__, ex)
raise Exception(msg) from ex
return obj
=====================================
src/hdmf/common/io/table.py
=====================================
@@ -1,7 +1,6 @@
from ...utils import docval, getargs
from ...build import ObjectMapper, BuildManager
from ...spec import Spec
-from ...container import Container
from ..table import DynamicTable, VectorIndex
from .. import register_map
@@ -23,7 +22,7 @@ class DynamicTableMap(ObjectMapper):
return container.colnames
@docval({"name": "spec", "type": Spec, "doc": "the spec to get the attribute value for"},
- {"name": "container", "type": Container, "doc": "the container to get the attribute value from"},
+ {"name": "container", "type": DynamicTable, "doc": "the container to get the attribute value from"},
{"name": "manager", "type": BuildManager, "doc": "the BuildManager used for managing this build"},
returns='the value of the attribute')
def get_attr_value(self, **kwargs):
=====================================
src/hdmf/common/table.py
=====================================
@@ -297,7 +297,10 @@ class DynamicTable(Container):
if col.get('required', False):
self.add_column(col['name'], col['description'],
index=col.get('index', False),
- table=col.get('table', False))
+ table=col.get('table', False),
+ # Pass through extra keyword arguments for add_column that subclasses may have added
+ **{k: col[k] for k in col.keys()
+ if k not in ['name', 'description', 'index', 'table', 'required']})
else: # create column name attributes (set to None) on the object even if column is not required
setattr(self, col['name'], None)
@@ -366,7 +369,11 @@ class DynamicTable(Container):
if data[col['name']] is not None:
self.add_column(col['name'], col['description'],
index=col.get('index', False),
- table=col.get('table', False))
+ table=col.get('table', False),
+ # Pass through extra keyword arguments for add_column that
+ # subclasses may have added
+ **{k: col[k] for k in col.keys()
+ if k not in ['name', 'description', 'index', 'table', 'required']})
extra_columns.remove(col['name'])
if extra_columns or missing_columns:
@@ -546,7 +553,7 @@ class DynamicTable(Container):
return self[key]
return default
- @docval({'name': 'exclude', 'type': set, 'doc': ' List of columns to exclude from the dataframe', 'default': None})
+ @docval({'name': 'exclude', 'type': set, 'doc': ' Set of columns to exclude from the dataframe', 'default': None})
def to_dataframe(self, **kwargs):
"""
Produce a pandas DataFrame containing this table's data.
@@ -716,6 +723,16 @@ class DynamicTableRegion(VectorData):
else:
raise ValueError("unrecognized argument: '%s'" % key)
+ def to_dataframe(self, **kwargs):
+ """
+ Convert the whole DynamicTableRegion to a pandas dataframe.
+
+ Keyword arguments are passed through to the to_dataframe method of DynamicTable that
+ is being referenced (i.e., self.table). This allows specification of the 'exclude'
+ parameter and any other parameters of DynamicTable.to_dataframe.
+ """
+ return self.table.to_dataframe(**kwargs).iloc[self.data[:]]
+
@property
def shape(self):
"""
@@ -723,3 +740,12 @@ class DynamicTableRegion(VectorData):
:return: Shape tuple with two integers indicating the number of rows and number of columns
"""
return (len(self.data), len(self.table.columns))
+
+ def __repr__(self):
+ cls = self.__class__
+ template = "%s %s.%s at 0x%d\n" % (self.name, cls.__module__, cls.__name__, id(self))
+ template += " Target table: %s %s.%s at 0x%d\n" % (self.table.name,
+ self.table.__class__.__module__,
+ self.table.__class__.__name__,
+ id(self.table))
+ return template
=====================================
src/hdmf/spec/namespace.py
=====================================
@@ -36,6 +36,8 @@ class SpecNamespace(dict):
__types_key = 'data_types'
+ UNVERSIONED = None # value representing missing version
+
@docval(*_namespace_args)
def __init__(self, **kwargs):
doc, full_name, name, version, date, author, contact, schema, catalog = \
@@ -48,8 +50,16 @@ class SpecNamespace(dict):
self['name'] = name
if full_name is not None:
self['full_name'] = full_name
+ if version == str(SpecNamespace.UNVERSIONED):
+ # the unversioned version may be written to file as a string and read from file as a string
+ warn("Loaded namespace '%s' is unversioned. Please notify the extension author." % name)
+ version = SpecNamespace.UNVERSIONED
if version is None:
- raise TypeError('SpecNamespace missing arg `version`. Please specify a version for the extension.')
+ # version is required on write -- see YAMLSpecWriter.write_namespace -- but can be None on read in order to
+ # be able to read older files with extensions that are missing the version key.
+ warn(("Loaded namespace '%s' is missing the required key 'version'. Version will be set to '%s'. "
+ "Please notify the extension author.") % (name, SpecNamespace.UNVERSIONED))
+ version = SpecNamespace.UNVERSIONED
self['version'] = version
if date is not None:
self['date'] = date
@@ -79,13 +89,16 @@ class SpecNamespace(dict):
@property
def author(self):
- """String or list of strings with the authors or None"""
+ """String or list of strings with the authors or None"""
return self.get('author', None)
@property
def version(self):
- """String, list, or tuple with the version or None """
- return self.get('version', None)
+ """
+ String, list, or tuple with the version or SpecNamespace.UNVERSIONED
+ if the version is missing or empty
+ """
+ return self.get('version', None) or SpecNamespace.UNVERSIONED
@property
def date(self):
@@ -422,7 +435,8 @@ class NamespaceCatalog:
catalog.register_spec(spec, spec_file)
included_types[s['namespace']] = tuple(types)
# construct namespace
- self.__namespaces[ns_name] = self.__spec_namespace_cls.build_namespace(catalog=catalog, **namespace)
+ ns = self.__spec_namespace_cls.build_namespace(catalog=catalog, **namespace)
+ self.__namespaces[ns_name] = ns
return included_types
@docval({'name': 'namespace_path', 'type': str, 'doc': 'the path to the file containing the namespaces(s) to load'},
=====================================
src/hdmf/spec/write.py
=====================================
@@ -45,9 +45,14 @@ class YAMLSpecWriter(SpecWriter):
yaml.dump(sorted_data, fd_write, Dumper=yaml.dumper.RoundTripDumper)
def write_namespace(self, namespace, path):
+ """Write the given namespace key-value pairs as YAML to the given path.
+
+ :param namespace: SpecNamespace holding the key-value pairs that define the namespace
+ :param path: File path to write the namespace to as YAML under the key 'namespaces'
+ """
with open(os.path.join(self.__outdir, path), 'w') as stream:
- ns = namespace
# Convert the date to a string if necessary
+ ns = namespace
if 'date' in namespace and isinstance(namespace['date'], datetime):
ns = copy.copy(ns) # copy the namespace to avoid side-effects
ns['date'] = ns['date'].isoformat()
@@ -55,7 +60,7 @@ class YAMLSpecWriter(SpecWriter):
def reorder_yaml(self, path):
"""
- Open a YAML file, load it as python data, sort the data, and write it back out to the
+ Open a YAML file, load it as python data, sort the data alphabetically, and write it back out to the
same path.
"""
with open(path, 'rb') as fd_read:
@@ -109,6 +114,11 @@ class NamespaceBuilder:
{'name': 'namespace_cls', 'type': type, 'doc': 'the SpecNamespace type', 'default': SpecNamespace})
def __init__(self, **kwargs):
ns_cls = popargs('namespace_cls', kwargs)
+ if kwargs['version'] is None:
+ # version is required on write as of HDMF 1.5. this check should prevent the writing of namespace files
+ # without a verison
+ raise ValueError("Namespace '%s' missing key 'version'. Please specify a version for the extension."
+ % kwargs['name'])
self.__ns_args = copy.deepcopy(kwargs)
self.__namespaces = OrderedDict()
self.__sources = OrderedDict()
@@ -237,9 +247,6 @@ def export_spec(ns_builder, new_data_types, output_dir):
warnings.warn('No data types specified. Exiting.')
return
- if not ns_builder.name:
- raise RuntimeError('Namespace name is required to export specs')
-
ns_path = ns_builder.name + '.namespace.yaml'
ext_path = ns_builder.name + '.extensions.yaml'
=====================================
src/hdmf/utils.py
=====================================
@@ -3,12 +3,17 @@ from abc import ABCMeta
import collections
import h5py
import numpy as np
+import warnings
+from enum import Enum
__macros = {
'array_data': [np.ndarray, list, tuple, h5py.Dataset],
'scalar_data': [str, int, float],
}
+# code to signify how to handle positional arguments in docval
+AllowPositional = Enum('AllowPositional', 'ALLOWED WARNING ERROR')
+
def docval_macro(macro):
"""Class decorator to add the class to a list of types associated with the key macro in the __macros dict
@@ -116,7 +121,8 @@ def __format_type(argtype):
raise ValueError("argtype must be a type, str, list, or tuple")
-def __parse_args(validator, args, kwargs, enforce_type=True, enforce_shape=True, allow_extra=False): # noqa: C901
+def __parse_args(validator, args, kwargs, enforce_type=True, enforce_shape=True, allow_extra=False, # noqa: C901
+ allow_positional=AllowPositional.ALLOWED):
"""
Internal helper function used by the docval decorator to parse and validate function arguments
@@ -129,14 +135,20 @@ def __parse_args(validator, args, kwargs, enforce_type=True, enforce_shape=True,
should be enforced if possible.
:param allow_extra: Boolean indicating whether extra keyword arguments are allowed (if False and extra keyword
arguments are specified, then an error is raised).
+ :param allow_positional: integer code indicating whether positional arguments are allowed:
+ AllowPositional.ALLOWED: positional arguments are allowed
+ AllowPositional.WARNING: return warning if positional arguments are supplied
+ AllowPositional.ERROR: return error if positional arguments are supplied
:return: Dict with:
* 'args' : Dict all arguments where keys are the names and values are the values of the arguments.
* 'errors' : List of string with error messages
"""
ret = dict()
+ syntax_errors = list()
type_errors = list()
value_errors = list()
+ future_warnings = list()
argsi = 0
extras = dict() # has to be initialized to empty here, to avoid spurious errors reported upon early raises
@@ -161,6 +173,14 @@ def __parse_args(validator, args, kwargs, enforce_type=True, enforce_shape=True,
% (len(validator), names, len(args) + len(kwargs), len(args), len(kwargs), sorted(kwargs))
)
+ if args:
+ if allow_positional == AllowPositional.WARNING:
+ msg = 'Positional arguments are discouraged and may be forbidden in a future release.'
+ future_warnings.append(msg)
+ elif allow_positional == AllowPositional.ERROR:
+ msg = 'Only keyword arguments (e.g., func(argname=value, ...)) are allowed.'
+ syntax_errors.append(msg)
+
# iterate through the docval specification and find a matching value in args / kwargs
it = iter(validator)
arg = next(it)
@@ -274,7 +294,8 @@ def __parse_args(validator, args, kwargs, enforce_type=True, enforce_shape=True,
# allow_extra needs to be tracked on a function so that fmt_docval_args doesn't strip them out
for key in extras.keys():
ret[key] = extras[key]
- return {'args': ret, 'type_errors': type_errors, 'value_errors': value_errors}
+ return {'args': ret, 'future_warnings': future_warnings, 'type_errors': type_errors, 'value_errors': value_errors,
+ 'syntax_errors': syntax_errors}
docval_idx_name = '__dv_idx__'
@@ -412,10 +433,12 @@ def docval(*validator, **options):
rtype = options.pop('rtype', None)
is_method = options.pop('is_method', True)
allow_extra = options.pop('allow_extra', False)
+ allow_positional = options.pop('allow_positional', True)
def dec(func):
_docval = _copy.copy(options)
_docval['allow_extra'] = allow_extra
+ _docval['allow_positional'] = allow_positional
func.__name__ = _docval.get('func_name', func.__name__)
func.__doc__ = _docval.get('doc', func.__doc__)
pos = list()
@@ -440,10 +463,17 @@ def docval(*validator, **options):
kwargs,
enforce_type=enforce_type,
enforce_shape=enforce_shape,
- allow_extra=allow_extra)
+ allow_extra=allow_extra,
+ allow_positional=allow_positional)
+
+ parse_warnings = parsed.get('future_warnings')
+ if parse_warnings:
+ msg = '%s: %s' % (func.__qualname__, ', '.join(parse_warnings))
+ warnings.warn(msg, FutureWarning)
for error_type, ExceptionType in (('type_errors', TypeError),
- ('value_errors', ValueError)):
+ ('value_errors', ValueError),
+ ('syntax_errors', SyntaxError)):
parse_err = parsed.get(error_type)
if parse_err:
msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
=====================================
src/hdmf/validate/validator.py
=====================================
@@ -6,7 +6,7 @@ from itertools import chain
from ..utils import docval, getargs, call_docval_func, pystr, get_data_shape
-from ..spec import Spec, AttributeSpec, GroupSpec, DatasetSpec, RefSpec
+from ..spec import Spec, AttributeSpec, GroupSpec, DatasetSpec, RefSpec, LinkSpec
from ..spec.spec import BaseStorageSpec, DtypeHelper
from ..spec import SpecNamespace
@@ -26,8 +26,10 @@ __additional = {
'uint8': ['uint16', 'uint32', 'uint64'],
'uint16': ['uint32', 'uint64'],
'uint32': ['uint64'],
+ 'utf': ['ascii']
}
+# if the spec dtype is a key in __allowable, then all types in __allowable[key] are valid
__allowable = dict()
for dt, dt_syn in __synonyms.items():
allow = copy(dt_syn)
@@ -404,6 +406,9 @@ class GroupValidator(BaseStorageValidator):
else:
self.__include_dts[spec.data_type_def] = spec
+ for spec in self.spec.links:
+ self.__include_dts[spec.data_type_inc] = spec
+
@docval({"name": "builder", "type": GroupBuilder, "doc": "the builder to validate"},
returns='a list of Errors', rtype=list)
def validate(self, **kwargs): # noqa: C901
@@ -432,7 +437,7 @@ class GroupValidator(BaseStorageValidator):
for bldr in dt_builders:
tmp = bldr
if isinstance(bldr, LinkBuilder):
- if inc_spec.linkable:
+ if isinstance(inc_spec, LinkSpec) or inc_spec.linkable:
tmp = bldr.builder
else:
ret.append(IllegalLinkError(self.get_spec_loc(inc_spec),
=====================================
tests/coloredtestrunner.py
=====================================
@@ -259,8 +259,9 @@ class ColoredTestResult(unittest.TextTestResult):
super().addSuccess(test)
output = self.complete_output()
self.result.append((0, test, output, ''))
- sys.stdout.write('.')
- sys.stdout.flush()
+ if self.verbosity > 1:
+ sys.stdout.write('.')
+ sys.stdout.flush()
if not hasattr(self, 'successes'):
self.successes = [test]
@@ -273,8 +274,9 @@ class ColoredTestResult(unittest.TextTestResult):
output = self.complete_output()
_, _exc_str = self.errors[-1]
self.result.append((2, test, output, _exc_str))
- sys.stdout.write('E')
- sys.stdout.flush()
+ if self.verbosity > 1:
+ sys.stdout.write('E')
+ sys.stdout.flush()
def addFailure(self, test, err):
self.failure_count += 1
@@ -282,8 +284,9 @@ class ColoredTestResult(unittest.TextTestResult):
output = self.complete_output()
_, _exc_str = self.failures[-1]
self.result.append((1, test, output, _exc_str))
- sys.stdout.write('F')
- sys.stdout.flush()
+ if self.verbosity > 1:
+ sys.stdout.write('F')
+ sys.stdout.flush()
def addSubTest(self, test, subtest, err):
if err is not None:
@@ -296,8 +299,9 @@ class ColoredTestResult(unittest.TextTestResult):
self.skip_count += 1
super().addSkip(test, reason)
self.complete_output()
- sys.stdout.write('s')
- sys.stdout.flush()
+ if self.verbosity > 1:
+ sys.stdout.write('s')
+ sys.stdout.flush()
def get_all_cases_run(self):
'''Return a list of each test case which failed or succeeded
=====================================
tests/unit/build_tests/test_io_map.py
=====================================
@@ -575,71 +575,239 @@ class TestConvertDtype(TestCase):
spec = DatasetSpec('an example dataset', RefSpec(reftype='object', target_type='int'), name='data')
self.assertTupleEqual(ObjectMapper.convert_dtype(spec, None), (None, 'object'))
- def test_convert_higher_precision(self):
- """Test that passing a data type with a precision <= specified returns the higher precision type"""
+ # do full matrix test of given value x and spec y, what does convert_dtype return?
+ def test_convert_to_64bit_spec(self):
+ """
+ Test that if given any value for a spec with a 64-bit dtype, convert_dtype will convert to the spec type.
+ Also test that if the given value is not the same as the spec, convert_dtype raises a warning.
+ """
spec_type = 'float64'
- value_types = ['float', 'float32', 'double', 'float64']
- self.convert_higher_precision_helper(spec_type, value_types)
+ value_types = ['double', 'float64']
+ self._test_convert_alias(spec_type, value_types)
+
+ spec_type = 'float64'
+ value_types = ['float', 'float32', 'long', 'int64', 'int', 'int32', 'int16', 'int8', 'uint64', 'uint',
+ 'uint32', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
spec_type = 'int64'
- value_types = ['long', 'int64', 'uint64', 'int', 'int32', 'int16', 'int8']
- self.convert_higher_precision_helper(spec_type, value_types)
+ value_types = ['long', 'int64']
+ self._test_convert_alias(spec_type, value_types)
- spec_type = 'int32'
- value_types = ['int32', 'int16', 'int8']
- self.convert_higher_precision_helper(spec_type, value_types)
+ spec_type = 'int64'
+ value_types = ['double', 'float64', 'float', 'float32', 'int', 'int32', 'int16', 'int8', 'uint64', 'uint',
+ 'uint32', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
- spec_type = 'int16'
- value_types = ['int16', 'int8']
- self.convert_higher_precision_helper(spec_type, value_types)
+ spec_type = 'uint64'
+ value_types = ['uint64']
+ self._test_convert_alias(spec_type, value_types)
+
+ spec_type = 'uint64'
+ value_types = ['double', 'float64', 'float', 'float32', 'long', 'int64', 'int', 'int32', 'int16', 'int8',
+ 'uint', 'uint32', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_float32_spec(self):
+ """Test conversion of various types to float32.
+ If given a value with precision > float32 and float base type, convert_dtype will keep the higher precision.
+ If given a value with 64-bit precision and different base type, convert_dtype will convert to float64.
+ If given a value that is float32, convert_dtype will convert to float32.
+ If given a value with precision <= float32, convert_dtype will convert to float32 and raise a warning.
+ """
+ spec_type = 'float32'
+ value_types = ['double', 'float64']
+ self._test_keep_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['long', 'int64', 'uint64']
+ expected_type = 'float64'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['float', 'float32']
+ self._test_convert_alias(spec_type, value_types)
+ value_types = ['int', 'int32', 'int16', 'int8', 'uint', 'uint32', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_int32_spec(self):
+ """Test conversion of various types to int32.
+ If given a value with precision > int32 and int base type, convert_dtype will keep the higher precision.
+ If given a value with 64-bit precision and different base type, convert_dtype will convert to int64.
+ If given a value that is int32, convert_dtype will convert to int32.
+ If given a value with precision <= int32, convert_dtype will convert to int32 and raise a warning.
+ """
+ spec_type = 'int32'
+ value_types = ['int64', 'long']
+ self._test_keep_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['double', 'float64', 'uint64']
+ expected_type = 'int64'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['int', 'int32']
+ self._test_convert_alias(spec_type, value_types)
+
+ value_types = ['float', 'float32', 'int16', 'int8', 'uint', 'uint32', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_uint32_spec(self):
+ """Test conversion of various types to uint32.
+ If given a value with precision > uint32 and uint base type, convert_dtype will keep the higher precision.
+ If given a value with 64-bit precision and different base type, convert_dtype will convert to uint64.
+ If given a value that is uint32, convert_dtype will convert to uint32.
+ If given a value with precision <= uint32, convert_dtype will convert to uint32 and raise a warning.
+ """
spec_type = 'uint32'
- value_types = ['uint32', 'uint16', 'uint8']
- self.convert_higher_precision_helper(spec_type, value_types)
+ value_types = ['uint64']
+ self._test_keep_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['double', 'float64', 'long', 'int64']
+ expected_type = 'uint64'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['uint', 'uint32']
+ self._test_convert_alias(spec_type, value_types)
+
+ value_types = ['float', 'float32', 'int', 'int32', 'int16', 'int8', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_int16_spec(self):
+ """Test conversion of various types to int16.
+ If given a value with precision > int16 and int base type, convert_dtype will keep the higher precision.
+ If given a value with 64-bit precision and different base type, convert_dtype will convert to int64.
+ If given a value with 32-bit precision and different base type, convert_dtype will convert to int32.
+ If given a value that is int16, convert_dtype will convert to int16.
+ If given a value with precision <= int16, convert_dtype will convert to int16 and raise a warning.
+ """
+ spec_type = 'int16'
+ value_types = ['long', 'int64', 'int', 'int32']
+ self._test_keep_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['double', 'float64', 'uint64']
+ expected_type = 'int64'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['float', 'float32', 'uint', 'uint32']
+ expected_type = 'int32'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['int16']
+ self._test_convert_alias(spec_type, value_types)
+
+ value_types = ['int8', 'uint16', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_uint16_spec(self):
+ """Test conversion of various types to uint16.
+ If given a value with precision > uint16 and uint base type, convert_dtype will keep the higher precision.
+ If given a value with 64-bit precision and different base type, convert_dtype will convert to uint64.
+ If given a value with 32-bit precision and different base type, convert_dtype will convert to uint32.
+ If given a value that is uint16, convert_dtype will convert to uint16.
+ If given a value with precision <= uint16, convert_dtype will convert to uint16 and raise a warning.
+ """
+ spec_type = 'uint16'
+ value_types = ['uint64', 'uint', 'uint32']
+ self._test_keep_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['double', 'float64', 'long', 'int64']
+ expected_type = 'uint64'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['float', 'float32', 'int', 'int32']
+ expected_type = 'uint32'
+ self._test_change_basetype_helper(spec_type, value_types, expected_type)
+
+ value_types = ['uint16']
+ self._test_convert_alias(spec_type, value_types)
+
+ value_types = ['int16', 'int8', 'uint8', 'bool']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ def test_convert_to_bool_spec(self):
+ """Test conversion of various types to bool.
+ If given a value with type bool, convert_dtype will convert to bool.
+ If given a value with type int8/uint8, convert_dtype will convert to bool and raise a warning.
+ Otherwise, convert_dtype will raise an error.
+ """
+ spec_type = 'bool'
+ value_types = ['bool']
+ self._test_convert_alias(spec_type, value_types)
- def convert_higher_precision_helper(self, spec_type, value_types):
- data = 2
+ value_types = ['uint8', 'int8']
+ self._test_convert_higher_precision_helper(spec_type, value_types)
+
+ value_types = ['double', 'float64', 'float', 'float32', 'long', 'int64', 'int', 'int32', 'int16', 'uint64',
+ 'uint', 'uint32', 'uint16']
+ self._test_convert_mismatch_helper(spec_type, value_types)
+
+ def _get_type(self, type_str):
+ return ObjectMapper._ObjectMapper__dtypes[type_str] # apply ObjectMapper mapping string to dtype
+
+ def _test_convert_alias(self, spec_type, value_types):
+ data = 1
spec = DatasetSpec('an example dataset', spec_type, name='data')
- match = (np.dtype(spec_type).type(data), np.dtype(spec_type))
+ match = (self._get_type(spec_type)(data), self._get_type(spec_type))
for dtype in value_types:
- value = np.dtype(dtype).type(data)
+ value = self._get_type(dtype)(data) # convert data to given dtype
with self.subTest(dtype=dtype):
ret = ObjectMapper.convert_dtype(spec, value)
self.assertTupleEqual(ret, match)
- self.assertIs(ret[0].dtype, match[1])
+ self.assertIs(ret[0].dtype.type, match[1])
- def test_keep_higher_precision(self):
- """Test that passing a data type with a precision >= specified return the given type"""
- spec_type = 'float'
- value_types = ['double', 'float64']
- self.keep_higher_precision_helper(spec_type, value_types)
-
- spec_type = 'int'
- value_types = ['int64']
- self.keep_higher_precision_helper(spec_type, value_types)
-
- spec_type = 'int8'
- value_types = ['long', 'int64', 'int', 'int32', 'int16']
- self.keep_higher_precision_helper(spec_type, value_types)
-
- spec_type = 'uint'
- value_types = ['uint64']
- self.keep_higher_precision_helper(spec_type, value_types)
-
- spec_type = 'uint8'
- value_types = ['uint64', 'uint32', 'uint', 'uint16']
- self.keep_higher_precision_helper(spec_type, value_types)
+ def _test_convert_higher_precision_helper(self, spec_type, value_types):
+ data = 1
+ spec = DatasetSpec('an example dataset', spec_type, name='data')
+ match = (self._get_type(spec_type)(data), self._get_type(spec_type))
+ for dtype in value_types:
+ value = self._get_type(dtype)(data) # convert data to given dtype
+ with self.subTest(dtype=dtype):
+ s = np.dtype(self._get_type(spec_type))
+ g = np.dtype(self._get_type(dtype))
+ msg = 'Value with data type %s is being converted to data type %s as specified.' % (g.name, s.name)
+ with self.assertWarnsWith(UserWarning, msg):
+ ret = ObjectMapper.convert_dtype(spec, value)
+ self.assertTupleEqual(ret, match)
+ self.assertIs(ret[0].dtype.type, match[1])
- def keep_higher_precision_helper(self, spec_type, value_types):
- data = 2
+ def _test_keep_higher_precision_helper(self, spec_type, value_types):
+ data = 1
spec = DatasetSpec('an example dataset', spec_type, name='data')
for dtype in value_types:
- value = np.dtype(dtype).type(data)
- match = (value, np.dtype(dtype))
+ value = self._get_type(dtype)(data)
+ match = (value, self._get_type(dtype))
with self.subTest(dtype=dtype):
ret = ObjectMapper.convert_dtype(spec, value)
self.assertTupleEqual(ret, match)
- self.assertIs(ret[0].dtype, match[1])
+ self.assertIs(ret[0].dtype.type, match[1])
+
+ def _test_change_basetype_helper(self, spec_type, value_types, exp_type):
+ data = 1
+ spec = DatasetSpec('an example dataset', spec_type, name='data')
+ match = (self._get_type(exp_type)(data), self._get_type(exp_type))
+ for dtype in value_types:
+ value = self._get_type(dtype)(data) # convert data to given dtype
+ with self.subTest(dtype=dtype):
+ s = np.dtype(self._get_type(spec_type))
+ e = np.dtype(self._get_type(exp_type))
+ g = np.dtype(self._get_type(dtype))
+ msg = ('Value with data type %s is being converted to data type %s (min specification: %s).'
+ % (g.name, e.name, s.name))
+ with self.assertWarnsWith(UserWarning, msg):
+ ret = ObjectMapper.convert_dtype(spec, value)
+ self.assertTupleEqual(ret, match)
+ self.assertIs(ret[0].dtype.type, match[1])
+
+ def _test_convert_mismatch_helper(self, spec_type, value_types):
+ data = 1
+ spec = DatasetSpec('an example dataset', spec_type, name='data')
+ for dtype in value_types:
+ value = self._get_type(dtype)(data) # convert data to given dtype
+ with self.subTest(dtype=dtype):
+ s = np.dtype(self._get_type(spec_type))
+ g = np.dtype(self._get_type(dtype))
+ msg = "expected %s, received %s - must supply %s" % (s.name, g.name, s.name)
+ with self.assertRaisesWith(ValueError, msg):
+ ObjectMapper.convert_dtype(spec, value)
def test_no_spec(self):
spec_type = None
=====================================
tests/unit/common/test_table.py
=====================================
@@ -282,23 +282,6 @@ class TestDynamicTable(TestCase):
with self.assertRaises(ValueError):
table.add_row({'bar': 60.0, 'foo': 6, 'baz': 'oryx', 'qax': -1}, None)
- def test_indexed_dynamic_table_region(self):
- table = self.with_columns_and_data()
- dynamic_table_region = DynamicTableRegion('dtr', [1, 2, 2], 'desc', table=table)
- fetch_ids = dynamic_table_region[:3].index.values
- self.assertListEqual(fetch_ids.tolist(), [1, 2, 2])
-
- def test_dynamic_table_iteration(self):
- table = self.with_columns_and_data()
- dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 3, 4], 'desc', table=table)
- for ii, item in enumerate(dynamic_table_region):
- self.assertTrue(table[ii].equals(item))
-
- def test_dynamic_table_region_shape(self):
- table = self.with_columns_and_data()
- dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 3, 4], 'desc', table=table)
- self.assertTupleEqual(dynamic_table_region.shape, (5, 3))
-
def test_nd_array_to_df(self):
data = np.array([[1, 1, 1], [2, 2, 2], [3, 3, 3]])
col = VectorData(name='data', description='desc', data=data)
@@ -322,6 +305,21 @@ class TestDynamicTable(TestCase):
self.assertTupleEqual(tuple(res.iloc[0]), (3, 30.0, 'bird'))
self.assertTupleEqual(tuple(res.iloc[1]), (5, 50.0, 'lizard'))
+ def test_repr(self):
+ table = self.with_spec()
+ expected = """with_spec hdmf.common.table.DynamicTable at 0x%d
+Fields:
+ colnames: ['foo' 'bar' 'baz']
+ columns: (
+ foo <class 'hdmf.common.table.VectorData'>,
+ bar <class 'hdmf.common.table.VectorData'>,
+ baz <class 'hdmf.common.table.VectorData'>
+ )
+ description: a test table
+"""
+ expected = expected % id(table)
+ self.assertEqual(str(table), expected)
+
class TestDynamicTableRoundTrip(H5RoundTripMixin, TestCase):
@@ -336,6 +334,137 @@ class TestDynamicTableRoundTrip(H5RoundTripMixin, TestCase):
return table
+class TestDynamicTableRegion(TestCase):
+
+ def setUp(self):
+ self.spec = [
+ {'name': 'foo', 'description': 'foo column'},
+ {'name': 'bar', 'description': 'bar column'},
+ {'name': 'baz', 'description': 'baz column'},
+ ]
+ self.data = [
+ [1, 2, 3, 4, 5],
+ [10.0, 20.0, 30.0, 40.0, 50.0],
+ ['cat', 'dog', 'bird', 'fish', 'lizard']
+ ]
+
+ def with_columns_and_data(self):
+ columns = [
+ VectorData(name=s['name'], description=s['description'], data=d)
+ for s, d in zip(self.spec, self.data)
+ ]
+ return DynamicTable("with_columns_and_data", 'a test table', columns=columns)
+
+ def test_indexed_dynamic_table_region(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [1, 2, 2], 'desc', table=table)
+ fetch_ids = dynamic_table_region[:3].index.values
+ self.assertListEqual(fetch_ids.tolist(), [1, 2, 2])
+
+ def test_dynamic_table_region_iteration(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 3, 4], 'desc', table=table)
+ for ii, item in enumerate(dynamic_table_region):
+ self.assertTrue(table[ii].equals(item))
+
+ def test_dynamic_table_region_shape(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 3, 4], 'desc', table=table)
+ self.assertTupleEqual(dynamic_table_region.shape, (5, 3))
+
+ def test_dynamic_table_region_to_dataframe(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region.to_dataframe()
+ self.assertListEqual(res.index.tolist(), [0, 1, 2, 2])
+ self.assertListEqual(res['foo'].tolist(), [1, 2, 3, 3])
+ self.assertListEqual(res['bar'].tolist(), [10.0, 20.0, 30.0, 30.0])
+ self.assertListEqual(res['baz'].tolist(), ['cat', 'dog', 'bird', 'bird'])
+
+ def test_dynamic_table_region_to_dataframe_exclude_cols(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region.to_dataframe(exclude=set(['baz', 'foo']))
+ self.assertListEqual(res.index.tolist(), [0, 1, 2, 2])
+ self.assertEqual(len(res.columns), 1)
+ self.assertListEqual(res['bar'].tolist(), [10.0, 20.0, 30.0, 30.0])
+
+ def test_dynamic_table_region_getitem_slice(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region[1:3]
+ self.assertListEqual(res.index.tolist(), [1, 2])
+ self.assertListEqual(res['foo'].tolist(), [2, 3])
+ self.assertListEqual(res['bar'].tolist(), [20.0, 30.0])
+ self.assertListEqual(res['baz'].tolist(), ['dog', 'bird'])
+
+ def test_dynamic_table_region_getitem_single_row_by_index(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region[2]
+ self.assertListEqual(res.index.tolist(), [2, ])
+ self.assertListEqual(res['foo'].tolist(), [3, ])
+ self.assertListEqual(res['bar'].tolist(), [30.0, ])
+ self.assertListEqual(res['baz'].tolist(), ['bird', ])
+
+ def test_dynamic_table_region_getitem_single_cell(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region[2, 'foo']
+ self.assertEqual(res, 3)
+ res = dynamic_table_region[1, 'baz']
+ self.assertEqual(res, 'dog')
+
+ def test_dynamic_table_region_getitem_slice_of_column(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ res = dynamic_table_region[0:3, 'foo']
+ self.assertListEqual(res, [1, 2, 3])
+ res = dynamic_table_region[1:3, 'baz']
+ self.assertListEqual(res, ['dog', 'bird'])
+
+ def test_dynamic_table_region_getitem_bad_index(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ with self.assertRaises(ValueError):
+ _ = dynamic_table_region['bad index']
+
+ def test_dynamic_table_region_table_prop(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ self.assertEqual(table, dynamic_table_region.table)
+
+ def test_dynamic_table_region_set_table_prop(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc')
+ dynamic_table_region.table = table
+ self.assertEqual(table, dynamic_table_region.table)
+
+ def test_dynamic_table_region_set_table_prop_to_none(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [0, 1, 2, 2], 'desc', table=table)
+ try:
+ dynamic_table_region.table = None
+ except AttributeError:
+ self.fail("DynamicTableRegion table setter raised AttributeError unexpectedly!")
+
+ def test_dynamic_table_region_set_with_bad_data(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [5, 1], 'desc') # index 5 is out of range
+ with self.assertRaises(IndexError):
+ dynamic_table_region.table = table
+ self.assertIsNone(dynamic_table_region.table)
+
+ def test_repr(self):
+ table = self.with_columns_and_data()
+ dynamic_table_region = DynamicTableRegion('dtr', [1, 2, 2], 'desc', table=table)
+ expected = """dtr hdmf.common.table.DynamicTableRegion at 0x%d
+ Target table: with_columns_and_data hdmf.common.table.DynamicTable at 0x%d
+"""
+ expected = expected % (id(dynamic_table_region), id(table))
+ self.assertEqual(str(dynamic_table_region), expected)
+
+
class TestElementIdentifiers(TestCase):
def test_identifier_search_single_list(self):
=====================================
tests/unit/spec_tests/test_load_namespace.py
=====================================
@@ -113,3 +113,113 @@ class TestSpecLoad(TestCase):
src_dsets = {s.name for s in self.ext_datasets}
ext_dsets = {s.name for s in es_spec.datasets}
self.assertSetEqual(src_dsets, ext_dsets)
+
+
+class TestSpecLoadEdgeCase(TestCase):
+
+ def setUp(self):
+ self.specs_path = 'test_load_namespace.specs.yaml'
+ self.namespace_path = 'test_load_namespace.namespace.yaml'
+
+ # write basically empty specs file
+ to_dump = {'groups': []}
+ with open(self.specs_path, 'w') as tmp:
+ yaml.safe_dump(json.loads(json.dumps(to_dump)), tmp, default_flow_style=False)
+
+ def tearDown(self):
+ if os.path.exists(self.namespace_path):
+ os.remove(self.namespace_path)
+ if os.path.exists(self.specs_path):
+ os.remove(self.specs_path)
+
+ def test_build_namespace_missing_version(self):
+ """Test that building/creating a SpecNamespace without a version works but raises a warning."""
+ # create namespace without version key
+ ns_dict = {
+ 'doc': 'a test namespace',
+ 'name': 'test_ns',
+ 'schema': [
+ {'source': self.specs_path}
+ ],
+ }
+ msg = ("Loaded namespace 'test_ns' is missing the required key 'version'. Version will be set to "
+ "'%s'. Please notify the extension author." % SpecNamespace.UNVERSIONED)
+ with self.assertWarnsWith(UserWarning, msg):
+ namespace = SpecNamespace.build_namespace(**ns_dict)
+
+ self.assertEqual(namespace.version, SpecNamespace.UNVERSIONED)
+
+ def test_load_namespace_none_version(self):
+ """Test that reading a namespace file without a version works but raises a warning."""
+ # create namespace with version key (remove it later)
+ ns_dict = {
+ 'doc': 'a test namespace',
+ 'name': 'test_ns',
+ 'schema': [
+ {'source': self.specs_path}
+ ],
+ 'version': '0.0.1'
+ }
+ namespace = SpecNamespace.build_namespace(**ns_dict)
+ namespace['version'] = None # work around lack of setter to remove version key
+
+ # write the namespace to file without version key
+ to_dump = {'namespaces': [namespace]}
+ with open(self.namespace_path, 'w') as tmp:
+ yaml.safe_dump(json.loads(json.dumps(to_dump)), tmp, default_flow_style=False)
+
+ # load the namespace from file
+ ns_catalog = NamespaceCatalog()
+ msg = ("Loaded namespace 'test_ns' is missing the required key 'version'. Version will be set to "
+ "'%s'. Please notify the extension author." % SpecNamespace.UNVERSIONED)
+ with self.assertWarnsWith(UserWarning, msg):
+ ns_catalog.load_namespaces(self.namespace_path)
+
+ self.assertEqual(ns_catalog.get_namespace('test_ns').version, SpecNamespace.UNVERSIONED)
+
+ def test_load_namespace_unversioned_version(self):
+ """Test that reading a namespace file with version=unversioned string works but raises a warning."""
+ # create namespace with version key (remove it later)
+ ns_dict = {
+ 'doc': 'a test namespace',
+ 'name': 'test_ns',
+ 'schema': [
+ {'source': self.specs_path}
+ ],
+ 'version': '0.0.1'
+ }
+ namespace = SpecNamespace.build_namespace(**ns_dict)
+ namespace['version'] = str(SpecNamespace.UNVERSIONED) # work around lack of setter to remove version key
+
+ # write the namespace to file without version key
+ to_dump = {'namespaces': [namespace]}
+ with open(self.namespace_path, 'w') as tmp:
+ yaml.safe_dump(json.loads(json.dumps(to_dump)), tmp, default_flow_style=False)
+
+ # load the namespace from file
+ ns_catalog = NamespaceCatalog()
+ msg = "Loaded namespace 'test_ns' is unversioned. Please notify the extension author."
+ with self.assertWarnsWith(UserWarning, msg):
+ ns_catalog.load_namespaces(self.namespace_path)
+
+ self.assertEqual(ns_catalog.get_namespace('test_ns').version, SpecNamespace.UNVERSIONED)
+
+ def test_missing_version_string(self):
+ """Test that the constant variable representing a missing version has not changed."""
+ self.assertIsNone(SpecNamespace.UNVERSIONED)
+
+ def test_get_namespace_missing_version(self):
+ """Test that SpecNamespace.version returns the constant for a missing version if version gets removed."""
+ # create namespace with version key (remove it later)
+ ns_dict = {
+ 'doc': 'a test namespace',
+ 'name': 'test_ns',
+ 'schema': [
+ {'source': self.specs_path}
+ ],
+ 'version': '0.0.1'
+ }
+ namespace = SpecNamespace.build_namespace(**ns_dict)
+ namespace['version'] = None # work around lack of setter to remove version key
+
+ self.assertEqual(namespace.version, SpecNamespace.UNVERSIONED)
=====================================
tests/unit/spec_tests/test_spec_write.py
=====================================
@@ -65,6 +65,25 @@ class TestSpec(TestCase):
nsstr = file.read()
self.assertEqual(nsstr, match_str)
+ def _test_namespace_file(self):
+ with open(self.namespace_path, 'r') as file:
+ match_str = \
+"""namespaces:
+- author: foo
+ contact: foo at bar.com
+ date: '%s'
+ doc: mydoc
+ full_name: My Laboratory
+ name: mylab
+ schema:
+ - doc: Extensions for my lab
+ source: mylab.extensions.yaml
+ title: Extensions for my lab
+ version: 0.0.1
+""" % self.date.isoformat() # noqa: E128
+ nsstr = file.read()
+ self.assertEqual(nsstr, match_str)
+
class TestNamespaceBuilder(TestSpec):
NS_NAME = 'test_ns'
@@ -88,25 +107,6 @@ class TestNamespaceBuilder(TestSpec):
self._test_namespace_file()
self._test_extensions_file()
- def _test_namespace_file(self):
- with open(self.namespace_path, 'r') as file:
- match_str = \
-"""namespaces:
-- author: foo
- contact: foo at bar.com
- date: '%s'
- doc: mydoc
- full_name: My Laboratory
- name: mylab
- schema:
- - doc: Extensions for my lab
- source: mylab.extensions.yaml
- title: Extensions for my lab
- version: 0.0.1
-""" % self.date.isoformat() # noqa: E128
- nsstr = file.read()
- self.assertEqual(nsstr, match_str)
-
def test_read_namespace(self):
ns_catalog = NamespaceCatalog()
ns_catalog.load_namespaces(self.namespace_path, resolve=True)
@@ -137,6 +137,18 @@ class TestNamespaceBuilder(TestSpec):
'source': 'mylab.extensions.yaml',
'title': 'Extensions for my lab'})
+ def test_missing_version(self):
+ """Test that creating a namespace builder without a version raises an error."""
+ msg = "Namespace '%s' missing key 'version'. Please specify a version for the extension." % self.ns_name
+ with self.assertRaisesWith(ValueError, msg):
+ self.ns_builder = NamespaceBuilder(doc="mydoc",
+ name=self.ns_name,
+ full_name="My Laboratory",
+ author="foo",
+ contact="foo at bar.com",
+ namespace_cls=SpecNamespace,
+ date=self.date)
+
class TestYAMLSpecWrite(TestSpec):
@@ -167,29 +179,11 @@ class TestYAMLSpecWrite(TestSpec):
def test_get_name(self):
self.assertEqual(self.ns_name, self.ns_builder.name)
- def _test_namespace_file(self):
- with open(self.namespace_path, 'r') as file:
- match_str = \
-"""namespaces:
-- author: foo
- contact: foo at bar.com
- date: '%s'
- doc: mydoc
- full_name: My Laboratory
- name: mylab
- schema:
- - doc: Extensions for my lab
- source: mylab.extensions.yaml
- title: Extensions for my lab
- version: 0.0.1
-""" % self.date.isoformat() # noqa: E128
- nsstr = file.read()
- self.assertEqual(nsstr, match_str)
-
class TestExportSpec(TestSpec):
def test_export(self):
+ """Test that export_spec writes the correct files."""
export_spec(self.ns_builder, self.data_types, '.')
self._test_namespace_file()
self._test_extensions_file()
@@ -201,8 +195,7 @@ class TestExportSpec(TestSpec):
os.remove(self.namespace_path)
def _test_namespace_file(self):
- with open(self.namespace_path, 'r') as nsfile:
- nsstr = nsfile.read()
+ with open(self.namespace_path, 'r') as file:
match_str = \
"""namespaces:
- author: foo
@@ -215,13 +208,10 @@ class TestExportSpec(TestSpec):
- source: mylab.extensions.yaml
version: 0.0.1
""" % self.date.isoformat() # noqa: E128
+ nsstr = file.read()
self.assertEqual(nsstr, match_str)
def test_missing_data_types(self):
+ """Test that calling export_spec on a namespace builder without data types raises a warning."""
with self.assertWarnsWith(UserWarning, 'No data types specified. Exiting.'):
export_spec(self.ns_builder, [], '.')
-
- def test_missing_name(self):
- self.ns_builder._NamespaceBuilder__ns_args['name'] = None
- with self.assertRaisesWith(RuntimeError, 'Namespace name is required to export specs'):
- export_spec(self.ns_builder, self.data_types, '.')
=====================================
tests/unit/test_io_hdf5_h5tools.py
=====================================
@@ -3,6 +3,7 @@ import unittest
import tempfile
import warnings
import numpy as np
+import h5py
from hdmf.utils import docval, getargs
from hdmf.data_utils import DataChunkIterator, InvalidDataIOError
@@ -1330,3 +1331,68 @@ class TestReadLink(TestCase):
bldr2 = read_io2.read_builder()
self.assertEqual(bldr2['link_to_link'].builder.source, self.target_path)
read_io2.close()
+
+
+class TestLoadNamespaces(TestCase):
+
+ def setUp(self):
+ self.manager = _get_manager()
+ self.path = get_temp_filepath()
+
+ def tearDown(self):
+ if os.path.exists(self.path):
+ os.remove(self.path)
+
+ def test_load_namespaces_none_version(self):
+ """Test that reading a file with a cached namespace and None version works but raises a warning."""
+ # Setup all the data we need
+ foo1 = Foo('foo1', [1, 2, 3, 4, 5], "I am foo1", 17, 3.14)
+ foobucket = FooBucket('test_bucket', [foo1])
+ foofile = FooFile([foobucket])
+
+ with HDF5IO(self.path, manager=self.manager, mode='w') as io:
+ io.write(foofile)
+
+ # make the file have group name "None" instead of "0.1.0" (namespace version is used as group name)
+ # and set the version key to "None"
+ with h5py.File(self.path, mode='r+') as f:
+ # rename the group
+ f.move('/specifications/' + CORE_NAMESPACE + '/0.1.0', '/specifications/' + CORE_NAMESPACE + '/None')
+
+ # replace the namespace dataset with a serialized dict without the version key
+ new_ns = ('{"namespaces":[{"doc":"a test namespace","schema":[{"source":"test"}],"name":"test_core",'
+ '"version":"None"}]}')
+ f['/specifications/' + CORE_NAMESPACE + '/None/namespace'][()] = new_ns
+
+ # load the namespace from file
+ ns_catalog = NamespaceCatalog()
+ msg = "Loaded namespace '%s' is unversioned. Please notify the extension author." % CORE_NAMESPACE
+ with self.assertWarnsWith(UserWarning, msg):
+ HDF5IO.load_namespaces(ns_catalog, self.path)
+
+ def test_load_namespaces_unversioned(self):
+ """Test that reading a file with a cached, unversioned version works but raises a warning."""
+ # Setup all the data we need
+ foo1 = Foo('foo1', [1, 2, 3, 4, 5], "I am foo1", 17, 3.14)
+ foobucket = FooBucket('test_bucket', [foo1])
+ foofile = FooFile([foobucket])
+
+ with HDF5IO(self.path, manager=self.manager, mode='w') as io:
+ io.write(foofile)
+
+ # make the file have group name "unversioned" instead of "0.1.0" (namespace version is used as group name)
+ # and remove the version key
+ with h5py.File(self.path, mode='r+') as f:
+ # rename the group
+ f.move('/specifications/' + CORE_NAMESPACE + '/0.1.0', '/specifications/' + CORE_NAMESPACE + '/unversioned')
+
+ # replace the namespace dataset with a serialized dict without the version key
+ new_ns = ('{"namespaces":[{"doc":"a test namespace","schema":[{"source":"test"}],"name":"test_core"}]}')
+ f['/specifications/' + CORE_NAMESPACE + '/unversioned/namespace'][()] = new_ns
+
+ # load the namespace from file
+ ns_catalog = NamespaceCatalog()
+ msg = ("Loaded namespace '%s' is missing the required key 'version'. Version will be set to "
+ "'%s'. Please notify the extension author." % (CORE_NAMESPACE, SpecNamespace.UNVERSIONED))
+ with self.assertWarnsWith(UserWarning, msg):
+ HDF5IO.load_namespaces(ns_catalog, self.path)
=====================================
tests/unit/utils_test/test_docval.py
=====================================
@@ -1,6 +1,6 @@
import numpy as np
-from hdmf.utils import docval, fmt_docval_args, get_docval, popargs
+from hdmf.utils import docval, fmt_docval_args, get_docval, popargs, AllowPositional
from hdmf.testing import TestCase
@@ -527,6 +527,38 @@ class TestDocValidator(TestCase):
self.assertEqual(res, np.bool_(True))
self.assertIsInstance(res, np.bool_)
+ def test_allow_positional_warn(self):
+ @docval({'name': 'arg1', 'type': bool, 'doc': 'this is a bool'}, allow_positional=AllowPositional.WARNING)
+ def method(self, **kwargs):
+ return popargs('arg1', kwargs)
+
+ # check that supplying a keyword arg is OK
+ res = method(self, arg1=True)
+ self.assertEqual(res, True)
+ self.assertIsInstance(res, bool)
+
+ # check that supplying a positional arg raises a warning
+ msg = ('TestDocValidator.test_allow_positional_warn.<locals>.method: '
+ 'Positional arguments are discouraged and may be forbidden in a future release.')
+ with self.assertWarnsWith(FutureWarning, msg):
+ method(self, True)
+
+ def test_allow_positional_error(self):
+ @docval({'name': 'arg1', 'type': bool, 'doc': 'this is a bool'}, allow_positional=AllowPositional.ERROR)
+ def method(self, **kwargs):
+ return popargs('arg1', kwargs)
+
+ # check that supplying a keyword arg is OK
+ res = method(self, arg1=True)
+ self.assertEqual(res, True)
+ self.assertIsInstance(res, bool)
+
+ # check that supplying a positional arg raises an error
+ msg = ('TestDocValidator.test_allow_positional_error.<locals>.method: '
+ 'Only keyword arguments (e.g., func(argname=value, ...)) are allowed.')
+ with self.assertRaisesWith(SyntaxError, msg):
+ method(self, True)
+
class TestDocValidatorChain(TestCase):
=====================================
tests/unit/validator_tests/test_validate.py
=====================================
@@ -1,6 +1,7 @@
from abc import ABCMeta, abstractmethod
from datetime import datetime
from dateutil.tz import tzlocal
+import numpy as np
from hdmf.spec import GroupSpec, AttributeSpec, DatasetSpec, SpecCatalog, SpecNamespace
from hdmf.build import GroupBuilder, DatasetBuilder
@@ -208,3 +209,86 @@ class TestNestedTypes(ValidatorTestBase):
results = self.vmap.validate(foo_builder)
self.assertEqual(len(results), 0)
+
+
+class TestDtypeValidation(TestCase):
+
+ def set_up_spec(self, dtype):
+ spec_catalog = SpecCatalog()
+ spec = GroupSpec('A test group specification with a data type',
+ data_type_def='Bar',
+ datasets=[DatasetSpec('an example dataset', dtype, name='data')],
+ attributes=[AttributeSpec('attr1', 'an example attribute', dtype)])
+ spec_catalog.register_spec(spec, 'test.yaml')
+ self.namespace = SpecNamespace(
+ 'a test namespace', CORE_NAMESPACE, [{'source': 'test.yaml'}], version='0.1.0', catalog=spec_catalog)
+ self.vmap = ValidatorMap(self.namespace)
+
+ def test_ascii_for_utf8(self):
+ """Test that validator allows ASCII data where UTF8 is specified."""
+ self.set_up_spec('text')
+ value = b'an ascii string'
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ self.assertEqual(len(results), 0)
+
+ def test_utf8_for_ascii(self):
+ """Test that validator does not allow UTF8 where ASCII is specified."""
+ self.set_up_spec('bytes')
+ value = 'a utf8 string'
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ result_strings = set([str(s) for s in results])
+ expected_errors = {"Bar/attr1 (my_bar.attr1): incorrect type - expected 'bytes', got 'utf'",
+ "Bar/data (my_bar/data): incorrect type - expected 'bytes', got 'utf'"}
+ self.assertEqual(result_strings, expected_errors)
+
+ def test_int64_for_int8(self):
+ """Test that validator allows int64 data where int8 is specified."""
+ self.set_up_spec('int8')
+ value = np.int64(1)
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ self.assertEqual(len(results), 0)
+
+ def test_int8_for_int64(self):
+ """Test that validator does not allow int8 data where int64 is specified."""
+ self.set_up_spec('int64')
+ value = np.int8(1)
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ result_strings = set([str(s) for s in results])
+ expected_errors = {"Bar/attr1 (my_bar.attr1): incorrect type - expected 'int64', got 'int8'",
+ "Bar/data (my_bar/data): incorrect type - expected 'int64', got 'int8'"}
+ self.assertEqual(result_strings, expected_errors)
+
+ def test_int64_for_numeric(self):
+ """Test that validator allows int64 data where numeric is specified."""
+ self.set_up_spec('numeric')
+ value = np.int64(1)
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ self.assertEqual(len(results), 0)
+
+ def test_bool_for_numeric(self):
+ """Test that validator does not allow bool data where numeric is specified."""
+ self.set_up_spec('numeric')
+ value = np.bool(1)
+ bar_builder = GroupBuilder('my_bar',
+ attributes={'data_type': 'Bar', 'attr1': value},
+ datasets=[DatasetBuilder('data', value)])
+ results = self.vmap.validate(bar_builder)
+ result_strings = set([str(s) for s in results])
+ expected_errors = {"Bar/attr1 (my_bar.attr1): incorrect type - expected 'numeric', got 'bool'",
+ "Bar/data (my_bar/data): incorrect type - expected 'numeric', got 'bool'"}
+ self.assertEqual(result_strings, expected_errors)
View it on GitLab: https://salsa.debian.org/med-team/hdmf/-/compare/7ea98b4d8b7b3a6c2f22f57c75365c2da09a1e8d...cbc9080261623a0c8591c4b16a3618ad6762217a
--
View it on GitLab: https://salsa.debian.org/med-team/hdmf/-/compare/7ea98b4d8b7b3a6c2f22f57c75365c2da09a1e8d...cbc9080261623a0c8591c4b16a3618ad6762217a
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20200514/7c3f6e50/attachment-0001.html>
More information about the debian-med-commit
mailing list