[Git][debian-gis-team/trollsift][upstream] New upstream version 0.4.0
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Sat Feb 5 08:28:56 GMT 2022
Antonio Valentino pushed to branch upstream at Debian GIS Project / trollsift
Commits:
0bfd2db5 by Antonio Valentino at 2022-02-05T08:07:43+00:00
New upstream version 0.4.0
- - - - -
10 changed files:
- .github/workflows/ci.yaml
- AUTHORS.md
- CHANGELOG.md
- RELEASING.md
- doc/source/usage.rst
- setup.cfg
- setup.py
- trollsift/parser.py
- trollsift/tests/unittests/test_parser.py
- trollsift/version.py
Changes:
=====================================
.github/workflows/ci.yaml
=====================================
@@ -9,7 +9,7 @@ jobs:
fail-fast: true
matrix:
os: ["windows-latest", "ubuntu-latest", "macos-latest"]
- python-version: ["3.7", "3.8", "3.9"]
+ python-version: ["3.8", "3.9", "3.10"]
env:
PYTHON_VERSION: ${{ matrix.python-version }}
=====================================
AUTHORS.md
=====================================
@@ -10,3 +10,4 @@ The following people have made contributions to this project:
- [Martin Raspaud (mraspaud)](https://github.com/mraspaud)
- [Hrobjartur Thorsteinsson (thorsteinssonh)](https://github.com/thorsteinssonh)
- [Stephan Finkensieper (sfinkens)](https://github.com/sfinkens)
+- [Paulo Medeiros (paulovcmedeiros)](https://github.com/paulovcmedeiros)
=====================================
CHANGELOG.md
=====================================
@@ -1,3 +1,24 @@
+## Version 0.4.0 (2022/02/03)
+
+### Issues Closed
+
+* [Issue 30](https://github.com/pytroll/trollsift/issues/30) - Problems with padding syntax ([PR 33](https://github.com/pytroll/trollsift/pull/33) by [@paulovcmedeiros](https://github.com/paulovcmedeiros))
+
+In this release 1 issue was closed.
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 33](https://github.com/pytroll/trollsift/pull/33) - Fix problems with type='' in string padding syntax ([30](https://github.com/pytroll/trollsift/issues/30))
+
+#### Features added
+
+* [PR 32](https://github.com/pytroll/trollsift/pull/32) - Add 'allow_partial' keyword to compose
+* [PR 31](https://github.com/pytroll/trollsift/pull/31) - Change tested Python versions to 3.8, 3.9 and 3.10
+* [PR 24](https://github.com/pytroll/trollsift/pull/24) - Skip Python2 support and require python 3.6 or higher
+
+
## Version 0.3.5 (2021/02/15)
### Issues Closed
=====================================
RELEASING.md
=====================================
@@ -1,12 +1,12 @@
-# Releasing SatPy
+# Releasing trollsift
-1. checkout master
+1. checkout main branch
2. pull from repo
3. run the unittests
4. run `loghub` and update the `CHANGELOG.md` file:
```
-loghub pytroll/trollsift -u <username> -st v0.8.0 -plg bug "Bugs fixed" -plg enhancement "Features added" -plg documentation "Documentation changes" -plg backwards-incompatibility "Backwards incompatible changes"
+loghub pytroll/trollsift --token $LOGHUB_GITHUB_TOKEN -st v0.8.0 -plg bug "Bugs fixed" -plg enhancement "Features added" -plg documentation "Documentation changes" -plg backwards-incompatibility "Backwards incompatible changes"
```
Don't forget to commit!
=====================================
doc/source/usage.rst
=====================================
@@ -44,17 +44,21 @@ The reverse operation is called 'compose', and is equivalent to the Python
string class format method. Here we take the filename pattern from earlier,
change the time stamp of the data, and write out a new file name,
-.. doctest::
- :hide:
- >>> p = Parser("/somedir/{directory}/hrpt_{platform:4s}{platnum:2s}_{time:%Y%m%d_%H%M}_{orbit:05d}.l1b")
- >>> data = p.parse("/somedir/otherdir/hrpt_noaa16_20140210_1004_69022.l1b")
-
-
>>> from datetime import datetime
- >>> data['time'] = datetime(2012, 1, 1, 1, 1)
+ >>>
+ >>> p = Parser("/somedir/{directory}/hrpt_{platform:4s}{platnum:2s}_{time:%Y%m%d_%H%M}_{orbit:05d}.l1b")
+ >>> data = {'directory': 'otherdir', 'platform': 'noaa', 'platnum': '16', 'time': datetime(2012, 1, 1, 1, 1), 'orbit': 69022}
>>> p.compose(data)
'/somedir/otherdir/hrpt_noaa16_20120101_0101_69022.l1b'
+It is also possible to compose only partially, i.e., compose by specifying values
+for only a subset of the parameters in the format string. Example:
+
+ >>> p = Parser("/somedir/{directory}/hrpt_{platform:4s}{platnum:2s}_{time:%Y%m%d_%H%M}_{orbit:05d}.l1b")
+ >>> data = {'directory':'my_dir'}
+ >>> p.compose(data, allow_partial=True)
+ '/somedir/my_dir/hrpt_{platform:4s}{platnum:2s}_{time:%Y%m%d_%H%M}_{orbit:05d}.l1b'
+
In addition to python's builtin string formatting functionality trollsift also
provides extra conversion options such as making all characters lowercase:
=====================================
setup.cfg
=====================================
@@ -2,7 +2,6 @@
description-file = README.md
[bdist_rpm]
-requires=python python-six
release=1
[bdist_wheel]
=====================================
setup.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2014, 2015
+# Copyright (c) 2014-2022 trollsift developers
#
# Author(s):
#
@@ -32,8 +32,9 @@ setup(name="trollsift",
cmdclass=versioneer.get_cmdclass(),
description='String parser/formatter',
long_description=README,
- author='Panu Lahtinen',
- author_email='panu.lahtinen at fmi.fi',
+ long_description_content_type='text/x-rst',
+ author='The Pytroll Team',
+ author_email='pytroll at googlegroups.com',
classifiers=["Development Status :: 5 - Production/Stable",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: GNU General Public License v3 " +
@@ -46,6 +47,7 @@ setup(name="trollsift",
packages=['trollsift'],
keywords=["string parsing", "string formatting", "pytroll"],
zip_safe=False,
+ python_requires='>=3.6',
install_requires=[],
tests_require=['pytest']
)
=====================================
trollsift/parser.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2014-2020 Trollsift Developers
+# Copyright (c) 2014-2022 Trollsift Developers
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -39,11 +39,22 @@ class Parser(object):
'''
return parse(self.fmt, stri, full_match=full_match)
- def compose(self, keyvals):
- '''Return string composed according to *fmt* string and filled
- with values with the corresponding keys in *keyvals* dictionary.
- '''
- return compose(self.fmt, keyvals)
+ def compose(self, keyvals, allow_partial=False):
+ """Compose format string *self.fmt* with parameters given in the *keyvals* dict.
+
+ Args:
+ keyvals (dict): "Parameter --> parameter value" map
+ allow_partial (bool): If True, then partial composition is allowed, i.e.,
+ not all parameters present in `fmt` need to be specified in `keyvals`.
+ Unspecified parameters will, in this case, be left unchanged.
+ (Default value = False).
+
+ Returns:
+ str: Result of formatting the *self.fmt* string with parameter values
+ extracted from the corresponding items in the *keyvals* dictionary.
+
+ """
+ return compose(fmt=self.fmt, keyvals=keyvals, allow_partial=allow_partial)
format = compose
@@ -152,13 +163,14 @@ spec_regexes['e'] = spec_regexes['f']
spec_regexes['E'] = spec_regexes['f']
spec_regexes['g'] = spec_regexes['f']
spec_regexes['X'] = spec_regexes['x']
-allow_multiple = ['c', 'd', 'o', 's', 'x', 'X']
+spec_regexes[''] = spec_regexes['s']
+allow_multiple = ['c', 'd', 'o', 's', '', 'x', 'X']
fixed_point_types = ['f', 'e', 'E', 'g']
# format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]
# https://docs.python.org/3.4/library/string.html#format-specification-mini-language
fmt_spec_regex = re.compile(
r'(?P<align>(?P<fill>.)?[<>=^])?(?P<sign>[\+\-\s])?(?P<pound>#)?(?P<zero>0)?(?P<width>\d+)?'
- r'(?P<comma>,)?(?P<precision>.\d+)?(?P<type>[bcdeEfFgGnosxX%])')
+ r'(?P<comma>,)?(?P<precision>.\d+)?(?P<type>[bcdeEfFgGnosxX%]?)')
def _get_fixed_point_regex(regex_dict, width, precision):
@@ -283,7 +295,7 @@ class RegexFormatter(string.Formatter):
if fill is None:
if width is not None and width[0] == '0':
fill = '0'
- elif ftype in ['s', 'd']:
+ elif ftype in ['s', '', 'd']:
fill = ' '
char_type = spec_regexes[ftype]
@@ -293,7 +305,7 @@ class RegexFormatter(string.Formatter):
width=width,
precision=precision
)
- if ftype == 's' and align and align.endswith('='):
+ if ftype in ('s', '') and align and align.endswith('='):
raise ValueError("Invalid format specification: '{}'".format(format_spec))
final_regex = char_type
if ftype in allow_multiple and (not width or width == '0'):
@@ -376,19 +388,15 @@ def _get_number_from_fmt(fmt):
def _convert(convdef, stri):
"""Convert the string *stri* to the given conversion definition *convdef*."""
- is_fixed_point = any([ftype in convdef for ftype in fixed_point_types])
if '%' in convdef:
result = dt.datetime.strptime(stri, convdef)
- elif 'd' in convdef or 's' in convdef or is_fixed_point:
- stri = _strip_padding(convdef, stri)
- if 'd' in convdef:
- result = int(stri)
- elif is_fixed_point:
- result = float(stri)
- else:
- result = stri
else:
- result = stri
+ result = _strip_padding(convdef, stri)
+ if 'd' in convdef:
+ result = int(result)
+ elif any(float_type_marker in convdef for float_type_marker in fixed_point_types):
+ result = float(result)
+
return result
@@ -446,9 +454,25 @@ def parse(fmt, stri, full_match=True):
return keyvals
-def compose(fmt, keyvals):
- """Convert parameters in `keyvals` to a string based on `fmt` string."""
- return formatter.format(fmt, **keyvals)
+def compose(fmt, keyvals, allow_partial=False):
+ """Compose format string *self.fmt* with parameters given in the *keyvals* dict.
+
+ Args:
+ fmt (str): Python format string to match against
+ keyvals (dict): "Parameter --> parameter value" map
+ allow_partial (bool): If True, then partial composition is allowed, i.e.,
+ not all parameters present in `fmt` need to be specified in `keyvals`.
+ Unspecified parameters will, in this case, be left unchanged.
+ (Default value = False).
+
+ Returns:
+ str: Result of formatting the *self.fmt* string with parameter values
+ extracted from the corresponding items in the *keyvals* dictionary.
+
+ """
+ if allow_partial:
+ return _partial_compose(fmt=fmt, keyvals=keyvals)
+ return _strict_compose(fmt=fmt, keyvals=keyvals)
DT_FMT = {
@@ -641,3 +665,51 @@ def purge():
"""
regex_formatter.format.cache_clear()
get_convert_dict.cache_clear()
+
+
+def _strict_compose(fmt, keyvals):
+ """Convert parameters in `keyvals` to a string based on `fmt` string."""
+ return formatter.format(fmt, **keyvals)
+
+
+def _partial_compose(fmt, keyvals):
+ """Convert parameters in `keyvals` to a string based on `fmt` string.
+
+ Similar to _strict_compose, but accepts partial composing, i.e., not all
+ parameters in `fmt` need to be specified in `keyvals`. Unspecified parameters
+ are left unchanged.
+
+ Args:
+ fmt (str): Python format string to match against
+ keyvals (dict): "Parameter --> parameter value" map
+
+ """
+ fmt, undefined_vars = _replace_undefined_params_with_placeholders(fmt, keyvals)
+ composed_string = _strict_compose(fmt=fmt, keyvals=keyvals)
+ for fmt_placeholder, fmt_specification in undefined_vars.items():
+ composed_string = composed_string.replace(fmt_placeholder, fmt_specification)
+
+ return composed_string
+
+
+def _replace_undefined_params_with_placeholders(fmt, keyvals=None):
+ """Replace with placeholders params in `fmt` not specified in `keyvals`."""
+ vars_left_undefined = get_convert_dict(fmt).keys()
+ if keyvals is not None:
+ vars_left_undefined -= keyvals.keys()
+
+ undefined_vars_placeholders_dict = {}
+ new_fmt = fmt
+ for var in sorted(vars_left_undefined):
+ matches = set(
+ match.group()
+ for match in re.finditer(rf"{{{re.escape(var)}([^\w{{}}].*?)*}}", new_fmt)
+ )
+ if len(matches) == 0:
+ raise ValueError(f"Could not capture definitions for {var} from {fmt}")
+ for var_specification in matches:
+ fmt_placeholder = f"({hex(hash(var_specification))})"
+ undefined_vars_placeholders_dict[fmt_placeholder] = var_specification
+ new_fmt = new_fmt.replace(var_specification, fmt_placeholder)
+
+ return new_fmt, undefined_vars_placeholders_dict
=====================================
trollsift/tests/unittests/test_parser.py
=====================================
@@ -126,6 +126,19 @@ class TestParser(unittest.TestCase):
'time': dt.datetime(2014, 2, 12, 14, 12),
'orbit': 12345})
+ def test_parse_string_padding_syntax_with_and_without_s(self):
+ """Test that, in string padding syntax, '' is equivalent to 's'.
+
+ From <https://docs.python.org/3.4/library/string.html#format-specification-mini-language>:
+ * Type 's': String format. This is the default type for strings and may be omitted.
+ * Type None: The same as 's'.
+ """
+ result = parse('{foo}/{bar:_<8}', 'baz/qux_____')
+ expected_result = parse('{foo}/{bar:_<8s}', 'baz/qux_____')
+ self.assertEqual(expected_result["foo"], "baz")
+ self.assertEqual(expected_result["bar"], "qux")
+ self.assertEqual(result, expected_result)
+
def test_parse_wildcards(self):
# Run
result = parse(
@@ -272,34 +285,6 @@ class TestParser(unittest.TestCase):
self.assertFalse(is_one2one(
"/somedir/{directory}/somedata_{platform:4s}_{time:%Y%d%m-%H%M}_{orbit:d}.l1b"))
- def test_compose(self):
- """Test the compose method's custom conversion options."""
- key_vals = {'a': 'this Is A-Test b_test c test'}
-
- new_str = compose("{a!c}", key_vals)
- self.assertEqual(new_str, 'This is a-test b_test c test')
- new_str = compose("{a!h}", key_vals)
- self.assertEqual(new_str, 'thisisatestbtestctest')
- new_str = compose("{a!H}", key_vals)
- self.assertEqual(new_str, 'THISISATESTBTESTCTEST')
- new_str = compose("{a!l}", key_vals)
- self.assertEqual(new_str, 'this is a-test b_test c test')
- new_str = compose("{a!R}", key_vals)
- self.assertEqual(new_str, 'thisIsATestbtestctest')
- new_str = compose("{a!t}", key_vals)
- self.assertEqual(new_str, 'This Is A-Test B_Test C Test')
- new_str = compose("{a!u}", key_vals)
- self.assertEqual(new_str, 'THIS IS A-TEST B_TEST C TEST')
- # builtin repr
- new_str = compose("{a!r}", key_vals)
- self.assertEqual(new_str, '\'this Is A-Test b_test c test\'')
- # no formatting
- new_str = compose("{a}", key_vals)
- self.assertEqual(new_str, 'this Is A-Test b_test c test')
- # bad formatter
- self.assertRaises(ValueError, compose, "{a!X}", key_vals)
- self.assertEqual(new_str, 'this Is A-Test b_test c test')
-
def test_greediness(self):
"""Test that the minimum match is parsed out.
@@ -322,9 +307,86 @@ class TestParser(unittest.TestCase):
self.assertEqual(exp, res_dict)
+class TestCompose:
+ """Test routines related to `compose` methods."""
+
+ @pytest.mark.parametrize('allow_partial', [False, True])
+ def test_compose(self, allow_partial):
+ """Test the compose method's custom conversion options."""
+ key_vals = {"a": "this Is A-Test b_test c test"}
+
+ new_str = compose("{a!c}", key_vals, allow_partial=allow_partial)
+ assert new_str == "This is a-test b_test c test"
+ new_str = compose("{a!h}", key_vals, allow_partial=allow_partial)
+ assert new_str == "thisisatestbtestctest"
+ new_str = compose("{a!H}", key_vals, allow_partial=allow_partial)
+ assert new_str == "THISISATESTBTESTCTEST"
+ new_str = compose("{a!l}", key_vals, allow_partial=allow_partial)
+ assert new_str == "this is a-test b_test c test"
+ new_str = compose("{a!R}", key_vals, allow_partial=allow_partial)
+ assert new_str == "thisIsATestbtestctest"
+ new_str = compose("{a!t}", key_vals, allow_partial=allow_partial)
+ assert new_str == "This Is A-Test B_Test C Test"
+ new_str = compose("{a!u}", key_vals, allow_partial=allow_partial)
+ assert new_str == "THIS IS A-TEST B_TEST C TEST"
+ # builtin repr
+ new_str = compose("{a!r}", key_vals, allow_partial=allow_partial)
+ assert new_str == "'this Is A-Test b_test c test'"
+ # no formatting
+ new_str = compose("{a}", key_vals, allow_partial=allow_partial)
+ assert new_str == "this Is A-Test b_test c test"
+ # bad formatter
+ with pytest.raises(ValueError):
+ new_str = compose("{a!X}", key_vals, allow_partial=allow_partial)
+ assert new_str == "this Is A-Test b_test c test"
+
+ def test_default_compose_is_strict(self):
+ """Make sure the default compose call does not accept partial composition."""
+ fmt = "{foo}_{bar}.qux"
+ with pytest.raises(KeyError):
+ _ = compose(fmt, {"foo": "foo"})
+
+ def test_partial_compose_simple(self):
+ """Test partial compose with a simple use case."""
+ fmt = "{variant:s}/{platform_name}_{start_time:%Y%m%d_%H%M}_{product}.{format}"
+ composed = compose(
+ fmt=fmt,
+ keyvals={"platform_name": "foo", "format": "bar"},
+ allow_partial=True
+ )
+ assert composed == "{variant:s}/foo_{start_time:%Y%m%d_%H%M}_{product}.bar"
+
+ def test_partial_compose_with_similarly_named_params(self):
+ """Test that partial compose handles well vars with common substrings in name."""
+ original_fmt = "{foo}{afooo}{fooo}.{bar}/{baz:%Y}/{baz:%Y%m%d_%H}/{baz:%Y}/{bar:d}"
+ composed = compose(fmt=original_fmt, keyvals={"afooo": "qux"}, allow_partial=True)
+ assert composed == "{foo}qux{fooo}.{bar}/{baz:%Y}/{baz:%Y%m%d_%H}/{baz:%Y}/{bar:d}"
+
+ def test_partial_compose_repeated_vars_with_different_formatting(self):
+ """Test partial compose with a fmt with repeated vars with different formatting."""
+ fmt = "/foo/{start_time:%Y%m}/bar/{baz}_{start_time:%Y%m%d_%H%M}.{format}"
+ composed = compose(fmt=fmt, keyvals={"format": "qux"}, allow_partial=True)
+ assert composed == "/foo/{start_time:%Y%m}/bar/{baz}_{start_time:%Y%m%d_%H%M}.qux"
+
+ @pytest.mark.parametrize(
+ 'original_fmt',
+ ["{}_{}", "{foo}{afooo}{fooo}.{bar}/{baz:%Y}/{baz:%Y%m%d_%H}/{baz:%Y}/{bar:d}"]
+ )
+ def test_partial_compose_is_identity_with_empty_keyvals(self, original_fmt):
+ """Test that partial compose leaves the input untouched if no keyvals at all."""
+ assert compose(fmt=original_fmt, keyvals={}, allow_partial=True) == original_fmt
+
+ def test_that_some_invalid_fmt_can_confuse_partial_compose(self):
+ """Test that a fmt with a weird char can confuse partial compose."""
+ fmt = "{foo?}_{bar}_{foo}.qux"
+ with pytest.raises(ValueError):
+ _ = compose(fmt=fmt, keyvals={}, allow_partial=True)
+
+
class TestParserFixedPoint:
"""Test parsing of fixed point numbers."""
+ @pytest.mark.parametrize('allow_partial_compose', [False, True])
@pytest.mark.parametrize(
('fmt', 'string', 'expected'),
[
@@ -355,7 +417,7 @@ class TestParserFixedPoint:
('{foo:7.2e}', '-1.23e4', -1.23e4)
]
)
- def test_match(self, fmt, string, expected):
+ def test_match(self, allow_partial_compose, fmt, string, expected):
"""Test cases expected to be matched."""
# Test parsed value
@@ -363,7 +425,7 @@ class TestParserFixedPoint:
assert parsed['foo'] == expected
# Test round trip
- composed = compose(fmt, {'foo': expected})
+ composed = compose(fmt, {'foo': expected}, allow_partial=allow_partial_compose)
parsed = parse(fmt, composed)
assert parsed['foo'] == expected
=====================================
trollsift/version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
- git_refnames = " (HEAD -> master, tag: v0.3.5)"
- git_full = "30caabdd6b30892380e0223a1f46394b835831a1"
- git_date = "2021-02-15 11:11:18 +0100"
+ git_refnames = " (HEAD -> main, tag: v0.4.0)"
+ git_full = "77c3efcc58eda33ae8ffa3579db811912e09de6a"
+ git_date = "2022-02-03 19:55:29 -0600"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
View it on GitLab: https://salsa.debian.org/debian-gis-team/trollsift/-/commit/0bfd2db576cb62430a61fbabf1ae038c860f198c
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/trollsift/-/commit/0bfd2db576cb62430a61fbabf1ae038c860f198c
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20220205/d34fea39/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list