[Git][debian-gis-team/trollsift][master] 4 commits: New upstream version 1.0.2
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Sat Oct 17 09:28:39 BST 2026
Antonio Valentino pushed to branch master at Debian GIS Project / trollsift
Commits:
6985ddde by Antonio Valentino at 2026-10-17T08:25:25+00:00
New upstream version 1.0.2
- - - - -
5efe9d68 by Antonio Valentino at 2026-10-17T08:25:25+00:00
Update upstream source from tag 'upstream/1.0.2'
Update to upstream version '1.0.2'
with Debian dir 821cd045a822a3973cd569eb04b5a76ff311dd4c
- - - - -
5e293b22 by Antonio Valentino at 2026-10-17T08:26:00+00:00
New upstream release
- - - - -
fff04cf5 by Antonio Valentino at 2026-10-17T08:26:40+00:00
Set distribution to unstable
- - - - -
15 changed files:
- − .gitattributes
- .github/workflows/ci.yaml
- .github/workflows/deploy-sdist.yaml
- .pre-commit-config.yaml
- + AGENTS.md
- CHANGELOG.md
- + CLAUDE.md
- − MANIFEST.in
- debian/changelog
- doc/rtd_environment.yaml
- doc/source/conf.py
- doc/source/usage.rst
- pyproject.toml
- trollsift/parser.py
- trollsift/tests/unittests/test_parser.py
Changes:
=====================================
.gitattributes deleted
=====================================
@@ -1 +0,0 @@
-trollsift/version.py export-subst
=====================================
.github/workflows/ci.yaml
=====================================
@@ -18,10 +18,10 @@ jobs:
steps:
- name: Checkout source
- uses: actions/checkout at v6
+ uses: actions/checkout at v7
- name: Set up Python ${{ matrix.python-version }}
- uses: actions/setup-python at v6
+ uses: actions/setup-python at v7
with:
python-version: ${{ matrix.python-version }}
@@ -35,14 +35,14 @@ jobs:
- name: Run unit tests
run: |
- pytest --cov=trollsift trollsift/tests --cov-report=xml
+ pytest --cov=trollsift trollsift --cov-report=xml
- name: Run mypy
run: |
mypy trollsift
- name: Upload unittest coverage to Codecov
- uses: codecov/codecov-action at v6
+ uses: codecov/codecov-action at v7
with:
flags: unittests
files: ./coverage.xml
=====================================
.github/workflows/deploy-sdist.yaml
=====================================
@@ -14,7 +14,7 @@ jobs:
steps:
- name: Checkout source
- uses: actions/checkout at v6
+ uses: actions/checkout at v7
- name: Create sdist
shell: bash -l {0}
@@ -24,4 +24,4 @@ jobs:
- name: Publish package to PyPI
if: github.event.action == 'published'
- uses: pypa/gh-action-pypi-publish at v1.14.0
+ uses: pypa/gh-action-pypi-publish at v1.14.2
=====================================
.pre-commit-config.yaml
=====================================
@@ -2,7 +2,7 @@ exclude: '^$'
fail_fast: false
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
- rev: 'v0.15.12'
+ rev: 'v0.16.10'
hooks:
- id: ruff
args: ["--fix"]
=====================================
AGENTS.md
=====================================
@@ -0,0 +1,153 @@
+# AGENTS.md
+
+Guide for AI agents working on **trollsift**. It records the things that are expensive to
+rediscover — invariants, deliberate design choices, and workflow traps — not things you can
+learn in thirty seconds from the file tree.
+
+## Orientation
+
+- Pure Python, **zero runtime dependencies**, Apache-2.0 (relicensed from GPL in v1.0.0).
+- **The entire library is `trollsift/parser.py`** (~730 lines) plus a 20-line `__init__.py`.
+ There is no other module. If you are looking for where something lives, it is in `parser.py`.
+- `trollsift/version.py` is **generated** by `hatch-vcs` from git tags and is gitignored.
+ Never edit or commit it. `pip install -e .` is required before the package imports.
+- Public API is exactly seven names: `Parser`, `StringFormatter`, `parse`, `compose`,
+ `globify`, `purge`, `validate`.
+- Purpose: parse and compose satellite granule filenames using Python format-string syntax.
+
+## Architecture
+
+Four pipelines, all in `parser.py`:
+
+- **Compose** — `compose()` → `_strict_compose()` = `formatter.format()`, where `formatter`
+ is the module-level `StringFormatter()` singleton. The `!c !l !u !t !R !h !H` conversions
+ are the *only* deviation from stdlib `str.format`.
+- **Partial compose** — `_partial_compose()` replaces each undefined `{var:spec}` occurrence
+ with a `hex(hash(...))` placeholder token, runs strict compose, then substitutes the
+ original `{var:spec}` text back. The result is still a valid format string, so partial
+ composes chain.
+- **Parse** — `parse()` → `get_convert_dict()` (field → spec) + `extract_values()` →
+ `regex_format()` (builds the regex via `RegexFormatter`) → `re.match` → `_convert()`
+ coerces each captured group to a Python type.
+- **Globify** — `globify()` → `GlobifyFormatter` singleton → the `DT_FMT` table maps
+ strftime directives to `?`/`*` runs.
+
+`DT_FMT` is **shared** between globify and parse: `_glob_datetime_directives()` uses its
+`?`/`*` values directly, while `_regex_datetime_directives()` derives `\d{N}` from
+`count("?")`. Editing `DT_FMT` changes both directions at once. Its `"%%"` entry is
+documentation only — every consumer first splits the spec on `%%` via
+`_apply_between_percent_escapes()`, so a directive replacement can never straddle a literal
+percent. That helper's `join` argument matters: the regex and glob passes rejoin with `%`
+because their output is finished, but globify's partial-datetime pass rejoins with `%%`
+because its output is still a format spec that gets a second pass.
+
+## Invariants — do not break these
+
+These are load-bearing for downstream projects. Changing any of them is a breaking change.
+
+- **Non-greedy parsing is a contract, not an accident.** `{a}_{b}` against `"abc_def_ghi"`
+ yields `abc` / `def_ghi`, not `abc_def` / `ghi`. Documented in `doc/source/usage.rst` and
+ the `RegexFormatter` docstring; enforced by `.*?` / `*?` throughout `format_spec_to_regex`.
+- **`parse()` and `extract_values()` raise plain `ValueError` on non-match.** Satpy catches
+ `ValueError` by name, and `validate()` relies on it. Do not introduce a custom exception.
+- **Default keyword values are API**: `parse(..., full_match=True)`,
+ `compose(..., allow_partial=False)` (which raises a bare `KeyError` for a missing key),
+ `globify(fmt, keyvals=None)`.
+- **`globify()` output must stay `fnmatch`-safe** — `*` and `?` only. Downstream feeds it to
+ `fnmatch.fnmatch`, `glob.iglob`, *and* `fsspec`'s `fs.glob`. Emitting character classes
+ like `[0-9]` would work with `glob` but silently change `fnmatch` and `fsspec` semantics.
+- **Type coercion on parse is API**: `:d` → `int`, `:x`/`:X` → base-16 `int`, `:o` → base-8,
+ `:b` → base-2, `f`/`e`/`E`/`g` → `float`, and any `%` spec → `datetime.datetime`. `_convert`
+ dispatches on the *parsed* type letter (`int_bases`, `fixed_point_types`), never on a
+ substring of the whole spec — otherwise a fill character misroutes it (`{foo:d>5s}` is a
+ string field, not an int).
+- **Padding/fill semantics** (`_strip_padding`, `_get_fill`) back hundreds of real patterns
+ such as `{x:_<6s}` and `{x:>04d}`. A width+fill spec deliberately degrades the regex to
+ `.{width}` and the fill is stripped afterwards during conversion. That two-step dance is
+ intentional — the regex cannot tell fill characters from content. `_strip_padding` takes
+ the already-parsed `fmt_spec_regex` groups, not the spec string, so `parse()` matches each
+ spec once. `=` alignment puts the padding after the sign (`_strip_sign_padding`), and a
+ `0` *flag* means a zero fill (`_get_padding_char`) — but only for `>`/`=` on numbers; see
+ the rough edge below.
+- **`StringFormatter.CONV_FUNCS`** (`c h H l t u`, plus `R` handled separately) is
+ user-facing: `R` removes `-`, `_`, `:` and space; `h` = R+lower, `H` = R+upper. End users
+ put these in Satpy `filename=` templates. Entries may be added, never removed or redefined.
+- **Names outside `__all__` that are de-facto public**: `get_convert_dict`, `extract_values`,
+ `is_one2one`, `regex_format`, and the module-level `formatter` / `globify_formatter`
+ singletons. External code imports these from `trollsift.parser`. Narrowing `__all__` in
+ v0.6.0 already caused a reported breakage (GH issue 84).
+
+## Format-spec behavior worth knowing before touching the regex code
+
+- Empty spec is equivalent to `s`; `s` maps to `\S`, so string fields never match whitespace.
+ A width+fill spec degrades to `.{width}`, which *does* match spaces.
+- Duplicate field names: identical specs emit a backreference `(?P=name)`; differing specs
+ raise `ValueError`.
+- Fixed-point width is a **minimum** (via a `(?=.{width})` lookahead); the decimal count is
+ **exact**. So `{foo:5.2f}` matches `"123.45"` and `" 1.23"` but not `"1.23"` or `"1.234"`.
+- Every character in `string.punctuation` is escaped in literal text **except `%`**, so
+ datetime directives survive. `format_spec_to_regex` strips backslashes before matching a
+ spec, because literal text has already been escaped by then.
+- Partial datetime globs: pass a `(value, "Ymd")` tuple to substitute only some directives
+ and leave the rest as `?`.
+- Parsing a time-only spec yields `datetime(1900, 1, 1, ...)` — plain `strptime` defaults.
+- **Caching**: module-level `lru_cache` on `regex_format` and `get_convert_dict`; `purge()`
+ clears both. `regex_format` constructs a **fresh** `RegexFormatter` on every cache miss on
+ purpose — the class holds mutable `_cached_fields` state and a shared instance broke under
+ concurrent use. Do not reintroduce a global `RegexFormatter` singleton.
+
+## Development workflow
+
+- **Install**: `pip install -e .` (required — `trollsift/version.py` is generated at build).
+- **Test**: `pytest` (testpaths is configured). CI runs `pytest --cov=trollsift trollsift/tests`.
+- **`filterwarnings = ["error"]`** — any warning fails a test. `--strict-markers` and
+ `--strict-config` are on, so never use a pytest marker without registering it.
+- **`mypy trollsift` runs in CI and there is no mypy config anywhere** — it runs on defaults.
+ This is easy to forget; run it locally before pushing.
+- **Lint**: `pre-commit run -a` (ruff + ruff-format, line length 120, Google-style
+ docstrings, **mccabe max-complexity 10**). `format_spec_to_regex` sits closest to the
+ complexity ceiling (9); past commits exist purely to get back under it. pre-commit.ci
+ runs on PRs but does **not** autofix them.
+- **Test layout**: `trollsift/tests/{unittests,integrationtests,regressiontests}/test_parser.py`.
+ No `conftest.py`. Older classes subclass `unittest.TestCase`; write new tests in plain
+ pytest style with `@pytest.mark.parametrize`, matching `TestCompose` / `TestParserFixedPoint`.
+- **CI matrix**: Windows/macOS/Linux × Python 3.10, 3.11, 3.13. Avoid path-separator
+ assumptions. Note `requires-python = ">=3.9"` is broader than what CI actually exercises.
+- **Docs**: `doc/source/usage.rst` is written entirely as doctests, but **nothing runs them in
+ CI**. After changing documented behavior, run `make doctest` in `doc/` by hand. Read the
+ Docs builds with `fail_on_warning: true`.
+- **Release**: see `RELEASING.md`. `CHANGELOG.md` is generated by `loghub` from GitHub PR
+ labels (`bug`, `enhancement`, `documentation`, `backwards-incompatibility`) — do not
+ hand-edit it. Tags are `vX.Y.Z`; PyPI publishing uses a trusted publisher on GitHub release.
+- **Commit style**: short imperative one-liners, no conventional-commit prefixes — e.g.
+ "Add mypy to CI", "Fix overly complex functions". Contributors add themselves to `AUTHORS.md`.
+
+## Downstream consumers (why compatibility matters)
+
+- **satpy** — the dominant consumer. `globify` + `parse` drive file discovery in
+ `satpy/readers/core/yaml_reader.py`; `Parser.compose` backs every writer's `filename=`;
+ `StringFormatter` renders NetCDF attribute templates in the AWIPS tiled writer; and a
+ `Parser` parses CF wavelength metadata. Roughly 130 reader YAML configs carry
+ `file_patterns` that trollsift both parses and globifies, exercising string widths, many
+ datetime layouts, `_<6s`-style fill/align, and zero-padded ints.
+- **trollflow2** — uses `compose`, including `compose("", ...)` on an empty pattern, and
+ installs trollsift from **git main** in its CI, so main-branch breakage surfaces there fast.
+- **uwsift** — uses `get_convert_dict`, `validate`, `parse`, the module-level `formatter`
+ singleton (iterating its `.parse()` tuples directly), and two-argument `globify(fmt, keyvals)`.
+- **pyspectral** — optional extra, uses `compose`.
+
+Rule of thumb: any behavior change to parsing or globifying deserves a Satpy sanity check and
+a `backwards-incompatibility` label on the PR.
+
+## Known rough edges
+
+Already known. Don't rediscover them, and don't "fix" them as a drive-by in an unrelated PR.
+
+- **Trailing zero padding is ambiguous and is deliberately not stripped.** `{foo:<05d}`
+ formats both `-12` and `-1200` as `"-1200"`; `{foo:^05d}` formats both `12` and `120` as
+ `"01200"`. There is no correct parse, so `_get_padding_char` infers a `"0"` pad from the
+ `0` flag only for the alignments that pad *before* the value (`>`, `=`), where leading
+ zeros are never significant. Note the *explicit* `0` fill spelling (`{foo:0<5d}`) does
+ strip, and so returns `-12` for `"-1200"` — long-standing behavior, left alone, and
+ inconsistent with the flag spelling on purpose. `is_one2one()` detects none of this: its
+ generated data fills the field width, so padding is never exercised.
=====================================
CHANGELOG.md
=====================================
@@ -1,3 +1,14 @@
+## Version 1.0.2 (2026/10/13)
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 110](https://github.com/pytroll/trollsift/pull/110) - Fix edge cases in conversions, padding, and percent literals
+
+In this release 1 pull request was closed.
+
+
## Version 1.0.1 (2026/06/01)
### Pull Requests Merged
=====================================
CLAUDE.md
=====================================
@@ -0,0 +1 @@
+ at AGENTS.md
=====================================
MANIFEST.in deleted
=====================================
@@ -1,6 +0,0 @@
-include doc/Makefile
-include doc/source/*
-include LICENSE.txt
-include README.rst
-include versioneer.py
-include trollsift/version.py
=====================================
debian/changelog
=====================================
@@ -1,3 +1,9 @@
+trollsift (1.0.2-1) unstable; urgency=medium
+
+ * New upstream release.
+
+ -- Antonio Valentino <antonio.valentino at tiscali.it> Sat, 17 Oct 2026 08:26:15 +0000
+
trollsift (1.0.1-1) unstable; urgency=medium
[ Bas Couwenberg ]
=====================================
doc/rtd_environment.yaml
=====================================
@@ -10,4 +10,4 @@ dependencies:
- sphinxcontrib-apidoc
- pip:
- graphviz
- - .. # relative path to the satpy project
+ - .. # relative path to the trollsift project
=====================================
doc/source/conf.py
=====================================
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
"""Build configuration file for trollsift's documentation."""
+import importlib.metadata
import sys
import os
@@ -44,10 +45,10 @@ copyright = "2014, Panu Lahtinen, Hrobjartur Thorsteinsson"
# |version| and |release|, also used in various other places throughout the
# built documents.
#
-# The short X.Y version.
-version = "0.1"
# The full version, including alpha/beta/rc tags.
-release = "0.1.0"
+release = importlib.metadata.version("trollsift")
+# The short X.Y version.
+version = ".".join(release.split(".")[:2])
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
=====================================
doc/source/usage.rst
=====================================
@@ -82,3 +82,47 @@ depending on your requirements you can call,
'/somedir/otherdir/hrpt_noaa16_20120101_0101_69022.l1b'
And achieve the exact same result as in the Parse object example above.
+
+gotchas
+-------
+
+padding
+^^^^^^^
+
+Fields can be padded with the usual Python format spec, and trollsift strips that
+padding back off when parsing:
+
+ >>> from trollsift import parse, compose
+ >>> fmt = "{platform:_<8s}_{orbit:05d}"
+ >>> compose(fmt, {'platform': 'noaa16', 'orbit': 69022})
+ 'noaa16___69022'
+ >>> parse(fmt, "noaa16___69022")
+ {'platform': 'noaa16', 'orbit': 69022}
+
+Padding a number with zeros on the *right* is the one case that can't be undone,
+because those zeros can't be told apart from the value's own. Both ``12`` and
+``12000`` are composed as ``"12000"`` here, so such a pattern is not invertible
+and parsing gives back the longer number:
+
+ >>> compose("{orbit:<05d}", {'orbit': 12})
+ '12000'
+ >>> parse("{orbit:<05d}", "12000")
+ {'orbit': 12000}
+
+The usual zero padding, on the left, has no such problem:
+
+ >>> compose("{orbit:05d}", {'orbit': 12})
+ '00012'
+ >>> parse("{orbit:05d}", "00012")
+ {'orbit': 12}
+
+literal percent signs
+^^^^^^^^^^^^^^^^^^^^^
+
+Inside a time field a ``%%`` means a literal percent sign, as it does in
+:meth:`~datetime.datetime.strftime`, rather than the start of a directive:
+
+ >>> compose("{time:%Y%m%d%%}", {'time': datetime(2014, 2, 10)})
+ '20140210%'
+ >>> parse("{time:%Y%m%d%%}", "20140210%")
+ {'time': datetime.datetime(2014, 2, 10, 0, 0)}
=====================================
pyproject.toml
=====================================
@@ -43,10 +43,10 @@ omit = ["trollsift/version.py"]
[tool.pytest.ini_options]
minversion = 6.0
-addopts = ["-ra", "--showlocals", "--strict-markers", "--strict-config"]
+addopts = ["-ra", "--showlocals", "--strict-markers", "--strict-config", "--doctest-modules"]
xfail_strict = true
log_cli_level = "info"
-testpaths = ["trollsift/tests"]
+testpaths = ["trollsift"]
filterwarnings = [
"error",
"ignore:numpy.ndarray size changed, may indicate binary incompatibility:RuntimeWarning",
@@ -57,7 +57,7 @@ line-length = 120
[tool.ruff.lint]
# See https://docs.astral.sh/ruff/rules/
-select = ["E", "W", "B", "D", "T10", "C90"]
+select = ["E", "W", "B", "D", "T10", "C90", "S"]
ignore = ["D101", "D102", "D103", "D104", "D105", "D106", "D107", "E203"]
[tool.ruff.lint.per-file-ignores]
=====================================
trollsift/parser.py
=====================================
@@ -12,7 +12,7 @@ import typing
if typing.TYPE_CHECKING:
from _typeshed import StrOrLiteralStr
from typing import Any
- from collections.abc import Iterable, Sequence, Mapping
+ from collections.abc import Callable, Iterable, Sequence, Mapping
class Parser:
@@ -92,10 +92,11 @@ class StringFormatter(string.Formatter):
hyphens. The added conversions are listed below and can be used in a
format string by prefixing them with an `!` like so:
+ >>> from trollsift.parser import StringFormatter
>>> fstr = "{!u}_{!l}"
>>> formatter = StringFormatter()
>>> formatter.format(fstr, "to_upper", "To_LowerCase")
- "TO_UPPER_to_lowercase"
+ 'TO_UPPER_to_lowercase'
- c: Make capitalized version of string (first character upper case, all lowercase after that) by executing the
parameter's `.capitalize()` method.
@@ -161,6 +162,10 @@ spec_regexes["X"] = spec_regexes["x"]
spec_regexes[""] = spec_regexes["s"]
allow_multiple = ["b", "c", "d", "o", "s", "", "x", "X"]
fixed_point_types = ["f", "e", "E", "g"]
+# integer types and the base they are written in
+int_bases = {"d": 10, "x": 16, "X": 16, "o": 8, "b": 2}
+# alignments that put the padding before the value
+leading_pad_aligns = (">", "=")
# format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]
# https://docs.python.org/3.4/library/string.html#format-specification-mini-language
fmt_spec_regex = re.compile(
@@ -188,16 +193,34 @@ def _get_fixed_point_regex(width: str | None, precision: str | None) -> str:
return spec_regexes["f"]
+def _apply_between_percent_escapes(format_spec: str, func: Callable[[str], str], join: str = "%") -> str:
+ """Apply ``func`` to each chunk of a datetime format spec between ``%%`` escapes.
+
+ A ``%%`` is a literal '%', so the chunks around it have to be rewritten one at a
+ time. Rewriting the whole spec at once lets a directive replacement consume a
+ character belonging to the literal: the ``%m`` in ``"%Y%%m"`` is not a month.
+
+ Args:
+ format_spec: Datetime format spec to rewrite (ex. ``"%Y%%m"``)
+ func: Called with each chunk of the spec between ``%%`` escapes
+ join: Put between the rewritten chunks in place of each ``%%``. Defaults to
+ a literal '%', for when the result is a finished regular expression or
+ glob pattern. Pass ``"%%"`` if the result is still a format spec.
+ """
+ return join.join(func(chunk) for chunk in format_spec.split("%%"))
+
+
class RegexFormatter(string.Formatter):
"""String formatter that converts a format string to a regular expression.
+ >>> from trollsift.parser import RegexFormatter, extract_values
>>> regex_formatter = RegexFormatter()
>>> regex_str = regex_formatter.format('{field_one:5d}_{field_two}')
Can also be used to extract values from a string given the format spec
for that string:
- >>> regex_formatter.extract_values('{field_one:5d}_{field_two}', '12345_sometext')
+ >>> extract_values('{field_one:5d}_{field_two}', '12345_sometext')
{'field_one': '12345', 'field_two': 'sometext'}
Note that the regular expressions generated by this class are specially
@@ -206,7 +229,7 @@ class RegexFormatter(string.Formatter):
the provided string, this class will prefer the shorter version of the
string in order to make the rest of the pattern match. For example:
- >>> regex_formatter.extract_values('{field_one}_{field_two}', 'abc_def_ghi')
+ >>> extract_values('{field_one}_{field_two}', 'abc_def_ghi')
{'field_one': 'abc', 'field_two': 'def_ghi'}
Note how `field_one` could have matched "abc_def", but the lower
@@ -270,17 +293,7 @@ class RegexFormatter(string.Formatter):
return key, self.UNPROVIDED_VALUE
def _regex_datetime(self, format_spec: str) -> str:
- replace_str = format_spec
- for fmt_key, fmt_val in DT_FMT.items():
- if fmt_key == "%%":
- # special case
- replace_str.replace("%%", "%")
- continue
- count = fmt_val.count("?")
- # either a series of numbers or letters/numbers
- regex = r"\d{{{:d}}}".format(count) if count else r"[^ \t\n\r\f\v\-_:]+"
- replace_str = replace_str.replace(fmt_key, regex)
- return replace_str
+ return _apply_between_percent_escapes(format_spec, _regex_datetime_directives)
def regex_field(self, field_name: str, value: Any, format_spec: str) -> str:
if value != self.UNPROVIDED_VALUE:
@@ -344,7 +357,6 @@ def format_spec_to_regex(field_name: str, format_spec: str) -> str:
def _get_fill(fill: str | None, width: str | None, ftype: str | None) -> str | None:
- # NOTE: does not properly handle `=` alignment
if fill is None:
if width is not None and width[0] == "0":
fill = "0"
@@ -400,48 +412,70 @@ def _convert(convdef: str, stri: str) -> Any:
"""Convert the string *stri* to the given conversion definition *convdef*."""
result: Any # force mypy type
if "%" in convdef:
- result = dt.datetime.strptime(stri, convdef)
- else:
- result = _strip_padding(convdef, stri)
- if "d" in convdef:
- result = int(result)
- elif "x" in convdef or "X" in convdef:
- result = int(result, 16)
- elif "o" in convdef:
- result = int(result, 8)
- elif "b" in convdef:
- result = int(result, 2)
- elif any(float_type_marker in convdef for float_type_marker in fixed_point_types):
- result = float(result)
-
+ return dt.datetime.strptime(stri, convdef)
+ # NOTE: check the parsed type, not the whole spec, so a fill character
+ # that happens to be a type letter (ex. "{foo:d>5s}") isn't mistaken for one
+ spec_match = fmt_spec_regex.match(convdef)
+ match_dict = spec_match.groupdict() if spec_match else {}
+ result = _strip_padding(match_dict, stri)
+ ftype = match_dict.get("type")
+ if ftype in int_bases:
+ # a field that was all padding held a zero
+ result = int(result or "0", int_bases[ftype])
+ elif ftype in fixed_point_types:
+ result = float(result)
return result
-def _strip_padding(convdef: str, stri: str) -> str:
+def _strip_padding(match_dict: Mapping[str, str | None], stri: str) -> str:
"""Strip padding from the given string.
Args:
- convdef: Conversion definition (indicates the padding)
+ match_dict: ``fmt_spec_regex`` groups of the conversion definition
+ (indicates the padding)
stri: String to be modified
"""
- regex_match = fmt_spec_regex.match(convdef)
- match_dict = regex_match.groupdict() if regex_match else {}
align = match_dict.get("align")
- pad = match_dict.get("fill")
if align:
# align character is the last one
align = align[-1]
- if align and align in "<>^" and not pad:
- pad = " "
+ pad = _get_padding_char(match_dict, align)
if align == ">":
stri = stri.lstrip(pad)
elif align == "<":
stri = stri.rstrip(pad)
elif align == "^":
stri = stri.strip(pad)
+ elif align == "=":
+ stri = _strip_sign_padding(stri, pad)
return stri
+def _get_padding_char(match_dict: Mapping[str, str | None], align: str | None) -> str | None:
+ """Get the character a value was padded with, or None if it wasn't padded.
+
+ A '0' *flag* (``{foo:>04d}``, as opposed to a '0' *fill* like ``{foo:0>4d}``)
+ pads numbers with zeros. It is only honored for the alignments that pad before
+ the value, and only for numbers: a trailing zero is significant, so stripping
+ it can't be undone (``{foo:<05d}`` formats both -12 and -1200 as "-1200").
+ """
+ pad = match_dict.get("fill")
+ if pad is None and align:
+ zero_padded = (
+ match_dict.get("zero") and align in leading_pad_aligns and match_dict.get("type") not in ("s", "", None)
+ )
+ pad = "0" if zero_padded else " "
+ return pad
+
+
+def _strip_sign_padding(stri: str, pad: str | None) -> str:
+ """Strip '='-aligned padding, which sits between the sign and the value."""
+ if stri[:1] in ("+", "-"):
+ # NOTE: a slice, as the value may be an empty string (ex. "{foo:*=d}")
+ return stri[0] + stri[1:].lstrip(pad)
+ return stri.lstrip(pad)
+
+
@lru_cache()
def get_convert_dict(fmt: str) -> dict[str, str]:
"""Retrieve parse definition from the format string `fmt`."""
@@ -516,10 +550,36 @@ DT_FMT = {
"%c": "*",
"%x": "*",
"%X": "*",
- "%%": "?",
+ # a literal '%'; the loops above never see it, as specs are split on it first
+ "%%": "%",
}
+def _regex_datetime_directives(format_spec: str) -> str:
+ """Replace strftime directives with regular expressions matching them."""
+ for fmt_key, fmt_val in DT_FMT.items():
+ count = fmt_val.count("?")
+ # either a series of numbers or letters/numbers
+ regex = r"\d{{{:d}}}".format(count) if count else r"[^ \t\n\r\f\v\-_:]+"
+ format_spec = format_spec.replace(fmt_key, regex)
+ return format_spec
+
+
+def _glob_datetime_directives(format_spec: str) -> str:
+ """Replace strftime directives with glob patterns matching them."""
+ for fmt_key, fmt_val in DT_FMT.items():
+ format_spec = format_spec.replace(fmt_key, fmt_val)
+ return format_spec
+
+
+def _substitute_datetime_directives(format_spec: str, value: dt.datetime, dt_fmt: str) -> str:
+ """Replace only the directives named in ``dt_fmt`` (ex. "Ymd") with ``value``'s fields."""
+ for fmt_letter in dt_fmt:
+ fmt = "%" + fmt_letter
+ format_spec = format_spec.replace(fmt, value.strftime(fmt))
+ return format_spec
+
+
class GlobifyFormatter(string.Formatter):
# special string to mark a parameter not being specified
UNPROVIDED_VALUE = "<trollsift unprovided value>"
@@ -539,18 +599,18 @@ class GlobifyFormatter(string.Formatter):
# specified with a tuple/list of 2 elements
# (value, partial format string)
value, dt_fmt = value
- for fmt_letter in dt_fmt:
- fmt = "%" + fmt_letter
- format_spec = format_spec.replace(fmt, value.strftime(fmt))
+ # NOTE: the result is still a format spec, so the escapes are kept
+ format_spec = _apply_between_percent_escapes(
+ format_spec,
+ lambda chunk: _substitute_datetime_directives(chunk, value, dt_fmt),
+ join="%%",
+ )
# Replace format spec with glob patterns (*, ?, etc)
if not format_spec:
return "*"
if "%" in format_spec:
- replace_str = format_spec
- for fmt_key, fmt_val in DT_FMT.items():
- replace_str = replace_str.replace(fmt_key, fmt_val)
- return replace_str
+ return _apply_between_percent_escapes(format_spec, _glob_datetime_directives)
if not re.search("[0-9]+", format_spec):
# non-integer type
return "*"
@@ -624,7 +684,7 @@ def _gen_data_for_spec(format_spec: str | None) -> int | str | dt.datetime:
else:
# clearly bad
raise ValueError(f"Bad format specification: {format_spec!r}")
- return random.randint(0, 99999999999999999) % (10**n)
+ return random.randint(0, 99999999999999999) % (10**n) # noqa: S311
# string type
if format_spec is None:
@@ -635,7 +695,7 @@ def _gen_data_for_spec(format_spec: str | None) -> int | str | dt.datetime:
n = 4
randstri = ""
for _ in range(n):
- randstri += random.choice(string.ascii_letters)
+ randstri += random.choice(string.ascii_letters) # noqa: S311
return randstri
=====================================
trollsift/tests/unittests/test_parser.py
=====================================
@@ -570,3 +570,137 @@ class TestParserFixedPoint:
)
def test_parse_integers(fmt, string, expected):
assert parse(fmt, string)["foo"] == expected
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "value", "string"),
+ [
+ ("{foo:d>5s}", "ab", "dddab"),
+ ("{foo:f>4s}", "ab", "ffab"),
+ ("{foo:b>4s}", "ab", "bbab"),
+ ("{foo:o>4s}", "ab", "ooab"),
+ ("{foo:e>6s}", "ab", "eeeeab"),
+ ("{foo:x<3s}", "ab", "abx"),
+ ("{foo:g<4s}", "ab", "abgg"),
+ ],
+)
+def test_parse_fill_character_that_looks_like_a_type(fmt, value, string):
+ """Check that a fill character that is also a type letter isn't converted as that type."""
+ assert compose(fmt, {"foo": value}) == string
+ assert parse(fmt, string) == {"foo": value}
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "string", "expected"),
+ [
+ ("{foo:f}", "1.5", 1.5),
+ ("{foo:e}", "1.5e+05", 150000.0),
+ ("{foo:E}", "1.5E+05", 150000.0),
+ ("{foo:g}", "1.5", 1.5),
+ ("{foo:5.2f}", "123.45", 123.45),
+ ],
+)
+def test_parse_floats(fmt, string, expected):
+ """Check that fixed point types are converted to floats."""
+ result = parse(fmt, string)["foo"]
+ assert isinstance(result, float)
+ assert result == expected
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "value", "string"),
+ [
+ # padding sits between the sign and the value with "=" alignment
+ ("{foo:*=5d}", 12, "***12"),
+ ("{foo:*=5d}", -12, "-**12"),
+ ("{foo:=5d}", 12, " 12"),
+ ("{foo:=5d}", -12, "- 12"),
+ ("{foo:0=5d}", -12, "-0012"),
+ ("{foo:+=5d}", 12, "+++12"),
+ ("{foo:*=5x}", 255, "***ff"),
+ ("{foo:*=8.2f}", 3.5, "****3.50"),
+ # a "0" flag pads with zeros, and ">" puts the sign after them
+ ("{foo:>04d}", -12, "0-12"),
+ ("{foo:>04d}", 0, "0000"),
+ ("{foo:>05d}", 12, "00012"),
+ ("{foo:>08.2f}", 0.5, "00000.50"),
+ ],
+)
+def test_parse_leading_padding(fmt, value, string):
+ """Check that padding placed before the value is stripped when parsing."""
+ assert compose(fmt, {"foo": value}) == string
+ assert parse(fmt, string) == {"foo": value}
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "string", "expected"),
+ [
+ ("{foo:<05d}", "-1200", -1200),
+ ("{foo:^05d}", "01200", 1200),
+ # an explicit "0" fill has always been stripped, even though it is
+ # just as ambiguous as the "0" flag above
+ ("{foo:0<5d}", "-1200", -12),
+ ],
+)
+def test_parse_trailing_zero_padding_is_ambiguous(fmt, string, expected):
+ """Check the parsing of trailing zeros, which can't be told from the value's own.
+
+ Both -12 and -1200 are formatted as "-1200" by ``{foo:<05d}``, so there is no
+ correct answer here. These cases only pin down what trollsift does today.
+ """
+ assert parse(fmt, string) == {"foo": expected}
+
+
+ at pytest.mark.parametrize(
+ ("string", "expected"),
+ [
+ ("ab", 0),
+ ("a12b", 12),
+ ("a-12b", -12),
+ ("a+12b", 12),
+ ],
+)
+def test_parse_padded_field_without_a_width(string, expected):
+ """Check that a padded field with no width, which may match nothing, is parsed.
+
+ A spec with an alignment but no width places no limit on how much it matches,
+ so the value can come back as an empty string and must not be indexed into.
+ """
+ assert parse("a{foo:*=d}b", string) == {"foo": expected}
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "string"),
+ [
+ ("{t:%Y%%m%d}", "2024%m04"),
+ ("{t:%Y%%m}", "2024%m"),
+ ("{t:%Y%%}", "2024%"),
+ ("{t:%%%Y}", "%2024"),
+ ("{t:%Y%%%%%m}", "2024%%03"),
+ ("{t:%d%%%m%%%Y}", "04%03%2024"),
+ ],
+)
+def test_parse_literal_percent_in_datetime(fmt, string):
+ """Check that a "%%" in a datetime spec is a literal percent, not a directive."""
+ spec = fmt[3:-1]
+ assert compose(fmt, {"t": dt.datetime(2024, 3, 4)}) == string
+ assert parse(fmt, string) == {"t": dt.datetime.strptime(string, spec)}
+
+
+ at pytest.mark.parametrize(
+ ("fmt", "keyvals", "expected"),
+ [
+ ("{t:%Y%%m%d}", None, "????%m??"),
+ ("{t:%Y%%}", None, "????%"),
+ ("{t:%%%Y}", None, "%????"),
+ ("{t:%Y%%%%%m}", None, "????%%??"),
+ # only the directives named in the tuple are filled in, and "m" names
+ # none here -- the "m" in the spec follows a literal percent
+ ("{t:%Y%%m%d}", {"t": (dt.datetime(2024, 3, 4), "m")}, "????%m??"),
+ ("{t:%Y%%m%d}", {"t": (dt.datetime(2024, 3, 4), "Y")}, "2024%m??"),
+ ("{t:%Y%%m%d}", {"t": dt.datetime(2024, 3, 4)}, "2024%m04"),
+ ],
+)
+def test_globify_literal_percent_in_datetime(fmt, keyvals, expected):
+ """Check that a "%%" in a datetime spec globs as a literal percent."""
+ assert globify(fmt, keyvals) == expected
View it on GitLab: https://salsa.debian.org/debian-gis-team/trollsift/-/compare/db5fa504727fe21a15f5c1399416d6956198daf3...fff04cf57e2ef9fdfe8ff8d0edac646645d83eeb
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/trollsift/-/compare/db5fa504727fe21a15f5c1399416d6956198daf3...fff04cf57e2ef9fdfe8ff8d0edac646645d83eeb
You're receiving this email because of your account on salsa.debian.org. Manage all notifications: https://salsa.debian.org/-/profile/notifications | Help: https://salsa.debian.org/help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20261017/32a2d80d/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list