[Git][debian-gis-team/pygeofilter][upstream] New upstream version 0.4.0
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Mon Jun 22 23:22:30 BST 2026
Antonio Valentino pushed to branch upstream at Debian GIS Project / pygeofilter
Commits:
12b92761 by Antonio Valentino at 2026-06-22T22:09:30+00:00
New upstream version 0.4.0
- - - - -
18 changed files:
- .github/workflows/main.yml
- − .github/workflows/release-please.yml
- + .github/workflows/release.yml
- CHANGELOG.md
- README.md
- pygeofilter/ast.py
- pygeofilter/backends/solr/evaluate.py
- pygeofilter/backends/sqlalchemy/evaluate.py
- pygeofilter/backends/sqlalchemy/filters.py
- pygeofilter/parsers/cql2_json/parser.py
- pygeofilter/parsers/cql2_text/grammar.lark
- pygeofilter/parsers/fes/base.py
- pygeofilter/parsers/iso8601.py
- pygeofilter/version.py
- tests/backends/solr/test_evaluate.py
- tests/backends/sqlalchemy/test_evaluate.py
- tests/parsers/cql2_json/test_parser.py
- tests/parsers/cql2_text/test_parser.py
Changes:
=====================================
.github/workflows/main.yml
=====================================
@@ -32,7 +32,7 @@ jobs:
- name: Install and run Elasticsearch 📦
uses: getong/elasticsearch-action at v1.2
with:
- elasticsearch version: '8.2.2'
+ elasticsearch version: '8.17.0'
host port: 9200
container port: 9200
host node port: 9300
@@ -41,7 +41,7 @@ jobs:
- name: Install and run Solr 📦
uses: OSGeo/solr-action at main
with:
- solr_version: 9.8.1
+ solr_version: 9.10
host_port: 8983
container_port: 8983
- name: Install and run OpenSearch 📦
=====================================
.github/workflows/release-please.yml deleted
=====================================
@@ -1,13 +0,0 @@
-on:
- push:
- branches:
- - main
-name: release-please
-jobs:
- release-please:
- runs-on: ubuntu-latest
- steps:
- - uses: googleapis/release-please-action at v4
- with:
- token: ${{ secrets.PAT_WORKFLOW }}
- release-type: python
=====================================
.github/workflows/release.yml
=====================================
@@ -0,0 +1,84 @@
+name: release
+
+on:
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "New version (e.g. 0.4.0)"
+ required: true
+
+permissions:
+ contents: write
+
+jobs:
+ release:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout at v4
+ with:
+ fetch-depth: 0
+ token: ${{ secrets.PAT_WORKFLOW }}
+
+ - name: Validate version format
+ run: |
+ if ! echo "${{ inputs.version }}" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
+ echo "::error::Version must be in X.Y.Z format"
+ exit 1
+ fi
+ if git tag -l "v${{ inputs.version }}" | grep -q .; then
+ echo "::error::Tag v${{ inputs.version }} already exists"
+ exit 1
+ fi
+
+ - name: Bump version.py
+ run: |
+ echo '__version__ = "${{ inputs.version }}"' > pygeofilter/version.py
+
+ - name: Update CHANGELOG.md
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ LAST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "")
+ if [ -n "$LAST_TAG" ]; then
+ NOTES=$(gh api repos/${{ github.repository }}/releases/generate-notes \
+ -f tag_name="v${{ inputs.version }}" \
+ -f previous_tag_name="$LAST_TAG" \
+ -f target_commitish="main" \
+ --jq .body)
+ else
+ NOTES=$(gh api repos/${{ github.repository }}/releases/generate-notes \
+ -f tag_name="v${{ inputs.version }}" \
+ -f target_commitish="main" \
+ --jq .body)
+ fi
+
+ HEADER="# Changelog"
+ DATE=$(date +%Y-%m-%d)
+ SECTION="## [${{ inputs.version }}](https://github.com/${{ github.repository }}/compare/${LAST_TAG}...v${{ inputs.version }}) ($DATE)"
+
+ {
+ echo "$HEADER"
+ echo
+ echo "$SECTION"
+ echo
+ echo "$NOTES"
+ echo
+ # append everything after the "# Changelog" header from the existing file
+ tail -n +2 CHANGELOG.md
+ } > CHANGELOG.new.md
+ mv CHANGELOG.new.md CHANGELOG.md
+
+ - name: Commit, tag, push
+ run: |
+ git config user.name "github-actions[bot]"
+ git config user.email "github-actions[bot]@users.noreply.github.com"
+ git add pygeofilter/version.py CHANGELOG.md
+ git commit -m "Release ${{ inputs.version }}"
+ git tag "v${{ inputs.version }}"
+ git push origin HEAD --tags
+
+ - name: Create GitHub Release
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ gh release create "v${{ inputs.version }}" --generate-notes
=====================================
CHANGELOG.md
=====================================
@@ -1,5 +1,25 @@
# Changelog
+## [0.4.0](https://github.com/geopython/pygeofilter/compare/v0.3.3...v0.4.0) (2026-06-08)
+
+## What's Changed
+* Added quotes to equality queries to avoid solr that solr fail if the … by @magnarem in https://github.com/geopython/pygeofilter/pull/136
+* Added escaping of In operator values to avoid solr error if the value… by @magnarem in https://github.com/geopython/pygeofilter/pull/137
+* update Elasticsearch version in CI by @tomkralidis in https://github.com/geopython/pygeofilter/pull/138
+* Add support for CQL2's CASEI (fixes #135) by @mikemahoney218-usgs in https://github.com/geopython/pygeofilter/pull/139
+* Fixes 05 2026 -Better spatial filter handling, some bugfixes by @magnarem in https://github.com/geopython/pygeofilter/pull/140
+* Fix in_(): quote string values to handle colons in Solr identifiers by @epifanio in https://github.com/geopython/pygeofilter/pull/141
+* Update CASEI implementation to support LIKE by @mikemahoney218-usgs in https://github.com/geopython/pygeofilter/pull/142
+* Allow NOT to take predicate in parens in CQL2 by @mikemahoney218-usgs in https://github.com/geopython/pygeofilter/pull/145
+* Add support for DATE in CQL2 Text by @mikemahoney218-usgs in https://github.com/geopython/pygeofilter/pull/148
+
+## New Contributors
+* @mikemahoney218-usgs made their first contribution in https://github.com/geopython/pygeofilter/pull/139
+* @epifanio made their first contribution in https://github.com/geopython/pygeofilter/pull/141
+
+**Full Changelog**: https://github.com/geopython/pygeofilter/compare/v0.3.3...v0.4.0
+
+
## [0.3.3](https://github.com/geopython/pygeofilter/compare/v0.3.2...v0.3.3) (2025-12-20)
=====================================
README.md
=====================================
@@ -204,6 +204,10 @@ Parsing fes query into AST
Equal(lhs=ATTRIBUTE title, rhs='birds')
```
+## Releasing
+
+Releases are done via the [release workflow](https://github.com/geopython/pygeofilter/actions/workflows/release.yml). A maintainer triggers it from the Actions tab, provides the new version number (e.g. `0.4.0`), and the workflow takes care of bumping `version.py`, updating `CHANGELOG.md` (auto-generated from merged PRs), tagging, creating a GitHub Release, and publishing to PyPI.
+
## Testing
For testing, several requirements must be satisfied. These can be installed, via pip:
=====================================
pygeofilter/ast.py
=====================================
@@ -32,7 +32,7 @@ from typing import ClassVar, List, Optional, Union
from . import values
AstType = Union["Node", values.ValueType, list]
-ScalarAstType = Union["Node", int, float]
+ScalarAstType = Union["Node", int, float, str]
SpatialAstType = Union["Node", values.SpatialValueType]
TemporalAstType = Union["Node", values.TemporalValueType]
ArrayAstType = Union["Node", List[AstType]]
@@ -225,7 +225,7 @@ class Like(Predicate):
"""Node class to represent a wildcard sting matching predicate."""
lhs: Node
- pattern: str
+ pattern: ScalarAstType
nocase: bool
wildcard: str
singlechar: str
=====================================
pygeofilter/backends/solr/evaluate.py
=====================================
@@ -32,12 +32,15 @@ Uses native Python to return dict of JSON request payload
"""
# pylint: disable=E1130,C0103,W0223
+
from datetime import date, datetime
-from typing import Optional
+from typing import Optional, Union
import shapely.wkt
+from dateutil import parser
from packaging.version import Version
from pygeoif import shape
+from pytz import UTC
from ... import ast, values
from ..evaluator import Evaluator, handle
@@ -45,9 +48,72 @@ from .util import like_to_wildcard
VERSION_9_8_1 = Version("9.8.1")
+
+def _split_query_and_filters(part):
+ """Return (query, filters) for a SolrDSLQuery-like part."""
+ if isinstance(part, SolrDSLQuery):
+ return part.get("query", "*:*"), list(part.get("filter", []))
+ return part, []
+
+
+def _invert_filter_query(filter_query):
+ """Invert a Solr filter expression.
+
+ For string filters, toggles a leading '-' prefix. For non-string filters
+ (e.g., bool dicts), wraps the filter in a bool.must_not structure.
+ """
+ if isinstance(filter_query, str):
+ return filter_query[1:] if filter_query.startswith("-") else f"-{filter_query}"
+ if isinstance(filter_query, dict) and "bool" in filter_query and "must_not" in filter_query["bool"]:
+ return {"bool": {"must": filter_query["bool"]["must_not"]}}
+ return {"bool": {"must_not": [filter_query]}}
+
+def _to_solr_date(value):
+ """Convert input date/datetime to Solr UTC datetime string: YYYY-MM-DDTHH:MM:SSZ.
+
+ Returns None for empty input.
+ Raises ValueError for unparseable input.
+ """
+ if value is None:
+ return None
+
+ if isinstance(value, int):
+ return value
+
+ if isinstance(value, float):
+ return value
+
+ if isinstance(value, datetime):
+ dt = value
+ elif isinstance(value, str):
+ text = value.strip()
+ if not text:
+ return None
+ try:
+ # dateutil handles many formats:
+ # 2024-11-04, 2024-11-04T10:00:00Z, 2024-11-04 10:00:00+01:00, etc.
+ dt = parser.isoparse(text)
+ except (ValueError, TypeError, OverflowError):
+ try:
+ dt = parser.parse(text)
+ except (ValueError, TypeError, OverflowError):
+ # Not a date-like string: keep term value as-is.
+ return value
+ else:
+ return value
+
+ # If no timezone is provided, assume UTC (adjust if your source is local time).
+ if dt.tzinfo is None:
+ dt = dt.replace(tzinfo=UTC)
+
+ dt_utc = dt.astimezone(UTC)
+
+ # Solr wants Zulu time with second precision.
+ return dt_utc.strftime("%Y-%m-%dT%H:%M:%SZ")
+
COMPARISON_OP_MAP = {
- ast.ComparisonOp.EQ: "{lhs}:{rhs}",
- ast.ComparisonOp.NE: "-{lhs}:{rhs}",
+ ast.ComparisonOp.EQ: "{lhs}:\"{rhs}\"",
+ ast.ComparisonOp.NE: "-{lhs}:\"{rhs}\"",
ast.ComparisonOp.GT: "{lhs}:{{{rhs} TO *]",
ast.ComparisonOp.GE: "{lhs}:[{rhs} TO *]",
ast.ComparisonOp.LT: "{lhs}:[* TO {rhs}}}",
@@ -63,11 +129,11 @@ ARITHMETIC_OP_MAP = {
class SolrDSLQuery(dict):
- def __init__(self, query="*:*", filters=None):
+ def __init__(self, query: Union[dict, str] = "*:*", filters=None):
"""
Initialize a Solr JSON DSL query object.
- :param query: The main query (default is '*:*').
+ :param query: The main query (default is "*:*").
:param filters: Optional filters to apply.
"""
super().__init__()
@@ -115,78 +181,91 @@ class SOLRDSLEvaluator(Evaluator):
@handle(ast.And)
def and_(self, _, lhs, rhs):
- """Joins two filter objects with an `and` operator."""
- # Extract the inner queries if lhs or rhs are SolrDSLQuery objects
- lhs = unwrap_query(lhs)
- rhs = unwrap_query(rhs)
-
- # Merge `must` and `must_not` properly
- combined_query = {"bool": {"must": []}}
-
- if "bool" in lhs and "must_not" in lhs["bool"]:
- # If `lhs` has a must_not clause, merge it
- combined_query["bool"]["must_not"] = lhs["bool"]["must_not"]
- lhs_must = {key: value for key, value in lhs["bool"].items() if key != "must_not"}
- if lhs_must:
- combined_query["bool"]["must"].append()
- else:
- # If `lhs` has no must_not clause, append it to must
- combined_query["bool"]["must"].append(lhs)
+ """Joins two filter objects with an `and` operator.
- if "bool" in rhs and "must_not" in rhs["bool"]:
- # If `rhs` has a must_not clause, merge it
- if "must_not" in combined_query["bool"]:
- combined_query["bool"]["must_not"].extend(rhs["bool"]["must_not"])
+ Spatial {!field ...} filters live in the SolrDSLQuery 'filter' key and must
+ not be merged into bool.must (they don't work correctly in the query position
+ for Geo3D fields). Non-spatial queries are combined in bool.must as before.
+ """
+ lhs_q = lhs.get("query", "*:*") if isinstance(lhs, SolrDSLQuery) else lhs
+ rhs_q = rhs.get("query", "*:*") if isinstance(rhs, SolrDSLQuery) else rhs
+ lhs_filters = list(lhs.get("filter", [])) if isinstance(lhs, SolrDSLQuery) else []
+ rhs_filters = list(rhs.get("filter", [])) if isinstance(rhs, SolrDSLQuery) else []
+ combined_filters = lhs_filters + rhs_filters
+
+ # Build must list from non-trivial (non-wildcard) query parts
+ must_parts = []
+ must_not_parts = []
+ for q in [lhs_q, rhs_q]:
+ if q == "*:*":
+ continue
+ if isinstance(q, dict) and "bool" in q:
+ if "must_not" in q["bool"]:
+ must_not_parts.extend(q["bool"]["must_not"])
+ if "must" in q["bool"]:
+ must_parts.extend(q["bool"]["must"])
else:
- combined_query["bool"]["must_not"] = rhs["bool"]["must_not"]
- rhs_must = {key: value for key, value in rhs["bool"].items() if key != "must_not"}
- if rhs_must:
- combined_query["bool"]["must"].append()
+ must_parts.append(q)
+
+ if not must_parts and not must_not_parts:
+ combined_q = "*:*"
+ elif not must_parts and must_not_parts:
+ combined_q = {"bool": {"must_not": must_not_parts}}
else:
- # If `rhs` has no must_not clause, append it to must
- combined_query["bool"]["must"].append(rhs)
+ combined_q = {"bool": {"must": must_parts}}
+ if must_not_parts:
+ combined_q["bool"]["must_not"] = must_not_parts
- return SolrDSLQuery(combined_query)
+ result = SolrDSLQuery(combined_q)
+ if combined_filters:
+ result["filter"] = combined_filters
+ return result
@handle(ast.Or)
def or_(self, _, lhs, rhs):
- # Extract the inner queries if lhs or rhs are SolrDSLQuery objects
- lhs = unwrap_query(lhs)
- rhs = unwrap_query(rhs)
-
- # Merge `must` and `must_not` properly
- combined_query = {"bool": {"should": []}}
-
- if "bool" in lhs and "must_not" in lhs["bool"]:
- # If `lhs` has a must_not clause, merge it
- combined_query["bool"]["must_not"] = lhs["bool"]["must_not"]
- lhs_must = {key: value for key, value in lhs["bool"].items() if key != "must_not"}
- if lhs_must:
- combined_query["bool"]["should"].append()
- else:
- # If `lhs` has no must_not clause, append it to must
- combined_query["bool"]["should"].append(lhs)
-
- if "bool" in rhs and "must_not" in rhs["bool"]:
- # If `rhs` has a must_not clause, merge it
- if "must_not" in combined_query["bool"]:
- combined_query["bool"]["must_not"].extend(rhs["bool"]["must_not"])
- else:
- combined_query["bool"]["must_not"] = rhs["bool"]["must_not"]
- rhs_must = {key: value for key, value in rhs["bool"].items() if key != "must_not"}
- if rhs_must:
- combined_query["bool"]["should"].append()
- else:
- # If `rhs` has no must_not clause, append it to must
- combined_query["bool"]["should"].append(rhs)
-
- return SolrDSLQuery(combined_query)
+ def to_or_clause(query_part, filter_parts):
+ def normalize_clause(clause):
+ if isinstance(clause, str) and clause.startswith("-"):
+ return {"bool": {"must": ["*:*"], "must_not": [clause[1:]]}}
+ if isinstance(clause, dict) and "bool" in clause:
+ bool_part = clause["bool"]
+ if "must_not" in bool_part and "must" not in bool_part and "should" not in bool_part:
+ normalized = dict(clause)
+ normalized["bool"] = dict(bool_part)
+ normalized["bool"]["must"] = ["*:*"]
+ return normalized
+ return clause
+
+ clauses = []
+ if query_part != "*:*":
+ clauses.append(normalize_clause(query_part))
+ clauses.extend(normalize_clause(clause) for clause in filter_parts)
+ if not clauses:
+ return "*:*"
+ if len(clauses) == 1:
+ return clauses[0]
+ return {"bool": {"must": clauses}}
+
+ lhs_q, lhs_filters = _split_query_and_filters(lhs)
+ rhs_q, rhs_filters = _split_query_and_filters(rhs)
+
+ lhs_clause = to_or_clause(lhs_q, lhs_filters)
+ rhs_clause = to_or_clause(rhs_q, rhs_filters)
+
+ # OR with a match-all branch is a no-op.
+ if lhs_clause == "*:*" or rhs_clause == "*:*":
+ return SolrDSLQuery("*:*")
+
+ # Keep OR in filter[] so spatial predicates remain in filter context.
+ or_filter = {"bool": {"should": [lhs_clause, rhs_clause]}}
+ return SolrDSLQuery("*:*", filters=[or_filter])
@handle(ast.LessThan, ast.LessEqual, ast.GreaterThan, ast.GreaterEqual)
def comparison(self, node, lhs, rhs):
"""
Creates a range query for comparison operators.
"""
+ rhs = _to_solr_date(rhs)
return SolrDSLQuery(f"{COMPARISON_OP_MAP[node.op]}".format(lhs=lhs, rhs=rhs))
@handle(ast.Between)
@@ -194,6 +273,8 @@ class SOLRDSLEvaluator(Evaluator):
"""
Creates a range query for between conditions.
"""
+ low = _to_solr_date(low)
+ high = _to_solr_date(high)
range_query = f"{lhs}:[{low} TO {high}]"
if node.not_:
# Negate the range query for NOT Between
@@ -205,7 +286,11 @@ class SOLRDSLEvaluator(Evaluator):
"""
Creates a terms query for `IN` conditions.
"""
- options_str = " OR ".join(str(option) for option in options)
+ def _quote(v):
+ if isinstance(v, str):
+ return '"' + v.replace('"', '\\"') + '"'
+ return str(v)
+ options_str = " OR ".join(_quote(option) for option in options)
terms_query = f"{lhs}:({options_str})"
if node.not_:
# Negate the terms query for NOT IN
@@ -250,16 +335,29 @@ class SOLRDSLEvaluator(Evaluator):
@handle(ast.Not)
def not_(self, _, sub):
"""Inverts a filter object."""
- # Extract the inner query if sub is a SolrDSLQuery
- sub_query = sub["query"] if isinstance(sub, SolrDSLQuery) else sub
+ if isinstance(sub, SolrDSLQuery):
+ sub_query, sub_filters = _split_query_and_filters(sub)
+
+ result = SolrDSLQuery("*:*")
+ if sub_filters:
+ result["filter"] = [_invert_filter_query(fq) for fq in sub_filters]
+
+ # A wildcard query contributes no restriction and does not need
+ # query-level negation.
+ if sub_query == "*:*":
+ return result
+
+ if isinstance(sub_query, dict) and "bool" in sub_query and "must_not" in sub_query["bool"]:
+ result["query"] = {"bool": {"must": sub_query["bool"]["must_not"]}}
+ return result
- # Handle the case where the sub-query is already a "must_not"
- if isinstance(sub_query, dict) and "bool" in sub_query and "must_not" in sub_query["bool"]:
- # If the sub-query is already a must_not, remove the negation
- return SolrDSLQuery({"bool": {"must": sub_query["bool"]["must_not"]}})
+ result["query"] = {"bool": {"must_not": [sub_query]}}
+ return result
- # Otherwise, create a new must_not clause
- return SolrDSLQuery({"bool": {"must_not": [sub_query]}})
+ # Non-SolrDSLQuery fallback.
+ if isinstance(sub, dict) and "bool" in sub and "must_not" in sub["bool"]:
+ return SolrDSLQuery({"bool": {"must": sub["bool"]["must_not"]}})
+ return SolrDSLQuery({"bool": {"must_not": [sub]}})
@handle(ast.Like)
def like(self, node: ast.Like, lhs):
@@ -293,10 +391,25 @@ class SOLRDSLEvaluator(Evaluator):
@handle(values.Geometry)
def geometry(self, node: values.Geometry):
"""Geometry values are converted to a Solr spatial query."""
- """Convert to wkt and make sure polygons are counter clockwise"""
geom_wkt = shape(node).wkt
geom = shapely.wkt.loads(geom_wkt)
- if geom.geom_type == "Polygon" or geom.geom_type == "MultiPolygon":
+ if geom.geom_type == "Polygon" or geom.geom_type =="MultiPolygon":
+ # Rectangular polygons (from BBox) must use ENVELOPE format for Geo3D.
+ # WKT polygons with coordinates at ±180/±90 are "coplanar" in 3D space
+ # (poles and antimeridian are degenerate points), causing Solr to reject
+ # or mishandle them. ENVELOPE is safe and correct for axis-aligned boxes.
+ coords = list(geom.exterior.coords)
+ if (
+ len(coords) == 5
+ and len({c[0] for c in coords[:-1]}) == 2
+ and len({c[1] for c in coords[:-1]}) == 2
+ ):
+ minx, miny, maxx, maxy = geom.bounds
+ # Global bbox covers the whole Earth — spatial predicate is a no-op.
+ if minx <= -180 and miny <= -90 and maxx >= 180 and maxy >= 90:
+ return None
+ if (minx <= -179.9999 and maxx >= 179.9999) or (miny <= -89.9999 and maxy >= 89.9999):
+ return f"ENVELOPE({minx}, {maxx}, {maxy}, {miny})"
geom = geom.reverse() if not geom.exterior.is_ccw else geom
return geom.wkt
@@ -305,6 +418,10 @@ class SOLRDSLEvaluator(Evaluator):
"""
Creates a term query for equality or inequality conditions.
"""
+ rhs = _to_solr_date(rhs)
+ if isinstance(rhs, str):
+ escaped_rhs = rhs.replace('"', '\\"')
+ rhs = f'"{escaped_rhs}"'
if node.op == ast.ComparisonOp.EQ:
# Use a term query for equality
return SolrDSLQuery(f"{lhs}:{rhs}")
@@ -355,19 +472,20 @@ class SOLRDSLEvaluator(Evaluator):
@handle(ast.GeometryIntersects, ast.GeometryDisjoint, ast.GeometryWithin, ast.GeometryContains, ast.GeometryEquals)
def spatial_comparison(self, node: ast.SpatialComparisonPredicate, lhs: str, rhs):
- """Creates a spatial query for the given spatial comparison
- predicate.
+ """Creates a spatial query for the given spatial comparison predicate.
+
+ Spatial {!field ...} queries MUST go into the Solr filter[] array, not the
+ main query field. When placed in query, Geo3D fields return wrong counts.
"""
- query = {}
- # Solr need capitalized first letter of operator
+ # rhs is None when geometry() detected a global bbox — no filter needed.
+ if rhs is None:
+ return SolrDSLQuery("*:*")
op = node.op.value.lower().capitalize()
+ geo_filter = f"{{!field f={lhs} v='Intersects({rhs})'}}"
if op == "Disjoint":
- geo_filter = f"{{!field f={lhs} v='Intersects({rhs})'}}"
- query = {"bool": {"must_not": [geo_filter]}}
- return SolrDSLQuery(query)
-
- query = f"{{!field f={lhs} v='{op}({rhs})'}}"
- return SolrDSLQuery(query)
+ return SolrDSLQuery("*:*", filters=[f"-{geo_filter}"])
+ geo_filter = f"{{!field f={lhs} v='{op}({rhs})'}}"
+ return SolrDSLQuery("*:*", filters=[geo_filter])
@handle(ast.BBox)
def bbox(self, node: ast.BBox, lhs):
@@ -376,7 +494,7 @@ class SOLRDSLEvaluator(Evaluator):
"""
bbox = self.envelope(values.Envelope(node.minx, node.maxx, node.miny, node.maxy))
query = f"{{!field f={lhs} v='Intersects({bbox})'}}"
- return SolrDSLQuery(query)
+ return SolrDSLQuery('*:*', filters=[query])
# @handle(ast.Arithmetic, subclasses=True)
# def arithmetic(self, node: ast.Arithmetic, lhs, rhs):
@@ -390,9 +508,13 @@ class SOLRDSLEvaluator(Evaluator):
@handle(values.Envelope)
def envelope(self, node: values.Envelope):
- """Envelope values are converted to an WKT ENVELOPE for Solr."""
- min_x = float(min(node.x1, node.x2))
- max_x = float(max(node.x1, node.x2))
+ """
+ Envelope values are converted to an WKT ENVELOPE for Solr.
+
+ If min_x > max_x, solr assume dateline crossing.
+ """
+ min_x = float(node.x1)
+ max_x = float(node.x2)
min_y = float(min(node.y1, node.y2))
max_y = float(max(node.y1, node.y2))
return f"ENVELOPE({min_x}, {max_x}, {max_y}, {min_y})"
=====================================
pygeofilter/backends/sqlalchemy/evaluate.py
=====================================
@@ -34,6 +34,8 @@ class SQLAlchemyFilterEvaluator(Evaluator):
@handle(ast.Like)
def like(self, node, lhs):
+ if (isinstance(node.pattern, ast.Function)):
+ node.pattern = filters.function_map[node.pattern.name](*node.pattern.arguments)
return filters.like(
lhs,
node.pattern,
@@ -112,10 +114,9 @@ class SQLAlchemyFilterEvaluator(Evaluator):
def arithmetic(self, node, lhs, rhs):
return filters.runop(lhs, rhs, node.op.value)
- # TODO: map functions
- # @handle(ast.FunctionExpressionNode)
- # def function(self, node, *arguments):
- # return self.function_map[node.name](*arguments)
+ @handle(ast.Function)
+ def function(self, node, *arguments):
+ return filters.function_map[node.name](*arguments)
@handle(*values.LITERALS)
def literal(self, node):
=====================================
pygeofilter/backends/sqlalchemy/filters.py
=====================================
@@ -27,6 +27,10 @@ def parse_geometry(geom: dict):
wkt = shape(geom).wkt
return func.ST_GeomFromEWKT(f"SRID={srid};{wkt}")
+# TODO: map functions
+function_map = {
+ "lower": func.lower
+}
# ------------------------------------------------------------------------------
# Filters
=====================================
pygeofilter/parsers/cql2_json/parser.py
=====================================
@@ -157,6 +157,9 @@ def walk_cql_json(node: JsonType): # noqa: C901
cast(List[ast.AstType], walk_cql_json(args[1])),
not_=False,
)
+
+ elif op == "casei":
+ return ast.Function("lower", [cast(ast.Node, walk_cql_json(args[0]))])
elif op in BINARY_OP_PREDICATES_MAP:
args = [cast(ast.Node, walk_cql_json(arg)) for arg in args]
=====================================
pygeofilter/parsers/cql2_text/grammar.lark
=====================================
@@ -33,9 +33,9 @@
?condition_1: predicate
| "NOT"i predicate -> not_
+ | "NOT"i "(" predicate ")" -> not_
| "(" condition ")"
-
?predicate: expression "=" expression -> eq
| expression "eq"i expression -> eq
| expression "<>" expression -> ne
@@ -51,8 +51,8 @@
| expression "gte"i expression -> gte
| expression "BETWEEN"i expression "AND"i expression -> between
| expression "NOT"i "BETWEEN"i expression "AND"i expression -> not_between
- | expression "LIKE"i SINGLE_QUOTED -> like
- | expression "NOT"i "LIKE"i SINGLE_QUOTED -> not_like
+ | expression "LIKE"i expression -> like
+ | expression "NOT"i "LIKE"i expression -> not_like
| expression "IN"i "(" expression ( "," expression )* ")" -> in_
| expression "NOT"i "IN"i "(" expression ( "," expression )* ")" -> not_in
| expression "IS"i "NULL"i -> null
@@ -113,9 +113,9 @@
func.2: attribute "(" expression ("," expression)* ")" -> function
-
?literal: timestamp
| interval
+ | date
| number
| BOOLEAN
| SINGLE_QUOTED
@@ -134,11 +134,11 @@ BOOLEAN.2: ( "TRUE"i | "FALSE"i)
DOUBLE_QUOTED: "\"" /.*?/ "\""
SINGLE_QUOTED: "'" /.*?/ "'"
+DATE: /[0-9]{4}-?[0-1][0-9]-?[0-3][0-9]/
DATETIME: /[0-9]{4}-?[0-1][0-9]-?[0-3][0-9][T ][0-2][0-9]:?[0-5][0-9]:?[0-5][0-9](\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})?/
?timestamp: "TIMESTAMP" "(" "'" DATETIME "'" ")"
?interval: "INTERVAL" "(" "'" DATETIME "'" "," "'" DATETIME "'" ")"
-
-
+?date: "DATE" "(" "'" DATE "'" ")"
attribute: /[a-zA-Z][a-zA-Z_:0-9.]+/
| DOUBLE_QUOTED
=====================================
pygeofilter/parsers/fes/base.py
=====================================
@@ -24,12 +24,18 @@ class FESBaseParser(XMLParser):
return predicate
@handle("And")
- def and_(self, node: Element, lhs, rhs):
- return ast.And(lhs, rhs)
+ def and_(self, node: Element, *args):
+ result = args[0]
+ for operand in args[1:]:
+ result = ast.And(result, operand)
+ return result
@handle("Or")
- def or_(self, node: Element, lhs, rhs):
- return ast.Or(lhs, rhs)
+ def or_(self, node: Element, *args):
+ result = args[0]
+ for operand in args[1:]:
+ result = ast.Or(result, operand)
+ return result
@handle("Not")
def not_(self, node: Element, lhs):
=====================================
pygeofilter/parsers/iso8601.py
=====================================
@@ -27,7 +27,7 @@
from lark import Transformer, v_args
-from ..util import parse_datetime, parse_duration
+from ..util import parse_datetime, parse_duration, parse_date
@v_args(meta=False, inline=True)
@@ -37,3 +37,6 @@ class ISO8601Transformer(Transformer):
def DURATION(self, duration):
return parse_duration(duration)
+
+ def DATE(self, date):
+ return parse_date(date)
=====================================
pygeofilter/version.py
=====================================
@@ -1 +1 @@
-__version__ = "0.3.3"
+__version__ = "0.4.0"
=====================================
tests/backends/solr/test_evaluate.py
=====================================
@@ -38,7 +38,8 @@ from pygeofilter.backends.solr.evaluate import SOLRDSLEvaluator, SolrDSLQuery
from pygeofilter.parsers.ecql import parse
from pygeofilter.util import parse_datetime
-SOLR_BASE_URL = "http://localhost:8983/solr/test" # replace with your Solr URL
+PORT=8983
+SOLR_BASE_URL = f"http://localhost:{PORT}/solr/test" # replace with your Solr URL
HEADERS = {
"Content-type": "application/json",
}
@@ -48,6 +49,7 @@ INPUT_DOCS = [
{
"id": "A",
"geometry_jts": "MULTIPOLYGON(((0 0, 0 5, 5 5,5 0,0 0)))",
+ "geometry_geo3d": "MULTIPOLYGON(((0 0, 0 5, 5 5,5 0,0 0)))",
"center": "POINT(2.5 2.5)",
"float_attribute": 0.0,
"int_attribute": 5,
@@ -58,6 +60,7 @@ INPUT_DOCS = [
{
"id": "B",
"geospatial_jts": "MULTIPOLYGON(((5 5, 5 10, 10 10,10 5,5 5)))",
+ "geospatial_geo3d": "MULTIPOLYGON(((5 5, 5 10, 10 10,10 5,5 5)))",
"center": "POINT(7.5 7.5)",
"float_attribute": 30.0,
"str_attribute": "this is another test",
@@ -71,19 +74,26 @@ INPUT_DOCS = [
@pytest.fixture(autouse=True, scope="session")
def prepare():
"""Prepare the Solr instance. Add the fields needed for testing"""
- # print('Preparing core')
- # Create a new core
- # res = requests.get('http://localhost:8983/solr/admin/cores?action=CREATE&name=test&configSet= /opt/solr/server/solr/configsets/_default/conf')
- # print(res)
# Add the field types
field_types = [
{
"name": "spatial_jts",
"class": "solr.SpatialRecursivePrefixTreeFieldType",
+ "geo": "true",
"autoIndex": "true",
"spatialContextFactory": "JTS",
"validationRule": "repairBuffer0",
- "distErrPct": "0.025",
+ "distErrPct": "0.015",
+ "maxDistErr": "0.001",
+ "distanceUnits": "kilometers",
+ },
+ {
+ "name": "spatial_geo3d",
+ "class": "solr.SpatialRecursivePrefixTreeFieldType",
+ "prefixTree": "s2",
+ "spatialContextFactory": "Geo3D",
+ "planetModel": "WGS84",
+ "distErrPct": "0.015",
"maxDistErr": "0.001",
"distanceUnits": "kilometers",
},
@@ -93,7 +103,7 @@ def prepare():
for field_type in field_types:
data = json.dumps({"add-field-type": field_type})
requests.post(
- "http://localhost:8983/api/cores/test/schema", headers={"Content-type": "application/json"}, data=data
+ f"http://localhost:{PORT}/api/cores/test/schema", headers={"Content-type": "application/json"}, data=data
)
# Define the fields to be added
@@ -105,6 +115,7 @@ def prepare():
{"name": "str_attribute", "type": "text_general"},
{"name": "center", "type": "location"},
{"name": "geometry_jts", "type": "spatial_jts", "multiValued": "false"},
+ {"name": "geometry_geo3d", "type": "spatial_geo3d", "multiValued": "false"},
{"name": "daterange_attribute", "type": "date_range"},
]
@@ -112,7 +123,9 @@ def prepare():
for field in fields:
data = json.dumps({"add-field": field})
requests.post(
- "http://localhost:8983/api/cores/test/schema", headers={"Content-type": "application/json"}, data=data
+ f"http://localhost:{PORT}/api/cores/test/schema",
+ headers={"Content-type": "application/json"},
+ data=data,
)
index = "ok"
yield index
@@ -238,13 +251,13 @@ def test_combination_like_not(data):
assert len(result) == 0
result = filter_(parse("str_attribute LIKE 'test' OR NOT str_attribute LIKE 'another'"))
- assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+ assert len(result) == 2
result = filter_(parse("str_attribute LIKE 'test' OR NOT str_attribute LIKE 'another'"))
- assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+ assert len(result) == 2
result = filter_(parse("NOT str_attribute LIKE 'another' OR str_attribute LIKE 'test'"))
- assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+ assert len(result) == 2
def test_in(data):
@@ -344,6 +357,45 @@ def test_attribute_mapping_fallback():
assert result2 == "subject_s"
+def test_spatial_disjoint_uses_negated_intersects_filter():
+ query = to_filter(parse("DISJOINT(geometry_jts, POLYGON((0 0, 1 0, 1 1, 0 1, 0 0)))"))
+ assert query["query"] == "*:*"
+ assert query["filter"] == ["-{!field f=geometry_jts v='Intersects(POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0)))'}"]
+
+
+def test_not_disjoint_inverts_filter_to_intersects():
+ query = to_filter(parse("NOT DISJOINT(geometry_jts, POLYGON((0 0, 1 0, 1 1, 0 1, 0 0)))"))
+ assert query["query"] == "*:*"
+ assert query["filter"] == ["{!field f=geometry_jts v='Intersects(POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0)))'}"]
+
+
+def test_equality_keeps_non_date_string_terms():
+ query = to_filter(parse("str_attribute = 'Space Borne Instrument'"))
+ assert query["query"] == 'str_attribute:"Space Borne Instrument"'
+
+
+def test_equality_normalizes_date_literals():
+ query = to_filter(parse("datetime_attribute = '2000-01-01'"))
+ assert query["query"] == 'datetime_attribute:"2000-01-01T00:00:00Z"'
+
+
+def test_or_keeps_spatial_in_filter_context():
+ query = to_filter(
+ parse("INTERSECTS(geometry_jts, POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))) OR str_attribute LIKE 'this is a test'")
+ )
+ assert query["query"] == "*:*"
+ assert query["filter"] == [
+ {
+ "bool": {
+ "should": [
+ "{!field f=geometry_jts v='Intersects(POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0)))'}",
+ 'str_attribute:"this is a test"',
+ ]
+ }
+ }
+ ]
+
+
# def test_array():
# result = filter_(
# ast.ArrayEquals(
@@ -406,6 +458,52 @@ def test_spatial(data):
result = filter_(parse("NOT DISJOINT(geometry_jts, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"))
assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+ result = filter_(parse("NOT INTERSECTS(geometry_jts, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"))
+ assert len(result) == 1 and result[0]["id"] is data[1]["id"]
+
+ result = filter_(
+ parse("DISJOINT(geometry_jts, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0))) OR str_attribute LIKE 'this is a test'")
+ )
+ assert len(result) == 2
+
+
+def test_spatial_geo3d(data):
+ result = filter_(parse("INTERSECTS(geometry_geo3d, ENVELOPE (0.0 1.0 0.0 1.0))"))
+ assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+
+ result = filter_(
+ parse(
+ "INTERSECTS(geometry_geo3d, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"
+ )
+ )
+ assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+
+ result = filter_(
+ parse("BBOX(center, 2, 2, 3, 3)"),
+ )
+ assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+
+ result = filter_(
+ parse(
+ "DISJOINT(geometry_geo3d, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"
+ )
+ )
+ assert len(result) == 1 and result[0]["id"] is data[1]["id"]
+
+ result = filter_(
+ parse(
+ "NOT DISJOINT(geometry_geo3d, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"
+ )
+ )
+ assert len(result) == 1 and result[0]["id"] is data[0]["id"]
+
+ result = filter_(
+ parse(
+ "NOT INTERSECTS(geometry_geo3d, POLYGON((0.0 0.0, 1.0 0.0, 1.0 1.0, 0.0 1.0, 0.0 0.0)))"
+ )
+ )
+ assert len(result) == 1 and result[0]["id"] is data[1]["id"]
+
# def test_arithmetic():
# result = filter_(
=====================================
tests/backends/sqlalchemy/test_evaluate.py
=====================================
@@ -18,7 +18,9 @@ from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.sql import func, select
from pygeofilter.backends.sqlalchemy.evaluate import to_filter
-from pygeofilter.parsers.ecql import parse
+from pygeofilter.parsers.ecql import parse as parse_ecql
+from pygeofilter.parsers.cql2_text import parse as parse_cql_text
+from pygeofilter.parsers.cql2_json import parse as parse_cql2_json
Base = declarative_base()
@@ -161,8 +163,8 @@ def db_session(setup_database, connection):
transaction.rollback()
-def evaluate(session, cql_expr, expected_ids, filter_option=None):
- ast = parse(cql_expr)
+def evaluate(session, cql_expr, expected_ids, filter_option=None, parser=parse_ecql):
+ ast = parser(cql_expr)
filters = to_filter(ast, FIELD_MAPPING, filter_option)
q = session.query(Record).join(RecordMeta).filter(filters)
@@ -272,6 +274,11 @@ def test_not_like_endswith(db_session):
def test_not_ilike_endswith(db_session):
evaluate(db_session, "strMetaAttribute NOT ILIKE '%b'", ("A",))
+def test_not_eq(db_session):
+ evaluate(db_session, "NOT(strAttribute = 'AAA')", ("B", ), None, parse_cql_text)
+
+def test_not_gt(db_session):
+ evaluate(db_session, "NOT(floatAttribute > 1)", ("A", ), None, parse_cql_text)
# (NOT) IN
@@ -294,9 +301,27 @@ def test_string_null(db_session):
def test_string_not_null(db_session):
evaluate(db_session, "intAttribute IS NOT NULL", ("A",))
+# CASEI
+def test_casei_equals(db_session):
+ evaluate(db_session, "CASEI(strAttribute) = CASEI('aaa')", ("A",), None, parse_cql_text)
+
+def test_casei_like(db_session):
+ evaluate(db_session, "CASEI(strAttribute) LIKE CASEI('aaa')", ("A",), None, parse_cql_text)
+
+def test_casei_notlike(db_session):
+ evaluate(db_session, "CASEI(strAttribute) NOT LIKE CASEI('aaa')", ("B", ), None, parse_cql_text)
+
+def test_casei_in(db_session):
+ evaluate(db_session, "CASEI(strAttribute) IN (CASEI('aaa'), CASEI('bbb'))", ("A", "B", ), None, parse_cql_text)
+
+def test_casei_json_like(db_session):
+ evaluate(db_session, '{"op": "like", "args": [ {"op": "casei", "args": [{"property": "strAttribute"}]}, {"op": "casei", "args": ["AAA"]} ] }', ("A", ), None, parse_cql2_json)
# temporal predicates
+def test_date_gte(db_session):
+ evaluate(db_session, "datetimeAttribute >= DATE('2000-01-01')", ("A", "B",), None, parse_cql_text)
+
def test_before(db_session):
evaluate(db_session, "datetimeAttribute BEFORE 2000-01-01T00:00:01Z", ("A",))
=====================================
tests/parsers/cql2_json/test_parser.py
=====================================
@@ -148,6 +148,25 @@ def test_attribute_is_null():
result = parse({"op": "isNull", "args": [{"property": "attr"}]})
assert result == ast.IsNull(ast.Attribute("attr"), False)
+def test_attribute_casei():
+ result = parse('{"op": "casei", "args": [{"property": "attr"}]}')
+ assert result == ast.Function("lower", [ast.Attribute("attr")])
+
+def test_literal_casei():
+ result = parse('{"op": "casei", "args": ["literal"]}')
+ assert result == ast.Function("lower", ["literal"])
+
+def test_like_casei():
+ result = parse('{"op": "like", "args": [ {"op": "casei", "args": [{"property": "stringattr"}]}, {"op": "casei", "args": ["AAA"]} ] }')
+ assert result == ast.Like(
+ ast.Function("lower", [ast.Attribute("stringattr")]),
+ ast.Function("lower", ["AAA"]),
+ nocase=False,
+ not_=False,
+ wildcard="%",
+ singlechar=".",
+ escapechar="\\",
+ )
def test_attribute_before():
result = parse(
=====================================
tests/parsers/cql2_text/test_parser.py
=====================================
@@ -1,4 +1,4 @@
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, date
from dateparser.timezone_parser import StaticTzInfo
@@ -201,6 +201,12 @@ def test_attribute_before():
datetime(2000, 1, 1, 0, 0, 1, tzinfo=StaticTzInfo("Z", timedelta(0))),
)
+def test_attribute_lt_date():
+ result = parse("attr < DATE('2000-01-01')")
+ assert result == ast.LessThan(
+ ast.Attribute("attr"),
+ date(2000, 1, 1),
+ )
def test_attribute_t_intersects():
# Using INTERVAL function with properly quoted timestamps
@@ -409,11 +415,52 @@ def test_nested_and_or():
assert result.rhs.rhs == ast.Equal(ast.Attribute("attr_d"), 4)
-def test_casei_function():
- result = parse("CASEI(provider) = 'coolsat'")
+def test_casei_equals():
+ result = parse("CASEI(provider) = CASEI('coolsat')")
# Assuming CASEI maps to 'lower' in the implementation
assert isinstance(result, ast.Equal)
assert isinstance(result.lhs, ast.Function)
+ assert isinstance(result.rhs, ast.Function)
+ assert result.lhs.name == "lower"
+ assert result.lhs.arguments == [ast.Attribute("provider")]
+ assert result.rhs.name == "lower"
+ assert result.rhs.arguments == ["coolsat"]
+
+
+def test_casei_like():
+ result = parse("CASEI(provider) LIKE CASEI('coolsat')")
+ # Assuming CASEI maps to 'lower' in the implementation
+ assert isinstance(result, ast.Like)
+ assert isinstance(result.lhs, ast.Function)
+ assert result.lhs.name == "lower"
+ assert result.lhs.arguments == [ast.Attribute("provider")]
+ assert result.pattern.name == "lower"
+ assert result.pattern.arguments == ["coolsat"]
+
+def test_casei_notlike():
+ result = parse("CASEI(provider) NOT LIKE CASEI('coolsat')")
+ # Assuming CASEI maps to 'lower' in the implementation
+ assert isinstance(result, ast.Like)
+ assert isinstance(result.lhs, ast.Function)
assert result.lhs.name == "lower"
assert result.lhs.arguments == [ast.Attribute("provider")]
- assert result.rhs == "coolsat"
+ assert result.pattern.name == "lower"
+ assert result.pattern.arguments == ["coolsat"]
+
+def test_not_gt():
+ result = parse("NOT(attr > 2)")
+ assert result == ast.Not(
+ ast.GreaterThan(ast.Attribute("attr"), 2)
+ )
+
+def test_not_lt():
+ result = parse("NOT(attr < 2)")
+ assert result == ast.Not(
+ ast.LessThan(ast.Attribute("attr"), 2)
+ )
+
+def test_not_eq():
+ result = parse("NOT(attr = 2)")
+ assert result == ast.Not(
+ ast.Equal(ast.Attribute("attr"), 2)
+ )
View it on GitLab: https://salsa.debian.org/debian-gis-team/pygeofilter/-/commit/12b92761e39f0d35e17f6a8e7c5dfe37ad7a687f
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pygeofilter/-/commit/12b92761e39f0d35e17f6a8e7c5dfe37ad7a687f
You're receiving this email because of your account on salsa.debian.org. Manage all notifications: https://salsa.debian.org/-/profile/notifications | Help: https://salsa.debian.org/help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20260622/d601713d/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list