[Git][debian-gis-team/eodag][master] 5 commits: New upstream version 4.0.2+ds
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Thu Mar 26 06:16:37 GMT 2026
Antonio Valentino pushed to branch master at Debian GIS Project / eodag
Commits:
11db4283 by Antonio Valentino at 2026-03-26T06:03:53+00:00
New upstream version 4.0.2+ds
- - - - -
27d0bbcb by Antonio Valentino at 2026-03-26T06:04:00+00:00
Update upstream source from tag 'upstream/4.0.2+ds'
Update to upstream version '4.0.2+ds'
with Debian dir 2b8dc73efd6e5cb423d872c9a6ae28ece8f0da35
- - - - -
61276891 by Antonio Valentino at 2026-03-26T06:10:04+00:00
New upstream release
- - - - -
2af8f282 by Antonio Valentino at 2026-03-26T06:10:10+00:00
Refresh patches
- - - - -
5008250e by Antonio Valentino at 2026-03-26T06:11:31+00:00
Set diatribution to unstable
- - - - -
21 changed files:
- CHANGES.rst
- debian/changelog
- debian/patches/0001-System-importlib.metadata.patch
- − debian/patches/0003-Fix-package-discoverywq.patch
- debian/patches/series
- docs/getting_started_guide/features_overview.ipynb
- docs/notebooks/tutos/tuto_meteoblue.ipynb
- docs/notebooks/tutos/tuto_wekeo.ipynb
- eodag/crunch.py
- eodag/plugins/base.py
- eodag/plugins/crunch/__init__.py
- eodag/plugins/crunch/base.py
- eodag/plugins/crunch/filter_latest_intersect.py
- eodag/plugins/crunch/filter_latest_tpl_name.py
- eodag/plugins/crunch/filter_overlap.py
- eodag/plugins/search/build_search_result.py
- eodag/plugins/search/cop_marine.py
- eodag/resources/providers.yml
- pyproject.toml
- tests/test_config.py
- + tests/units/test_crunch.py
Changes:
=====================================
CHANGES.rst
=====================================
@@ -3,6 +3,56 @@ Release history
===============
+v4.0.2 (2026-03-23)
+===================
+
+Bug Fixes
+---------
+
+* **crunch**: Minor fixes and improved crunches test coverage (`#2001`_, `c638016`_)
+
+* **plugins**: Order of values of EcmwfSearch bbox is N-W-S-E (`#2056`_, `b56678c`_)
+
+* **plugins**: Replace deprecated items_per_page in CopMarineSearch (`#2100`_, `b2df124`_)
+
+* **providers**: Queried orbit number parsing (`#2104`_, `bb03961`_)
+
+Build System
+------------
+
+* Fixed deprecated project.license expression (`#2105`_, `b595765`_)
+
+* Updated package discovery pattern (`#2103`_, `64b22c7`_)
+
+Documentation
+-------------
+
+* Tutorials typos fixes (`#1996`_, `5026718`_)
+
+Performance Improvements
+------------------------
+
+* **plugins**: Stop fetching when search page limit is reached on CopMarineSearch (`#2014`_,
+ `a6eaa0a`_)
+
+.. _#1996: https://github.com/CS-SI/eodag/pull/1996
+.. _#2001: https://github.com/CS-SI/eodag/pull/2001
+.. _#2014: https://github.com/CS-SI/eodag/pull/2014
+.. _#2056: https://github.com/CS-SI/eodag/pull/2056
+.. _#2100: https://github.com/CS-SI/eodag/pull/2100
+.. _#2103: https://github.com/CS-SI/eodag/pull/2103
+.. _#2104: https://github.com/CS-SI/eodag/pull/2104
+.. _#2105: https://github.com/CS-SI/eodag/pull/2105
+.. _5026718: https://github.com/CS-SI/eodag/commit/5026718b7d812e04cb630fe28726bb124babbf37
+.. _64b22c7: https://github.com/CS-SI/eodag/commit/64b22c70a2cf0c37ecf6ad681daaa40734b4b815
+.. _a6eaa0a: https://github.com/CS-SI/eodag/commit/a6eaa0ac30a38934b22453e9c0eae990cedabe56
+.. _b2df124: https://github.com/CS-SI/eodag/commit/b2df12454fe565e64a3b270a0490c09f394a9e59
+.. _b56678c: https://github.com/CS-SI/eodag/commit/b56678cf67f581a6ae095fcbbdc227215cc51ce2
+.. _b595765: https://github.com/CS-SI/eodag/commit/b595765b397772e7010bc550c4f0f4d516e5c4d0
+.. _bb03961: https://github.com/CS-SI/eodag/commit/bb03961083f3815a0787cb98882e3dd2a63e9daa
+.. _c638016: https://github.com/CS-SI/eodag/commit/c6380163b38e18f8181b9f521ee38524ec8c34ff
+
+
v4.0.1 (2026-03-18)
===================
=====================================
debian/changelog
=====================================
@@ -1,3 +1,12 @@
+eodag (4.0.2+ds-1) unstable; urgency=medium
+
+ * New usptream release.
+ * debian/patches:
+ - Drop 0003-Fix-package-discoverywq.patch, applied upstream.
+ - Refresh remaining patches.
+
+ -- Antonio Valentino <antonio.valentino at tiscali.it> Thu, 26 Mar 2026 06:10:59 +0000
+
eodag (4.0.1+ds-1) unstable; urgency=medium
* New upstream release.
=====================================
debian/patches/0001-System-importlib.metadata.patch
=====================================
@@ -33,10 +33,10 @@ index 0acf4ed..fecee64 100644
):
try:
diff --git a/pyproject.toml b/pyproject.toml
-index 6e6666f..dd560d4 100644
+index a4495bb..01cb109 100644
--- a/pyproject.toml
+++ b/pyproject.toml
-@@ -38,7 +38,6 @@ dependencies = [
+@@ -39,7 +39,6 @@ dependencies = [
"botocore",
"click",
"geojson",
=====================================
debian/patches/0003-Fix-package-discoverywq.patch deleted
=====================================
@@ -1,22 +0,0 @@
-From: Antonio Valentino <antonio.valentino at tiscali.it>
-Date: Sat, 21 Mar 2026 17:35:57 +0000
-Subject: Fix package discovery
-
-Forwarded: https://github.com/CS-SI/eodag/pull/2103
----
- pyproject.toml | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/pyproject.toml b/pyproject.toml
-index dd560d4..a252562 100644
---- a/pyproject.toml
-+++ b/pyproject.toml
-@@ -198,7 +198,7 @@ StacListAssets = "eodag.plugins.search.stac_list_assets:StacListAssets"
- include-package-data = true
-
- [tool.setuptools.packages.find]
--exclude = ["*.tests", "*.tests.*", "tests.*", "tests"]
-+include = ["eodag*"]
-
- [tool.setuptools.package-data]
- "*" = ["LICENSE", "NOTICE", "py.typed"]
=====================================
debian/patches/series
=====================================
@@ -1,3 +1,2 @@
0001-System-importlib.metadata.patch
0002-no-mypy-boto3-s3.patch
-0003-Fix-package-discoverywq.patch
=====================================
docs/getting_started_guide/features_overview.ipynb
=====================================
@@ -30,6 +30,11 @@
"## Configure"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": []
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -56,7 +61,7 @@
"source": [
"Now we will configure `eodag` to be able to download using *PEPS*. For that we need to fill our credentials:\n",
"\n",
- "* in the user configuration file `~/.config/eodag.eodag.yml`:\n",
+ "* in the user configuration file `~/.config/eodag/eodag.yml`:\n",
"\n",
"```yaml\n",
"peps:\n",
=====================================
docs/notebooks/tutos/tuto_meteoblue.ipynb
=====================================
@@ -50,8 +50,9 @@
}
],
"source": [
- "tomorrow = (datetime.date.today() + datetime.timedelta(days=1)).isoformat()\n",
- "after_tomorrow = (datetime.date.today() + datetime.timedelta(days=2)).isoformat()\n",
+ "today = datetime.datetime.now(datetime.timezone.utc).date()\n",
+ "tomorrow = (today + datetime.timedelta(days=1)).isoformat()\n",
+ "after_tomorrow = (today + datetime.timedelta(days=2)).isoformat()\n",
"aoi_bbox = [-2, 42, 3, 45]\n",
"\n",
"products_from_collection = dag.search(\n",
=====================================
docs/notebooks/tutos/tuto_wekeo.ipynb
=====================================
@@ -3845,7 +3845,7 @@
" geom=[0.25, 43.2, 2.8, 43.9],\n",
" provider=\"wekeo_ecmwf\",\n",
" **{\n",
- " 'ecmwf:variable': 'wildfire_flux_of_total_carbon_in_aerosols',\n",
+ " 'ecmwf:variable': ['wildfire_flux_of_total_carbon_in_aerosols'],\n",
" 'ecmwf:data_format': 'grib',\n",
" }\n",
")\n",
=====================================
eodag/crunch.py
=====================================
@@ -17,8 +17,18 @@
# limitations under the License.
"""Crunch filters import gateway"""
-from .plugins.crunch.filter_date import FilterDate # noqa
-from .plugins.crunch.filter_latest_intersect import FilterLatestIntersect # noqa
-from .plugins.crunch.filter_latest_tpl_name import FilterLatestByName # noqa
-from .plugins.crunch.filter_overlap import FilterOverlap # noqa
-from .plugins.crunch.filter_property import FilterProperty # noqa
+from .plugins.crunch import (
+ FilterDate,
+ FilterLatestByName,
+ FilterLatestIntersect,
+ FilterOverlap,
+ FilterProperty,
+)
+
+__all__ = [
+ "FilterDate",
+ "FilterLatestIntersect",
+ "FilterLatestByName",
+ "FilterOverlap",
+ "FilterProperty",
+]
=====================================
eodag/plugins/base.py
=====================================
@@ -66,9 +66,12 @@ class PluginTopic(metaclass=EODAGPluginMount):
def __repr__(self) -> str:
config = getattr(self, "config", None)
+ priority = ""
+ if config is not None and hasattr(config, "priority"):
+ priority = str(config.priority) # is an int
return "{}(provider={}, priority={}, topic={})".format(
self.__class__.__name__,
getattr(self, "provider", ""),
- config.priority if config else "",
+ priority,
self.__class__.mro()[-3].__name__,
)
=====================================
eodag/plugins/crunch/__init__.py
=====================================
@@ -16,3 +16,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""EODAG plugins.crunch package"""
+from .filter_date import Crunch, FilterDate
+from .filter_latest_intersect import FilterLatestIntersect
+from .filter_latest_tpl_name import FilterLatestByName
+from .filter_overlap import FilterOverlap
+from .filter_property import FilterProperty
+
+__all__ = [
+ "Crunch",
+ "FilterDate",
+ "FilterLatestIntersect",
+ "FilterLatestByName",
+ "FilterOverlap",
+ "FilterProperty",
+]
=====================================
eodag/plugins/crunch/base.py
=====================================
@@ -17,6 +17,7 @@
# limitations under the License
from __future__ import annotations
+from abc import abstractmethod
from typing import TYPE_CHECKING, Any, Optional
from eodag.config import PluginConfig
@@ -32,10 +33,11 @@ class Crunch(PluginTopic):
:param config: Crunch configuration
"""
- def __init__(self, config: Optional[dict[str, Any]]) -> None:
+ def __init__(self, config: Optional[dict[str, Any]] = None) -> None:
self.config = PluginConfig()
self.config.__dict__ = config if config is not None else {}
+ @abstractmethod
def proceed(
self, products: list[EOProduct], **search_params: Any
) -> list[EOProduct]:
=====================================
eodag/plugins/crunch/filter_latest_intersect.py
=====================================
@@ -66,7 +66,7 @@ class FilterLatestIntersect(Crunch):
:returns: The filtered products
"""
logger.debug("Start filtering for latest products")
- if not products:
+ if not products or (isinstance(products, list) and len(products) == 0):
return []
# Warning: May crash if start_datetime is not in the appropriate format
products.sort(key=self.sort_product_by_start_date, reverse=True)
=====================================
eodag/plugins/crunch/filter_latest_tpl_name.py
=====================================
@@ -42,8 +42,12 @@ class FilterLatestByName(Crunch):
NAME_PATTERN_CONSTRAINT = re.compile(r"\(\?P<tileid>\\d\{6\}\)")
- def __init__(self, config: dict[str, Any]) -> None:
+ def __init__(self, config: Optional[dict[str, Any]] = None) -> None:
+ if config is None:
+ config = {}
super(FilterLatestByName, self).__init__(config)
+ if "name_pattern" not in config:
+ raise ValidationError('Required parameter "name_pattern" in config')
name_pattern = config.pop("name_pattern")
if not self.NAME_PATTERN_CONSTRAINT.search(name_pattern):
raise ValidationError(
=====================================
eodag/plugins/crunch/filter_overlap.py
=====================================
@@ -63,7 +63,13 @@ class FilterOverlap(Crunch):
"geometry not found in cruncher arguments, filtering disabled."
)
return products
- minimum_overlap = float(self.config.__dict__.get("minimum_overlap", "0"))
+ try:
+ minimum_overlap = max(
+ float(self.config.__dict__.get("minimum_overlap", "0")), 0
+ )
+ except ValueError:
+ minimum_overlap = 0
+
contains = self.config.__dict__.get("contains", False)
intersects = self.config.__dict__.get("intersects", False)
within = self.config.__dict__.get("within", False)
@@ -85,7 +91,7 @@ class FilterOverlap(Crunch):
logger.debug("Minimum overlap is: {} %".format(minimum_overlap))
logger.debug("Initial requested extent area: %s", search_geom.area)
- if search_geom.area == 0:
+ if search_geom.area == 0.0:
logger.debug(
"No product can overlap a requested extent that is not a polygon (i.e with area=0)"
)
=====================================
eodag/plugins/search/build_search_result.py
=====================================
@@ -263,6 +263,12 @@ def _update_properties_from_element(
"minItems": 4,
"additionalItems": False,
"items": [
+ {
+ "type": "number",
+ "maximum": 90,
+ "minimum": -90,
+ "description": "North border of the bounding box",
+ },
{
"type": "number",
"maximum": 180,
@@ -281,12 +287,6 @@ def _update_properties_from_element(
"minimum": -180,
"description": "East border of the bounding box",
},
- {
- "type": "number",
- "maximum": 90,
- "minimum": -90,
- "description": "North border of the bounding box",
- },
],
}
)
=====================================
eodag/plugins/search/cop_marine.py
=====================================
@@ -328,6 +328,8 @@ class CopMarineSearch(StaticStacSearch):
start_index = limit * (token - 1) + 1
num_total = 0
for i, dataset_item in enumerate(datasets_items_list):
+ if len(products) >= limit and not prep.count:
+ break
# Filter by geometry
if "id" not in kwargs and geometry:
dataset_geom = get_geometry_from_various(**dataset_item)
@@ -496,6 +498,9 @@ class CopMarineSearch(StaticStacSearch):
if product:
products.append(product)
current_object = item_key
+ if len(products) >= limit and not prep.count:
+ stop_search = True
+ break
search_params = (
kwargs
=====================================
eodag/resources/providers.yml
=====================================
@@ -715,8 +715,8 @@
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platform#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{gsd}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{providers}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:absolute_orbit}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:relative_orbit}')"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:absolute_orbit})"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:relative_orbit})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{sat:orbit_state#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {eo:cloud_cover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'instrumentShortName' and att/OData.CSC.StringAttribute/Value eq '{instruments#csv_list}')"
@@ -1996,11 +1996,11 @@
license: '$.properties.license'
# OpenSearch Parameters for Product Search (Table 5)
product:acquisition_type: '$.properties.acquisitionType'
- sat:absolute_orbit:
+ sat:relative_orbit:
- 'orbitNumber'
- '$.properties.orbitNumber'
- sat:relative_orbit:
- - absoluteOrbitNumber'
+ sat:absolute_orbit:
+ - 'absoluteOrbitNumber'
- '$.properties.absoluteOrbitNumber'
sat:orbit_state:
- 'orbitDirection={sat:orbit_state#to_title}'
@@ -2358,8 +2358,8 @@
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platform#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{gsd}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{providers}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:absolute_orbit}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:relative_orbit}')"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:absolute_orbit})"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:relative_orbit})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{sat:orbit_state#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {eo:cloud_cover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'instrumentShortName' and att/OData.CSC.StringAttribute/Value eq '{instruments#csv_list}')"
@@ -3973,8 +3973,8 @@
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platform#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{gsd}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{providers}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:absolute_orbit}')"
- - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq '{sat:relative_orbit}')"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:absolute_orbit})"
+ - "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.StringAttribute/Value eq {sat:relative_orbit})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{sat:orbit_state#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {eo:cloud_cover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'instrumentShortName' and att/OData.CSC.StringAttribute/Value eq '{instruments#csv_list}')"
=====================================
pyproject.toml
=====================================
@@ -9,7 +9,8 @@ readme = {file = "README.rst", content-type = "text/x-rst"}
authors = [
{name = "CS GROUP - France", email = "eodag at csgroup.space"}
]
-license = {text = "Apache-2.0"}
+license = "Apache-2.0"
+license-files = ["LICENSE"]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
@@ -199,13 +200,13 @@ StacListAssets = "eodag.plugins.search.stac_list_assets:StacListAssets"
include-package-data = true
[tool.setuptools.packages.find]
-exclude = ["*.tests", "*.tests.*", "tests.*", "tests"]
+include = ["eodag*"]
[tool.setuptools.package-data]
"*" = ["LICENSE", "NOTICE", "py.typed"]
[tool.setuptools_scm]
-fallback_version = "4.0.2.dev0"
+fallback_version = "4.0.3.dev0"
[tool.isort]
multi_line_output = 3
=====================================
tests/test_config.py
=====================================
@@ -23,7 +23,7 @@ from importlib.resources import files as res_files
from io import StringIO
from tempfile import TemporaryDirectory
-import yaml.parser
+import yaml
from eodag.api.provider import ProvidersDict
from eodag.config import PluginConfig
=====================================
tests/units/test_crunch.py
=====================================
@@ -0,0 +1,686 @@
+# -*- coding: utf-8 -*-
+# Copyright 2026, CS GROUP - France, http://www.c-s.fr
+#
+# This file is part of EODAG project
+# https://www.github.com/CS-SI/EODAG
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import unittest
+from collections import UserList
+from typing import Any
+from unittest import mock
+
+import dateutil
+import shapely
+from shapely import Polygon, geometry
+from shapely.errors import ShapelyError
+
+from eodag.api.product import EOProduct
+from eodag.api.search_result import SearchResult
+from eodag.crunch import (
+ FilterDate,
+ FilterLatestByName,
+ FilterLatestIntersect,
+ FilterOverlap,
+ FilterProperty,
+)
+from eodag.plugins.crunch.base import Crunch
+from eodag.utils import DEFAULT_SHAPELY_GEOMETRY
+from eodag.utils.exceptions import ValidationError
+
+
+class TestPluginCrunch(unittest.TestCase):
+
+ __product_id: int = 0
+
+ def __fake_search_result(
+ self, products_properties: list[dict[str:Any]]
+ ) -> SearchResult:
+ """Mock search result"""
+ search_results = []
+ for product_properties in products_properties:
+ TestPluginCrunch.__product_id += 1
+ properties = {
+ "id": "fake_{}".format(TestPluginCrunch.__product_id),
+ "geometry": DEFAULT_SHAPELY_GEOMETRY,
+ }
+ for key in product_properties:
+ properties[key] = product_properties[key]
+
+ search_results.append(
+ EOProduct(
+ provider="fake_provider",
+ properties=properties,
+ collection="fake_collection",
+ )
+ )
+
+ return SearchResult(UserList(search_results))
+
+ def test_crunch_base(self):
+ """Crunch base test"""
+ crunch = Crunch({})
+ products: list[EOProduct] = []
+ try:
+ crunch.proceed(products)
+ self.fail("Abstract class Crunch must not be instantiable")
+ except NotImplementedError:
+ pass
+ except Exception as error:
+ self.fail(
+ "Unexpected error when try to instanciate abstract class Crunch: {error}".format(
+ error=error
+ )
+ )
+
+ def test_crunch_filterdate(self):
+ """Crunch FilterDate test"""
+ # No products to filter
+ search_results: SearchResult = self.__fake_search_result([])
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-15", end="2025-01-15"))
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Some products to filter
+ search_results = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-16", "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-16", "end_datetime": "2025-01-17"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-17"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-18"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-19"},
+ ]
+ )
+ self.assertEqual(len(search_results), 5)
+
+ # Full filter
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-15", end="2025-01-15"))
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-16 00:00:00", end="2025-01-16 23:59:59"))
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-16", end="2025-01-16"))
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-16", end="2025-01-17"))
+ )
+
+ self.assertEqual(len(filtered_result), 3)
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-16", end="2025-01-18"))
+ )
+ self.assertEqual(len(filtered_result), 4)
+
+ # Partial filter
+
+ filtered_result = search_results.crunch(FilterDate(dict(start="2025-01-17")))
+ self.assertEqual(len(filtered_result), 3)
+
+ filtered_result = search_results.crunch(FilterDate(dict(end="2025-01-18")))
+ self.assertEqual(len(filtered_result), 4)
+
+ # No filters
+ filtered_result = search_results.crunch(FilterDate())
+ self.assertEqual(len(filtered_result), 5)
+
+ # Wrong filter (invalid date order)
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-17", end="2025-01-16"))
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Invalid filter
+ try:
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="invalid_date", end="2025-01-18"))
+ )
+ self.fail("Must not let pass invalid date format")
+ except dateutil.parser._parser.ParserError:
+ pass
+
+ try:
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-16", end="wrong_date"))
+ )
+ self.fail("Must not let pass invalid date format")
+ except dateutil.parser._parser.ParserError:
+ pass
+
+ # Uncomplete products
+ search_results = self.__fake_search_result(
+ [
+ {"start_datetime": None, "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-16", "end_datetime": "2025-01-17"},
+ {"start_datetime": "2025-01-17", "end_datetime": None},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-18"},
+ {"start_datetime": None, "end_datetime": None},
+ ]
+ )
+
+ filtered_result = search_results.crunch(
+ FilterDate(dict(start="2025-01-15", end="2025-01-15"))
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ filtered_result = search_results.crunch(FilterDate(dict(start="2025-01-17")))
+ self.assertEqual(len(filtered_result), 4)
+
+ filtered_result = search_results.crunch(FilterDate(dict(end="2025-01-18")))
+ self.assertEqual(len(filtered_result), 5)
+
+ def test_crunch_filterdate_sort(self):
+ """Crunch FilterDate.sort_product_by_start_date test"""
+
+ # sort_product_by_start_date
+ products = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "end_datetime": "2025-01-18"},
+ {"start_datetime": "2025-01-16", "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-17"},
+ ]
+ )
+ products.sort(key=FilterDate.sort_product_by_start_date)
+ self.assertEqual(products[0].properties["start_datetime"], "2025-01-16")
+ self.assertEqual(products[1].properties["start_datetime"], "2025-01-17")
+ self.assertEqual(products[2].properties["start_datetime"], "2025-01-18")
+
+ products.sort(key=FilterDate.sort_product_by_start_date, reverse=True)
+ self.assertEqual(products[0].properties["start_datetime"], "2025-01-18")
+ self.assertEqual(products[1].properties["start_datetime"], "2025-01-17")
+ self.assertEqual(products[2].properties["start_datetime"], "2025-01-16")
+
+ products = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "end_datetime": "2025-01-18"},
+ {"start_datetime": None, "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-17"},
+ ]
+ )
+ products.sort(key=FilterDate.sort_product_by_start_date, reverse=False)
+ self.assertEqual(products[0].properties.get("start_datetime", None), None)
+ self.assertEqual(
+ products[1].properties.get("start_datetime", None), "2025-01-17"
+ )
+ self.assertEqual(
+ products[2].properties.get("start_datetime", None), "2025-01-18"
+ )
+
+ def test_crunch_latest_intersect(self):
+ """Crunch FilterLatestIntersect test"""
+
+ # No products to filter
+ search_results: SearchResult = self.__fake_search_result([])
+ filtered_result = search_results.crunch(FilterLatestIntersect())
+ self.assertEqual(len(filtered_result), 0)
+
+ # Some product filter
+ search_results = self.__fake_search_result(
+ [
+ {"geometry": geometry.box(10, 20, 15, 30)},
+ {"geometry": geometry.box(10, 30, 15, 40)},
+ {"geometry": geometry.box(15, 20, 20, 30)},
+ {"geometry": geometry.box(15, 30, 20, 40)},
+ ]
+ )
+
+ # No filter
+ filtered_result = search_results.crunch(FilterLatestIntersect())
+ self.assertEqual(len(filtered_result), 4)
+
+ # invalid filter geometry, ignore filter
+ filtered_result = search_results.crunch(
+ FilterLatestIntersect(), geometry="hell_wrong_geometry"
+ )
+ self.assertEqual(len(filtered_result), 4)
+
+ # Mismatch filter
+
+ filtered_result = search_results.crunch(
+ FilterLatestIntersect(), geometry=geometry.box(0, 0, 0, 0)
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(
+ FilterLatestIntersect(),
+ geometry={"lonmin": 0, "latmin": 0, "lonmax": 0, "latmax": 0},
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(
+ FilterLatestIntersect(),
+ geometry=Polygon(((0, 0), (0, 1), (1, 1), (1, 0), (0, 0))),
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ def test_crunch_latest_intersect_sort(self):
+ """Crunch FilterLatestIntersect.sort_product_by_start_date test"""
+
+ # sort_product_by_start_date
+ products = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "end_datetime": "2025-01-18"},
+ {"start_datetime": "2025-01-16", "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-17"},
+ ]
+ )
+ products.sort(key=FilterLatestIntersect.sort_product_by_start_date)
+ self.assertEqual(products[0].properties["start_datetime"], "2025-01-16")
+ self.assertEqual(products[1].properties["start_datetime"], "2025-01-17")
+ self.assertEqual(products[2].properties["start_datetime"], "2025-01-18")
+
+ products.sort(
+ key=FilterLatestIntersect.sort_product_by_start_date, reverse=True
+ )
+ self.assertEqual(products[0].properties["start_datetime"], "2025-01-18")
+ self.assertEqual(products[1].properties["start_datetime"], "2025-01-17")
+ self.assertEqual(products[2].properties["start_datetime"], "2025-01-16")
+
+ products = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "end_datetime": "2025-01-18"},
+ {"start_datetime": None, "end_datetime": "2025-01-16"},
+ {"start_datetime": "2025-01-17", "end_datetime": "2025-01-17"},
+ ]
+ )
+ products.sort(
+ key=FilterLatestIntersect.sort_product_by_start_date, reverse=False
+ )
+ self.assertEqual(products[0].properties.get("start_datetime", None), None)
+ self.assertEqual(
+ products[1].properties.get("start_datetime", None), "2025-01-17"
+ )
+ self.assertEqual(
+ products[2].properties.get("start_datetime", None), "2025-01-18"
+ )
+
+ def test_crunch_lastestbyname(self):
+ """Crunch FilterLatestByName test"""
+
+ # Missing parameter
+ search_results: SearchResult = self.__fake_search_result([])
+ try:
+ _ = search_results.crunch(FilterLatestByName())
+ self.fail('FilterLatestByName require parameter "name_pattern"')
+ except ValidationError:
+ pass
+
+ # Invalid pattern
+ try:
+ _ = search_results.crunch(
+ FilterLatestByName({"name_pattern": "hell_pattern!"})
+ )
+ self.fail('FilterLatestByName require valid parameter "name_pattern"')
+ except ValidationError:
+ pass
+
+ # No products
+ filter_products = search_results.crunch(
+ FilterLatestByName({"name_pattern": "(?P<tileid>\\d{6})"})
+ )
+ self.assertEqual(len(filter_products), 0)
+
+ # Some products to filter
+ search_results = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "title": "000001"},
+ {"start_datetime": "2025-01-18", "title": "000002"},
+ {"start_datetime": "2025-01-19", "title": "000003"},
+ {"start_datetime": "2025-01-19", "title": "000004"},
+ {"start_datetime": "2025-01-20", "title": "000005"},
+ ]
+ )
+ filter_products = search_results.crunch(
+ FilterLatestByName({"name_pattern": "(?P<tileid>\\d{6})"})
+ )
+ self.assertEqual(len(filter_products), 5)
+
+ # Some products to filter with mixed title
+ search_results = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "title": "malformed"},
+ {"start_datetime": "2025-01-18", "title": "000002"},
+ {"start_datetime": "2025-01-19", "title": "malformed"},
+ {"start_datetime": "2025-01-19", "title": "000004"},
+ {"start_datetime": "2025-01-20", "title": "malformed"},
+ ]
+ )
+ filter_products = search_results.crunch(
+ FilterLatestByName({"name_pattern": "(?P<tileid>\\d{6})"})
+ )
+ self.assertEqual(len(filter_products), 2)
+
+ # Some products to filter with mixed title, repeated formatted title
+ search_results = self.__fake_search_result(
+ [
+ {"start_datetime": "2025-01-18", "title": "malformed"},
+ {"start_datetime": "2025-01-18", "title": "000002"},
+ {"start_datetime": "2025-01-19", "title": "malformed"},
+ {"start_datetime": "2025-01-19", "title": "000002"},
+ {"start_datetime": "2025-01-20", "title": "malformed"},
+ ]
+ )
+ filter_products = search_results.crunch(
+ FilterLatestByName({"name_pattern": "(?P<tileid>\\d{6})"})
+ )
+ self.assertEqual(len(filter_products), 1)
+
+ def test_crunch_overlap(self):
+ """Crunch FilterOverlap test"""
+
+ # No products, no configuration
+ search_results: SearchResult = self.__fake_search_result([])
+ filtered_result = search_results.crunch(FilterOverlap())
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(
+ FilterOverlap(), geometry=geometry.box(0, 0, 0, 0)
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Invalid configuration
+ search_results = self.__fake_search_result(
+ [
+ {"geometry": geometry.box(10, 20, 15, 30)},
+ {"geometry": geometry.box(10, 30, 15, 40)},
+ {"geometry": geometry.box(15, 20, 20, 30)},
+ {"geometry": geometry.box(15, 30, 20, 40)},
+ ]
+ )
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": -1, # min: 0
+ "contains": False,
+ "intersects": False,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(0, 0, 0, 0),
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": "hell_minimum!",
+ "contains": False,
+ "intersects": False,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(0, 0, 0, 0),
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Parameters exclusion (contains, intersects, within)
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": True,
+ "intersects": True,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(0, 0, 0, 0),
+ )
+ self.assertEqual(len(filtered_result), 4)
+
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": True,
+ "intersects": False,
+ "within": True,
+ }
+ ),
+ geometry=geometry.box(0, 0, 0, 0),
+ )
+ self.assertEqual(len(filtered_result), 4)
+
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": False,
+ "intersects": True,
+ "within": True,
+ }
+ ),
+ geometry=geometry.box(0, 0, 0, 0),
+ )
+ self.assertEqual(len(filtered_result), 4)
+
+ # Intersect
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 95,
+ "contains": False,
+ "intersects": True,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(12, 25, 13, 35),
+ )
+ self.assertEqual(len(filtered_result), 2)
+
+ # Contains
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": True,
+ "intersects": False,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(12, 25, 13, 35),
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Contains
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": True,
+ "intersects": False,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(12, 21, 13, 29),
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ # Within
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 100,
+ "contains": False,
+ "intersects": False,
+ "within": True,
+ }
+ ),
+ geometry=geometry.box(9, 19, 16, 31),
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ # Search geom area = 0
+ geom = Polygon(((10, 20), (15, 30), (15, 20), (10, 30), (10, 20)))
+ self.assertFalse(geom.is_valid)
+ search_results = self.__fake_search_result(
+ [
+ {"geometry": geometry.box(10, 20, 15, 30)},
+ {"geometry": geometry.box(10, 30, 15, 40)},
+ {"geometry": geometry.box(15, 20, 20, 30)},
+ {"geometry": geometry.box(15, 30, 20, 40)},
+ ]
+ )
+ filtered_result = search_results.crunch(
+ FilterOverlap(
+ {
+ "minimum_overlap": 50,
+ "contains": False,
+ "intersects": True,
+ "within": False,
+ }
+ ),
+ geometry=geometry.box(10, 30, 10, 30),
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ # Product invalid geometry (butterfly shape) recoverable
+ geom = Polygon(((10, 20), (15, 30), (15, 20), (10, 30), (10, 20)))
+ self.assertFalse(geom.is_valid)
+ search_results = self.__fake_search_result([{"geometry": geom}])
+ filter = FilterOverlap(
+ {
+ "minimum_overlap": 50,
+ "contains": False,
+ "intersects": True,
+ "within": False,
+ }
+ )
+
+ # Disable search intersection to test invalid geom
+ for index in range(0, len(search_results)):
+ search_results[index].search_intersection = None
+
+ filtered_result = search_results.crunch(
+ filter, geometry=geometry.box(10, 30, 20, 40)
+ )
+ self.assertEqual(len(filtered_result), 1)
+
+ # Force fail search_geom intersection
+ def shapely_intersect_forced_exception(a, b, grid_size=None, **kwargs):
+ print("Mocked: no shapely.intersection allowed")
+ raise ShapelyError("No intersection allowed")
+
+ with mock.patch.object(
+ shapely, "intersection", new=shapely_intersect_forced_exception
+ ):
+ filtered_result = search_results.crunch(
+ filter, geometry=geometry.box(10, 30, 20, 40)
+ )
+ self.assertEqual(len(filtered_result), 0)
+
+ def test_crunch_property(self):
+ """Crunch FilterProperty test"""
+
+ # No configuration
+ search_results = self.__fake_search_result(
+ [
+ {"myproperty": 1},
+ {"myproperty": 2},
+ {"myproperty": 3},
+ {"myproperty": 4},
+ {"myproperty": 5},
+ ]
+ )
+ filtered_result = search_results.crunch(FilterProperty({}))
+ self.assertEqual(len(filtered_result), 5)
+
+ # Partial configuration
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": 3}))
+ self.assertEqual(len(filtered_result), 1)
+
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": 6}))
+ self.assertEqual(len(filtered_result), 0)
+
+ # Wrong configuration
+ filtered_result = search_results.crunch(
+ FilterProperty({"myproperty": 6, "operator": "hell"})
+ )
+ self.assertEqual(len(filtered_result), 5)
+
+ # Full configuration
+ for test in [
+ {"value": 2, "operator": "lt", "expect_results": 1},
+ {"value": 2, "operator": "le", "expect_results": 2},
+ {"value": 2, "operator": "eq", "expect_results": 1},
+ {"value": 2, "operator": "ne", "expect_results": 4},
+ {"value": 2, "operator": "ge", "expect_results": 4},
+ {"value": 2, "operator": "gt", "expect_results": 3},
+ {"value": 4, "operator": "lt", "expect_results": 3},
+ {"value": 4, "operator": "le", "expect_results": 4},
+ {"value": 4, "operator": "eq", "expect_results": 1},
+ {"value": 4, "operator": "ne", "expect_results": 4},
+ {"value": 4, "operator": "ge", "expect_results": 2},
+ {"value": 4, "operator": "gt", "expect_results": 1},
+ ]:
+ filtered_result = search_results.crunch(
+ FilterProperty(
+ {"myproperty": test["value"], "operator": test["operator"]}
+ )
+ )
+ self.assertEqual(len(filtered_result), test["expect_results"])
+
+ # Multitypes data
+ search_results = self.__fake_search_result(
+ [
+ {"myproperty": 1},
+ {"myproperty": "2"},
+ {"myproperty": bool},
+ {"myproperty": None},
+ {"myproperty": 5},
+ ]
+ )
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": 3}))
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": 2}))
+ self.assertEqual(len(filtered_result), 0)
+
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": "2"}))
+ self.assertEqual(len(filtered_result), 1)
+
+ # Mismatch filter value type makes no filter wihtout error
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": True}))
+ self.assertEqual(len(filtered_result), 1)
+
+ # Invalid filter value type makes no results
+ filtered_result = search_results.crunch(FilterProperty({"myproperty": None}))
+ self.assertEqual(len(filtered_result), 0)
+
+ # Multitypes data with operator set
+ for value in [42, "myvalue", True]:
+ for test in [
+ {"operator": "lt", "expect_match": False}, # Lower (False)
+ {"operator": "le", "expect_match": True}, # Lower(False) + Equal(True)
+ {"operator": "eq", "expect_match": True}, # Equal(True)
+ {"operator": "ne", "expect_match": False}, # Non Equal(False)
+ {
+ "operator": "ge",
+ "expect_match": True,
+ }, # Greater(False) + Equal(True)
+ {"operator": "gt", "expect_match": False}, # Greater(False)
+ ]:
+ search_results = self.__fake_search_result([{"myproperty": value}])
+ filtered_result = search_results.crunch(
+ FilterProperty({"myproperty": value, "operator": test["operator"]})
+ )
+ self.assertEqual(len(filtered_result) == 1, test["expect_match"])
View it on GitLab: https://salsa.debian.org/debian-gis-team/eodag/-/compare/ccb95fca5b9786b7a0a3b6bc39be60da445766d9...5008250e71b7fab18ea38b4ca0ea4781e9392762
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/eodag/-/compare/ccb95fca5b9786b7a0a3b6bc39be60da445766d9...5008250e71b7fab18ea38b4ca0ea4781e9392762
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20260326/42c7faf7/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list