[Git][debian-gis-team/xarray-sentinel][master] 6 commits: tmp
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Fri Nov 18 13:36:42 GMT 2022
Antonio Valentino pushed to branch master at Debian GIS Project / xarray-sentinel
Commits:
c7b7ffec by Antonio Valentino at 2022-08-26T11:31:23+00:00
tmp
- - - - -
04d44442 by Antonio Valentino at 2022-11-18T13:24:03+00:00
New upstream version 0.9.5+ds
- - - - -
deb8e968 by Antonio Valentino at 2022-11-18T13:24:13+00:00
Update upstream source from tag 'upstream/0.9.5+ds'
Update to upstream version '0.9.5+ds'
with Debian dir 46c84a3670db975fa7ab4a41dd27cd41df4ec7ef
- - - - -
e365cd20 by Antonio Valentino at 2022-11-18T13:24:46+00:00
New upstream release
- - - - -
db0127a5 by Antonio Valentino at 2022-11-18T13:27:36+00:00
Drop patches applied upstream
- - - - -
a73a4b08 by Antonio Valentino at 2022-11-18T13:28:01+00:00
Set distribution to unstable
- - - - -
16 changed files:
- .github/workflows/on-push.yml
- README.md
- debian/changelog
- − debian/patches/0001-Shebang.patch
- − debian/patches/0002-Package-data.patch
- − debian/patches/series
- debian/rules
- environment.yml
- notebooks/download_data.sh
- pyproject.toml
- setup.cfg
- tests/test_20_sentinel1.py
- tests/test_30_sentinel1_fsspec.py
- xarray_sentinel/conventions.py
- xarray_sentinel/esa_safe.py
- xarray_sentinel/sentinel1.py
Changes:
=====================================
.github/workflows/on-push.yml
=====================================
@@ -34,7 +34,7 @@ jobs:
steps:
- uses: actions/checkout at v3
- name: Install Conda environment with Micromamba
- uses: mamba-org/provision-with-micromamba at v12
+ uses: mamba-org/provision-with-micromamba at v14
with:
environment-file: environment.yml
environment-name: DEVELOP
@@ -53,7 +53,7 @@ jobs:
run: |
make doc-test COV_REPORT=xml
- name: Upload code coverage to Codecov
- uses: codecov/codecov-action at v3.1.0
+ uses: codecov/codecov-action at v3.1.1
type-check:
needs: [unit-tests]
@@ -65,7 +65,7 @@ jobs:
steps:
- uses: actions/checkout at v3
- name: Install Conda environment with Micromamba
- uses: mamba-org/provision-with-micromamba at v12
+ uses: mamba-org/provision-with-micromamba at v14
with:
environment-file: environment.yml
environment-name: DEVELOP
@@ -99,7 +99,7 @@ jobs:
steps:
- uses: actions/checkout at v3
- name: Install Conda environment with Micromamba
- uses: mamba-org/provision-with-micromamba at v12
+ uses: mamba-org/provision-with-micromamba at v14
with:
environment-file: environment${{ matrix.extra }}.yml
environment-name: DEVELOP${{ matrix.extra }}
@@ -115,7 +115,7 @@ jobs:
run: |
make test COV_REPORT=xml
- name: Upload code coverage to Codecov
- uses: codecov/codecov-action at v3.1.0
+ uses: codecov/codecov-action at v3.1.1
distribution:
runs-on: ubuntu-latest
=====================================
README.md
=====================================
@@ -23,8 +23,8 @@ with the following functionalities:
- reads several metadata elements:
satellite orbit and attitude, ground control points, radiometric calibration look up tables,
Doppler centroid estimation and more
-- reads uncompressed and compressed SAFE data products on the local computer or
- on a network via [*fsspec*](https://filesystem-spec.readthedocs.io) - **depends on rasterio >= 1.3.0**
+- (partially broken, see [#127](https://github.com/bopen/xarray-sentinel/issues/127)) reads uncompressed and compressed SAFE data products on the local computer or
+ on a network via [*fsspec*](https://filesystem-spec.readthedocs.io)
- supports larger-than-memory and distributed data access via [*Dask*](https://dask.org) and
[*rioxarray*](https://corteva.github.io/rioxarray) /
[*rasterio*](https://rasterio.readthedocs.io) / [*GDAL*](https://gdal.org)
@@ -86,7 +86,7 @@ Attributes: ...
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
- ... ...
+ ...
start_time: 2021-04-01T15:28:55.111501
stop_time: 2021-04-01T15:29:14.277650
group: /
@@ -126,7 +126,7 @@ Attributes: ...
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
- ... ...
+ ...
geospatial_lon_min: 42.772483374347
geospatial_lon_max: 43.75770573943618
group: /S3/VH
@@ -188,7 +188,7 @@ Attributes: ...
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
- ... ...
+ ...
stop_time: 2021-04-01T15:29:14.277650
group: /S3/VH/calibration
Conventions: CF-1.8
@@ -263,7 +263,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
- ... ...
+ ...
geospatial_lon_min: -61.94949110259839
geospatial_lon_max: -60.24826879672774
group: /IW1/HH
@@ -297,7 +297,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
- ... ...
+ ...
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
@@ -328,7 +328,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 42768
relative_orbit_number: 171
- ... ...
+ ...
group: /IW1/HH
Conventions: CF-1.8
history: created by xarray_sentinel-...
@@ -364,7 +364,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 26269
relative_orbit_number: 168
- ... ...
+ ...
geospatial_lon_max: 12.093126130070317
group: /IW1/VH
azimuth_anx_time: 2210.634453
@@ -388,29 +388,14 @@ Coordinates:
line (azimuth_time) int64 dask.array<chunksize=(2048,), meta=np.ndarray>
* azimuth_time (azimuth_time) datetime64[ns] 2021-04-01T15:28:55.11150...
* slant_range_time (slant_range_time) float64 0.005273 0.005273 ... 0.005303
-Attributes:
+Attributes: ...
family_name: SENTINEL-1
number: A
mode: SM
swaths: ['S3']
orbit_number: 37258
relative_orbit_number: 86
- pass: ASCENDING
- ascending_node_time: 2021-04-01T13:53:42.874198
- transmitter_receiver_polarisations: ['VV', 'VH']
- product_type: SLC
- start_time: 2021-04-01T15:28:55.111501
- stop_time: 2021-04-01T15:29:14.277650
- radar_frequency: 5.40500045433435
- azimuth_pixel_spacing: 3.55338
- range_pixel_spacing: 2.246363
- product_first_line_utc_time: 2021-04-01T15:28:55.111501
- product_last_line_utc_time: 2021-04-01T15:29:14.277650
- azimuth_time_interval: 0.0005194923129469381
- image_slant_range_time: 0.005272617843915159
- range_sampling_rate: 66728395.09333333
- incidence_angle_mid_swath: 32.03479766845703
- geospatial_bounds: POLYGON((43.75770573943618 -12.01571...
+ ...
geospatial_lat_min: -12.17883496921861
geospatial_lat_max: -10.85986742252814
geospatial_lon_min: 42.772483374347
@@ -450,7 +435,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 26269
relative_orbit_number: 168
- ... ...
+ ...
number_of_bursts: 9
lines_per_burst: 1501
group: /IW1/VH
@@ -480,7 +465,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 26269
relative_orbit_number: 168
- ... ...
+ ...
number_of_bursts: 9
lines_per_burst: 1501
group: /IW1/VH
@@ -518,7 +503,7 @@ Attributes: ...
swaths: ['IW1', 'IW2', 'IW3']
orbit_number: 26269
relative_orbit_number: 168
- ... ...
+ ...
number_of_bursts: 9
lines_per_burst: 1501
group: /IW1/VH
=====================================
debian/changelog
=====================================
@@ -1,3 +1,12 @@
+xarray-sentinel (0.9.5+ds-1) unstable; urgency=medium
+
+ * New upstream release.
+ * debian/patches:
+ - drop 0001-Shebang.patch and 0002-Package-data.patch,
+ applied upstream.
+
+ -- Antonio Valentino <antonio.valentino at tiscali.it> Fri, 18 Nov 2022 13:27:45 +0000
+
xarray-sentinel (0.9.3+ds-4) unstable; urgency=medium
* Skip tests requiring large memory.
=====================================
debian/patches/0001-Shebang.patch deleted
=====================================
@@ -1,18 +0,0 @@
-From: Antonio Valentino <antonio.valentino at tiscali.it>
-Date: Sun, 7 Aug 2022 10:56:55 +0000
-Subject: Shebang
-
-Forwarded: https://github.com/bopen/xarray-sentinel/pull/119
----
- notebooks/download_data.sh | 1 +
- 1 file changed, 1 insertion(+)
-
-diff --git a/notebooks/download_data.sh b/notebooks/download_data.sh
-index 30a446d..45d706c 100755
---- a/notebooks/download_data.sh
-+++ b/notebooks/download_data.sh
-@@ -1,3 +1,4 @@
-+#!/bin/sh
-
- mkdir -p data
-
=====================================
debian/patches/0002-Package-data.patch deleted
=====================================
@@ -1,23 +0,0 @@
-From: Antonio Valentino <antonio.valentino at tiscali.it>
-Date: Sun, 7 Aug 2022 10:59:50 +0000
-Subject: Package data
-
-Forwarded: https://github.com/bopen/xarray-sentinel/pull/120
----
- setup.cfg | 3 +++
- 1 file changed, 3 insertions(+)
-
-diff --git a/setup.cfg b/setup.cfg
-index 7d3542d..481af02 100644
---- a/setup.cfg
-+++ b/setup.cfg
-@@ -32,6 +32,9 @@ install_requires =
- xmlschema
- include_package_data = True
-
-+[options.package_data]
-+xarray_sentinel = resources/sentinel1/*.xsd
-+
- [options.entry_points]
- xarray.backends =
- sentinel-1 = xarray_sentinel.xarray_backends:Sentinel1Backend
=====================================
debian/patches/series deleted
=====================================
@@ -1,2 +0,0 @@
-0001-Shebang.patch
-0002-Package-data.patch
=====================================
debian/rules
=====================================
@@ -1,5 +1,5 @@
#! /usr/bin/make -f
-
+export DH_VERBOSE=1
include /usr/share/dpkg/pkg-info.mk
export PYBUILD_NAME=xarray-sentinel
=====================================
environment.yml
=====================================
@@ -7,6 +7,7 @@ dependencies:
- mypy
- netcdf4
- numpy >= 1.22.0
+- pandas-stubs
- pytest
- pytest-cov
- rasterio >= 1.3.0
=====================================
notebooks/download_data.sh
=====================================
@@ -1,3 +1,4 @@
+#!/bin/sh
mkdir -p data
=====================================
pyproject.toml
=====================================
@@ -8,7 +8,6 @@ requires = [
[tool.coverage.run]
branch = true
-omit = ["setup.py"]
[tool.isort]
profile = "black"
=====================================
setup.cfg
=====================================
@@ -32,6 +32,9 @@ install_requires =
xmlschema
include_package_data = True
+[options.package_data]
+xarray_sentinel = resources/sentinel1/*.xsd
+
[options.entry_points]
xarray.backends =
sentinel-1 = xarray_sentinel.xarray_backends:Sentinel1Backend
@@ -46,15 +49,9 @@ plugins = numpy.typing.mypy_plugin
[mypy-cfchecker]
ignore_missing_imports = True
-[mypy-dask]
-ignore_missing_imports = True
-
[mypy-fsspec]
ignore_missing_imports = True
-[mypy-pandas]
-ignore_missing_imports = True
-
[mypy-rasterio]
ignore_missing_imports = True
=====================================
tests/test_20_sentinel1.py
=====================================
@@ -121,6 +121,9 @@ def test_open_coordinate_conversion_dataset() -> None:
assert isinstance(res, xr.Dataset)
assert set(res.coords) == {"azimuth_time", "degree"}
+ with pytest.raises(TypeError):
+ sentinel1.open_coordinate_conversion_dataset(SLC_IW1_VV_annotation)
+
def test_open_gcp_dataset() -> None:
expected_geospatial_bounds = (
=====================================
tests/test_30_sentinel1_fsspec.py
=====================================
@@ -2,7 +2,6 @@ import pathlib
import fsspec
import pytest
-import rasterio
import xarray as xr
from xarray_sentinel import sentinel1
@@ -96,9 +95,7 @@ def test_open_dataset_zip_metadata() -> None:
assert res.dims == {"axis": 3, "azimuth_time": 17} # type: ignore
- at pytest.mark.xfail(
- rasterio.__version__ < "1.3.0", reason="full fsspec support needs rasterio > 1.3.0"
-)
+ at pytest.mark.xfail
def test_open_dataset_zip_data() -> None:
zip_path = (
DATA_FOLDER
=====================================
xarray_sentinel/conventions.py
=====================================
@@ -116,9 +116,9 @@ VARIABLE_ATTRIBUTES = {
def update_attributes(ds: xr.Dataset, group: str = "") -> xr.Dataset:
# NOTE: keep the version in sync with the capabilities of CF compliance checkers
ds.attrs["Conventions"] = "CF-1.8"
- ds.attrs.update(GROUP_ATTRIBUTES.get(group, {})) # type: ignore
+ ds.attrs.update(GROUP_ATTRIBUTES.get(group, {}))
ds.attrs["history"] = f"created by xarray_sentinel-{__version__}"
for var in ds.variables:
attrs = VARIABLE_ATTRIBUTES.get(str(var), {})
- ds.variables[var].attrs.update(attrs) # type: ignore
+ ds.variables[var].attrs.update(attrs)
return ds
=====================================
xarray_sentinel/esa_safe.py
=====================================
@@ -44,7 +44,7 @@ def parse_tag(
) -> Dict[str, Any]:
schema = cached_sentinel1_schemas(schema_type)
if hasattr(xml_path, "seek"):
- xml_path.seek(0) # type: ignore
+ xml_path.seek(0)
xml_tree = ElementTree.parse(xml_path)
tag_dict: Any = schema.decode(xml_tree, query, validation=validation)
assert isinstance(tag_dict, dict), f"{type(tag_dict)} is not dict"
=====================================
xarray_sentinel/sentinel1.py
=====================================
@@ -182,6 +182,8 @@ def open_coordinate_conversion_dataset(
coordinate_conversion = esa_safe.parse_tag_as_list(
annotation_path, ".//coordinateConversionList/coordinateConversion"
)
+ if len(coordinate_conversion) == 0:
+ raise TypeError("coordinateConversion tag not present in annotations")
gr0 = []
sr0 = []
@@ -203,15 +205,14 @@ def open_coordinate_conversion_dataset(
coords: Dict[str, Any] = {}
data_vars: Dict[str, Any] = {}
- if srgrCoefficients:
- coords["azimuth_time"] = [np.datetime64(dt) for dt in azimuth_time]
- coords["degree"] = list(range(len(srgrCoefficients[0])))
+ coords["azimuth_time"] = [np.datetime64(dt) for dt in azimuth_time]
+ coords["degree"] = list(range(len(srgrCoefficients[0])))
- data_vars["gr0"] = ("azimuth_time", gr0)
- data_vars["sr0"] = ("azimuth_time", sr0)
- data_vars["slant_range_time"] = ("azimuth_time", slant_range_time)
- data_vars["srgrCoefficients"] = (("azimuth_time", "degree"), srgrCoefficients)
- data_vars["grsrCoefficients"] = (("azimuth_time", "degree"), grsrCoefficients)
+ data_vars["gr0"] = ("azimuth_time", gr0)
+ data_vars["sr0"] = ("azimuth_time", sr0)
+ data_vars["slant_range_time"] = ("azimuth_time", slant_range_time)
+ data_vars["srgrCoefficients"] = (("azimuth_time", "degree"), srgrCoefficients)
+ data_vars["grsrCoefficients"] = (("azimuth_time", "degree"), grsrCoefficients)
return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)
@@ -269,7 +270,7 @@ def open_gcp_dataset(
footprint = get_footprint_linestring(ds.azimuth_time, ds.slant_range_time, ds)
geospatial_attrs = make_geospatial_attributes(footprint)
- ds.attrs.update(geospatial_attrs) # type: ignore
+ ds.attrs.update(geospatial_attrs)
return ds
@@ -351,35 +352,14 @@ def open_attitude_dataset(
return ds
-def open_orbit_dataset(
- annotation: esa_safe.PathOrFileType, attrs: Dict[str, Any] = {}
+def make_orbit(
+ azimuth_time: List[Any],
+ positions: List[List[Any]],
+ velocities: List[List[Any]],
+ attrs: Dict[str, Any] = {},
) -> xr.Dataset:
- orbits = esa_safe.parse_tag_as_list(annotation, ".//orbit")
-
- reference_system = orbits[0]["frame"]
- variables = ["position", "velocity"]
- data: Dict[str, List[Any]] = {var: [[], [], []] for var in variables}
- azimuth_time: List[Any] = []
- for orbit in orbits:
- azimuth_time.append(orbit["time"])
- data["position"][0].append(orbit["position"]["x"])
- data["position"][1].append(orbit["position"]["y"])
- data["position"][2].append(orbit["position"]["z"])
- data["velocity"][0].append(orbit["velocity"]["x"])
- data["velocity"][1].append(orbit["velocity"]["y"])
- data["velocity"][2].append(orbit["velocity"]["z"])
- if orbit["frame"] != reference_system:
- warnings.warn(
- "reference_system is not consistent in all the state vectors. "
- )
- reference_system = None
-
- position = xr.Variable(data=data["position"], dims=("axis", "azimuth_time")) # type: ignore
- velocity = xr.Variable(data=data["velocity"], dims=("axis", "azimuth_time")) # type: ignore
-
- attrs = attrs.copy()
- if reference_system is not None:
- attrs.update({"reference_system": reference_system})
+ position = xr.Variable(data=positions, dims=("axis", "azimuth_time")) # type: ignore
+ velocity = xr.Variable(data=velocities, dims=("axis", "azimuth_time")) # type: ignore
ds = xr.Dataset(
data_vars={"position": position, "velocity": velocity},
@@ -395,6 +375,36 @@ def open_orbit_dataset(
return ds
+def open_orbit_dataset(
+ annotation: esa_safe.PathOrFileType, attrs: Dict[str, Any] = {}
+) -> xr.Dataset:
+ orbits = esa_safe.parse_tag_as_list(annotation, ".//orbit")
+
+ attrs = attrs.copy()
+ reference_system = orbits[0]["frame"]
+ if reference_system is not None:
+ attrs.update({"reference_system": reference_system})
+
+ azimuth_times: List[Any] = []
+ positions: List[List[Any]] = [[], [], []]
+ velocities: List[List[Any]] = [[], [], []]
+ for orbit in orbits:
+ azimuth_times.append(orbit["time"])
+ positions[0].append(orbit["position"]["x"])
+ positions[1].append(orbit["position"]["y"])
+ positions[2].append(orbit["position"]["z"])
+ velocities[0].append(orbit["velocity"]["x"])
+ velocities[1].append(orbit["velocity"]["y"])
+ velocities[2].append(orbit["velocity"]["z"])
+ if orbit["frame"] != reference_system:
+ warnings.warn(
+ "reference_system is not consistent in all the state vectors. "
+ )
+ reference_system = None
+
+ return make_orbit(azimuth_times, positions, velocities, attrs)
+
+
def open_dc_estimate_dataset(
annotation: esa_safe.PathOrFileType, attrs: Dict[str, Any] = {}
) -> xr.Dataset:
@@ -511,7 +521,13 @@ def open_rasterio_dataarray(
raise FileNotFoundError(str(ex))
raise
else:
- arr = xr.open_dataarray(fs.open(measurement), engine="rasterio", chunks=chunks)
+ # FIXME: rioxarray / rasterio do not support opening a file object any more, so
+ # full fsspec support via `fs.open(measurement)` is now broken and we fall back
+ # to hope that rasterio remote protocols works with fsspec urlpaths
+ maybe_rasterio_urlpath = fs.unstrip_protocol(measurement)
+ arr = xr.open_dataarray(
+ maybe_rasterio_urlpath, engine="rasterio", chunks=chunks
+ )
return arr
@@ -653,22 +669,22 @@ def open_pol_dataset(
arr = arr.swap_dims(swap_dims)
if gcp:
- attrs.update(gcp.attrs) # type: ignore
+ attrs.update(gcp.attrs)
arr.attrs.update(attrs)
- arr.encoding.update(encoding) # type: ignore
+ arr.encoding.update(encoding)
return xr.Dataset(attrs=attrs, data_vars={"measurement": arr})
def find_bursts_index(
pol_dataset: DataArrayOrDataset,
- azimuth_anx_time: float,
+ azimuth_anx_seconds: float,
use_center: bool = False,
) -> int:
lines_per_burst = pol_dataset.attrs["lines_per_burst"]
anx_datetime = np.datetime64(pol_dataset.attrs["ascending_node_time"])
- azimuth_anx_time = pd.Timedelta(azimuth_anx_time, unit="s")
+ azimuth_anx_time = pd.Timedelta(azimuth_anx_seconds, unit="s")
if use_center:
azimuth_anx_time_center = (
pol_dataset.azimuth_time[lines_per_burst // 2 :: lines_per_burst]
@@ -756,7 +772,7 @@ def crop_burst_dataset(
if gcp is not None:
footprint = get_footprint_linestring(ds.azimuth_time, ds.slant_range_time, gcp)
geospatial_attrs = make_geospatial_attributes(footprint)
- ds.attrs.update(geospatial_attrs) # type: ignore
+ ds.attrs.update(geospatial_attrs)
if "burst_ids" in ds.attrs:
ds.attrs["burst_id"] = ds.attrs["burst_ids"][burst_index]
View it on GitLab: https://salsa.debian.org/debian-gis-team/xarray-sentinel/-/compare/96847dbbd5e077b7db2126ea84e3cd4f583f1c85...a73a4b08c6cc275305078c102a8b4bfa89f9fafb
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/xarray-sentinel/-/compare/96847dbbd5e077b7db2126ea84e3cd4f583f1c85...a73a4b08c6cc275305078c102a8b4bfa89f9fafb
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20221118/2b519bc9/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list