[Git][debian-gis-team/satpy][upstream] New upstream version 0.53.0
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Sat Nov 23 19:14:47 GMT 2024
Antonio Valentino pushed to branch upstream at Debian GIS Project / satpy
Commits:
270deb06 by Antonio Valentino at 2024-11-09T16:55:13+00:00
New upstream version 0.53.0
- - - - -
27 changed files:
- .git_archival.txt
- .github/workflows/ci.yaml
- .github/workflows/deploy-sdist.yaml
- .pre-commit-config.yaml
- AUTHORS.md
- CHANGELOG.md
- satpy/enhancements/__init__.py
- satpy/etc/composites/abi.yaml
- satpy/etc/composites/ahi.yaml
- satpy/etc/composites/ami.yaml
- satpy/etc/composites/vii.yaml
- satpy/etc/composites/viirs.yaml
- satpy/etc/readers/mersi_ll_l1b.yaml
- satpy/etc/readers/modis_l1b.yaml
- satpy/modifiers/angles.py
- satpy/modifiers/atmosphere.py
- satpy/readers/hdfeos_base.py
- satpy/readers/hrit_jma.py
- satpy/readers/sar_c_safe.py
- satpy/readers/viirs_edr.py
- satpy/tests/enhancement_tests/test_enhancements.py
- satpy/tests/reader_tests/test_ahi_hrit.py
- satpy/tests/reader_tests/test_viirs_edr.py
- satpy/tests/test_composites.py
- satpy/tests/test_modifiers.py
- satpy/tests/writer_tests/test_cf.py
- satpy/writers/cf_writer.py
Changes:
=====================================
.git_archival.txt
=====================================
@@ -1,4 +1,4 @@
-node: 6814f7531746651537797f6c5dd9315c5d55e0fe
-node-date: 2024-10-23T12:46:22+02:00
-describe-name: v0.52.1
-ref-names: tag: v0.52.1
+node: bd29bb18a0ecf28dc49c7851057b8a67bccae03f
+node-date: 2024-11-08T16:40:56+01:00
+describe-name: v0.53.0
+ref-names: HEAD -> main, tag: v0.53.0
=====================================
.github/workflows/ci.yaml
=====================================
@@ -42,6 +42,8 @@ jobs:
python-version: ${{ matrix.python-version }}
activate-environment: test-environment
channels: conda-forge
+ conda-remove-defaults: true
+ channel-priority: strict
- name: Set cache environment variables
shell: bash -l {0}
=====================================
.github/workflows/deploy-sdist.yaml
=====================================
@@ -23,7 +23,7 @@ jobs:
- name: Publish package to PyPI
if: github.event.action == 'published'
- uses: pypa/gh-action-pypi-publish at v1.10.2
+ uses: pypa/gh-action-pypi-publish at v1.11.0
with:
user: __token__
password: ${{ secrets.pypi_password }}
=====================================
.pre-commit-config.yaml
=====================================
@@ -3,7 +3,7 @@ fail_fast: false
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
- rev: 'v0.6.9'
+ rev: 'v0.7.2'
hooks:
- id: ruff
- repo: https://github.com/pre-commit/pre-commit-hooks
@@ -19,7 +19,7 @@ repos:
- id: bandit
args: [--ini, .bandit]
- repo: https://github.com/pre-commit/mirrors-mypy
- rev: 'v1.11.2' # Use the sha / tag you want to point at
+ rev: 'v1.13.0' # Use the sha / tag you want to point at
hooks:
- id: mypy
additional_dependencies:
=====================================
AUTHORS.md
=====================================
@@ -38,6 +38,8 @@ The following people have made contributions to this project:
- [Gerrit Holl (gerritholl)](https://github.com/gerritholl) - Deutscher Wetterdienst
- [David Hoese (djhoese)](https://github.com/djhoese)
- [Marc Honnorat (honnorat)](https://github.com/honnorat)
+- [Chung-Hsiang Horng(chorng)](https://github.com/chorng)
+- [Mario Hros (k3a)](https://github.com/k3a)
- [Lloyd Hughes (system123)](https://github.com/system123)
- [Sara Hörnquist (shornqui)](https://github.com/shornqui)
- [Mikhail Itkin (mitkin)](https://github.com/mitkin)
=====================================
CHANGELOG.md
=====================================
@@ -1,3 +1,38 @@
+## Version 0.53.0 (2024/11/08)
+
+### Issues Closed
+
+* [Issue 2960](https://github.com/pytroll/satpy/issues/2960) - netcdf4 version causes error ([PR 2961](https://github.com/pytroll/satpy/pull/2961) by [@sfinkens](https://github.com/sfinkens))
+* [Issue 2952](https://github.com/pytroll/satpy/issues/2952) - Altitude, LandCover, and LandSeaMask are missing in the `mersi_ll_l1b` reader for FY3E L1B ([PR 2953](https://github.com/pytroll/satpy/pull/2953) by [@chorng](https://github.com/chorng))
+* [Issue 2948](https://github.com/pytroll/satpy/issues/2948) - "Missing" platform abbreviation causes unexpected error when loading data array in Scene ([PR 2949](https://github.com/pytroll/satpy/pull/2949) by [@joleenf](https://github.com/joleenf))
+
+In this release 3 issues were closed.
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 2971](https://github.com/pytroll/satpy/pull/2971) - Pin flexparser before it breaks pint
+* [PR 2970](https://github.com/pytroll/satpy/pull/2970) - Remove rayleigh correction on VIIRS false_color for I02 band
+* [PR 2968](https://github.com/pytroll/satpy/pull/2968) - Remove unneeded call to private scipy function in SAR reader
+* [PR 2965](https://github.com/pytroll/satpy/pull/2965) - Fix MODIS readers chunking compatibility with newer dask
+* [PR 2961](https://github.com/pytroll/satpy/pull/2961) - Fix CF writer crashing with netcdf development version ([2960](https://github.com/pytroll/satpy/issues/2960))
+* [PR 2957](https://github.com/pytroll/satpy/pull/2957) - Bugfix the VIIRS lowres version of the day-microphysics.
+* [PR 2956](https://github.com/pytroll/satpy/pull/2956) - Fix cira stretch upcasting the data
+* [PR 2954](https://github.com/pytroll/satpy/pull/2954) - Fix Rayleigh correction to use the same datatype as the input data
+* [PR 2950](https://github.com/pytroll/satpy/pull/2950) - Fix dtype promotion in `SunZenithReduction`
+* [PR 2949](https://github.com/pytroll/satpy/pull/2949) - Add more platforms to VIIRS EDR reader ([2948](https://github.com/pytroll/satpy/issues/2948))
+* [PR 2930](https://github.com/pytroll/satpy/pull/2930) - Fix data type when getting a line offset for a segmented hrit_jma
+
+#### Features added
+
+* [PR 2973](https://github.com/pytroll/satpy/pull/2973) - Remove flexparser pinning
+* [PR 2953](https://github.com/pytroll/satpy/pull/2953) - Add altitude, landcover, and landseamask to mersi_ll_l1b reader ([2952](https://github.com/pytroll/satpy/issues/2952))
+* [PR 2946](https://github.com/pytroll/satpy/pull/2946) - Update MODIS L1b reader with additional geoinfo datasets
+
+In this release 14 pull requests were closed.
+
+
## Version 0.52.1 (2024/10/23)
### Issues Closed
=====================================
satpy/enhancements/__init__.py
=====================================
@@ -219,11 +219,12 @@ def cira_stretch(img, **kwargs):
@exclude_alpha
def _cira_stretch(band_data):
- log_root = np.log10(0.0223)
+ dtype = band_data.dtype
+ log_root = np.log10(0.0223, dtype=dtype)
denom = (1.0 - log_root) * 0.75
band_data *= 0.01
band_data = band_data.clip(np.finfo(float).eps)
- band_data = np.log10(band_data)
+ band_data = np.log10(band_data, dtype=dtype)
band_data -= log_root
band_data /= denom
return band_data
=====================================
satpy/etc/composites/abi.yaml
=====================================
@@ -36,7 +36,7 @@ composites:
- name: C02
modifiers: [sunz_corrected, rayleigh_corrected_crefl]
- name: C03
- modifiers: [sunz_corrected, rayleigh_corrected_crefl]
+ modifiers: [sunz_corrected]
standard_name: toa_bidirectional_reflectance
green_raw:
@@ -710,7 +710,7 @@ composites:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: C03
- modifiers: [sunz_corrected, rayleigh_corrected]
+ modifiers: [sunz_corrected]
- name: C02
modifiers: [sunz_corrected, rayleigh_corrected]
- name: green
=====================================
satpy/etc/composites/ahi.yaml
=====================================
@@ -214,11 +214,11 @@ composites:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- wavelength: 1.63
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
- wavelength: 0.85
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
- wavelength: 0.635
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
high_resolution_band: blue
standard_name: natural_color
@@ -303,34 +303,6 @@ composites:
- name: B01
standard_name: true_color_reproduction_color_stretch
-# true_color_reducedsize_land:
-# compositor: !!python/name:satpy.composites.GenericCompositor
-# prerequisites:
-# - wavelength: 0.65
-# modifiers: [reducer4, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_land]
-# - wavelength: 0.51
-# modifiers: [reducer2, vegetation_corrected_reduced, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_land]
-# - wavelength: 0.46
-# modifiers: [reducer2, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_land]
-# standard_name: true_color
-#
-# true_color_reducedsize_marine_tropical:
-# compositor: !!python/name:satpy.composites.GenericCompositor
-# prerequisites:
-# - wavelength: 0.65
-# modifiers: [reducer4, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_marine_tropical]
-# - wavelength: 0.51
-# modifiers: [reducer2, vegetation_corrected_reduced, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_marine_tropical]
-# - wavelength: 0.46
-# modifiers: [reducer2, effective_solar_pathlength_corrected,
-# rayleigh_corrected_reducedsize_marine_tropical]
-# standard_name: true_color
-
day_microphysics_eum:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
=====================================
satpy/etc/composites/ami.yaml
=====================================
@@ -137,11 +137,11 @@ composites:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: NR016
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
- name: VI008
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
- name: VI006
- modifiers: [sunz_corrected] #, rayleigh_corrected]
+ modifiers: [sunz_corrected]
high_resolution_band: blue
standard_name: natural_color
=====================================
satpy/etc/composites/vii.yaml
=====================================
@@ -87,7 +87,6 @@ composites:
- compositor: !!python/name:satpy.composites.DifferenceCompositor
prerequisites:
- name: 'vii_3740'
- modifiers: [ co2_corrected ]
- name: 'vii_10690'
- compositor: !!python/name:satpy.composites.DifferenceCompositor
prerequisites:
=====================================
satpy/etc/composites/viirs.yaml
=====================================
@@ -232,7 +232,7 @@ composites:
modifiers: [sunz_corrected, rayleigh_corrected]
optional_prerequisites:
- name: I02
- modifiers: [sunz_corrected_iband, rayleigh_corrected_iband]
+ modifiers: [sunz_corrected_iband]
standard_name: false_color
high_resolution_band: green
@@ -405,7 +405,7 @@ composites:
- name: M07
modifiers: [sunz_corrected]
- name: M12
- modifiers: [nir_reflectance]
+ modifiers: [nir_reflectance_lowres]
- M15
standard_name: day_microphysics
=====================================
satpy/etc/readers/mersi_ll_l1b.yaml
=====================================
@@ -272,3 +272,27 @@ datasets:
coordinates: [longitude, latitude]
file_type: mersi_ll_l1b_1000_geo
file_key: Geolocation/MoonAzimuth
+ altitude:
+ name: altitude
+ units: degree
+ standard_name: altitude
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: mersi_ll_l1b_1000_geo
+ file_key: Geolocation/Altitude
+ landcover:
+ name: landcover
+ units: degree
+ standard_name: landcover
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: mersi_ll_l1b_1000_geo
+ file_key: Geolocation/LandCover
+ landseamask:
+ name: landseamask
+ units: degree
+ standard_name: landseamask
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: mersi_ll_l1b_1000_geo
+ file_key: Geolocation/LandSeaMask
=====================================
satpy/etc/readers/modis_l1b.yaml
=====================================
@@ -486,6 +486,34 @@ datasets:
coordinates: [longitude, latitude]
file_type: [hdf_eos_geo, hdf_eos_data_1000m]
+ landsea_mask:
+ name: landsea_mask
+ sensor: modis
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: [hdf_eos_geo]
+
+ height:
+ name: height
+ sensor: modis
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: [hdf_eos_geo]
+
+ range:
+ name: range
+ sensor: modis
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: [hdf_eos_geo]
+
+ waterpresent:
+ name: waterpresent
+ sensor: modis
+ resolution: 1000
+ coordinates: [longitude, latitude]
+ file_type: [hdf_eos_geo]
+
file_types:
hdf_eos_data_250m:
=====================================
satpy/modifiers/angles.py
=====================================
@@ -572,7 +572,6 @@ def sunzen_reduction(data: da.Array,
return da.map_blocks(_sunzen_reduction_ndarray, data, sunz, limit, max_sza, strength,
meta=np.array((), dtype=data.dtype), chunks=data.chunks)
-
def _sunzen_reduction_ndarray(data: np.ndarray,
sunz: np.ndarray,
limit: float,
@@ -584,7 +583,7 @@ def _sunzen_reduction_ndarray(data: np.ndarray,
# invert the reduction factor such that minimum reduction is done at `limit` and gradually increases towards max_sza
with np.errstate(invalid="ignore"): # we expect space pixels to be invalid
- reduction_factor = 1. - np.log(reduction_factor + 1) / np.log(2)
+ reduction_factor = 1. - np.log2(reduction_factor + 1)
# apply non-linearity to the reduction factor for a non-linear reduction of the signal. This can be used for a
# slower or faster transision to higher/lower fractions at the ndvi extremes. If strength equals 1.0, this
=====================================
satpy/modifiers/atmosphere.py
=====================================
@@ -77,7 +77,9 @@ class PSPRayleighReflectance(ModifierBase):
projectables = projectables + (optional_datasets or [])
if len(projectables) != 6:
vis, red = self.match_data_arrays(projectables)
- sata, satz, suna, sunz = get_angles(vis)
+ # Adjust the angle data precision to match the data
+ # This does not affect the accuracy visibly
+ sata, satz, suna, sunz = [d.astype(vis.dtype) for d in get_angles(vis)]
else:
vis, red, sata, satz, suna, sunz = self.match_data_arrays(projectables)
# First make sure the two azimuth angles are in the range 0-360:
@@ -97,7 +99,7 @@ class PSPRayleighReflectance(ModifierBase):
aerosol_type = self.attrs.get("aerosol_type", "marine_clean_aerosol")
reduce_lim_low = abs(self.attrs.get("reduce_lim_low", 70))
reduce_lim_high = abs(self.attrs.get("reduce_lim_high", 105))
- reduce_strength = np.clip(self.attrs.get("reduce_strength", 0), 0, 1)
+ reduce_strength = np.clip(self.attrs.get("reduce_strength", 0), 0, 1).astype(vis.dtype)
logger.info("Removing Rayleigh scattering with atmosphere '%s' and "
"aerosol type '%s' for '%s'",
=====================================
satpy/readers/hdfeos_base.py
=====================================
@@ -238,7 +238,7 @@ class HDFEOSBaseFileReader(BaseFileHandler):
return normalize_low_res_chunks(
(1,) * num_nonyx_dims + ("auto", -1),
var_shape,
- (1,) * num_nonyx_dims + (scan_length_250m, -1),
+ (1,) * num_nonyx_dims + (scan_length_250m, var_shape[-1]),
(1,) * num_nonyx_dims + (res_multiplier, res_multiplier),
np.float32,
)
@@ -333,6 +333,10 @@ class HDFEOSGeoReader(HDFEOSBaseFileReader):
"satellite_zenith_angle": ("SensorZenith", "Sensor_Zenith"),
"solar_azimuth_angle": ("SolarAzimuth", "SolarAzimuth"),
"solar_zenith_angle": ("SolarZenith", "Solar_Zenith"),
+ "water_present": "WaterPresent",
+ "landsea_mask": "Land/SeaMask",
+ "height": "Height",
+ "range": "Range",
}
def __init__(self, filename, filename_info, filetype_info, **kwargs):
=====================================
satpy/readers/hrit_jma.py
=====================================
@@ -345,8 +345,8 @@ class HRITJMAFileHandler(HRITFileHandler):
if self.is_segmented:
# loff in the file specifies the offset of the full disk image
# centre (1375/2750 for VIS/IR)
- segment_number = self.mda["segment_sequence_number"] - 1
- loff -= (self.mda["total_no_image_segm"] - segment_number - 1) * nlines
+ segment_number = int(self.mda["segment_sequence_number"]) - 1
+ loff -= (int(self.mda["total_no_image_segm"]) - segment_number - 1) * nlines
elif self.area_id in (NORTH_HEMIS, SOUTH_HEMIS):
# loff in the file specifies the start line of the half disk image
# in the full disk image
=====================================
satpy/readers/sar_c_safe.py
=====================================
@@ -511,15 +511,14 @@ def intp(grid_x, grid_y, interpolator):
def interpolate_xarray_linear(xpoints, ypoints, values, shape, chunks=CHUNK_SIZE):
"""Interpolate linearly, generating a dask array."""
- from scipy.interpolate.interpnd import LinearNDInterpolator, _ndim_coords_from_arrays
+ from scipy.interpolate.interpnd import LinearNDInterpolator
if isinstance(chunks, (list, tuple)):
vchunks, hchunks = chunks
else:
vchunks, hchunks = chunks, chunks
-
- points = _ndim_coords_from_arrays(np.vstack((np.asarray(ypoints, dtype=np.uint16),
- np.asarray(xpoints, dtype=np.uint16))).T)
+ points = np.vstack((np.asarray(ypoints, dtype=np.uint16),
+ np.asarray(xpoints, dtype=np.uint16))).T
interpolator = LinearNDInterpolator(points, values)
=====================================
satpy/readers/viirs_edr.py
=====================================
@@ -197,9 +197,12 @@ class VIIRSJRRFileHandler(BaseFileHandler):
platform_path = self.filename_info["platform_shortname"]
platform_dict = {"NPP": "Suomi-NPP",
"JPSS-1": "NOAA-20",
+ "SNPP": "Suomi-NPP",
"J01": "NOAA-20",
+ "N20": "NOAA-20",
"JPSS-2": "NOAA-21",
- "J02": "NOAA-21"}
+ "J02": "NOAA-21",
+ "N21": "NOAA-21"}
return platform_dict[platform_path.upper()]
def available_datasets(self, configured_datasets=None):
=====================================
satpy/tests/enhancement_tests/test_enhancements.py
=====================================
@@ -34,30 +34,55 @@ from satpy.enhancements import create_colormap, on_dask_array, on_separate_bands
def run_and_check_enhancement(func, data, expected, **kwargs):
"""Perform basic checks that apply to multiple tests."""
+ pre_attrs = data.attrs
+ img = _get_enhanced_image(func, data, **kwargs)
+
+ _assert_image(img, pre_attrs, func.__name__, "palettes" in kwargs)
+ _assert_image_data(img, expected)
+
+
+def _get_enhanced_image(func, data, **kwargs):
from trollimage.xrimage import XRImage
- pre_attrs = data.attrs
img = XRImage(data)
func(img, **kwargs)
+ return img
+
+
+def _assert_image(img, pre_attrs, func_name, has_palette):
+ assert isinstance(img.data, xr.DataArray)
assert isinstance(img.data.data, da.Array)
+
old_keys = set(pre_attrs.keys())
# It is OK to have "enhancement_history" added
new_keys = set(img.data.attrs.keys()) - {"enhancement_history"}
# In case of palettes are used, _FillValue is added.
# Colorize doesn't add the fill value, so ignore that
- if "palettes" in kwargs and func.__name__ != "colorize":
+ if has_palette and func_name != "colorize":
assert "_FillValue" in new_keys
# Remove it from further comparisons
new_keys = new_keys - {"_FillValue"}
assert old_keys == new_keys
- res_data_arr = img.data
- assert isinstance(res_data_arr, xr.DataArray)
- assert isinstance(res_data_arr.data, da.Array)
- res_data = res_data_arr.data.compute() # mimics what xrimage geotiff writing does
+
+def _assert_image_data(img, expected, dtype=None):
+ # Compute the data to mimic what xrimage geotiff writing does
+ res_data = img.data.data.compute()
assert not isinstance(res_data, da.Array)
np.testing.assert_allclose(res_data, expected, atol=1.e-6, rtol=0)
+ if dtype:
+ assert img.data.dtype == dtype
+ assert res_data.dtype == dtype
+
+
+def run_and_check_enhancement_with_dtype(func, data, expected, **kwargs):
+ """Perform basic checks that apply to multiple tests."""
+ pre_attrs = data.attrs
+ img = _get_enhanced_image(func, data, **kwargs)
+
+ _assert_image(img, pre_attrs, func.__name__, "palettes" in kwargs)
+ _assert_image_data(img, expected, dtype=data.dtype)
def identical_decorator(func):
@@ -109,14 +134,15 @@ class TestEnhancementStretch:
exp_data = exp_data[np.newaxis, :, :]
run_and_check_enhancement(_enh_func, in_data, exp_data)
- def test_cira_stretch(self):
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_cira_stretch(self, dtype):
"""Test applying the cira_stretch."""
from satpy.enhancements import cira_stretch
expected = np.array([[
[np.nan, -7.04045974, -7.04045974, 0.79630132, 0.95947296],
- [1.05181359, 1.11651012, 1.16635571, 1.20691137, 1.24110186]]])
- run_and_check_enhancement(cira_stretch, self.ch1, expected)
+ [1.05181359, 1.11651012, 1.16635571, 1.20691137, 1.24110186]]], dtype=dtype)
+ run_and_check_enhancement_with_dtype(cira_stretch, self.ch1.astype(dtype), expected)
def test_reinhard(self):
"""Test the reinhard algorithm."""
=====================================
satpy/tests/reader_tests/test_ahi_hrit.py
=====================================
@@ -75,8 +75,8 @@ class TestHRITJMAFileHandler(unittest.TestCase):
proj_h8 = b"GEOS(140.70) "
proj_mtsat2 = b"GEOS(145.00) "
proj_name = proj_h8 if platform == "Himawari-8" else proj_mtsat2
- return {"image_segm_seq_no": segno,
- "total_no_image_segm": numseg,
+ return {"image_segm_seq_no": np.uint8(segno),
+ "total_no_image_segm": np.uint8(numseg),
"projection_name": proj_name,
"projection_parameters": {
"a": 6378169.00,
@@ -85,10 +85,10 @@ class TestHRITJMAFileHandler(unittest.TestCase):
},
"cfac": 10233128,
"lfac": 10233128,
- "coff": coff,
- "loff": loff,
- "number_of_columns": ncols,
- "number_of_lines": nlines,
+ "coff": np.int32(coff),
+ "loff": np.int32(loff),
+ "number_of_columns": np.uint16(ncols),
+ "number_of_lines": np.uint16(nlines),
"image_data_function": idf,
"image_observation_time": self._get_acq_time(nlines)}
=====================================
satpy/tests/reader_tests/test_viirs_edr.py
=====================================
@@ -468,7 +468,8 @@ class TestVIIRSJRRReader:
[
("npp", "Suomi-NPP"),
("JPSS-1", "NOAA-20"),
- ("J01", "NOAA-20")
+ ("J01", "NOAA-20"),
+ ("n21", "NOAA-21")
])
def test_get_platformname(self, surface_reflectance_file, filename_platform, exp_shortname):
"""Test finding start and end times of granules."""
=====================================
satpy/tests/test_composites.py
=====================================
@@ -257,20 +257,28 @@ class TestRatioSharpenedCompositors:
with pytest.raises(ValueError, match="SelfSharpenedRGB requires at least one high resolution band, not 'None'"):
comp((self.ds1, self.ds2, self.ds3))
- def test_basic_no_high_res(self):
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_basic_no_high_res(self, dtype):
"""Test that three datasets can be passed without optional high res."""
from satpy.composites import RatioSharpenedRGB
comp = RatioSharpenedRGB(name="true_color")
- res = comp((self.ds1, self.ds2, self.ds3))
+ res = comp((self.ds1.astype(dtype), self.ds2.astype(dtype), self.ds3.astype(dtype)))
assert res.shape == (3, 2, 2)
+ assert res.dtype == dtype
+ assert res.values.dtype == dtype
- def test_basic_no_sharpen(self):
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_basic_no_sharpen(self, dtype):
"""Test that color None does no sharpening."""
from satpy.composites import RatioSharpenedRGB
comp = RatioSharpenedRGB(name="true_color", high_resolution_band=None)
- res = comp((self.ds1, self.ds2, self.ds3), optional_datasets=(self.ds4,))
+ res = comp((self.ds1.astype(dtype), self.ds2.astype(dtype), self.ds3.astype(dtype)),
+ optional_datasets=(self.ds4.astype(dtype),))
assert res.shape == (3, 2, 2)
+ assert res.dtype == dtype
+ assert res.values.dtype == dtype
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize(
("high_resolution_band", "neutral_resolution_band", "exp_r", "exp_g", "exp_b"),
[
@@ -300,22 +308,26 @@ class TestRatioSharpenedCompositors:
np.array([[1.0, 1.0], [np.nan, 1.0]], dtype=np.float64))
]
)
- def test_ratio_sharpening(self, high_resolution_band, neutral_resolution_band, exp_r, exp_g, exp_b):
+ def test_ratio_sharpening(self, high_resolution_band, neutral_resolution_band, exp_r, exp_g, exp_b, dtype):
"""Test RatioSharpenedRGB by different groups of high_resolution_band and neutral_resolution_band."""
from satpy.composites import RatioSharpenedRGB
comp = RatioSharpenedRGB(name="true_color", high_resolution_band=high_resolution_band,
neutral_resolution_band=neutral_resolution_band)
- res = comp((self.ds1, self.ds2, self.ds3), optional_datasets=(self.ds4,))
+ res = comp((self.ds1.astype(dtype), self.ds2.astype(dtype), self.ds3.astype(dtype)),
+ optional_datasets=(self.ds4.astype(dtype),))
assert "units" not in res.attrs
assert isinstance(res, xr.DataArray)
assert isinstance(res.data, da.Array)
+ assert res.dtype == dtype
data = res.values
np.testing.assert_allclose(data[0], exp_r, rtol=1e-5)
np.testing.assert_allclose(data[1], exp_g, rtol=1e-5)
np.testing.assert_allclose(data[2], exp_b, rtol=1e-5)
+ assert res.dtype == dtype
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize(
("exp_shape", "exp_r", "exp_g", "exp_b"),
[
@@ -325,17 +337,19 @@ class TestRatioSharpenedCompositors:
np.array([[16 / 3, 16 / 3], [16 / 3, 0]], dtype=np.float64))
]
)
- def test_self_sharpened_basic(self, exp_shape, exp_r, exp_g, exp_b):
+ def test_self_sharpened_basic(self, exp_shape, exp_r, exp_g, exp_b, dtype):
"""Test that three datasets can be passed without optional high res."""
from satpy.composites import SelfSharpenedRGB
comp = SelfSharpenedRGB(name="true_color")
- res = comp((self.ds1, self.ds2, self.ds3))
- data = res.values
+ res = comp((self.ds1.astype(dtype), self.ds2.astype(dtype), self.ds3.astype(dtype)))
+ assert res.dtype == dtype
+ data = res.values
assert data.shape == exp_shape
np.testing.assert_allclose(data[0], exp_r, rtol=1e-5)
np.testing.assert_allclose(data[1], exp_g, rtol=1e-5)
np.testing.assert_allclose(data[2], exp_b, rtol=1e-5)
+ assert data.dtype == dtype
class TestDifferenceCompositor(unittest.TestCase):
=====================================
satpy/tests/test_modifiers.py
=====================================
@@ -135,29 +135,46 @@ class TestSunZenithCorrector:
assert res.dtype == res_np.dtype
assert "y" not in res.coords
assert "x" not in res.coords
+ if as_32bit:
+ assert res.dtype == np.float32
- def test_basic_lims_not_provided(self, sunz_ds1):
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_basic_lims_not_provided(self, sunz_ds1, dtype):
"""Test custom limits when SZA isn't provided."""
from satpy.modifiers.geometry import SunZenithCorrector
comp = SunZenithCorrector(name="sza_test", modifiers=tuple(), correction_limit=90)
- res = comp((sunz_ds1,), test_attr="test")
- np.testing.assert_allclose(res.values, np.array([[66.853262, 68.168939], [66.30742, 67.601493]]))
-
+ res = comp((sunz_ds1.astype(dtype),), test_attr="test")
+ expected = np.array([[66.853262, 68.168939], [66.30742, 67.601493]], dtype=dtype)
+ values = res.values
+ np.testing.assert_allclose(values, expected, rtol=1e-5)
+ assert res.dtype == dtype
+ assert values.dtype == dtype
+
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
- def test_basic_default_provided(self, data_arr, sunz_sza):
+ def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
"""Test default limits when SZA is provided."""
from satpy.modifiers.geometry import SunZenithCorrector
comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
- res = comp((data_arr, sunz_sza), test_attr="test")
- np.testing.assert_allclose(res.values, np.array([[22.401667, 22.31777], [22.437503, 22.353533]]))
-
+ res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
+ expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
+ values = res.values
+ np.testing.assert_allclose(values, expected)
+ assert res.dtype == dtype
+ assert values.dtype == dtype
+
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
- def test_basic_lims_provided(self, data_arr, sunz_sza):
+ def test_basic_lims_provided(self, data_arr, sunz_sza, dtype):
"""Test custom limits when SZA is provided."""
from satpy.modifiers.geometry import SunZenithCorrector
comp = SunZenithCorrector(name="sza_test", modifiers=tuple(), correction_limit=90)
- res = comp((data_arr, sunz_sza), test_attr="test")
- np.testing.assert_allclose(res.values, np.array([[66.853262, 68.168939], [66.30742, 67.601493]]))
+ res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
+ expected = np.array([[66.853262, 68.168939], [66.30742, 67.601493]], dtype=dtype)
+ values = res.values
+ np.testing.assert_allclose(values, expected, rtol=1e-5)
+ assert res.dtype == dtype
+ assert values.dtype == dtype
def test_imcompatible_areas(self, sunz_ds2, sunz_sza):
"""Test sunz correction on incompatible areas."""
@@ -179,18 +196,28 @@ class TestSunZenithReducer:
cls.custom = SunZenithReducer(name="sza_reduction_test_custom", modifiers=tuple(),
correction_limit=70, max_sza=95, strength=3.0)
- def test_default_settings(self, sunz_ds1, sunz_sza):
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_default_settings(self, sunz_ds1, sunz_sza, dtype):
"""Test default settings with sza data available."""
- res = self.default((sunz_ds1, sunz_sza), test_attr="test")
- np.testing.assert_allclose(res.values,
- np.array([[0.02916261, 0.02839063], [0.02949383, 0.02871911]]),
- rtol=1e-5)
-
- def test_custom_settings(self, sunz_ds1, sunz_sza):
+ res = self.default((sunz_ds1.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
+ expected = np.array([[0.02916261, 0.02839063], [0.02949383, 0.02871911]], dtype=dtype)
+ assert res.dtype == dtype
+ values = res.values
+ assert values.dtype == dtype
+ np.testing.assert_allclose(values,
+ expected,
+ rtol=2e-5)
+
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
+ def test_custom_settings(self, sunz_ds1, sunz_sza, dtype):
"""Test custom settings with sza data available."""
- res = self.custom((sunz_ds1, sunz_sza), test_attr="test")
- np.testing.assert_allclose(res.values,
- np.array([[0.01041319, 0.01030033], [0.01046164, 0.01034834]]),
+ res = self.custom((sunz_ds1.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
+ expected = np.array([[0.01041319, 0.01030033], [0.01046164, 0.01034834]], dtype=dtype)
+ assert res.dtype == dtype
+ values = res.values
+ assert values.dtype == dtype
+ np.testing.assert_allclose(values,
+ expected,
rtol=1e-5)
def test_invalid_max_sza(self, sunz_ds1, sunz_sza):
@@ -502,6 +529,7 @@ class TestPSPRayleighReflectance:
})
return input_band, red_band, angle1, angle1, angle1, angle1
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize(
("name", "wavelength", "resolution", "aerosol_type", "reduce_lim_low", "reduce_lim_high", "reduce_strength",
"exp_mean", "exp_unique"),
@@ -521,7 +549,7 @@ class TestPSPRayleighReflectance:
]
)
def test_rayleigh_corrector(self, name, wavelength, resolution, aerosol_type, reduce_lim_low, reduce_lim_high,
- reduce_strength, exp_mean, exp_unique):
+ reduce_strength, exp_mean, exp_unique, dtype):
"""Test PSPRayleighReflectance with fake data."""
from satpy.modifiers.atmosphere import PSPRayleighReflectance
ray_cor = PSPRayleighReflectance(name=name, atmosphere="us-standard", aerosol_types=aerosol_type,
@@ -535,42 +563,48 @@ class TestPSPRayleighReflectance:
assert ray_cor.attrs["reduce_strength"] == reduce_strength
input_band, red_band, *_ = self._create_test_data(name, wavelength, resolution)
- res = ray_cor([input_band, red_band])
+ res = ray_cor([input_band.astype(dtype), red_band.astype(dtype)])
assert isinstance(res, xr.DataArray)
assert isinstance(res.data, da.Array)
+ assert res.dtype == dtype
data = res.values
unique = np.unique(data[~np.isnan(data)])
np.testing.assert_allclose(np.nanmean(data), exp_mean, rtol=1e-5)
assert data.shape == (3, 5)
np.testing.assert_allclose(unique, exp_unique, rtol=1e-5)
+ assert data.dtype == dtype
+ @pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize("as_optionals", [False, True])
- def test_rayleigh_with_angles(self, as_optionals):
+ def test_rayleigh_with_angles(self, as_optionals, dtype):
"""Test PSPRayleighReflectance with angles provided."""
from satpy.modifiers.atmosphere import PSPRayleighReflectance
aerosol_type = "rayleigh_only"
ray_cor = PSPRayleighReflectance(name="B01", atmosphere="us-standard", aerosol_types=aerosol_type)
- prereqs, opt_prereqs = self._get_angles_prereqs_and_opts(as_optionals)
+ prereqs, opt_prereqs = self._get_angles_prereqs_and_opts(as_optionals, dtype)
with mock.patch("satpy.modifiers.atmosphere.get_angles") as get_angles:
res = ray_cor(prereqs, opt_prereqs)
get_angles.assert_not_called()
assert isinstance(res, xr.DataArray)
assert isinstance(res.data, da.Array)
+ assert res.dtype == dtype
data = res.values
unique = np.unique(data[~np.isnan(data)])
np.testing.assert_allclose(unique, np.array([-75.0, -37.71298492, 31.14350754]), rtol=1e-5)
assert data.shape == (3, 5)
+ assert data.dtype == dtype
- def _get_angles_prereqs_and_opts(self, as_optionals):
+ def _get_angles_prereqs_and_opts(self, as_optionals, dtype):
wavelength = (0.45, 0.47, 0.49)
resolution = 1000
input_band, red_band, *angles = self._create_test_data("B01", wavelength, resolution)
- prereqs = [input_band, red_band]
+ prereqs = [input_band.astype(dtype), red_band.astype(dtype)]
opt_prereqs = []
+ angles = [a.astype(dtype) for a in angles]
if as_optionals:
opt_prereqs = angles
else:
=====================================
satpy/tests/writer_tests/test_cf.py
=====================================
@@ -528,12 +528,19 @@ class TestNetcdfEncodingKwargs:
assert f["test-array"].dtype == expected["dtype"]
assert f["test-array"].encoding["complevel"] == expected["complevel"]
- def test_warning_if_backends_dont_match(self, scene, filename, monkeypatch):
+ @pytest.mark.parametrize(
+ "versions",
+ [
+ {"netCDF4": "1.5.0", "libnetcdf": "4.9.1-development"},
+ {"netCDF4": "1.6.0", "libnetcdf": "invalid-version"}
+ ]
+ )
+ def test_warning_if_backends_dont_match(self, scene, filename, monkeypatch, versions):
"""Test warning if backends don't match."""
import netCDF4
with monkeypatch.context() as m:
- m.setattr(netCDF4, "__version__", "1.5.0")
- m.setattr(netCDF4, "__netcdf4libversion__", "4.9.1")
+ m.setattr(netCDF4, "__version__", versions["netCDF4"])
+ m.setattr(netCDF4, "__netcdf4libversion__", versions["libnetcdf"])
with pytest.warns(UserWarning, match=r"Backend version mismatch"):
scene.save_datasets(filename=filename, writer="cf")
=====================================
satpy/writers/cf_writer.py
=====================================
@@ -158,7 +158,7 @@ import warnings
import numpy as np
import xarray as xr
-from packaging.version import Version
+from packaging.version import InvalidVersion, Version
from satpy.cf.coords import EPOCH # noqa: F401 (for backward compatibility)
from satpy.writers import Writer
@@ -390,8 +390,26 @@ def _backend_versions_match():
def _get_backend_versions():
import netCDF4
+ libnetcdf_version = _parse_libnetcdf_version(
+ netCDF4.__netcdf4libversion__
+ )
return {
"netCDF4": Version(netCDF4.__version__),
- "libnetcdf": Version(netCDF4.__netcdf4libversion__),
+ "libnetcdf": libnetcdf_version,
"xarray": Version(xr.__version__)
}
+
+
+def _parse_libnetcdf_version(version_str):
+ # Make libnetcdf development version compatible with PEP440
+ version_str = version_str.replace("development", "dev")
+ try:
+ return Version(version_str)
+ except InvalidVersion:
+ warnings.warn(
+ f"Unable to parse netcdf-c version {version_str}, "
+ f"using 0.0.0 as fallback",
+ UserWarning,
+ stacklevel=3
+ )
+ return Version("0.0.0")
View it on GitLab: https://salsa.debian.org/debian-gis-team/satpy/-/commit/270deb06076375f3ba116d695e033ae480250d36
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/satpy/-/commit/270deb06076375f3ba116d695e033ae480250d36
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20241123/96b35af2/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list