[Git][debian-gis-team/pyspectral][master] 7 commits: New upstream version 0.12.0+ds
Antonio Valentino (@antonio.valentino)
gitlab at salsa.debian.org
Wed Oct 12 07:37:46 BST 2022
Antonio Valentino pushed to branch master at Debian GIS Project / pyspectral
Commits:
7ce84d3f by Antonio Valentino at 2022-10-12T05:43:45+00:00
New upstream version 0.12.0+ds
- - - - -
11dd8db4 by Antonio Valentino at 2022-10-12T05:43:48+00:00
Update upstream source from tag 'upstream/0.12.0+ds'
Update to upstream version '0.12.0+ds'
with Debian dir 9ba0d057e06f9bf9076d93c72327a8f12f9b1ac7
- - - - -
99a8d0f1 by Antonio Valentino at 2022-10-12T05:49:45+00:00
New upstream release
- - - - -
09ed2d2b by Antonio Valentino at 2022-10-12T05:50:27+00:00
Update d/copyright
- - - - -
382659ed by Antonio Valentino at 2022-10-12T05:52:35+00:00
Add build-dep on python3-responses
- - - - -
e8fa1fec by Antonio Valentino at 2022-10-12T06:09:17+00:00
Add autopkgtests configuration (d/tests)
- - - - -
5ff0c47b by Antonio Valentino at 2022-10-12T06:09:36+00:00
Set distribution to unstable
- - - - -
28 changed files:
- .pre-commit-config.yaml
- CHANGELOG.md
- + CHANGELOG_RSR_DATA.rst
- continuous_integration/environment.yaml
- debian/changelog
- debian/control
- debian/copyright
- + debian/tests/control
- + debian/tests/python3
- doc/rayleigh_correction.rst
- + pyproject.toml
- pyspectral/bandnames.py
- pyspectral/etc/pyspectral.yaml
- pyspectral/raw_reader.py
- pyspectral/rayleigh.py
- pyspectral/rsr_reader.py
- pyspectral/tests/test_rayleigh.py
- pyspectral/tests/test_rsr_reader.py
- pyspectral/tests/test_utils.py
- pyspectral/utils.py
- rsr_convert_scripts/README.rst
- rsr_convert_scripts/agri_rsr.py
- + rsr_convert_scripts/avhrr1_rsr.py
- rsr_convert_scripts/fci_rsr.py
- + rsr_convert_scripts/msu_gsa_reader.py
- rsr_convert_scripts/seviri_rsr.py
- rsr_convert_scripts/viirs_rsr.py
- setup.py
Changes:
=====================================
.pre-commit-config.yaml
=====================================
@@ -6,3 +6,11 @@ repos:
hooks:
- id: flake8
additional_dependencies: [flake8-docstrings, flake8-debugger, flake8-bugbear]
+- repo: https://github.com/pycqa/isort
+ rev: 5.10.1
+ hooks:
+ - id: isort
+ language_version: python3
+ci:
+ # To trigger manually, comment on a pull request with "pre-commit.ci autofix"
+ autofix_prs: false
=====================================
CHANGELOG.md
=====================================
@@ -1,3 +1,38 @@
+## Version <v0.12.0> (2022/10/11)
+
+### Issues Closed
+
+* [Issue 127](https://github.com/pytroll/pyspectral/issues/127) - AVHRR instrument naming - allow avhrr-# ([PR 136](https://github.com/pytroll/pyspectral/pull/136) by [@adybbroe](https://github.com/adybbroe))
+
+In this release 1 issue was closed.
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 159](https://github.com/pytroll/pyspectral/pull/159) - Revert np.nan_to_num usage on rayleigh correction array
+* [PR 158](https://github.com/pytroll/pyspectral/pull/158) - Fix RSR band name retrieval for unconfigured sensors
+* [PR 157](https://github.com/pytroll/pyspectral/pull/157) - Fix noaa6 rsr
+* [PR 153](https://github.com/pytroll/pyspectral/pull/153) - Fix a minor formatting bug in the atm-correction RTD pages
+* [PR 136](https://github.com/pytroll/pyspectral/pull/136) - Fix avhrr instrument naming ([127](https://github.com/pytroll/pyspectral/issues/127))
+
+#### Features added
+
+* [PR 156](https://github.com/pytroll/pyspectral/pull/156) - Add support for NOAA-21 VIIRS
+* [PR 152](https://github.com/pytroll/pyspectral/pull/152) - EPS-SG VII (METimage) channel names added
+* [PR 150](https://github.com/pytroll/pyspectral/pull/150) - Add AGRI bands to the BANDNAMES dict
+* [PR 148](https://github.com/pytroll/pyspectral/pull/148) - Updated the FCI spectral responses as of April 2022
+* [PR 145](https://github.com/pytroll/pyspectral/pull/145) - Add converter for Arctica-M N1 RSR data.
+* [PR 136](https://github.com/pytroll/pyspectral/pull/136) - Fix avhrr instrument naming ([127](https://github.com/pytroll/pyspectral/issues/127))
+
+#### Documentation changes
+
+* [PR 154](https://github.com/pytroll/pyspectral/pull/154) - Cleanup RSR and LUT download functions for clearer usage
+* [PR 153](https://github.com/pytroll/pyspectral/pull/153) - Fix a minor formatting bug in the atm-correction RTD pages
+
+In this release 13 pull requests were closed.
+
+
## Version <v0.11.0> (2022/03/01)
=====================================
CHANGELOG_RSR_DATA.rst
=====================================
@@ -0,0 +1,59 @@
+Changelog for the Relative Spectral Response data
+=================================================
+
+Version <v1.2.0> (Tue Sep 27 21:08:13 2022)
+-------------------------------------------
+
+ * Added VIIRS RSR for JPSS-2/NOAA-21:
+ Based on the J2_VIIRS_RSR_DAWG_At-Launch_Public_Release_V2_Jul2019.zip
+ The data read are the Detector wise files under J2_VIIRS_Detector_RSR_V2:
+
+ J2_VIIRS_RSR_DNBLGS_Detector_Fused_V2FS.txt
+ J2_VIIRS_RSR_I1_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_I2_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_I3_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_I4_Detector_V2F.txt
+ J2_VIIRS_RSR_I5_Detector_V2F.txt
+ J2_VIIRS_RSR_M10_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M11_Detector_V2F.txt
+ J2_VIIRS_RSR_M12_Detector_V2F.txt
+ J2_VIIRS_RSR_M13_Detector_V2F.txt
+ J2_VIIRS_RSR_M14_Detector_V2F.txt
+ J2_VIIRS_RSR_M15_Detector_V2F.txt
+ J2_VIIRS_RSR_M16A_Detector_V2F.txt
+ J2_VIIRS_RSR_M1_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M2_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M3_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M4_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M5_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M6_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M7_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M8_Detector_Fused_V2F.txt
+ J2_VIIRS_RSR_M9_Detector_WVCOR_Fused_V2F.txt
+
+ The two files here were not considered:
+ J2_VIIRS_RSR_DNBMGS_Detector_Fused_V2FS.txt
+ J2_VIIRS_RSR_M16B_Detector_V2F.txt
+
+ * Correted errors concerning NOAA-6 channel 1 and 2. These proved to be wrong
+ (a factor of 10 scale error), as the files they were based on were
+ wrong. For AVHRR-1 we have been using the ascii files from NOAA STAR
+ (https://www.star.nesdis.noaa.gov/smcd/spb/fwu/homepage/AVHRR/spec_resp_func/index.html). Example
+ of the start of the file for NOAA-6 channel-1:
+
+ %> cat NOAA_6_A103C001.txt
+ Wavelegth (nm) Normalized RSF
+ 5600.000000 0.071000
+ 5700.000000 0.449000
+ 5800.000000 0.739000
+ 5900.000000 0.813000
+ 6000.000000 0.806000
+ 6200.000000 0.919000
+ 6400.000000 1.000000
+ ...
+
+ Here we have instead used the xls files on which the NOAA-STAR files are based: AVHRR1_SRF_only.xls
+
+ For TIROS-N there seem to be a typo in the wavelengths array for channel-1
+ on TIROS-N: A 640 nm should most likely have been 840 nm. This has been
+ corrected in the hdf5 file.
=====================================
continuous_integration/environment.yaml
=====================================
@@ -8,13 +8,9 @@ dependencies:
- toolz
- Cython
- sphinx
- - cartopy
- - pillow
- matplotlib
- scipy
- pyyaml
- - pyproj
- - pyresample
- coveralls
- coverage
- codecov
@@ -22,22 +18,13 @@ dependencies:
- netcdf4
- h5py
- h5netcdf
- - imageio
- pyhdf
- mock
- - libtiff
- - geoviews
- - zarr
- - python-eccodes
- - geoviews
- pytest
- pytest-cov
- - fsspec
- - pylibtiff
+ - responses
- appdirs
- python-geotiepoints
- pip
- pip:
- trollsift
- - trollimage
- - pyorbital
\ No newline at end of file
=====================================
debian/changelog
=====================================
@@ -1,13 +1,17 @@
-pyspectral (0.11.0+ds-2) UNRELEASED; urgency=medium
+pyspectral (0.12.0+ds-1) unstable; urgency=medium
[ Bas Couwenberg ]
- * Team upload.
* Bump Standards-Version to 4.6.1, no changes.
[ Antonio Valentino ]
+ * New upstream release.
* Fix d/copyright formatting.
+ * Update d/copyright file.
+ * debian/control:
+ - add build-dependency on python3-responses.
+ * Add autopkgtests configuration (d/tests).
- -- Bas Couwenberg <sebastic at debian.org> Tue, 21 Jun 2022 07:19:24 +0200
+ -- Antonio Valentino <antonio.valentino at tiscali.it> Wed, 12 Oct 2022 06:09:28 +0000
pyspectral (0.11.0+ds-1) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -17,6 +17,7 @@ Build-Depends: debhelper-compat (= 12),
python3-numpy,
python3-pytest,
python3-requests,
+ python3-responses,
python3-trollsift,
python3-scipy,
python3-setuptools,
=====================================
debian/copyright
=====================================
@@ -6,12 +6,10 @@ Source: https://github.com/pytroll/pyspectral
unnecessary CI configuration files.
Also two broken Git LFS references are removed from the archive.
Files-Excluded: pyspectral/data/modis
- appveyor*
.gitchangelog.rc
.github
.landscape.yaml
.stickler.yml
- .travis.yml
doc/_static/mersi2_rsr_band_0040_0070.png
doc/_static/mersi2_rsr_band_0040_0070_missingbands.png
=====================================
debian/tests/control
=====================================
@@ -0,0 +1,2 @@
+Tests: python3
+Depends: @, @builddeps@, python3-all
=====================================
debian/tests/python3
=====================================
@@ -0,0 +1,12 @@
+#!/bin/sh
+set -efu
+
+PYS=${PYS:-"$(py3versions --supported 2>/dev/null)"}
+TESTPKG=${TESTPKG:-pyspectral}
+
+cd "$AUTOPKGTEST_TMP"
+
+for py in $PYS; do
+ echo "=== $py ==="
+ $py -m pytest --pyargs ${TESTPKG} -k "not test_rayleigh"
+done
=====================================
doc/rayleigh_correction.rst
=====================================
@@ -48,7 +48,7 @@ As the Rayleigh scattering, which is the dominating part we are correcting for
under normal situations (when there is no excessive pollution or aerosols in
the line of sight) is proportional to :math:`\frac{1}{{\lambda}^4}` the
effective wavelength is derived by convolution of the spectral response with
-:math:`\frac{1}{{\lambda}^4}`.
+:math:`\frac{1}{{\lambda}^4}`.
To get the atmospheric contribution for an arbitrary band, first the
solar zenith, satellite zenith and the sun-satellite azimuth difference angles
@@ -80,9 +80,9 @@ provide a more gentle scaling in cases of high reflectances (above 20%):
In case you want to bypass the reading of the sensor response functions or you have
a sensor for which there are no RSR data available in PySpectral it is still possible
-to derive an atmospheric correction for that band. All what is needed is the effective
-wavelength of the band, given in micrometers (`:math: \mu m`). This wavelength is
-normally calculated by PySPectral from the RSR data when passing the name of the band
+to derive an atmospheric correction for that band. All that is needed is the effective
+wavelength of the band, given in micrometers (:math:`\mu m`). This wavelength is
+normally calculated by PySpectral from the RSR data when passing the name of the band
as above.
>>> from pyspectral.rayleigh import Rayleigh
@@ -139,5 +139,3 @@ should not be used for any scientific analysis.
.. _Satpy: http://www.github.com/pytroll/satpy
.. _zenodo: https://doi.org/10.5281/zenodo.1288441
.. _`scientific paper`: https://doi.org/10.3390/rs10040560
-
-
=====================================
pyproject.toml
=====================================
@@ -0,0 +1,7 @@
+[tool.isort]
+sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]
+profile = "black"
+skip_gitignore = true
+default_section = "THIRDPARTY"
+known_first_party = "pyspectral"
+line_length = 120
=====================================
pyspectral/bandnames.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
-# Copyright (c) 2021 Adam.Dybbroe
+# Copyright (c) 2021, 2022 Adam.Dybbroe
# Author(s):
@@ -73,6 +73,7 @@ for chan_num in range(1, 37):
# MODIS RSR files were made before 'chX' became standard in pyspectral
BANDNAMES['modis'] = {str(chan_num): str(chan_num) for chan_num in range(1, 37)}
+
BANDNAMES['seviri'] = {'VIS006': 'VIS0.6',
'VIS008': 'VIS0.8',
'IR_016': 'NIR1.6',
@@ -127,6 +128,22 @@ BANDNAMES['abi'] = {'C01': 'ch1',
'C16': 'ch16'
}
+BANDNAMES['agri'] = {'C01': 'ch1',
+ 'C02': 'ch2',
+ 'C03': 'ch3',
+ 'C04': 'ch4',
+ 'C05': 'ch5',
+ 'C06': 'ch6',
+ 'C07': 'ch7',
+ 'C08': 'ch8',
+ 'C09': 'ch9',
+ 'C10': 'ch10',
+ 'C11': 'ch11',
+ 'C12': 'ch12',
+ 'C13': 'ch13',
+ 'C14': 'ch14',
+ }
+
BANDNAMES['ahi'] = {'B01': 'ch1',
'B02': 'ch2',
'B03': 'ch3',
@@ -163,22 +180,22 @@ BANDNAMES['ami'] = {'VI004': 'ch1',
'IR133': 'ch16'
}
-BANDNAMES['fci'] = {'vis_04': 'ch1',
- 'vis_05': 'ch2',
- 'vis_06': 'ch3',
- 'vis_08': 'ch4',
- 'vis_09': 'ch5',
- 'nir_13': 'ch6',
- 'nir_16': 'ch7',
- 'nir_22': 'ch8',
- 'ir_38': 'ch9',
- 'wv_63': 'ch10',
- 'wv_73': 'ch11',
- 'ir_87': 'ch12',
- 'ir_97': 'ch13',
- 'ir_105': 'ch14',
- 'ir_123': 'ch15',
- 'ir_133': 'ch16'
+BANDNAMES['fci'] = {'vis_04': 'VIS0.4',
+ 'vis_05': 'VIS0.5',
+ 'vis_06': 'VIS0.6_HR',
+ 'vis_08': 'VIS0.8',
+ 'vis_09': 'VIS0.9',
+ 'nir_13': 'NIR1.3',
+ 'nir_16': 'NIR1.6',
+ 'nir_22': 'NIR2.2_HR',
+ 'ir_38': 'IR3.8_HR',
+ 'wv_63': 'WV6.3',
+ 'wv_73': 'WV7.3',
+ 'ir_87': 'IR8.7',
+ 'ir_97': 'IR9.7',
+ 'ir_105': 'IR10.5_HR',
+ 'ir_123': 'IR12.3',
+ 'ir_133': 'IR13.3'
}
BANDNAMES['slstr'] = {'S1': 'ch1',
@@ -193,3 +210,25 @@ BANDNAMES['slstr'] = {'S1': 'ch1',
'F1': 'ch7',
'F2': 'ch8',
}
+
+BANDNAMES['VII'] = {'vii_443': 'ch1',
+ 'vii_555': 'ch2',
+ 'vii_668': 'ch3',
+ 'vii_752': 'ch4',
+ 'vii_763': 'ch5',
+ 'vii_865': 'ch6',
+ 'vii_914': 'ch7',
+ 'vii_1240': 'ch8',
+ 'vii_1375': 'ch9',
+ 'vii_1640': 'ch10',
+ 'vii_2250': 'ch11',
+ 'vii_3740': 'ch12',
+ 'vii_3959': 'ch13',
+ 'vii_4050': 'ch14',
+ 'vii_6725': 'ch15',
+ 'vii_7325': 'ch16',
+ 'vii_8540': 'ch17',
+ 'vii_10690': 'ch18',
+ 'vii_12020': 'ch19',
+ 'vii_13345': 'ch20',
+ }
=====================================
pyspectral/etc/pyspectral.yaml
=====================================
@@ -146,6 +146,8 @@ download_from_internet: True
# Sentinel-3A-olci:
# path: /path/to/original/sentinel-3a/olci/data
+# Arctica-M-N1-msu-gsa:
+# path: /path/to/original/Arctica_M_N1_SRF.xlsx
# Sentinel-2A-msi:
# path: /path/to/original/sentinel-2a/msi/data/S2-SRF_COPE-GSEG-EOPG-TN-15-0007_3.0.xlsx
=====================================
pyspectral/raw_reader.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2016-2019 Pytroll developers
+# Copyright (c) 2016-2022 Pytroll developers
#
# Author(s):
#
@@ -25,6 +25,7 @@
import os
import logging
from pyspectral.config import get_config
+from pathlib import Path
LOG = logging.getLogger(__name__)
@@ -48,14 +49,15 @@ class InstrumentRSR(object):
self.output_dir = None
self.path = None
+ self.filename = None
self.options = {}
- self.requested_band_filename = None
def _get_options_from_config(self):
"""Get configuration settings from configuration file."""
options = get_config()
self.output_dir = options.get('rsr_dir', './')
- self.path = options[self.platform_name + '-' + self.instrument]['path']
+ self.path = Path(options[self.platform_name + '-' + self.instrument]['path'])
+ self.filename = options[self.platform_name + '-' + self.instrument].get('filename')
self.options = options
def _get_bandfilenames(self):
=====================================
pyspectral/rayleigh.py
=====================================
@@ -267,7 +267,7 @@ class Rayleigh(RayleighConfigBaseClass):
dtype=res.dtype,
chunks=getattr(res, "chunks", None))
- res = np.clip(np.nan_to_num(res), 0, 100)
+ res = np.clip(res, 0, 100)
if compute:
res = res.compute()
return res
@@ -349,22 +349,23 @@ def get_reflectance_lut_from_file(lut_filename):
return azidiff, satellite_zenith_secant, sun_zenith_secant
-def check_and_download(**kwargs):
- """
- Download atm correction LUT tables if they are not up to date already.
+def check_and_download(dry_run=False, aerosol_types=None):
+ """Download atm correction LUT tables if they are not up-to-date already.
Do a check for the version of the atmospheric correction LUTs and attempt
downloading only if needed.
"""
- aerosol_types = kwargs.get('aerosol_types', AEROSOL_TYPES)
- dry_run = kwargs.get('dry_run', False)
-
+ aerosol_types = aerosol_types or AEROSOL_TYPES
+ needed_aerosol_types = []
for aerosol_type in aerosol_types:
atmcorr = RayleighConfigBaseClass(aerosol_type)
if atmcorr.lutfiles_version_uptodate:
LOG.info("Atm correction LUTs, for aerosol distribution %s, already the latest!",
aerosol_type)
else:
- # Download
- download_luts(aerosol_type=aerosol_type, dry_run=dry_run)
+ needed_aerosol_types.append(aerosol_type)
+
+ # Download
+ if needed_aerosol_types:
+ download_luts(aerosol_types=needed_aerosol_types, dry_run=dry_run)
=====================================
pyspectral/rsr_reader.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2014-2020 Pytroll developers
+# Copyright (c) 2014-2022 Pytroll developers
#
# Author(s):
#
@@ -23,21 +23,20 @@
"""Reading the spectral responses in the internal pyspectral hdf5 format."""
+import logging
import os
-import numpy as np
from glob import glob
from os.path import expanduser
-from pyspectral.config import get_config
-from pyspectral.utils import WAVE_NUMBER
-from pyspectral.utils import WAVE_LENGTH
-from pyspectral.utils import BANDNAMES
-from pyspectral.utils import (INSTRUMENTS, download_rsr)
-from pyspectral.utils import (RSR_DATA_VERSION_FILENAME, RSR_DATA_VERSION)
-from pyspectral.utils import (convert2wavenumber, get_central_wave)
-from pyspectral.utils import convert2str
+import numpy as np
+
+from pyspectral.config import get_config
+from pyspectral.utils import (BANDNAMES, INSTRUMENTS, RSR_DATA_VERSION,
+ RSR_DATA_VERSION_FILENAME, WAVE_LENGTH,
+ WAVE_NUMBER, check_and_adjust_instrument_name,
+ convert2str, convert2wavenumber, download_rsr,
+ get_central_wave)
-import logging
LOG = logging.getLogger(__name__)
OSCAR_PLATFORM_NAMES = {'eos-2': 'EOS-Aqua',
@@ -48,15 +47,19 @@ OSCAR_PLATFORM_NAMES = {'eos-2': 'EOS-Aqua',
class RSRDict(dict):
+ """Helper dict-like class to handle multiple names for band keys."""
+
def __init__(self, instrument=None):
+ """Initialize dict and primary instrument name."""
self.instrument = instrument
dict.__init__(self)
def __getitem__(self, key):
+ """Get value either directly or fallback to pre-configured 'standard' names.."""
try:
val = dict.__getitem__(self, key)
except KeyError:
- if key in BANDNAMES[self.instrument]:
+ if self.instrument in BANDNAMES and key in BANDNAMES[self.instrument]:
val = dict.__getitem__(self, BANDNAMES[self.instrument][key])
elif key in BANDNAMES['generic']:
val = dict.__getitem__(self, BANDNAMES['generic'][key])
@@ -112,7 +115,7 @@ class RSRDataBaseClass(object):
class RelativeSpectralResponse(RSRDataBaseClass):
"""Container for the relative spectral response functions for various satellite imagers."""
- def __init__(self, platform_name=None, instrument=None, **kwargs):
+ def __init__(self, platform_name=None, instrument=None, filename=None):
"""Initialize the class instance.
Create the instance either from platform name and instrument or from
@@ -123,32 +126,22 @@ class RelativeSpectralResponse(RSRDataBaseClass):
self.platform_name = platform_name
self.instrument = instrument
- self.filename = None
- self.rsr.instrument = self.instrument
- if not self.instrument or not self.platform_name:
- if 'filename' in kwargs:
- self.filename = kwargs['filename']
- else:
+ self.filename = filename
+ if not self.filename:
+ if not self.instrument or not self.platform_name:
raise AttributeError(
- "platform name and sensor or filename must be specified")
- else:
- self._check_instrument()
+ "Either platform name and sensor, or filename, must be specified")
- if not self.filename:
+ self._check_instrument()
self._get_filename()
self._check_filename_exist()
self.load()
+ self.rsr.instrument = self.instrument
def _check_instrument(self):
"""Check and try fix instrument name if needed."""
- instr = INSTRUMENTS.get(self.platform_name, self.instrument.lower())
- if instr != self.instrument.lower():
- self.instrument = instr
- LOG.warning("Inconsistent instrument/satellite input - " +
- "instrument set to %s", self.instrument)
-
- self.instrument = self.instrument.lower().replace('/', '')
+ self.instrument = check_and_adjust_instrument_name(self.platform_name, self.instrument)
def _get_filename(self):
"""Get the rsr filname from platform and instrument names, and download if not available."""
@@ -178,7 +171,6 @@ class RelativeSpectralResponse(RSRDataBaseClass):
def _check_filename_exist(self):
"""Check that the file exist."""
-
if not os.path.exists(self.filename) or not os.path.isfile(self.filename):
errmsg = ('pyspectral RSR file does not exist! Filename = ' +
str(self.filename))
@@ -334,26 +326,15 @@ class RelativeSpectralResponse(RSRDataBaseClass):
self.set_band_central_wavelength_per_detector(h5f, bandname, dname)
-def check_and_download(**kwargs):
+def check_and_download(dest_dir=None, dry_run=False):
"""Do a check for the version and attempt downloading only if needed."""
- dry_run = kwargs.get('dry_run', False)
- dest_dir = kwargs.get('dest_dir', None)
-
rsr = RSRDataBaseClass()
if rsr.rsr_data_version_uptodate:
LOG.info("RSR data already the latest!")
else:
- if dest_dir:
- download_rsr(dest_dir=dest_dir, dry_run=dry_run)
- else:
- download_rsr(dry_run=dry_run)
-
-
-def main():
- """Main."""
- modis = RelativeSpectralResponse('EOS-Terra', 'modis')
- del(modis)
+ download_rsr(dest_dir=dest_dir, dry_run=dry_run)
if __name__ == "__main__":
- main()
+ modis = RelativeSpectralResponse('EOS-Terra', 'modis')
+ del modis
=====================================
pyspectral/tests/test_rayleigh.py
=====================================
@@ -23,11 +23,13 @@
"""Unittest for the rayleigh correction utilities."""
import contextlib
import os
+import unittest
+
import numpy as np
import dask.array as da
import h5py
import pytest
-from pyspectral import rayleigh
+from pyspectral import rayleigh, utils
from pyspectral.tests.data import (
TEST_RAYLEIGH_LUT,
TEST_RAYLEIGH_AZID_COORD,
@@ -42,6 +44,7 @@ TEST_RAYLEIGH_RESULT1 = np.array([10.339923, 8.64748], dtype='float32')
TEST_RAYLEIGH_RESULT2 = np.array([9.653559, 8.464997], dtype='float32')
TEST_RAYLEIGH_RESULT3 = np.array([5.488735, 8.533125], dtype='float32')
TEST_RAYLEIGH_RESULT4 = np.array([0.0, 8.64748], dtype='float32')
+TEST_RAYLEIGH_RESULT5 = np.array([9.653559, np.nan], dtype='float32')
TEST_RAYLEIGH_RESULT_R1 = np.array([16.66666667, 20.83333333, 25.], dtype='float32')
TEST_RAYLEIGH_RESULT_R2 = np.array([0., 6.25, 12.5], dtype='float32')
@@ -313,28 +316,25 @@ class TestRayleigh:
np.testing.assert_allclose(refl_corr2, refl_corr3)
@patch('pyspectral.rayleigh.da', None)
- def test_get_reflectance(self, fake_lut_hdf5):
+ @pytest.mark.parametrize(
+ ("sun_zenith", "sat_zenith", "azidiff", "redband_refl", "exp_result"),
+ [
+ (np.array([67., 32.]), np.array([45., 18.]), np.array([150., 110.]), np.array([14., 5.]),
+ TEST_RAYLEIGH_RESULT1),
+ (np.array([60., 20.]), np.array([49., 26.]), np.array([140., 130.]), np.array([12., 8.]),
+ TEST_RAYLEIGH_RESULT2),
+ (np.array([60., 20.]), np.array([49., 26.]), np.array([140., 130.]), np.array([12., np.nan]),
+ TEST_RAYLEIGH_RESULT5),
+ ]
+ )
+ def test_get_reflectance(self, fake_lut_hdf5, sun_zenith, sat_zenith, azidiff, redband_refl, exp_result):
"""Test getting the reflectance correction."""
- sun_zenith = np.array([67., 32.])
- sat_zenith = np.array([45., 18.])
- azidiff = np.array([150., 110.])
- redband_refl = np.array([14., 5.])
rayl = _create_rayleigh()
with mocked_rsr():
refl_corr = rayl.get_reflectance(
sun_zenith, sat_zenith, azidiff, 'ch3', redband_refl)
- np.testing.assert_allclose(refl_corr, TEST_RAYLEIGH_RESULT1)
-
- sun_zenith = np.array([60., 20.])
- sat_zenith = np.array([49., 26.])
- azidiff = np.array([140., 130.])
- redband_refl = np.array([12., 8.])
- with mocked_rsr():
- refl_corr = rayl.get_reflectance(
- sun_zenith, sat_zenith, azidiff, 'ch3', redband_refl)
-
assert isinstance(refl_corr, np.ndarray)
- np.testing.assert_allclose(refl_corr, TEST_RAYLEIGH_RESULT2)
+ np.testing.assert_allclose(refl_corr, exp_result)
@patch('pyspectral.rayleigh.da', None)
def test_get_reflectance_no_rsr(self, fake_lut_hdf5):
@@ -389,3 +389,50 @@ class TestRayleigh:
# test LUT doesn't have a subartic_summer file
with pytest.raises(IOError):
rayleigh.Rayleigh('UFO', 'unknown', atmosphere='subarctic summer')
+
+
+ at pytest.mark.parametrize(
+ ("version", "exp_download"),
+ [
+ (None, False),
+ ("v0.0.0", True),
+ ],
+)
+def test_check_and_download(tmp_path, version, exp_download):
+ """Test that check_and_download only downloads when necessary."""
+ from pyspectral.rayleigh import check_and_download
+ with _fake_lut_dir(tmp_path, version), unittest.mock.patch("pyspectral.rayleigh.download_luts") as download:
+ check_and_download()
+ if exp_download:
+ download.assert_called()
+ else:
+ download.assert_not_called()
+
+
+ at contextlib.contextmanager
+def _fake_lut_dir(tmp_path, lut_version):
+ with _fake_get_config(tmp_path):
+ for aerosol_type in utils.AEROSOL_TYPES:
+ atype_version_fn = utils.ATM_CORRECTION_LUT_VERSION[aerosol_type]["filename"]
+ atype_version = utils.ATM_CORRECTION_LUT_VERSION[aerosol_type]["version"]
+ atype_subdir = tmp_path / aerosol_type
+ atype_subdir.mkdir()
+ version_filename = str(atype_subdir / atype_version_fn)
+ with open(version_filename, "w") as version_file:
+ version_file.write(lut_version or atype_version)
+ yield
+
+
+ at contextlib.contextmanager
+def _fake_get_config(tmp_path):
+ def _get_config():
+ return {
+ "rayleigh_dir": str(tmp_path),
+ "rsr_dir": str(tmp_path),
+ "download_from_internet": True,
+ }
+ with unittest.mock.patch("pyspectral.rayleigh.get_config") as get_config, \
+ unittest.mock.patch("pyspectral.utils.get_config") as get_config2:
+ get_config.side_effect = _get_config
+ get_config2.side_effect = _get_config
+ yield
=====================================
pyspectral/tests/test_rsr_reader.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2017-2021 Pytroll developers
+# Copyright (c) 2017-2022 Pytroll developers
#
# Author(s):
#
@@ -22,18 +22,19 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Unit testing the generic rsr hdf5 reader."""
-
+import contextlib
import os.path
-import numpy as np
-import xarray as xr
import unittest
from unittest.mock import patch
+
+import numpy as np
import pytest
+import xarray as xr
from pyspectral.rsr_reader import RelativeSpectralResponse, RSRDict
-from pyspectral.utils import WAVE_NUMBER
-from pyspectral.utils import RSR_DATA_VERSION
from pyspectral.tests.unittest_helpers import assertNumpyArraysEqual
+from pyspectral.utils import (RSR_DATA_VERSION, RSR_DATA_VERSION_FILENAME,
+ WAVE_NUMBER)
TEST_RSR = {'20': {}, }
TEST_RSR['20']['det-1'] = {}
@@ -87,8 +88,12 @@ class TestRsrReader(unittest.TestCase):
@patch('pyspectral.rsr_reader.RelativeSpectralResponse.load')
@patch('pyspectral.rsr_reader.download_rsr')
@patch('pyspectral.rsr_reader.RelativeSpectralResponse._get_rsr_data_version')
- def test_rsr_response(self, get_rsr_version, download_rsr, load, isfile, exists):
- """Test the RelativeSpectralResponse class initialisation."""
+ def test_get_rsr_from_platform_name_and_instrument(self, get_rsr_version,
+ download_rsr, load, isfile, exists):
+ """Test the RelativeSpectralResponse class initialisation.
+
+ Test getting the rsr from platform_name and instrument.
+ """
load.return_code = None
download_rsr.return_code = None
isfile.return_code = True
@@ -100,6 +105,7 @@ class TestRsrReader(unittest.TestCase):
test_rsr = RelativeSpectralResponse('GOES-16')
test_rsr = RelativeSpectralResponse(instrument='ABI')
+ with patch('pyspectral.rsr_reader.get_config', return_value=TEST_CONFIG):
test_rsr = RelativeSpectralResponse('GOES-16', 'AbI')
self.assertEqual(test_rsr.platform_name, 'GOES-16')
self.assertEqual(test_rsr.instrument, 'abi')
@@ -114,6 +120,22 @@ class TestRsrReader(unittest.TestCase):
self.assertEqual(
test_rsr.filename, os.path.join(TEST_RSR_DIR, 'rsr_abi_GOES-16.h5'))
+ @patch('os.path.exists')
+ @patch('os.path.isfile')
+ @patch('pyspectral.rsr_reader.RelativeSpectralResponse.load')
+ @patch('pyspectral.rsr_reader.download_rsr')
+ @patch('pyspectral.rsr_reader.RelativeSpectralResponse._get_rsr_data_version')
+ def test_get_rsr_from_filename(self, get_rsr_version, download_rsr, load, isfile, exists):
+ """Test the RelativeSpectralResponse class initialisation.
+
+ Test getting the rsr from filename only.
+ """
+ load.return_code = None
+ download_rsr.return_code = None
+ isfile.return_code = True
+ exists.return_code = True
+ get_rsr_version.return_code = RSR_DATA_VERSION
+
with patch('pyspectral.rsr_reader.get_config', return_value=TEST_CONFIG):
test_rsr = RelativeSpectralResponse(
filename=os.path.join(TEST_RSR_DIR, 'rsr_abi_GOES-16.h5'))
@@ -241,7 +263,7 @@ class TestPopulateRSRObject(unittest.TestCase):
_ = RelativeSpectralResponse('MyPlatform')
exception_raised = exec_info.value
- expected_value = 'platform name and sensor or filename must be specified'
+ expected_value = 'Either platform name and sensor, or filename, must be specified'
self.assertEqual(str(exception_raised), expected_value)
@patch('pyspectral.rsr_reader.RelativeSpectralResponse.load')
@@ -449,3 +471,49 @@ class TestPopulateRSRObject(unittest.TestCase):
# Check exception raised if incorrect band name given
with self.assertRaises(KeyError):
print('d', test_rsr['VIS030'])
+
+ def test_rsr_unconfigured_sensor(self):
+ """Test RSRDict finds generic band conversions when specific sensor is not configured."""
+ test_rsr = RSRDict(instrument="i dont exist")
+ test_rsr["ch1"] = 2
+ assert test_rsr['1'] == 2
+
+
+ at pytest.mark.parametrize(
+ ("version", "exp_download"),
+ [
+ (RSR_DATA_VERSION, False),
+ ("v1.0.0", True),
+ ],
+)
+def test_check_and_download(tmp_path, version, exp_download):
+ """Test that check_and_download only downloads when necessary."""
+ from pyspectral.rsr_reader import check_and_download
+ with _fake_rsr_dir(tmp_path, version), unittest.mock.patch("pyspectral.rsr_reader.download_rsr") as download:
+ check_and_download()
+ if exp_download:
+ download.assert_called()
+ else:
+ download.assert_not_called()
+
+
+ at contextlib.contextmanager
+def _fake_rsr_dir(tmp_path, rsr_version):
+ with _fake_get_config(tmp_path):
+ version_filename = str(tmp_path / RSR_DATA_VERSION_FILENAME)
+ with open(version_filename, "w") as version_file:
+ version_file.write(rsr_version)
+ yield
+
+
+ at contextlib.contextmanager
+def _fake_get_config(tmp_path):
+ def _get_config():
+ return {
+ "rayleigh_dir": str(tmp_path),
+ "rsr_dir": str(tmp_path),
+ "download_from_internet": True,
+ }
+ with unittest.mock.patch("pyspectral.rsr_reader.get_config") as get_config:
+ get_config.side_effect = _get_config
+ yield
=====================================
pyspectral/tests/test_utils.py
=====================================
@@ -1,7 +1,8 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2014-2021 Pytroll developers
+# Copyright (c) 2014-2022 Pytroll developers
+
#
# Author(s):
#
@@ -22,16 +23,25 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Do the unit testing for the utils library."""
-
-import pytest
+import contextlib
+import logging
+import os
+import re
+import tarfile
+import tempfile
import unittest
+import warnings
+from io import BytesIO
+
import numpy as np
+import pytest
+import responses
from pyspectral import utils
-from pyspectral.utils import np2str, bytes2string
-
+from pyspectral.utils import bytes2string, check_and_adjust_instrument_name, np2str
TEST_RSR = {'20': {}, }
+TEST_RSR2 = {'20': {}, }
TEST_RSR['20']['det-1'] = {}
TEST_RSR['20']['det-1']['central_wavelength'] = 3.75
TEST_RSR['20']['det-1']['wavelength'] = np.array([
@@ -52,6 +62,8 @@ TEST_RSR['20']['det-1']['response'] = np.array([
1., 0.92676, 0.67429, 0.44715, 0.27762, 0.14852,
0.07141, 0.04151, 0.02925, 0.02085, 0.01414, 0.01], dtype='float32')
+TEST_RSR2['20'] = TEST_RSR['20']['det-1']
+
RESULT_RSR = {'20': {}}
RESULT_RSR['20']['det-1'] = {}
RESULT_RSR['20']['det-1']['wavenumber'] = np.array([
@@ -76,6 +88,20 @@ RESULT_RSR['20']['det-1']['response'] = np.array([
dtype='float32')
+class _MockedRSR:
+ def __init__(self, usedet=True):
+ self.instrument = 'test_sensor'
+ if usedet:
+ self.tmp_rsr = TEST_RSR
+ else:
+ self.tmp_rsr = TEST_RSR2
+ self.output_dir = tempfile.mkdtemp()
+
+ def __call__(self, chan, plat):
+ self.rsr = self.tmp_rsr[chan]
+ return self
+
+
class RsrTestData(object):
"""Container for the RSR test datasets."""
@@ -128,6 +154,26 @@ class TestUtils(unittest.TestCase):
wvn = newrsr['20']['det-1']['wavenumber']
self.assertTrue(np.allclose(wvn_res, wvn))
+ def test_convert2hdf5(self):
+ """Test the conversion utility from original RSR to HDF5."""
+ mocked_rsr_nodet = _MockedRSR(usedet=False)
+ utils.convert2hdf5(mocked_rsr_nodet, 'Test_SAT', ['20'])
+ fname = f'{mocked_rsr_nodet.output_dir}/rsr_test_sensor_Test_SAT.h5'
+ self.assertTrue(os.path.exists(fname))
+ try:
+ os.remove(fname)
+ except OSError:
+ pass
+
+ mocked_rsr_det = _MockedRSR(usedet=True)
+ utils.convert2hdf5(mocked_rsr_det, 'Test_SAT', ['20'], detectors=['det-1'])
+ fname = f'{mocked_rsr_det.output_dir}/rsr_test_sensor_Test_SAT.h5'
+ self.assertTrue(os.path.exists(fname))
+ try:
+ os.remove(fname)
+ except OSError:
+ pass
+
def test_get_bandname_from_wavelength(self):
"""Test the right bandname is found provided the wavelength in micro meters."""
bname = utils.get_bandname_from_wavelength('abi', 0.4, self.rsr.rsr)
@@ -181,74 +227,196 @@ class TestUtils(unittest.TestCase):
np.testing.assert_allclose(wvl_range, expected_range)
-def test_np2str_byte_object():
- """Test the np2str function on a byte object."""
- # byte object
- npstring = np.string_('hey')
- assert np2str(npstring) == 'hey'
-
-
-def test_np2str_single_element_array():
- """Test the np2str function."""
- # single element numpy array
- npstring = np.string_('Hej')
- np_arr = np.array([npstring])
- assert np2str(np_arr) == 'Hej'
-
-
-def test_np2str_single_element_scalar():
- """Test the np2str function on a scalar."""
- # scalar numpy array
- npstring = np.string_('hej')
- np_arr = np.array(npstring)
- assert np2str(np_arr) == 'hej'
-
-
-def test_np2str_multi_element():
- """Test the np2str function on a multi-element array."""
- # multi-element array
- npstring = np.string_('hej')
- npstring = np.array([npstring, npstring])
- with pytest.raises(ValueError):
- _ = np2str(npstring)
-
-
-def test_np2str_scalar():
- """Test the np2str function inputting a scalar value."""
- # non-array-non-string
- with pytest.raises(ValueError):
- _ = np2str(5)
-
-
-def test_np2str_pure_string():
- """Test the np2str function inputting a pure string."""
- # pure string
- pure_str = 'HEJ'
- assert np2str(pure_str) is pure_str
-
-
-def test_bytes2string_bytes_string():
- """Test the bytes2string function inputting a bytes string."""
- # bytes string
- pure_str = b'Hello'
- assert bytes2string(pure_str) == 'Hello'
-
-
-def test_bytes2string_pure_string():
- """Test the bytes2string function inputting a pure string."""
- # pure string
- pure_str = 'Hello'
- assert bytes2string(pure_str) == 'Hello'
-
-
-def test_bytes2string_numpy_string():
- """Test the bytes2string function inputting numpy string."""
- npstring = np.string_('HELLO')
- assert bytes2string(npstring) == 'HELLO'
-
-
-def test_bytes2string_numpy_string_array():
- """Test the bytes2string function inputting numpy string array."""
- npstring = np.string_('HELLO')
- np_arr = np.array(npstring)
- assert bytes2string(np_arr) == np_arr
+ at pytest.mark.parametrize(
+ ("input_value", "exp_except"),
+ [
+ (np.string_("hey"), False),
+ (np.array([np.string_("hey")]), False),
+ (np.array(np.string_("hey")), False),
+ ("hey", False),
+ (np.array([np.string_("hey"), np.string_("hey")]), True),
+ (5, True),
+ ],
+)
+def test_np2str(input_value, exp_except):
+ """Test the np2str function on different inputs."""
+ if exp_except:
+ with pytest.raises(ValueError):
+ np2str(input_value)
+ else:
+ assert np2str(input_value) == "hey"
+
+
+ at pytest.mark.parametrize(
+ ("input_value", "exp_result"),
+ [
+ (b"Hello", "Hello"),
+ ("Hello", "Hello"),
+ (np.string_("Hello"), "Hello"),
+ (np.array(np.string_("Hello")), np.array(np.string_("Hello"))),
+ ]
+)
+def test_bytes2string(input_value, exp_result):
+ """Test the bytes2string function with different inputs."""
+ assert bytes2string(input_value) == exp_result
+
+
+def test_download_rsr_dry_run():
+ """Test dry run of the download_rsr function."""
+ with unittest.mock.patch("pyspectral.utils.requests") as requests_mock:
+ utils.download_rsr(dry_run=True)
+ requests_mock.assert_not_called()
+
+
+def test_download_rsr(tmp_path):
+ """Test basic usage of the download_rsr function."""
+ tar_data = _create_fake_rsr_tarball_bytes()
+ with responses.RequestsMock() as rsps:
+ rsps.add(
+ "GET",
+ utils.HTTP_PYSPECTRAL_RSR,
+ body=tar_data,
+ status=200,
+ content_type="application/octet-stream",
+ headers={
+ "Content-Length": str(len(tar_data)),
+ },
+ )
+ utils.download_rsr(dest_dir=str(tmp_path))
+ assert os.path.isfile(tmp_path / "PYSPECTRAL_RSR_VERSION")
+
+
+def _create_fake_rsr_tarball_bytes():
+ file_obj = BytesIO()
+
+ with tarfile.open(mode="w:gz", fileobj=file_obj) as rsr_tar:
+ info = tarfile.TarInfo(utils.RSR_DATA_VERSION_FILENAME)
+ info.size = len(utils.RSR_DATA_VERSION)
+ rsr_tar.addfile(info, BytesIO(utils.RSR_DATA_VERSION.encode("ascii")))
+
+ file_obj.seek(0)
+ return file_obj.read()
+
+
+def test_download_luts_dry_run():
+ """Test a dry run of the download_luts function."""
+ with unittest.mock.patch("pyspectral.utils.requests") as requests_mock:
+ utils.download_luts(dry_run=True)
+ requests_mock.assert_not_called()
+
+
+ at pytest.mark.parametrize(
+ ("aerosol_types", "aerosol_type", "exp_warning"),
+ [
+ (None, None, False),
+ (["desert_aerosol"], None, False),
+ (None, "desert_aerosol", True),
+ ]
+)
+def test_download_luts(tmp_path, aerosol_types, aerosol_type, exp_warning):
+ """Test basic usage of the download_luts function."""
+ atypes_to_create = _atypes_to_create(aerosol_types, aerosol_type)
+ tar_data = _create_fake_lut_tarball_bytes(aerosol_types=atypes_to_create)
+ with responses.RequestsMock() as rsps:
+ rsps.add(
+ "GET",
+ re.compile(re.escape(utils.LUT_URL_PREFIX) + r"_.+\.tgz"),
+ body=tar_data,
+ status=200,
+ content_type="application/octet-stream",
+ headers={
+ "Content-Length": str(len(tar_data)),
+ },
+ )
+ with _fake_get_config(tmp_path), warnings.catch_warnings(record=True) as warns:
+ utils.download_luts(aerosol_types=aerosol_types, aerosol_type=aerosol_type)
+ _check_user_warning(warns, exp_warning)
+ _check_expected_aerosol_files(atypes_to_create, tmp_path)
+
+
+def _atypes_to_create(aerosol_types, aerosol_type):
+ if aerosol_types:
+ return aerosol_types
+ if aerosol_type is None:
+ return utils.AEROSOL_TYPES
+ return [aerosol_type]
+
+
+def _create_fake_lut_tarball_bytes(aerosol_types):
+ file_obj = BytesIO()
+
+ with tarfile.open(mode="w:gz", fileobj=file_obj) as rsr_tar:
+ for aerosol_type in aerosol_types:
+ version_fn = utils.ATM_CORRECTION_LUT_VERSION[aerosol_type]["filename"]
+ version_value = utils.ATM_CORRECTION_LUT_VERSION[aerosol_type]["version"]
+ info = tarfile.TarInfo(version_fn)
+ info.size = len(version_value)
+ rsr_tar.addfile(info, BytesIO(version_value.encode("ascii")))
+
+ file_obj.seek(0)
+ return file_obj.read()
+
+
+ at contextlib.contextmanager
+def _fake_get_config(tmp_path):
+ def _get_config():
+ return {
+ "rayleigh_dir": str(tmp_path),
+ "rsr_dir": str(tmp_path),
+ }
+ with unittest.mock.patch("pyspectral.utils.get_config") as get_config:
+ get_config.side_effect = _get_config
+ yield
+
+
+def _check_user_warning(warns, exp_warning):
+ all_user_warnings = [warn_msg for warn_msg in warns if issubclass(warn_msg.category, UserWarning)]
+ all_matching_warnings = [warn_msg for warn_msg in all_user_warnings if "aerosol_type" in str(warn_msg.message)]
+ assert len(all_matching_warnings) == (1 if exp_warning else 0)
+
+
+def _check_expected_aerosol_files(atypes_to_create, tmp_path):
+ for aerosol_type in utils.AEROSOL_TYPES:
+ atype_fn = tmp_path / aerosol_type / utils.ATM_CORRECTION_LUT_VERSION[aerosol_type]["filename"]
+ if aerosol_type in atypes_to_create:
+ assert atype_fn.is_file()
+ else:
+ assert not atype_fn.is_file()
+
+
+ at pytest.mark.parametrize(
+ ("platform_name", "input_value_sensor", "exp_sensor_name"),
+ [
+ ('Metop-C', 'avhrr/3', 'avhrr3'),
+ ('NOAA-19', 'avhrr/3', 'avhrr3'),
+ ('NOAA-12', 'avhrr/2', 'avhrr2'),
+ ('TIROS-N', 'avhrr/1', 'avhrr1'),
+ ('Metop-C', 'avhrr-3', 'avhrr3'),
+ ('NOAA-12', 'avhrr-2', 'avhrr2'),
+ ('TIROS-N', 'avhrr-1', 'avhrr1'),
+ ],
+)
+def test_check_and_adjust_instrument_name_instrument_okay(platform_name, input_value_sensor, exp_sensor_name):
+ """Test the checking and adjusting of the instrument name."""
+ res = check_and_adjust_instrument_name(platform_name, input_value_sensor)
+ assert res == exp_sensor_name
+
+
+ at pytest.mark.parametrize(
+ ("platform_name", "input_value_sensor", "exp_sensor_name"),
+ [
+ ('GOES-16', 'Idontknow', 'abi'),
+ ('FY-4B', 'unknown', 'agri'),
+ ('Meteosat-12', 'seviri', 'fci'),
+ ],
+)
+def test_check_and_adjust_instrument_name_instrument_name_inconsistent(caplog,
+ platform_name,
+ input_value_sensor,
+ exp_sensor_name):
+ """Test the checking and adjusting of the instrument name."""
+ with caplog.at_level(logging.WARNING):
+ _ = check_and_adjust_instrument_name(platform_name, input_value_sensor)
+
+ log_output = "Inconsistent instrument/satellite input - instrument set to {instr}".format(instr=exp_sensor_name)
+ assert log_output in caplog.text
=====================================
pyspectral/utils.py
=====================================
@@ -19,11 +19,23 @@
"""Utility functions."""
-import os
import logging
+import os
+import tarfile
+import warnings
+
import numpy as np
-from pyspectral.config import get_config
+import requests
+
from pyspectral.bandnames import BANDNAMES
+from pyspectral.config import get_config
+
+TQDM_LOADED = True
+try:
+ from tqdm import tqdm
+except ImportError:
+ TQDM_LOADED = False
+
LOG = logging.getLogger(__name__)
@@ -52,6 +64,7 @@ INSTRUMENTS = {'NOAA-19': 'avhrr/3',
'NOAA-20': 'viirs',
'EOS-Aqua': 'modis',
'EOS-Terra': 'modis',
+ 'FY-3E': 'mersi-2',
'FY-3D': 'mersi-2',
'FY-3C': 'virr',
'FY-3B': 'virr',
@@ -61,14 +74,25 @@ INSTRUMENTS = {'NOAA-19': 'avhrr/3',
'Meteosat-9': 'seviri',
'Meteosat-8': 'seviri',
'FY-4A': 'agri',
+ 'FY-4B': 'agri',
'GEO-KOMPSAT-2A': 'ami',
- 'MTG-I1': 'fci'
+ 'MTG-I1': 'fci',
+ 'Meteosat-12': 'fci',
+ 'GOES-16': 'abi',
+ 'GOES-17': 'abi',
+ 'GOES-18': 'abi',
+ 'Arctica-M-N1': 'msu-gsa'
}
-HTTP_PYSPECTRAL_RSR = "https://zenodo.org/record/6026563/files/pyspectral_rsr_data.tgz"
+INSTRUMENT_TRANSLATION_DASH2SLASH = {'avhrr-1': 'avhrr/1',
+ 'avhrr-2': 'avhrr/2',
+ 'avhrr-3': 'avhrr/3'}
+
+
+HTTP_PYSPECTRAL_RSR = "https://zenodo.org/record/7116653/files/pyspectral_rsr_data.tgz"
RSR_DATA_VERSION_FILENAME = "PYSPECTRAL_RSR_VERSION"
-RSR_DATA_VERSION = "v1.0.18"
+RSR_DATA_VERSION = "v1.2.0"
ATM_CORRECTION_LUT_VERSION = {}
ATM_CORRECTION_LUT_VERSION['antarctic_aerosol'] = {'version': 'v1.0.1',
@@ -94,6 +118,7 @@ ATM_CORRECTION_LUT_VERSION['urban_aerosol'] = {'version': 'v1.0.1',
ATM_CORRECTION_LUT_VERSION['rayleigh_only'] = {'version': 'v1.0.1',
'filename': 'PYSPECTRAL_ATM_CORR_LUT_RO'}
+#: Aerosol types available as downloadable LUTs for rayleigh correction
AEROSOL_TYPES = ['antarctic_aerosol', 'continental_average_aerosol',
'continental_clean_aerosol', 'continental_polluted_aerosol',
'desert_aerosol', 'marine_clean_aerosol',
@@ -105,15 +130,15 @@ ATMOSPHERES = {'subarctic summer': 4, 'subarctic winter': 5,
'tropical': 8, 'us-standard': 9}
HTTPS_RAYLEIGH_LUTS = {}
-URL_PREFIX = "https://zenodo.org/record/1288441/files/pyspectral_atm_correction_luts"
+LUT_URL_PREFIX = "https://zenodo.org/record/1288441/files/pyspectral_atm_correction_luts"
for atype in AEROSOL_TYPES:
name = {'rayleigh_only': 'no_aerosol'}.get(atype, atype)
- url = "{prefix}_{name}.tgz".format(prefix=URL_PREFIX, name=name)
+ url = "{prefix}_{name}.tgz".format(prefix=LUT_URL_PREFIX, name=name)
HTTPS_RAYLEIGH_LUTS[atype] = url
def get_rayleigh_lut_dir(aerosol_type):
- """Get the rayleight LUT directory for the specified aerosol type."""
+ """Get the rayleigh LUT directory for the specified aerosol type."""
conf = get_config()
local_rayleigh_dir = conf.get('rayleigh_dir')
return os.path.join(local_rayleigh_dir, aerosol_type)
@@ -229,7 +254,7 @@ def sort_data(x_vals, y_vals):
return x_vals, y_vals
-def convert2hdf5(ClassIn, platform_name, bandnames, scale=1e-06):
+def convert2hdf5(ClassIn, platform_name, bandnames, scale=1e-06, detectors=None):
"""Retrieve original RSR data and convert to internal hdf5 format.
*scale* is the number which has to be multiplied to the wavelength data in
@@ -253,121 +278,137 @@ def convert2hdf5(ClassIn, platform_name, bandnames, scale=1e-06):
for chname in bandnames:
sensor = ClassIn(chname, platform_name)
grp = h5f.create_group(chname)
- wvl = sensor.rsr['wavelength'][~np.isnan(sensor.rsr['wavelength'])]
- rsp = sensor.rsr['response'][~np.isnan(sensor.rsr['wavelength'])]
- grp.attrs['central_wavelength'] = get_central_wave(wvl, rsp)
- arr = sensor.rsr['wavelength']
+
+ # If multiple detectors, assume all have same wavelength range in SRF.
+ if detectors is not None:
+ wvl = sensor.rsr[detectors[0]]['wavelength'][~np.isnan(sensor.rsr[detectors[0]]['wavelength'])]
+ arr = sensor.rsr[detectors[0]]['wavelength']
+ grp.attrs['number_of_detectors'] = len(detectors)
+ else:
+ wvl = sensor.rsr['wavelength'][~np.isnan(sensor.rsr['wavelength'])]
+ arr = sensor.rsr['wavelength']
+
+ # Save wavelengths to file
dset = grp.create_dataset('wavelength', arr.shape, dtype='f')
dset.attrs['unit'] = 'm'
dset.attrs['scale'] = scale
dset[...] = arr
- arr = sensor.rsr['response']
- dset = grp.create_dataset('response', arr.shape, dtype='f')
- dset[...] = arr
-
-def download_rsr(**kwargs):
+ # Now to do the responses
+ if detectors is None:
+ rsp = sensor.rsr['response'][~np.isnan(sensor.rsr['wavelength'])]
+ grp.attrs['central_wavelength'] = get_central_wave(wvl, rsp)
+ arr = sensor.rsr['response']
+ dset = grp.create_dataset('response', arr.shape, dtype='f')
+ dset[...] = arr
+ else:
+ for cur_det in detectors:
+ det_grp = grp.create_group(cur_det)
+ rsp = sensor.rsr[cur_det]['response'][~np.isnan(sensor.rsr[cur_det]['wavelength'])]
+ det_grp.attrs['central_wavelength'] = get_central_wave(wvl, rsp)
+ arr = sensor.rsr[cur_det]['response']
+ dset = det_grp.create_dataset('response', arr.shape, dtype='f')
+ dset[...] = arr
+
+
+def download_rsr(dest_dir=None, dry_run=False):
"""Download the relative spectral response functions.
- Download the pre-compiled hdf5 formatet relative spectral response functions
- from the internet
+ Download the pre-compiled HDF5 formatted relative spectral response
+ functions from the internet as tarballs, extracts them, then deletes
+ the tarball.
- """
- import tarfile
- import requests
- TQDM_LOADED = True
- try:
- from tqdm import tqdm
- except ImportError:
- TQDM_LOADED = False
+ See :func:`pyspectral.rsr_reader.check_and_download` for a "smart" version
+ of this process that only downloads the necessary files.
+
+ Args:
+ dest_dir (str): Path to put the temporary tarball and extracted RSR
+ files.
+ dry_run (bool): If True, don't actually download files, only log what
+ URLs would be downloaded. Defaults to False.
+ """
config = get_config()
local_rsr_dir = config.get('rsr_dir')
- dest_dir = kwargs.get('dest_dir', local_rsr_dir)
- dry_run = kwargs.get('dry_run', False)
+ dest_dir = dest_dir or local_rsr_dir
LOG.info("Download RSR files and store in directory %s", dest_dir)
-
filename = os.path.join(dest_dir, "pyspectral_rsr_data.tgz")
- LOG.debug("Get data. URL: %s", HTTP_PYSPECTRAL_RSR)
+ LOG.debug("RSR URL: %s", HTTP_PYSPECTRAL_RSR)
LOG.debug("Destination = %s", dest_dir)
if dry_run:
return
- response = requests.get(HTTP_PYSPECTRAL_RSR)
- if TQDM_LOADED:
- with open(filename, "wb") as handle:
- for data in tqdm(response.iter_content()):
- handle.write(data)
- else:
- with open(filename, "wb") as handle:
- for data in response.iter_content():
- handle.write(data)
-
- tar = tarfile.open(filename)
- tar.extractall(dest_dir)
- tar.close()
- os.remove(filename)
+ _download_tarball_and_extract(HTTP_PYSPECTRAL_RSR, filename, dest_dir)
-def download_luts(**kwargs):
- """Download the luts from internet."""
- import tarfile
- import requests
- TQDM_LOADED = True
- try:
- from tqdm import tqdm
- except ImportError:
- TQDM_LOADED = False
+def download_luts(aerosol_types=None, dry_run=False, aerosol_type=None):
+ """Download the luts from internet.
- dry_run = kwargs.get('dry_run', False)
+ See :func:`pyspectral.rayleigh.check_and_download` for a "smart" version
+ of this process that only downloads the necessary files.
- if 'aerosol_type' in kwargs:
- if isinstance(kwargs['aerosol_type'], (list, tuple, set)):
- aerosol_types = kwargs['aerosol_type']
- else:
- aerosol_types = [kwargs['aerosol_type'], ]
- else:
- aerosol_types = HTTPS_RAYLEIGH_LUTS.keys()
+ Args:
+ aerosol_types (Iterable): Aerosol types to download the LUTs for..
+ Defaults to all aerosol types. See :data:`AEROSOL_TYPES` for the
+ full list.
+ dry_run (bool): If True, don't actually download files, only log what
+ URLs would be downloaded. Defaults to False.
+ aerosol_type (str): Deprecated.
- chunk_size = 1024 * 1024 # 1 MB
+ """
+ aerosol_types = _get_aerosol_types(aerosol_types, aerosol_type)
for subname in aerosol_types:
-
LOG.debug('Aerosol type: %s', subname)
- http = HTTPS_RAYLEIGH_LUTS[subname]
- LOG.debug('URL = %s', http)
+ lut_tarball_url = HTTPS_RAYLEIGH_LUTS[subname]
+ LOG.debug('Atmospheric LUT URL = %s', lut_tarball_url)
subdir_path = get_rayleigh_lut_dir(subname)
- try:
- LOG.debug('Create directory: %s', subdir_path)
- if not dry_run:
- os.makedirs(subdir_path)
- except OSError:
- if not os.path.isdir(subdir_path):
- raise
-
+ LOG.debug('Create directory: %s', subdir_path)
+ if not dry_run:
+ os.makedirs(subdir_path, exist_ok=True)
if dry_run:
continue
- response = requests.get(http)
- total_size = int(response.headers['content-length'])
+ local_tarball_pathname = os.path.join(subdir_path, "pyspectral_rayleigh_correction_luts.tgz")
+ _download_tarball_and_extract(lut_tarball_url, local_tarball_pathname, subdir_path)
+
- filename = os.path.join(
- subdir_path, "pyspectral_rayleigh_correction_luts.tgz")
- if TQDM_LOADED:
- with open(filename, "wb") as handle:
- for data in tqdm(iterable=response.iter_content(chunk_size=chunk_size),
- total=(int(total_size / chunk_size + 0.5)), unit='kB'):
- handle.write(data)
+def _get_aerosol_types(aerosol_types, aerosol_type):
+ if aerosol_type is not None:
+ warnings.warn("'aerosol_type' is deprecated, use 'aerosol_types' instead.", UserWarning)
+ if isinstance(aerosol_type, (list, tuple, set)):
+ aerosol_types = aerosol_type
else:
- with open(filename, "wb") as handle:
- for data in response.iter_content():
- handle.write(data)
+ aerosol_types = [aerosol_type]
+ elif aerosol_types is None:
+ aerosol_types = list(HTTPS_RAYLEIGH_LUTS.keys())
+ return aerosol_types
- tar = tarfile.open(filename)
- tar.extractall(subdir_path)
- tar.close()
- os.remove(filename)
+
+def _download_tarball_and_extract(tarball_url, local_pathname, extract_dir):
+ chunk_size = 1024 * 1024 # 1 MB
+ response = requests.get(tarball_url)
+ total_size = int(response.headers['content-length'])
+
+ with open(local_pathname, "wb") as handle:
+ for data in _tqdm_or_iter(response.iter_content(chunk_size=chunk_size),
+ total=(int(total_size / chunk_size + 0.5)),
+ unit='kB'):
+ handle.write(data)
+
+ tar = tarfile.open(local_pathname)
+ tar.extractall(extract_dir)
+ tar.close()
+ os.remove(local_pathname)
+
+
+def _tqdm_or_iter(an_iterable, **tqdm_kwargs):
+ """Wrap an iterable with tqdm if it is available, otherwise return the iterable."""
+ if TQDM_LOADED:
+ return tqdm(iterable=an_iterable, **tqdm_kwargs)
+ else:
+ return an_iterable
def debug_on():
@@ -481,3 +522,34 @@ def bytes2string(var):
if isinstance(var, bytes):
return var.decode('utf-8')
return var
+
+
+def check_and_adjust_instrument_name(platform_name, instrument):
+ """Check instrument name and try fix if inconsistent.
+
+ It checks against the possible listed instrument names for each platform.
+ It also makes an adjustment replacing names like avhrr/1 with avhrr1,
+ removing the '/'.
+ """
+ instr = INSTRUMENTS.get(platform_name, instrument.lower())
+ if not are_instruments_identical(instr, instrument.lower()):
+ instrument = instr
+ LOG.warning("Inconsistent instrument/satellite input - instrument set to %s",
+ instrument)
+
+ return instrument.lower().replace('/', '').replace('-', '')
+
+
+def are_instruments_identical(name1, name2):
+ """Given two instrument names check if they are both describing the same instrument.
+
+ Takes care of the case of AVHRR where the internal pyspectral naming
+ (following WMO Oscar) is is with a slash as in 'avhrr/1', but where a
+ naming using a dash instead is equally accepted, as in 'avhrr-1'.
+ """
+ if name1 == name2:
+ return True
+ if name1 == INSTRUMENT_TRANSLATION_DASH2SLASH.get(name2):
+ return True
+
+ return False
=====================================
rsr_convert_scripts/README.rst
=====================================
@@ -174,7 +174,12 @@ the pyspectral.yaml file:
.. code::
- %> python msu_gs_reader.py
+ %> python msu_gsa_reader.py
+
+Converts RSRs for the MSU-GS/A sensors aboard Arctica-M N1 satellite.
+RSRs were retrieved from Roshydromet. Filenames look like:
+
+``rtcoef_electro-l_2_msugs_srf_ch01.txt``
Converts RSRs for the MSU-GS sensor aboard Electro-L N2. RSRs were retrieved from the NWP-SAF.
Filenames look like:
=====================================
rsr_convert_scripts/agri_rsr.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2018, 2019 Pytroll developers
+# Copyright (c) 2018, 2019, 2022 Pytroll developers
#
# Author(s):
#
@@ -21,8 +21,9 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-"""Read the FY-4A AGRI relative spectral responses. Data from
-http://fy4.nsmc.org.cn/portal/cn/fycv/srf.html
+"""Read the FY-4A AGRI relative spectral responses.
+
+Data from http://fy4.nsmc.org.cn/portal/cn/fycv/srf.html
"""
import os
import numpy as np
@@ -31,30 +32,45 @@ from pyspectral.utils import convert2hdf5 as tohdf5
from pyspectral.raw_reader import InstrumentRSR
from pyspectral.utils import logging_on, get_logger
-FY4A_BAND_NAMES = ['ch1', 'ch2', 'ch3', 'ch4', 'ch5', 'ch6', 'ch7', 'ch8',
- 'ch9', 'ch10', 'ch11', 'ch12', 'ch13', 'ch14']
-BANDNAME_SCALE2MICROMETERS = {'ch1': 0.001,
- 'ch2': 0.001,
- 'ch3': 0.001,
- 'ch4': 1.0,
- 'ch5': 1.0,
- 'ch6': 1.0,
- 'ch7': 1.0,
- 'ch8': 1.0,
- 'ch9': 1.0,
- 'ch10': 1.0,
- 'ch11': 1.0,
- 'ch12': 1.0,
- 'ch13': 1.0,
- 'ch14': 1.0}
+FY4_AGRI_BAND_NAMES = ['ch1', 'ch2', 'ch3', 'ch4', 'ch5', 'ch6', 'ch7', 'ch8',
+ 'ch9', 'ch10', 'ch11', 'ch12', 'ch13', 'ch14', 'ch15']
+BANDNAME_SCALE2MICROMETERS = {'FY-4A': {'ch1': 0.001,
+ 'ch2': 0.001,
+ 'ch3': 0.001,
+ 'ch4': 1.0,
+ 'ch5': 1.0,
+ 'ch6': 1.0,
+ 'ch7': 1.0,
+ 'ch8': 1.0,
+ 'ch9': 1.0,
+ 'ch10': 1.0,
+ 'ch11': 1.0,
+ 'ch12': 1.0,
+ 'ch13': 1.0,
+ 'ch14': 1.0},
+ 'FY-4B': {'ch1': 0.001,
+ 'ch2': 0.001,
+ 'ch3': 0.001,
+ 'ch4': 0.001,
+ 'ch5': 0.001,
+ 'ch6': 0.001,
+ 'ch7': 1.0,
+ 'ch8': 1.0,
+ 'ch9': 1.0,
+ 'ch10': 1.0,
+ 'ch11': 1.0,
+ 'ch12': 1.0,
+ 'ch13': 1.0,
+ 'ch14': 1.0,
+ 'ch15': 1.0}}
class AGRIRSR(InstrumentRSR):
- """Container for the FY-4 AGRI RSR data"""
+ """Container for the FY-4 AGRI RSR data."""
def __init__(self, bandname, platform_name):
- """Initialise the FY-4 AGRI relative spectral response data"""
- super(AGRIRSR, self).__init__(bandname, platform_name, FY4A_BAND_NAMES)
+ """Initialise the FY-4 AGRI relative spectral response data."""
+ super(AGRIRSR, self).__init__(bandname, platform_name, FY4_AGRI_BAND_NAMES)
self.instrument = INSTRUMENTS.get(platform_name, 'agri')
@@ -64,7 +80,7 @@ class AGRIRSR(InstrumentRSR):
LOG.debug("Filenames: %s", str(self.filenames))
if self.filenames[bandname] and os.path.exists(self.filenames[bandname]):
self.requested_band_filename = self.filenames[bandname]
- scale = BANDNAME_SCALE2MICROMETERS.get(bandname)
+ scale = BANDNAME_SCALE2MICROMETERS[platform_name].get(bandname)
if scale:
self._load(scale=scale)
else:
@@ -79,9 +95,9 @@ class AGRIRSR(InstrumentRSR):
self.filename = self.requested_band_filename
def _load(self, scale=0.001):
- """Load the AGRI RSR data for the band requested
+ """Load the AGRI RSR data for the band requested.
- Wavelength is given in nanometers.
+ Wavelength is given in nanometers.
"""
data = np.genfromtxt(self.requested_band_filename,
unpack=True,
@@ -95,14 +111,14 @@ class AGRIRSR(InstrumentRSR):
self.rsr = {'wavelength': wavelength, 'response': response}
-def main():
- """Main"""
- for platform_name in ["FY-4A", ]:
- tohdf5(AGRIRSR, platform_name, FY4A_BAND_NAMES)
+def convert_agri():
+ """Read original AGRI RSR data and convert to common Pyspectral hdf5 format."""
+ for platform_name in ["FY-4A", "FY-4B"]:
+ tohdf5(AGRIRSR, platform_name, FY4_AGRI_BAND_NAMES)
if __name__ == "__main__":
LOG = get_logger(__name__)
logging_on()
- main()
+ convert_agri()
=====================================
rsr_convert_scripts/avhrr1_rsr.py
=====================================
@@ -0,0 +1,191 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2022 Pytroll developers
+#
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+"""Read the NOAA AVHRR/1 relative spectral response functions.
+
+Data from NOAA: AVHRR1_SRF_only.xls
+"""
+
+import os
+from xlrd import open_workbook
+from pyspectral.config import get_config
+from pyspectral.utils import get_central_wave
+import numpy as np
+import pkg_resources
+import logging
+import h5py
+
+
+LOG = logging.getLogger(__name__)
+
+AVHRR_BAND_NAMES = {'avhrr/1': ['ch1', 'ch2', 'ch3', 'ch4']}
+AVHRR1_SATELLITES = ['TIROS-N', 'NOAA-6', 'NOAA-8', 'NOAA-10']
+
+DATA_PATH = pkg_resources.resource_filename('pyspectral', 'data/')
+
+CHANNEL_NAMES = {'A101C001': 'ch1',
+ 'A101C002': 'ch2',
+ 'A101C003': 'ch3',
+ 'A101C004': 'ch4',
+ 'A102C001': 'ch1',
+ 'A102C002': 'ch2',
+ 'A102C003': 'ch3',
+ 'A102C004': 'ch4',
+ 'A103C001': 'ch1',
+ 'A103C002': 'ch2',
+ 'A103C003': 'ch3',
+ 'A103C004': 'ch4',
+ 'A104C001': 'ch1',
+ 'A104C002': 'ch2',
+ 'A104C003': 'ch3',
+ 'A104C004': 'ch4'}
+
+
+class AvhrrRSR():
+ """Container for the NOAA AVHRR-1 RSR data."""
+
+ def __init__(self, wavespace='wavelength'):
+ """Initialize the AVHRR-1 RSR class."""
+ options = get_config()
+
+ self.avhrr_path = options['avhrr/1'].get('path')
+ if not os.path.exists(self.avhrr_path):
+ self.avhrr1_path = os.path.join(
+ DATA_PATH, options['avhrr/1'].get('filename'))
+
+ self.output_dir = options.get('rsr_dir', './')
+
+ self.rsr = {}
+ for satname in AVHRR1_SATELLITES:
+ self.rsr[satname] = {}
+ for chname in AVHRR_BAND_NAMES['avhrr/1']:
+ self.rsr[satname][chname] = {'wavelength': None, 'response': None}
+
+ self._load()
+ self.wavespace = wavespace
+ if wavespace not in ['wavelength', 'wavenumber']:
+ raise AttributeError("wavespace has to be either " +
+ "'wavelength' or 'wavenumber'!")
+
+ self.unit = 'micrometer'
+ if wavespace == 'wavenumber':
+ # Convert to wavenumber:
+ self.convert2wavenumber()
+
+ def _load(self, scale=1.0):
+ """Load the AVHRR RSR data for the band requested."""
+ wb_ = open_workbook(self.avhrr_path)
+
+ sheet_names = []
+ for sheet in wb_.sheets():
+ if sheet.name in ['Kleespies Data', ]:
+ print("Skip sheet...")
+ continue
+
+ ch_name = CHANNEL_NAMES.get(sheet.name.strip())
+ if not ch_name:
+ break
+
+ sheet_names.append(sheet.name.strip())
+
+ header = sheet.col_values(0, start_rowx=0, end_rowx=2)
+ platform_name = header[0].strip("# ")
+ unit = header[1].split("Wavelength (")[1].strip(")")
+
+ scale = get_scale_from_unit(unit)
+
+ wvl = sheet.col_values(0, start_rowx=2)
+ is_comment = True
+ idx = 0
+ while is_comment:
+ item = wvl[::-1][idx]
+ if isinstance(item, str):
+ idx = idx+1
+ else:
+ break
+
+ ndim = len(wvl) - idx
+ wvl = wvl[0:ndim]
+
+ if platform_name == "TIROS-N":
+ wvl = adjust_typo_avhrr1_srf_only_xls_file(platform_name, wvl)
+
+ response = sheet.col_values(1, start_rowx=2, end_rowx=2+ndim)
+
+ wavelength = np.array(wvl) * scale
+ response = np.array(response)
+
+ self.rsr[platform_name][ch_name]['wavelength'] = wavelength
+ self.rsr[platform_name][ch_name]['response'] = response
+
+
+def adjust_typo_avhrr1_srf_only_xls_file(platform_name, wvl):
+ """Adjust typo in wavelength: 640 should most certainly have been 840 in the AVHRR1_SRF_only.xls."""
+ epsilon = 0.01
+ for idx, wavel in enumerate(wvl[1::]):
+ if wvl[idx] > wavel and abs(wavel-640.0) < epsilon:
+ wvl[idx+1] = 840.0
+ return wvl
+
+
+def get_scale_from_unit(unit):
+ """Get the scaling factor to go from unit to micrometers."""
+ unit2scale = {'Å': 0.0001, 'nm': 0.001}
+ return unit2scale.get(unit)
+
+
+def generate_avhrr1_file(avhrr1, platform_name):
+ """Generate the relative response functions for one AVHRR-1 ensor.
+
+ Format is the pyspectral internal common format.
+ """
+ filename = os.path.join(avhrr1.output_dir, "rsr_avhrr1_{sat}.h5".format(sat=platform_name))
+
+ with h5py.File(filename, "w") as h5f:
+
+ h5f.attrs['description'] = 'Relative Spectral Responses for AVHRR/1'
+ h5f.attrs['platform_name'] = platform_name
+ bandnames = avhrr1.rsr[platform_name].keys()
+ h5f.attrs['band_names'] = [str(key) for key in bandnames]
+
+ for chname in bandnames:
+ grp = h5f.create_group(chname)
+
+ wvl = avhrr1.rsr[platform_name][chname]['wavelength']
+ rsp = avhrr1.rsr[platform_name][chname]['response']
+ grp.attrs['central_wavelength'] = get_central_wave(wvl, rsp)
+ arr = avhrr1.rsr[platform_name][chname]['wavelength']
+ dset = grp.create_dataset('wavelength', arr.shape, dtype='f')
+ dset.attrs['unit'] = 'm'
+ dset.attrs['scale'] = 1e-06
+ dset[...] = arr
+ arr = avhrr1.rsr[platform_name][chname]['response']
+ dset = grp.create_dataset('response', arr.shape, dtype='f')
+ dset[...] = arr
+
+
+def run_avhrr1():
+ """Create the AVHRR-1 relative spectral response files."""
+ avhrr_obj = AvhrrRSR()
+ for platform_name in ["NOAA-10", "NOAA-8", "NOAA-6", "TIROS-N"]:
+ generate_avhrr1_file(avhrr_obj, platform_name)
+
+
+if __name__ == "__main__":
+ run_avhrr1()
=====================================
rsr_convert_scripts/fci_rsr.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2020 Pytroll developers
+# Copyright (c) 2020-2022 Pytroll developers
#
# Author(s):
#
@@ -22,63 +22,50 @@
"""Read the MTG FCI relative spectral response functions.
-Data from EUMETSAT NWP-SAF:
-https://nwpsaf.eu/downloads/rtcoef_rttov12/ir_srf/rtcoef_mtg_1_fci_srf.html
+Data from EUMETSAT:
+https://sftp.eumetsat.int/public/folder/UsCVknVOOkSyCdgpMimJNQ/User-Materials/MTGUP/Materials/FCI-SRF_Apr2022/
"""
-import os
import logging
-import numpy as np
+from netCDF4 import Dataset
from pyspectral.utils import convert2hdf5 as tohdf5
from pyspectral.raw_reader import InstrumentRSR
-
+from pyspectral.bandnames import BANDNAMES
LOG = logging.getLogger(__name__)
-FCI_BAND_NAMES = ['ch1', 'ch2', 'ch3', 'ch4', 'ch5', 'ch6', 'ch7', 'ch8', 'ch9', 'ch10',
- 'ch11', 'ch12', 'ch13', 'ch14', 'ch15', 'ch16']
+FCI_BAND_NAMES = list(BANDNAMES['fci'].values())
class FciRSR(InstrumentRSR):
"""Container for the MTG FCI RSR data."""
def __init__(self, bandname, platform_name):
- """Setup the MTG FCI RSR data container."""
- super(FciRSR, self).__init__(bandname, platform_name, FCI_BAND_NAMES)
+ """MTG FCI RSR data container."""
+ super().__init__(bandname, platform_name)
self.instrument = 'fci'
-
self._get_options_from_config()
- self._get_bandfilenames()
-
- LOG.debug("Filenames: %s", str(self.filenames))
- if self.filenames[bandname] and os.path.exists(self.filenames[bandname]):
- self.requested_band_filename = self.filenames[bandname]
- self._load()
- else:
- LOG.warning("Couldn't find an existing file for this band: %s",
- str(self.bandname))
+ LOG.debug("Filename with all bands: %s", str(self.filename))
+ self._load()
- # To be compatible with VIIRS....
- self.filename = self.requested_band_filename
-
- def _load(self, scale=10000.0):
+ def _load(self, scale=1000000.0):
"""Load the FCI RSR data for the band requested."""
- data = np.genfromtxt(self.requested_band_filename,
- unpack=True,
- names=['wavenumber',
- 'response'],
- skip_header=4)
+ LOG.debug("File: %s", str(self.filename))
+
+ ncf = Dataset(self.path / self.filename, 'r')
- # Data are wavenumbers in cm-1:
- wavelength = 1. / data['wavenumber'] * scale
- response = data['response']
+ wvl = ncf.variables['wavelength'][:] * scale
+ resp = ncf.variables['srf'][:]
- self.rsr = {'wavelength': wavelength[::-1], 'response': response[::-1]}
+ bandnames = ncf.variables['channel_id'][:]
+ for idx, band_name in enumerate(bandnames):
+ if band_name == self.bandname:
+ self.rsr = {'wavelength': wvl[:, idx], 'response': resp[:, idx]}
def main():
- """Main function creating the internal Pyspectral hdf5 output for FCI."""
+ """Create the internal Pyspectral hdf5 output for FCI."""
for platform_name in ["Meteosat-12", 'MTG-I1']:
tohdf5(FciRSR, platform_name, FCI_BAND_NAMES)
=====================================
rsr_convert_scripts/msu_gsa_reader.py
=====================================
@@ -0,0 +1,106 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2022 Pytroll developers
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+"""Read the Arctica-M N1 MSU-GS/A spectral response functions.
+Data from Roshydromet via personal communication.
+"""
+
+from pyspectral.utils import convert2hdf5 as tohdf5
+from pyspectral.raw_reader import InstrumentRSR
+from pyspectral.config import get_config
+import pandas as pd
+import logging
+
+LOG = logging.getLogger(__name__)
+
+# Names of the individual bands
+MSUGSA_BAND_NAMES = ['ch1', 'ch2', 'ch3', 'ch4', 'ch5',
+ 'ch6', 'ch7', 'ch8', 'ch9', 'ch10']
+
+# Set up VIS and IR bands, needed for selecting sheet in RSR file
+VISBANDS = {'ch1', 'ch2', 'ch3'}
+IRBANDS = {'ch4', 'ch5', 'ch6', 'ch7', 'ch8', 'ch9', 'ch10'}
+
+#: Default time format
+_DEFAULT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S'
+
+#: Default log format
+_DEFAULT_LOG_FORMAT = '[%(levelname)s: %(asctime)s : %(name)s] %(message)s'
+
+
+class MsugsaRSR(InstrumentRSR):
+ """Container for the Arctica-M-N1 MSU-GS/A relative spectral response data"""
+
+ def __init__(self, bandname, platform_name):
+
+ super(MsugsaRSR, self).__init__(
+ bandname, platform_name, MSUGSA_BAND_NAMES)
+
+ self.instrument = 'msu-gsa'
+ self.platform_name = platform_name
+
+ options = get_config()
+
+ self.msugsa_path = options[
+ self.platform_name + '-' + self.instrument].get('path')
+
+ self.output_dir = options.get('rsr_dir', './')
+
+ self._load()
+
+ def _load(self, scale=10000.0):
+ """Load the MSU-GS/A RSR data for the band requested"""
+ detectors = {}
+ # The Arctica satellites have two instruments on them. Pyspectral isn't set up
+ # to handle this, so instead we call them separate detectors.
+ for detnum in (1, 2):
+ if self.bandname in VISBANDS:
+ data = pd.read_excel(self.msugsa_path, sheet_name=f'MSU-{detnum} vis', skiprows=2,
+ names=['wvl', 'ch1', 'ch2', 'ch3'])
+ elif self.bandname in IRBANDS:
+ data = pd.read_excel(self.msugsa_path, sheet_name=f'MSU-{detnum} IR', skiprows=2,
+ names=['wvl', 'ch4', 'ch5', 'ch6', 'ch7', 'ch8', 'ch9', 'ch10'])
+ else:
+ raise KeyError(f"{self.bandname} is not a valid VIS or IR band!")
+ wavelength = data['wvl']
+ response = data[self.bandname]
+
+ detectors[f'det-{detnum}'] = {'wavelength': wavelength, 'response': response}
+
+ self.rsr = detectors
+
+
+def main():
+ """Main"""
+ for platform_name in ['Arctica-M-N1', ]:
+ tohdf5(MsugsaRSR, platform_name, MSUGSA_BAND_NAMES, detectors=['det-1', 'det-2'])
+
+
+if __name__ == "__main__":
+ import sys
+ LOG = logging.getLogger('msu_gsa_rsr')
+ handler = logging.StreamHandler(sys.stderr)
+
+ formatter = logging.Formatter(fmt=_DEFAULT_LOG_FORMAT,
+ datefmt=_DEFAULT_TIME_FORMAT)
+ handler.setFormatter(formatter)
+ handler.setLevel(logging.DEBUG)
+ LOG.setLevel(logging.DEBUG)
+ LOG.addHandler(handler)
+
+ main()
=====================================
rsr_convert_scripts/seviri_rsr.py
=====================================
@@ -2,7 +2,7 @@
#
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2013-2018, 2020 Pytroll developers
+# Copyright (c) 2013-2022 Pytroll developers
#
# Author(s):
#
@@ -22,9 +22,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-"""Interface to the SEVIRI spectral response functions for all four MSG
-satellites
-"""
+"""Interface to the SEVIRI spectral response functions for all four MSG satellites."""
import os
from xlrd import open_workbook
@@ -38,17 +36,14 @@ DATA_PATH = pkg_resources.resource_filename('pyspectral', 'data/')
class Seviri(object):
-
- """Class for Seviri RSR"""
+ """Class for Seviri RSR."""
def __init__(self, wavespace='wavelength'):
"""
- Read the seviri relative spectral responses for all channels and all
- MSG satellites.
+ Read the seviri relative spectral responses for all channels and all MSG satellites.
Optional input: 'wavespace'. Equals 'wavelength' (units of micron's) on
default. Can be 'wavenumber' in which case the unit is in cm-1.
-
"""
options = get_config()
@@ -79,7 +74,7 @@ class Seviri(object):
self.get_centrals()
def _load(self, filename=None):
- """Read the SEVIRI rsr data"""
+ """Read the SEVIRI rsr data."""
if not filename:
filename = self.seviri_path
@@ -161,7 +156,7 @@ class Seviri(object):
self.rsr[ch_name]['wavelength'] = wvl
def convert2wavenumber(self):
- """Convert from wavelengths to wavenumber"""
+ """Convert from wavelengths to wavenumber."""
for chname in self.rsr.keys():
elems = [k for k in self.rsr[chname].keys()]
for sat in elems:
@@ -184,8 +179,11 @@ class Seviri(object):
self.unit = 'cm-1'
def get_centrals(self):
- """Get the central wavenumbers or central wavelengths of all channels,
- depending on the given 'wavespace'
+ """Get the central wavenumbers or central wavelengths of all channels.
+
+ The unit (wavelength or wavenumber) depends on the 'wave space' given by
+ the attribute *wavespace*.
+
"""
result = {}
for chname in self.rsr.keys():
@@ -213,15 +211,14 @@ class Seviri(object):
def get_central_wave(wavl, resp):
- """Calculate the central wavelength or the central wavenumber,
- depending on what is input
- """
+ """Calculate the central wavelength (or the central wavenumber if inputs are wave numbers)."""
return np.trapz(resp * wavl, wavl) / np.trapz(resp, wavl)
def generate_seviri_file(seviri, platform_name):
- """Generate the pyspectral internal common format relative response
- function file for one SEVIRI
+ """Generate the relative response function file for one SEVIRI sensor.
+
+ The file format is the pyspectral internal common format.
"""
import h5py
@@ -257,14 +254,9 @@ def generate_seviri_file(seviri, platform_name):
dset[...] = arr
-def main():
- """Main"""
+if __name__ == "__main__":
sev_obj = Seviri()
for satnum in [8, 9, 10, 11]:
generate_seviri_file(sev_obj, 'Meteosat-{0:d}'.format(satnum))
print("Meteosat-{0:d} done...".format(satnum))
-
-
-if __name__ == "__main__":
- main()
=====================================
rsr_convert_scripts/viirs_rsr.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Copyright (c) 2013-2020 Pytroll developers
+# Copyright (c) 2013-2022 Pytroll developers
#
# Author(s):
#
@@ -21,9 +21,11 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-"""Interface to VIIRS relative spectral responses"""
+"""Interface to read original VIIRS relative spectral responses and write to hdf5 file."""
import os
+import sys
+import h5py
import numpy as np
from pyspectral.utils import get_central_wave
from pyspectral.config import get_config
@@ -37,29 +39,8 @@ VIIRS_BAND_NAMES = ['M1', 'M2', 'M3', 'M4', 'M5',
'I1', 'I2', 'I3', 'I4', 'I5',
'DNB']
-N_HEADER_LINES = {'Suomi-NPP': 1, 'NOAA-20': 5}
-N_HEADER_LINES_DNB = {'Suomi-NPP': 1, 'NOAA-20': 6}
-
# JPSS-1: Band det SS wvl_nm RSR SNR SNR_Thresh QF
# UAID/Swp
-NAMES1 = {'Suomi-NPP': ['bandname',
- 'detector',
- 'subsample',
- 'wavelength',
- 'band_avg_snr',
- 'asr',
- 'response',
- 'quality_flag',
- 'xtalk_flag'],
- 'NOAA-20': ['bandname',
- 'detector',
- 'subsample',
- 'wavelength',
- 'response',
- 'snr',
- 'snr_thresh',
- 'qf',
- 'uaid_swp']}
DTYPE1 = {'Suomi-NPP': [('bandname', '|S3'),
('detector', '<i4'),
@@ -71,6 +52,15 @@ DTYPE1 = {'Suomi-NPP': [('bandname', '|S3'),
('quality_flag', '<i4'),
('xtalk_flag', '<i4')],
'NOAA-20': [('bandname', '|S3'),
+ ('detector', '<i4'),
+ ('subsample', '<i4'),
+ ('wavelength', '<f8'),
+ ('response', '<f8'),
+ ('snr', '<f8'),
+ ('snr_thresh', '<f8'),
+ ('qf', '<i4'),
+ ('uaid_swp', '<i4')],
+ 'NOAA-21': [('bandname', '|S3'),
('detector', '<i4'),
('subsample', '<i4'),
('wavelength', '<f8'),
@@ -104,10 +94,10 @@ _DEFAULT_LOG_FORMAT = '[%(levelname)s: %(asctime)s : %(name)s] %(message)s'
class ViirsRSR(object):
-
- """Container for the (S-NPP/JPSS) VIIRS RSR data"""
+ """Container for the (S-NPP/JPSS) VIIRS RSR data."""
def __init__(self, bandname, platform_name):
+ """Initialize the VIIRS RSR object."""
self.platform_name = platform_name
self.bandname = bandname
self.filename = None
@@ -124,7 +114,7 @@ class ViirsRSR(object):
self._load()
def _get_bandfilenames(self, **options):
- """Get filename for each band"""
+ """Get filename for each band."""
conf = options[self.platform_name + '-viirs']
rootdir = conf['rootdir']
@@ -138,8 +128,7 @@ class ViirsRSR(object):
filename, {'bandname': band})
def _get_bandfile(self, **options):
- """Get the VIIRS rsr filename"""
-
+ """Get the VIIRS rsr filename."""
# Need to understand why there are A&B files for band M16. FIXME!
# Anyway, the absolute response differences are small, below 0.05
@@ -153,21 +142,16 @@ class ViirsRSR(object):
self.filename = path
def _load(self, scale=0.001):
- """Load the VIIRS RSR data for the band requested"""
- if self.bandname == 'DNB':
- header_lines_to_skip = N_HEADER_LINES_DNB[self.platform_name]
- else:
- header_lines_to_skip = N_HEADER_LINES[self.platform_name]
+ """Load the VIIRS RSR data for the band requested."""
+ header_lines_to_skip = get_header_lines2skip(self.filename, self.platform_name)
try:
data = np.genfromtxt(self.filename,
- unpack=True, skip_header=header_lines_to_skip,
- names=NAMES1[self.platform_name],
+ skip_header=header_lines_to_skip,
dtype=DTYPE1[self.platform_name])
except ValueError:
data = np.genfromtxt(self.filename,
- unpack=True, skip_header=N_HEADER_LINES[
- self.platform_name],
+ unpack=True, skip_header=header_lines_to_skip,
names=NAMES2[self.platform_name],
dtype=DTYPE2[self.platform_name])
@@ -186,11 +170,20 @@ class ViirsRSR(object):
self.rsr = detectors
-def main():
- """Main"""
- import sys
- import h5py
+def get_header_lines2skip(filename, platform_name):
+ """Check the file nd find the number of header lines to skip."""
+ with open(filename, 'r') as fpt:
+ nlines_hd = 0
+ line = '% '
+ while line.startswith('%'):
+ line = fpt.readline()
+ nlines_hd = nlines_hd + 1
+
+ return nlines_hd - 1
+
+def create_viirs_rsr(platform_name):
+ """Create the VIIRS RSR functions and save to a pyspectral formattet hdf5 file."""
handler = logging.StreamHandler(sys.stderr)
formatter = logging.Formatter(fmt=_DEFAULT_LOG_FORMAT,
@@ -200,8 +193,6 @@ def main():
LOG.setLevel(logging.DEBUG)
LOG.addHandler(handler)
- platform_name = "NOAA-20"
- # platform_name = "Suomi-NPP"
viirs = ViirsRSR('M1', platform_name)
filename = os.path.join(viirs.output_dir,
"rsr_viirs_{0}.h5".format(platform_name))
@@ -219,7 +210,7 @@ def main():
grp.attrs['number_of_detectors'] = len(viirs.rsr.keys())
# Loop over each detector to check if the sampling wavelengths are
# identical:
- det_names = viirs.rsr.keys()
+ det_names = list(viirs.rsr.keys())
wvl = viirs.rsr[det_names[0]]['wavelength']
wvl, idx = np.unique(wvl, return_index=True)
wvl_is_constant = True
@@ -262,5 +253,5 @@ def main():
if __name__ == "__main__":
- LOG = logging.getLogger('viirs_rsr')
- main()
+ platform_name = "NOAA-21"
+ create_viirs_rsr(platform_name)
=====================================
setup.py
=====================================
@@ -1,7 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
-# Copyright (c) 2013-2021 Pytroll
+# Copyright (c) 2013-2022 Pytroll
# Author(s):
@@ -43,13 +43,11 @@ try:
except IOError:
long_description = ''
-requires = ['docutils>=0.3', 'numpy>=1.5.1', 'scipy>=0.14',
- 'python-geotiepoints>=1.1.1',
- 'h5py>=2.5', 'requests', 'pyyaml',
- 'appdirs']
+requires = ['docutils>=0.3', 'numpy', 'scipy', 'python-geotiepoints>=1.1.1',
+ 'h5py>=2.5', 'requests', 'pyyaml', 'appdirs']
dask_extra = ['dask[array]']
-test_requires = ['pyyaml', 'dask[array]', 'xlrd', 'pytest', 'xarray']
+test_requires = ['pyyaml', 'dask[array]', 'xlrd', 'pytest', 'xarray', 'responses']
NAME = 'pyspectral'
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyspectral/-/compare/6476c4318bb1d0253a3e02b7b55958956534a8d8...5ff0c47b27c2e27a9bd4559216ff2022a596dd76
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyspectral/-/compare/6476c4318bb1d0253a3e02b7b55958956534a8d8...5ff0c47b27c2e27a9bd4559216ff2022a596dd76
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20221012/b8f54fa2/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list