[Git][debian-gis-team/mapproxy][upstream] New upstream version 1.14.0
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Wed Nov 24 14:44:08 GMT 2021
Bas Couwenberg pushed to branch upstream at Debian GIS Project / mapproxy
Commits:
4413f702 by Bas Couwenberg at 2021-11-24T15:03:34+01:00
New upstream version 1.14.0
- - - - -
26 changed files:
- + .github/workflows/test.yml
- − .travis.yml
- CHANGES.txt
- + DEVELOPMENT.md
- doc/conf.py
- doc/configuration.rst
- doc/mapproxy_util.rst
- doc/seed.rst
- doc/sources.rst
- mapproxy/cache/tile.py
- mapproxy/config/loader.py
- mapproxy/config/spec.py
- mapproxy/image/__init__.py
- mapproxy/response.py
- mapproxy/script/conf/app.py
- mapproxy/seed/script.py
- mapproxy/seed/seeder.py
- mapproxy/seed/spec.py
- mapproxy/source/error.py
- + mapproxy/test/system/fixture/tileservice_refresh.yaml
- + mapproxy/test/system/test_refresh.py
- + mapproxy/test/system/test_response_headers.py
- mapproxy/test/system/test_seed.py
- mapproxy/util/times.py
- requirements-tests.txt
- setup.py
Changes:
=====================================
.github/workflows/test.yml
=====================================
@@ -0,0 +1,73 @@
+name: Running mapproxy tests
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - master
+ pull_request:
+ branches:
+ - master
+
+jobs:
+ test:
+ runs-on: ubuntu-latest
+ services:
+ redis-server:
+ image: redis
+ ports:
+ - 6379:6379
+ couchdb:
+ image: couchdb:2
+ ports:
+ - 5984:5984
+ riak:
+ image: basho/riak-kv
+ ports:
+ - 8087:8087
+ - 8098:8098
+
+ strategy:
+ matrix:
+ python-version: [2.7, 3.6, 3.7, 3.8, 3.9]
+
+ env:
+ MAPPROXY_TEST_COUCHDB: 'http://localhost:5984'
+ MAPPROXY_TEST_REDIS: '127.0.0.1:6379'
+ MAPPROXY_TEST_RIAK_HTTP: 'http://localhost:8098'
+ MAPPROXY_TEST_RIAK_PBC: 'pbc://localhost:8087'
+ # do not load /etc/boto.cfg with Python 3 incompatible plugin
+ # https://github.com/travis-ci/travis-ci/issues/5246#issuecomment-166460882
+ BOTO_CONFIG: '/doesnotexist'
+
+ steps:
+ - name: Install packages
+ run: |
+ sudo apt update
+ sudo apt install proj-bin libgeos-dev libgdal-dev libxslt1-dev libxml2-dev build-essential python-dev libjpeg-dev zlib1g-dev libfreetype6-dev protobuf-compiler libprotoc-dev -y
+
+ - name: Checkout sources
+ uses: actions/checkout at v2
+
+ - name: Use python ${{ matrix.python-version }}
+ uses: actions/setup-python at v2
+ with:
+ python-version: ${{ matrix.python-version }}
+
+ - name: Cache python deps 💾
+ uses: actions/cache at v2
+ with:
+ path: ~/.cache/pip
+ key: ${{ runner.OS }}-python-${{ hashFiles('**/requirements-tests.txt') }}
+ restore-keys: |
+ ${{ runner.OS }}-python-
+ ${{ runner.OS }}-
+
+ - name: Install dependencies ⏬
+ run: |
+ pip install -r requirements-tests.txt
+ if [[ ${{ matrix.python-version }} = 2.7 || ${{ matrix.python-version }} = 3.8 ]]; then pip install -U "Pillow!=8.3.0,!=8.3.1"; fi
+ pip freeze
+
+ - name: Run tests 🏗️
+ run: pytest mapproxy
=====================================
.travis.yml deleted
=====================================
@@ -1,63 +0,0 @@
-language: python
-
-python:
- - "2.7"
- - "3.5"
- - "3.6"
- - "3.7"
- - "3.8"
-
-services:
- - couchdb
- - redis-server
- - docker
-
-addons:
- apt:
- packages:
- - proj-bin
- - libgeos-dev
- - libgdal-dev
- - libxslt1-dev
- - libxml2-dev
- - build-essential
- - python-dev
- - libjpeg-dev
- - zlib1g-dev
- - libfreetype6-dev
- - protobuf-compiler
- - libprotoc-dev
-
-env:
- global:
- - MAPPROXY_TEST_COUCHDB=http://127.0.0.1:5984
- - MAPPROXY_TEST_REDIS=127.0.0.1:6379
- - MAPPROXY_TEST_RIAK_HTTP=http://localhost:8098
- - MAPPROXY_TEST_RIAK_PBC=pbc://localhost:8087
-
- # do not load /etc/boto.cfg with Python 3 incompatible plugin
- # https://github.com/travis-ci/travis-ci/issues/5246#issuecomment-166460882
- - BOTO_CONFIG=/doesnotexist
-
-matrix:
- include:
- # Test 2.7 and 3.8 also with latest Pillow version
- - python: "2.7"
- env: USE_LATEST_PILLOW=1
- - python: "3.8"
- env: USE_LATEST_PILLOW=1
-
-cache:
- directories:
- - $HOME/.cache/pip
-
-before_install:
- - docker run --detach --rm --publish 8087:8087 --publish 8098:8098 basho/riak-kv
-
-install:
- - "pip install -r requirements-tests.txt"
- - "if [[ $USE_LATEST_PILLOW = '1' ]]; then pip install -U Pillow; fi"
- - "pip freeze"
-
-script:
- - pytest mapproxy
=====================================
CHANGES.txt
=====================================
@@ -1,6 +1,21 @@
Nightly
~~~~~~~~~~~~~~~~~
+1.14.0 2021-11-24
+~~~~~~~~~~~~~~~~~
+
+Improvements:
+
+- Refresh while serving (#518).
+- Enabled commandline option `skip uncached` (#515)
+- Several dependencies updated
+- Support for python 3.5 has been dropped because of its EOL, 3.9 has been added
+
+Fixes:
+
+- Several minor bugfixes
+- Security fix to avoid potential web cache poisoning.
+
1.13.2 2021-07-14
~~~~~~~~~~~~~~~~~
=====================================
DEVELOPMENT.md
=====================================
@@ -0,0 +1,83 @@
+Dev Setup
+=========
+
+* Create parent directory for source, applications and the virtual env
+* Clone source into directory mapproxy: `git clone `
+* Install dependencies: https://mapproxy.org/docs/latest/install.html#install-dependencies
+* Create virtualenv: `python3.6 -m venv ./venv`
+* Activate virtualenv: `source venv/bin/activate`
+* Install mapproxy: `pip install -e mapproxy/`
+* Install dev dependencies: `pip install -r mapproxy/requirements-tests.txt`
+* Run tests:
+ * `cd mapproxy`
+ * `pytest mapproxy`
+ * Run single test: `pytest mapproxy/test/unit/test_grid.py -v`
+* Create an application: `mapproxy-util create -t base-config apps/base`
+
+* Start a dev server in debug mode: `mapproxy-util serve-develop apps/base/mapproxy.yaml --debug`
+
+
+Coding Style
+------------
+
+PEP8: https://www.python.org/dev/peps/pep-0008/
+
+
+Debugging
+---------
+
+ - With PyCharm:
+ * Attach to dev server with https://www.jetbrains.com/help/pycharm/attaching-to-local-process.html
+
+ - With ipython:
+ * `pip install ipython ipdb`
+
+ - With Visual Studio Code:
+ * After creating a virtual env and mapproxy configuration:
+ * Create a `launch.json` file in the project-root/.vscode directory with the following content:
+ ```
+ {
+ "version": "0.2.0",
+ "configurations": [
+ {
+ "name": "Debug local mapproxy",
+ "type": "python",
+ "request": "launch",
+ "program": ".venv/bin/mapproxy-util",
+ "args": ["serve-develop", "-b", ":1234", "config/mapproxy.yaml"],
+ "console": "integratedTerminal",
+ "autoReload": {
+ "enable": true
+ }
+ }
+ ]
+ }
+ ```
+
+ * Then start debugging by hitting `F5`.
+
+
+Some more details in the documentation
+--------------------------------------
+
+See https://mapproxy.org/docs/latest/development.html
+
+
+Some incomplete notes about the structure of the software
+---------------------------------------------------------
+
+A mapproxy app decides on the request-URL which handler it starts. There exist different handlers for WMS, WMTS.
+
+Incoming http requests are transformed into own request objects (for example `WMSRequest`).
+
+The class `TileManager` decides if tiles are served from cache or from a source.
+
+All caches need to implement the interface `TileCacheBase`.
+
+The code in `config/` builds mapproxy out of a configuration. `config/spec.py` validates the config.
+
+The sources live in `source/` which in turn use low-level functions from `client/` to request the data.
+
+The file `layer.py` merges/clips/transforms tiles.
+
+The whole of MapProxy is stateless apart from the chache which uses locks on file system level.
=====================================
doc/conf.py
=====================================
@@ -49,9 +49,9 @@ copyright = u'Oliver Tonnhofer, Omniscale'
# built documents.
#
# The short X.Y version.
-version = '1.13'
+version = '1.14'
# The full version, including alpha/beta/rc tags.
-release = '1.13.2'
+release = '1.14.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
=====================================
doc/configuration.rst
=====================================
@@ -513,6 +513,43 @@ To trigger the rescaling behaviour, a tile needs to be missing in the cache and
Tiles created by the ``upscale_tiles`` or ``downscale_tiles`` option are only stored in the cache if this option is set to true.
+``refresh_before``
+"""""""""""""""""""
+
+Here you can force MapProxy to refresh tiles from the source while serving if they are found to be expired.
+The validity conditions are the same as for seeding:
+
+Explanation::
+
+ # absolute as ISO time
+ refresh_before:
+ time: 2010-10-21T12:35:00
+
+ # relative from the time of the tile request
+ refresh_before:
+ weeks: 1
+ days: 7
+ hours: 4
+ minutes: 15
+
+ # modification time of a given file
+ refresh_before:
+ mtime: path/to/file
+
+Example
+~~~~~~~~
+
+::
+
+ caches:
+ osm_cache:
+ grids: ['osm_grid']
+ sources: [OSM]
+ disable_storage: false
+ refresh_before:
+ days: 1
+
+
``disable_storage``
""""""""""""""""""""
=====================================
doc/mapproxy_util.rst
=====================================
@@ -73,7 +73,7 @@ base-config:
log-ini:
- Creates an example logging configuration. You need to pass the target filename to the command.
+ Creates an example logging configuration. You need to pass the target filename to the command (i.e. `my-app/log.ini`).
wsgi-app:
Creates an example server script for the given MapProxy configuration (:option:`--f/--mapproxy-conf<mapproxy-util create -f>`) . You need to pass the target filename to the command.
@@ -105,6 +105,10 @@ You need to pass the MapProxy configuration as an argument. The server will auto
The server address where the HTTP server should listen for incomming connections. Can be a port (``:8080``), a host (``localhost``) or both (``localhost:8081``). The default is ``localhost:8080``. You need to use ``0.0.0.0`` to be able to connect to the server from external clients.
+.. cmdoption:: --debug
+
+ The server outputs debug logging information to the console.
+
Example
-------
=====================================
doc/seed.rst
=====================================
@@ -69,6 +69,16 @@ Options
This will simulate the seed/cleanup process without requesting, creating or removing any tiles.
+.. option:: -l N, --skip-geoms-for-last-levels N
+
+ This will skip checking the intersections between tiles and seed geometries on the last N levels.
+
+.. option:: --skip-uncached
+
+ This will seed only tiles which are already in the cache. This option is interesting in combination with
+ the configuration entry `refresh_before`_ to refresh regularly the existing tiles and to avoid loading
+ all available tiles.
+
.. option:: --summary
Print a summary of all seeding and cleanup tasks and exit.
=====================================
doc/sources.rst
=====================================
@@ -242,6 +242,12 @@ Each status code takes the following options:
You need to enable ``transparent`` for your source, if you use ``on_error`` responses with transparency.
+``authorize_stale``
+
+ Set this to ``True`` if MapProxy should serve in priority stale tiles present in cache. If the specified source error occurs, MapProxy will serve a stale tile which is still in cache instead of the error reponse, even if the tile in cache should be refreshed according to refresh_before date. Otherwise (``False``) MapProxy will serve the unicolor error response defined by the error handler if the source is faulty and the tile is not in cache, or is stale.
+
+You need to enable ``transparent`` for your source, if you use ``on_error`` responses with transparency.
+
::
my_tile_source:
@@ -250,6 +256,10 @@ You need to enable ``transparent`` for your source, if you use ``on_error`` resp
url: http://localhost:8080/service?
layers: base
on_error:
+ 404:
+ response: 'transparent'
+ cache: False
+ authorize_stale: True
500:
response: '#ede9e3'
cache: False
=====================================
mapproxy/cache/tile.py
=====================================
@@ -44,8 +44,10 @@ from mapproxy.image.opts import ImageOptions
from mapproxy.image.merge import merge_images
from mapproxy.image.tile import TileSplitter, TiledImage
from mapproxy.layer import MapQuery, BlankImage
+from mapproxy.source import SourceError
from mapproxy.util import async_
-from mapproxy.util.py import reraise
+from mapproxy.util.py import reraise, reraise_exception
+import sys
class TileManager(object):
@@ -76,6 +78,7 @@ class TileManager(object):
self.sources = sources
self.minimize_meta_requests = minimize_meta_requests
self._expire_timestamp = None
+ self._refresh_before = {}
self.pre_store_filter = pre_store_filter or []
self.concurrent_tile_creators = concurrent_tile_creators
self.tile_creator_class = tile_creator_class or TileCreator
@@ -202,7 +205,9 @@ class TileManager(object):
max_mtime = self.expire_timestamp(tile)
if cached and max_mtime is not None:
self.cache.load_tile_metadata(tile)
- stale = tile.timestamp < max_mtime
+ # file time stamp must be rounded to integer since time conversion functions
+ # mktime and timetuple strip decimals from seconds
+ stale = int(tile.timestamp) <= max_mtime
if stale:
cached = False
return cached
@@ -228,6 +233,9 @@ class TileManager(object):
:note: Returns _expire_timestamp by default.
"""
+ if self._refresh_before:
+ from mapproxy.seed.config import before_timestamp_from_options
+ return before_timestamp_from_options(self._refresh_before)
return self._expire_timestamp
def apply_tile_filter(self, tile):
@@ -318,6 +326,12 @@ class TileCreator(object):
"""
return self.tile_mgr.is_cached(tile)
+ def is_stale(self, tile):
+ """
+ Return True if the tile exists in cache and is expired.
+ """
+ return self.tile_mgr.is_stale(tile)
+
def create_tiles(self, tiles):
if not self.sources:
return []
@@ -362,8 +376,20 @@ class TileCreator(object):
self.tile_mgr.request_format, dimensions=self.dimensions)
with self.tile_mgr.lock(tile):
if not self.is_cached(tile):
- source = self._query_sources(query)
+ try:
+ source = self._query_sources(query)
+ # if source is not available, try to serve tile in cache
+ except SourceError as e:
+ if self.is_stale(tile):
+ self.cache.load_tile(tile)
+ else:
+ reraise_exception(e, sys.exc_info())
if not source: return []
+ if source.authorize_stale and self.is_stale(tile):
+ # The configuration authorises blank tiles generated by the error_handler
+ # to be replaced by stale tiles from cache.
+ self.cache.load_tile(tile)
+ return [tile]
if self.tile_mgr.image_opts != source.image_opts:
# call as_buffer to force conversion into cache format
source.as_buffer(self.tile_mgr.image_opts)
=====================================
mapproxy/config/loader.py
=====================================
@@ -611,11 +611,12 @@ class SourceConfiguration(ConfigurationBase):
raise ConfigurationError("invalid error code %r in on_error", status_code)
cacheable = response_conf.get('cache', False)
color = response_conf.get('response', 'transparent')
+ authorize_stale = response_conf.get('authorize_stale', False)
if color == 'transparent':
color = (255, 255, 255, 0)
else:
color = parse_color(color)
- error_handler.add_handler(status_code, color, cacheable)
+ error_handler.add_handler(status_code, color, cacheable, authorize_stale)
return error_handler
@@ -1574,6 +1575,8 @@ class CacheConfiguration(ConfigurationBase):
cache_rescaled_tiles=cache_rescaled_tiles,
rescale_tiles=rescale_tiles,
)
+ if self.conf['name'] in self.context.caches:
+ mgr._refresh_before = self.context.caches[self.conf['name']].conf.get('refresh_before', {})
extent = merge_layer_extents(sources)
if extent.is_default:
extent = map_extent_from_grid(tile_grid)
=====================================
mapproxy/config/spec.py
=====================================
@@ -36,6 +36,16 @@ def validate_options(conf_dict):
else:
return [], True
+time_spec = {
+ 'seconds': number(),
+ 'minutes': number(),
+ 'hours': number(),
+ 'days': number(),
+ 'weeks': number(),
+ 'time': anything(),
+ 'mtime': str(),
+}
+
coverage = recursive({
'polygons': str(),
'polygons_srs': str(),
@@ -179,6 +189,7 @@ on_error = {
anything(): {
required('response'): one_of([int], str),
'cache': bool,
+ 'authorize_stale': bool
}
}
@@ -419,6 +430,7 @@ mapproxy_yaml_spec = {
'cache_rescaled_tiles': bool(),
'upscale_tiles': int(),
'downscale_tiles': int(),
+ 'refresh_before': time_spec,
'watermark': {
'text': string_type,
'font_size': number(),
@@ -588,8 +600,7 @@ mapproxy_yaml_spec = {
}
})])
),
- # `parts` can be used for partial configurations that are referenced
- # from other sections (e.g. coverages, dimensions, etc.)
+ # `parts` can be used for partial configurations that are referenced
+ # from other sections (e.g. coverages, dimensions, etc.)
'parts': anything(),
}
-
=====================================
mapproxy/image/__init__.py
=====================================
@@ -89,7 +89,24 @@ class GeoReference(object):
return tags
-class ImageSource(object):
+class BaseImageSource(object):
+ """
+ Virtual parent class for ImageSource and BlankImageSource
+ """
+ def __init__(self):
+ raise Exception("Virtual class BaseImageSource, cannot be instanciated.")
+
+ def as_image(self):
+ raise Exception("Virtual class BaseImageSource, method as_image cannot be called.")
+
+ def as_buffer(self, image_opts=None, format=None, seekable=False):
+ raise Exception("Virtual class BaseImageSource, method as_buffer cannot be called.")
+
+ def close_buffers(self):
+ pass
+
+
+class ImageSource(BaseImageSource):
"""
This class wraps either a PIL image, a file-like object, or a file name.
You can access the result as an image (`as_image` ) or a file-like buffer
@@ -111,6 +128,7 @@ class ImageSource(object):
self._size = size
self.cacheable = cacheable
self.georef = georef
+ self.authorize_stale = False
@property
def source(self):
@@ -238,7 +256,7 @@ def SubImageSource(source, size, offset, image_opts, cacheable=True):
img.paste(subimg, offset)
return ImageSource(img, size=size, image_opts=new_image_opts, cacheable=cacheable)
-class BlankImageSource(object):
+class BlankImageSource(BaseImageSource):
"""
ImageSource for transparent or solid-color images.
Implements optimized as_buffer() method.
@@ -249,6 +267,7 @@ class BlankImageSource(object):
self._buf = None
self._img = None
self.cacheable = cacheable
+ self.authorize_stale = False
def as_image(self):
if not self._img:
=====================================
mapproxy/response.py
=====================================
@@ -42,6 +42,12 @@ class Response(object):
content_type = self.default_content_type
self.headers['Content-type'] = content_type
+ if content_type.startswith(('text/', 'application/')):
+ # Capability documents can be dependent on the value of a few X-headers.
+ # Tell this caching proxies via the Vary HTTP header. This also prevents
+ # malicious cache poisoning.
+ self.headers['Vary'] = 'X-Script-Name, X-Forwarded-Host, X-Forwarded-Proto'
+
def _status_set(self, status):
if isinstance(status, int):
status = status_code(status)
=====================================
mapproxy/script/conf/app.py
=====================================
@@ -71,7 +71,10 @@ def write_header(f, capabilities):
@contextmanager
def file_or_stdout(name):
if name == '-':
- yield codecs.getwriter('utf-8')(sys.stdout)
+ if hasattr(sys.stdout, 'buffer'):
+ yield codecs.getwriter('utf-8')(sys.stdout.buffer)
+ else:
+ yield codecs.getwriter('utf-8')(sys.stdout)
else:
with open(name, 'wb') as f:
yield codecs.getwriter('utf-8')(f)
=====================================
mapproxy/seed/script.py
=====================================
@@ -105,6 +105,11 @@ class SeedScript(object):
metavar="N",
help="do not check for intersections between tiles"
" and seed geometries on the last N levels")
+ parser.add_option("--skip-uncached",
+ action="store_true", dest="skip_uncached", default=False,
+ help="only treat tiles which are already present in the cache."
+ " This can be used with the configuration entry `refresh_before`"
+ " to refresh only the existing cache.")
parser.add_option("--summary",
action="store_true", dest="summary", default=False,
help="print summary with all seeding tasks and exit."
@@ -243,7 +248,8 @@ class SeedScript(object):
progress_store=progress)
seed(seed_tasks, progress_logger=logger, dry_run=options.dry_run,
concurrency=options.concurrency, cache_locker=cache_locker,
- skip_geoms_for_last_levels=options.geom_levels)
+ skip_geoms_for_last_levels=options.geom_levels,
+ skip_uncached=options.skip_uncached)
if cleanup_tasks:
print('========== Cleanup tasks ==========')
print('Start cleanup process (%d task%s)' % (
=====================================
mapproxy/seed/seeder.py
=====================================
@@ -477,7 +477,7 @@ class CleanupTask(object):
return NONE
def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
- progress_logger=None, cache_locker=None):
+ progress_logger=None, cache_locker=None, skip_uncached=False):
if cache_locker is None:
cache_locker = DummyCacheLocker()
@@ -496,7 +496,7 @@ def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
start_progress = None
seed_progress = SeedProgress(old_progress_identifier=start_progress)
seed_task(task, concurrency, dry_run, skip_geoms_for_last_levels, progress_logger,
- seed_progress=seed_progress)
+ seed_progress=seed_progress, skip_uncached=skip_uncached)
except CacheLockedError:
print(' ...cache is locked, skipping')
active_tasks = [task] + active_tasks[:-1]
@@ -505,7 +505,7 @@ def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
def seed_task(task, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
- progress_logger=None, seed_progress=None):
+ progress_logger=None, seed_progress=None, skip_uncached=False):
if task.coverage is False:
return
if task.refresh_timestamp is not None:
@@ -518,7 +518,11 @@ def seed_task(task, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
tile_worker_pool = TileWorkerPool(task, TileSeedWorker, dry_run=dry_run,
size=concurrency, progress_logger=progress_logger)
- tile_walker = TileWalker(task, tile_worker_pool, handle_uncached=True,
+ # If the configuration requests to only refresh tiles which are already in cache,
+ # tile walker parameters shall be adapted
+ handle_stale = skip_uncached
+ handle_uncached = not skip_uncached
+ tile_walker = TileWalker(task, tile_worker_pool, handle_uncached=handle_uncached, handle_stale=handle_stale,
skip_geoms_for_last_levels=skip_geoms_for_last_levels, progress_logger=progress_logger,
seed_progress=seed_progress,
work_on_metatiles=work_on_metatiles,
=====================================
mapproxy/seed/spec.py
=====================================
@@ -17,7 +17,7 @@ from mapproxy.util.ext.dictspec.validator import validate, ValidationError
from mapproxy.util.ext.dictspec.spec import one_off, anything, number
from mapproxy.util.ext.dictspec.spec import required
-from mapproxy.config.spec import coverage
+from mapproxy.config.spec import coverage, time_spec
def validate_seed_conf(conf_dict):
"""
@@ -31,16 +31,6 @@ def validate_seed_conf(conf_dict):
else:
return [], True
-time_spec = {
- 'seconds': number(),
- 'minutes': number(),
- 'hours': number(),
- 'days': number(),
- 'weeks': number(),
- 'time': anything(),
- 'mtime': str(),
-}
-
from_to_spec = {
'from': number(),
'to': number(),
=====================================
mapproxy/source/error.py
=====================================
@@ -20,19 +20,20 @@ class HTTPSourceErrorHandler(object):
def __init__(self):
self.response_error_codes = {}
- def add_handler(self, http_code, color, cacheable=False):
- self.response_error_codes[http_code] = (color, cacheable)
+ def add_handler(self, http_code, color, cacheable=False, authorize_stale=False):
+ self.response_error_codes[http_code] = (color, cacheable, authorize_stale)
def handle(self, status_code, query):
color = cacheable = None
if status_code in self.response_error_codes:
- color, cacheable = self.response_error_codes[status_code]
+ color, cacheable, authorize_stale = self.response_error_codes[status_code]
elif 'other' in self.response_error_codes:
- color, cacheable = self.response_error_codes['other']
+ color, cacheable, authorize_stale = self.response_error_codes['other']
else:
return None
transparent = len(color) == 4
image_opts = ImageOptions(bgcolor=color, transparent=transparent)
img_source = BlankImageSource(query.size, image_opts, cacheable=cacheable)
- return img_source
\ No newline at end of file
+ img_source.authorize_stale = authorize_stale
+ return img_source
=====================================
mapproxy/test/system/fixture/tileservice_refresh.yaml
=====================================
@@ -0,0 +1,59 @@
+globals:
+ cache:
+ base_dir: cache_data/
+ meta_size: [1, 1]
+ meta_buffer: 0
+
+services:
+ tms:
+ # origin: 'nw'
+
+layers:
+ - name: wms_cache
+ title: Direct Layer
+ sources: [wms_cache]
+
+ - name: wms_cache_isotime
+ title: Direct Layer
+ sources: [wms_cache_isotime]
+
+ - name: wms_cache_png
+ title: Direct Layer
+ sources: [wms_cache_png]
+
+caches:
+ wms_cache:
+ format: image/jpeg
+ sources: [wms_source]
+ refresh_before:
+ seconds: 1
+
+ wms_cache_isotime:
+ format: image/jpeg
+ sources: [wms_source]
+ refresh_before:
+ time: "2009-02-15T23:31:30"
+
+ wms_cache_png:
+ format: image/png
+ sources: [wms_source]
+ refresh_before:
+ seconds: 1
+
+sources:
+ wms_source:
+ type: wms
+ req:
+ url: http://localhost:42423/service
+ layers: bar
+ on_error:
+ 404:
+ response: 'transparent'
+ cache: False
+ 405:
+ response: '#ff0000'
+ cache: False
+ 406:
+ response: 'transparent'
+ cache: False
+ authorize_stale: True
=====================================
mapproxy/test/system/test_refresh.py
=====================================
@@ -0,0 +1,207 @@
+# This file is part of the MapProxy project.
+# Copyright (C) 2010-2012 Omniscale <http://omniscale.de>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import time
+import hashlib
+
+from io import BytesIO
+import pytest
+
+from mapproxy.compat.image import Image
+from mapproxy.test.image import is_jpeg, tmp_image
+from mapproxy.test.http import mock_httpd
+from mapproxy.test.system import SysTest
+from mapproxy.util.times import timestamp_from_isodate
+
+ at pytest.fixture(scope="module")
+def config_file():
+ return "tileservice_refresh.yaml"
+
+
+class TestRefresh(SysTest):
+
+ def test_refresh_tile_1s(self, app, cache_dir):
+ with tmp_image((256, 256), format="jpeg") as img:
+ expected_req = (
+ {
+ "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+ "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+ "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+ "&WIDTH=256"
+ },
+ {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ file_path = cache_dir.join(
+ "wms_cache_EPSG900913/01/000/000/000/000/000/000.jpeg"
+ )
+ assert file_path.check()
+ t1 = file_path.mtime()
+ # file_path.remove()
+ # assert not file_path.check()
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ assert file_path.check()
+ t2 = file_path.mtime()
+ # tile is expired after 1 sec, so it will be fetched again from mock server
+ time.sleep(1.2)
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ assert file_path.check()
+ t3 = file_path.mtime()
+ assert t2 == t1
+ assert t3 > t2
+
+ def test_refresh_tile_mtime(self, app, cache_dir):
+ with tmp_image((256, 256), format="jpeg") as img:
+ expected_req = (
+ {
+ "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+ "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+ "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+ "&WIDTH=256"
+ },
+ {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ file_path = cache_dir.join(
+ "wms_cache_isotime_EPSG900913/01/000/000/000/000/000/000.jpeg"
+ )
+ assert file_path.check()
+ timestamp = timestamp_from_isodate("2009-02-15T23:31:30")
+ file_path.setmtime(timestamp + 1.2)
+ t1 = file_path.mtime()
+ resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ t2 = file_path.mtime()
+ file_path.setmtime(timestamp - 1.2)
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ assert file_path.check()
+ t3 = file_path.mtime()
+ assert t2 == t1
+ assert t3 > t2
+
+ def test_refresh_tile_source_error_no_stale(self, app, cache_dir):
+ source_request = {
+ "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fpng"
+ "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+ "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+ "&WIDTH=256"
+ }
+ with tmp_image((256, 256), format="png") as img:
+ expected_req = (
+ source_request,
+ {"body": img.read(), "headers": {"content-type": "image/png"}},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+ assert resp.content_type == "image/png"
+ img.seek(0)
+ assert resp.body == img.read()
+ resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+ assert resp.content_type == "image/png"
+ img.seek(0)
+ assert resp.body == img.read()
+ # tile is expired after 1 sec, so it will be requested again from mock server
+ time.sleep(1.2)
+ expected_req = (
+ source_request,
+ {"body": "", "status": 404},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+ assert resp.content_type == "image/png"
+ # error handler for 404 does not authorise stale tiles, so transparent tile will be rendered
+ resp_img = Image.open(BytesIO(resp.body))
+ # check response transparency
+ assert resp_img.getbands() == ('R', 'G', 'B', 'A')
+ assert resp_img.getextrema()[3] == (0, 0)
+
+ expected_req = (
+ source_request,
+ {"body": "", "status": 405},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+ assert resp.content_type == "image/png"
+ # error handler for 405 does not authorise stale tiles, so red tile will be rendered
+ resp_img = Image.open(BytesIO(resp.body))
+ # check response red color
+ assert resp_img.getbands() == ('R', 'G', 'B')
+ assert resp_img.getextrema() == ((255, 255), (0, 0), (0, 0))
+
+ def test_refresh_tile_source_error_stale(self, app, cache_dir):
+ with tmp_image((256, 256), format="jpeg") as img:
+ expected_req = (
+ {
+ "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+ "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+ "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+ "&WIDTH=256"
+ },
+ {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ img.seek(0)
+ assert resp.body == img.read()
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ img.seek(0)
+ assert resp.body == img.read()
+ # tile is expired after 1 sec, so it will be fetched again from mock server
+ time.sleep(1.2)
+ expected_req = (
+ {
+ "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+ "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+ "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+ "&WIDTH=256"
+ },
+ {"body": "", "status": 406},
+ )
+ with mock_httpd(
+ ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+ ):
+ resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+ assert resp.content_type == "image/jpeg"
+ # Check that initial non empty img is served as a stale tile
+ img.seek(0)
+ assert resp.body == img.read()
=====================================
mapproxy/test/system/test_response_headers.py
=====================================
@@ -0,0 +1,54 @@
+# This file is part of the MapProxy project.
+# Copyright (C) 2011 Omniscale <http://omniscale.de>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+
+import pytest
+
+from mapproxy.test.system import SysTest
+
+
+class TestResponseHeaders(SysTest):
+ """
+ Check if the vary header is set for text / xml content like capabilities
+ """
+ @pytest.fixture(scope='class')
+ def config_file(self):
+ return 'auth.yaml'
+
+ def test_tms(self, app):
+ resp = app.get('http://localhost/tms')
+ assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+ def test_wms(self, app):
+ resp = app.get('http://localhost/service?SERVICE=WMS&REQUEST=GetCapabilities'
+ '&VERSION=1.1.0')
+ assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+ def test_wmts(self, app):
+ resp = app.get('http://localhost/service?SERVICE=WMTS&REQUEST=GetCapabilities')
+ assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+ def test_restful_wmts(self, app):
+ resp = app.get('http://localhost/wmts/1.0.0/WMTSCapabilities.xml')
+ assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+ def test_no_endpoint(self, app):
+ resp = app.get('http://localhost/service?')
+ assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+ def test_image_response(self, app):
+ resp = app.get('http://localhost/tms/1.0.0/layer1a/EPSG900913/0/0/0.png')
+ assert resp.vary == None
=====================================
mapproxy/test/system/test_seed.py
=====================================
@@ -86,11 +86,42 @@ class SeedTestBase(SeedTestEnvironment):
{'body': img_data, 'headers': {'content-type': 'image/png'}})
with mock_httpd(('localhost', 42423), [expected_req]):
with local_base_config(self.mapproxy_conf.base_config):
- seed_conf = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
+ seed_conf = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
tasks, cleanup_tasks = seed_conf.seeds(['one']), seed_conf.cleanups()
seed(tasks, dry_run=False)
cleanup(cleanup_tasks, verbose=False, dry_run=False)
+ def test_seed_skip_uncached(self):
+ with tmp_image((256, 256), format='png') as img:
+ img_data = img.read()
+ with local_base_config(self.mapproxy_conf.base_config):
+ expected_req = ({'path': r'/service?LAYERS=foo&SERVICE=WMS&FORMAT=image%2Fpng'
+ '&REQUEST=GetMap&VERSION=1.1.1&bbox=-180.0,-90.0,180.0,90.0'
+ '&width=256&height=128&srs=EPSG:4326'},
+ {'body': img_data, 'headers': {'content-type': 'image/png'}})
+ seed_conf = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
+ tasks, cleanup_tasks = seed_conf.seeds(['one']), seed_conf.cleanups()
+
+ # tile not in cache => skipped by seeder
+ seed(tasks, dry_run=False, skip_uncached=True)
+ assert not self.tile_exists((0, 0, 0))
+
+ with mock_httpd(('localhost', 42423), [expected_req]):
+ # force tile generation in cache (via skip_uncached=False)
+ seed(tasks, dry_run=False, skip_uncached=False)
+ assert self.tile_exists((0, 0, 0))
+
+ # no refresh since tile is not older than 1 day (cf. config seed.yaml)
+ seed(tasks, dry_run=False, skip_uncached=True)
+
+ # create stale tile (older than 1 day)
+ self.make_tile((0, 0, 0), timestamp=time.time() - (60*60*25))
+ with mock_httpd(('localhost', 42423), [expected_req]):
+ # check that old tile in cache is refreshed
+ seed(tasks, dry_run=False, skip_uncached=True)
+ assert self.tile_exists((0, 0, 0))
+ cleanup(cleanup_tasks, verbose=False, dry_run=False)
+
def test_reseed_uptodate(self):
# tile already there.
self.make_tile((0, 0, 0))
=====================================
mapproxy/util/times.py
=====================================
@@ -72,4 +72,4 @@ def timestamp_from_isodate(isodate):
date = isodate
else:
date = datetime.datetime.strptime(isodate, "%Y-%m-%dT%H:%M:%S")
- return mktime(date.timetuple())
\ No newline at end of file
+ return mktime(date.timetuple())
=====================================
requirements-tests.txt
=====================================
@@ -1,8 +1,8 @@
-Jinja2==2.11.2
+Jinja2==2.11.3
MarkupSafe==1.1.1
Pillow==6.2.2;python_version<"3.0"
-Pillow==7.2.0;python_version>="3.0"
-PyYAML==5.3.1
+Pillow==8.2.0;python_version>="3.0"
+PyYAML==5.4
Shapely==1.7.0
WebOb==1.8.6
WebTest==2.0.35
@@ -17,7 +17,7 @@ certifi==2020.6.20
cffi==1.14.2
cfn-lint==0.35.0
chardet==3.0.4
-cryptography==3.0
+cryptography==3.3.2
decorator==4.4.2
docker==4.3.0
docutils==0.15.2
@@ -33,7 +33,7 @@ jsonpickle==1.4.1
jsonpointer==2.0
jsonschema==3.2.0
junit-xml==1.9
-lxml==4.5.2
+lxml==4.6.3
mock==3.0.5;python_version<"3.6"
mock==4.0.2;python_version>="3.6"
more-itertools==5.0.0;python_version<"3.0"
@@ -43,7 +43,7 @@ networkx==2.2;python_version<"3.0"
networkx==2.4;python_version>="3.0"
packaging==20.4
pluggy==0.13.1
-py==1.9.0
+py==1.10.0
pyasn1==0.4.8
pycparser==2.20
pyparsing==2.2.2;python_version<"3.0"
@@ -61,7 +61,7 @@ requests==2.24.0
responses==0.10.16
riak==2.7.0
rsa==4.5;python_version<"3.0"
-rsa==4.6;python_version>="3.0"
+rsa==4.7;python_version>="3.0"
s3transfer==0.3.3
six==1.15.0
soupsieve==1.9.6;python_version<"3.0"
=====================================
setup.py
=====================================
@@ -21,11 +21,11 @@ def package_installed(pkg):
# depend on PIL if it is installed, otherwise
# require Pillow
if package_installed('Pillow'):
- install_requires.append('Pillow !=2.4.0')
+ install_requires.append('Pillow !=2.4.0,!=8.3.0,!=8.3.1')
elif package_installed('PIL'):
install_requires.append('PIL>=1.1.6,<1.2.99')
else:
- install_requires.append('Pillow !=2.4.0')
+ install_requires.append('Pillow !=2.4.0,!=8.3.0,!=8.3.1')
if platform.python_version_tuple() < ('2', '6'):
# for mapproxy-seed
@@ -54,7 +54,7 @@ def long_description(changelog_releases=10):
setup(
name='MapProxy',
- version="1.13.2",
+ version="1.14.0",
description='An accelerating proxy for tile and web map services',
long_description=long_description(7),
author='Oliver Tonnhofer',
View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/commit/4413f702a4ec2fc89d1e2ae1f5cae57f5ac5baf4
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/commit/4413f702a4ec2fc89d1e2ae1f5cae57f5ac5baf4
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20211124/5300307d/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list