[Git][debian-gis-team/mapproxy][master] 5 commits: New upstream version 1.14.0

Bas Couwenberg (@sebastic) gitlab at salsa.debian.org
Wed Nov 24 14:44:00 GMT 2021



Bas Couwenberg pushed to branch master at Debian GIS Project / mapproxy


Commits:
4413f702 by Bas Couwenberg at 2021-11-24T15:03:34+01:00
New upstream version 1.14.0
- - - - -
8288c9f5 by Bas Couwenberg at 2021-11-24T15:03:41+01:00
Update upstream source from tag 'upstream/1.14.0'

Update to upstream version '1.14.0'
with Debian dir 2cbd18d112348fe9d3ab1c92b9b58ed01f60116a
- - - - -
6e0024bf by Bas Couwenberg at 2021-11-24T15:04:57+01:00
New upstream release.

- - - - -
4b6452e4 by Bas Couwenberg at 2021-11-24T15:34:26+01:00
Update lintian overrides.

- - - - -
668f1c62 by Bas Couwenberg at 2021-11-24T15:34:26+01:00
Set distribution to unstable.

- - - - -


28 changed files:

- + .github/workflows/test.yml
- − .travis.yml
- CHANGES.txt
- + DEVELOPMENT.md
- debian/changelog
- − debian/python3-mapproxy.lintian-overrides
- doc/conf.py
- doc/configuration.rst
- doc/mapproxy_util.rst
- doc/seed.rst
- doc/sources.rst
- mapproxy/cache/tile.py
- mapproxy/config/loader.py
- mapproxy/config/spec.py
- mapproxy/image/__init__.py
- mapproxy/response.py
- mapproxy/script/conf/app.py
- mapproxy/seed/script.py
- mapproxy/seed/seeder.py
- mapproxy/seed/spec.py
- mapproxy/source/error.py
- + mapproxy/test/system/fixture/tileservice_refresh.yaml
- + mapproxy/test/system/test_refresh.py
- + mapproxy/test/system/test_response_headers.py
- mapproxy/test/system/test_seed.py
- mapproxy/util/times.py
- requirements-tests.txt
- setup.py


Changes:

=====================================
.github/workflows/test.yml
=====================================
@@ -0,0 +1,73 @@
+name: Running mapproxy tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    services:
+      redis-server:
+        image: redis
+        ports:
+          - 6379:6379
+      couchdb:
+        image: couchdb:2
+        ports:
+          - 5984:5984
+      riak:
+        image: basho/riak-kv
+        ports:
+          - 8087:8087
+          - 8098:8098
+      
+    strategy:
+      matrix:
+        python-version: [2.7, 3.6, 3.7, 3.8, 3.9]
+
+    env:
+      MAPPROXY_TEST_COUCHDB: 'http://localhost:5984'
+      MAPPROXY_TEST_REDIS: '127.0.0.1:6379'
+      MAPPROXY_TEST_RIAK_HTTP: 'http://localhost:8098'
+      MAPPROXY_TEST_RIAK_PBC: 'pbc://localhost:8087'
+      # do not load /etc/boto.cfg with Python 3 incompatible plugin
+      # https://github.com/travis-ci/travis-ci/issues/5246#issuecomment-166460882
+      BOTO_CONFIG: '/doesnotexist'
+
+    steps:
+    - name: Install packages
+      run: |
+        sudo apt update
+        sudo apt install proj-bin libgeos-dev libgdal-dev libxslt1-dev libxml2-dev build-essential python-dev libjpeg-dev zlib1g-dev libfreetype6-dev protobuf-compiler libprotoc-dev -y
+
+    - name: Checkout sources
+      uses: actions/checkout at v2
+
+    - name: Use python ${{ matrix.python-version }}
+      uses: actions/setup-python at v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Cache python deps 💾
+      uses: actions/cache at v2
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.OS }}-python-${{ hashFiles('**/requirements-tests.txt') }}
+        restore-keys: |
+          ${{ runner.OS }}-python-
+          ${{ runner.OS }}-
+
+    - name: Install dependencies ⏬
+      run: |
+        pip install -r requirements-tests.txt
+        if [[ ${{ matrix.python-version }} = 2.7 || ${{ matrix.python-version }} = 3.8 ]]; then pip install -U "Pillow!=8.3.0,!=8.3.1"; fi
+        pip freeze
+
+    - name: Run tests 🏗️
+      run: pytest mapproxy


=====================================
.travis.yml deleted
=====================================
@@ -1,63 +0,0 @@
-language: python
-
-python:
-  - "2.7"
-  - "3.5"
-  - "3.6"
-  - "3.7"
-  - "3.8"
-
-services:
-  - couchdb
-  - redis-server
-  - docker
-
-addons:
-  apt:
-    packages:
-      - proj-bin
-      - libgeos-dev
-      - libgdal-dev
-      - libxslt1-dev
-      - libxml2-dev
-      - build-essential
-      - python-dev
-      - libjpeg-dev
-      - zlib1g-dev
-      - libfreetype6-dev
-      - protobuf-compiler
-      - libprotoc-dev
-
-env:
-  global:
-    - MAPPROXY_TEST_COUCHDB=http://127.0.0.1:5984
-    - MAPPROXY_TEST_REDIS=127.0.0.1:6379
-    - MAPPROXY_TEST_RIAK_HTTP=http://localhost:8098
-    - MAPPROXY_TEST_RIAK_PBC=pbc://localhost:8087
-
-    # do not load /etc/boto.cfg with Python 3 incompatible plugin
-    # https://github.com/travis-ci/travis-ci/issues/5246#issuecomment-166460882
-    - BOTO_CONFIG=/doesnotexist
-
-matrix:
-  include:
-    # Test 2.7 and 3.8 also with latest Pillow version
-    - python: "2.7"
-      env: USE_LATEST_PILLOW=1
-    - python: "3.8"
-      env: USE_LATEST_PILLOW=1
-
-cache:
-  directories:
-    - $HOME/.cache/pip
-
-before_install:
-    - docker run --detach --rm --publish 8087:8087 --publish 8098:8098 basho/riak-kv
-
-install:
-    - "pip install -r requirements-tests.txt"
-    - "if [[ $USE_LATEST_PILLOW = '1' ]]; then pip install -U Pillow; fi"
-    - "pip freeze"
-
-script:
-    - pytest mapproxy


=====================================
CHANGES.txt
=====================================
@@ -1,6 +1,21 @@
 Nightly
 ~~~~~~~~~~~~~~~~~
 
+1.14.0 2021-11-24
+~~~~~~~~~~~~~~~~~
+
+Improvements:
+
+- Refresh while serving (#518).
+- Enabled commandline option `skip uncached` (#515)
+- Several dependencies updated
+- Support for python 3.5 has been dropped because of its EOL, 3.9 has been added
+
+Fixes:
+
+- Several minor bugfixes
+- Security fix to avoid potential web cache poisoning.
+
 1.13.2 2021-07-14
 ~~~~~~~~~~~~~~~~~
 


=====================================
DEVELOPMENT.md
=====================================
@@ -0,0 +1,83 @@
+Dev Setup
+=========
+
+* Create parent directory for source, applications and the virtual env
+* Clone source into directory mapproxy: `git clone `
+* Install dependencies: https://mapproxy.org/docs/latest/install.html#install-dependencies
+* Create virtualenv: `python3.6 -m venv ./venv`
+* Activate virtualenv: `source venv/bin/activate`
+* Install mapproxy: `pip install -e mapproxy/`
+* Install dev dependencies: `pip install -r mapproxy/requirements-tests.txt`
+* Run tests:
+    * `cd mapproxy`
+    * `pytest mapproxy`
+    * Run single test: `pytest mapproxy/test/unit/test_grid.py -v`
+* Create an application: `mapproxy-util create -t base-config apps/base`
+
+* Start a dev server in debug mode: `mapproxy-util serve-develop apps/base/mapproxy.yaml --debug`
+
+
+Coding Style
+------------
+
+PEP8: https://www.python.org/dev/peps/pep-0008/
+
+
+Debugging
+---------
+
+  - With PyCharm:
+    * Attach to dev server with https://www.jetbrains.com/help/pycharm/attaching-to-local-process.html
+
+  - With ipython:
+    * `pip install ipython ipdb`
+
+  - With Visual Studio Code:
+    * After creating a virtual env and mapproxy configuration:
+    * Create a `launch.json` file in the project-root/.vscode directory with the following content:
+    ```
+    {
+      "version": "0.2.0",
+      "configurations": [
+        {
+          "name": "Debug local mapproxy",
+          "type": "python",
+          "request": "launch",
+          "program": ".venv/bin/mapproxy-util",
+          "args": ["serve-develop", "-b", ":1234", "config/mapproxy.yaml"],
+          "console": "integratedTerminal",
+          "autoReload": {
+            "enable": true
+          }
+        }
+      ]
+    }
+    ```
+
+    * Then start debugging by hitting `F5`.
+
+
+Some more details in the documentation
+--------------------------------------
+
+See https://mapproxy.org/docs/latest/development.html
+
+
+Some incomplete notes about the structure of the software
+---------------------------------------------------------
+
+A mapproxy app decides on the request-URL which handler it starts. There exist different handlers for WMS, WMTS.
+
+Incoming http requests are transformed into own request objects (for example `WMSRequest`).
+
+The class `TileManager` decides if tiles are served from cache or from a source.
+
+All caches need to implement the interface `TileCacheBase`.
+
+The code in `config/` builds mapproxy out of a configuration. `config/spec.py` validates the config.
+
+The sources live in `source/` which in turn use low-level functions from `client/` to request the data.
+
+The file `layer.py` merges/clips/transforms tiles.
+
+The whole of MapProxy is stateless apart from the chache which uses locks on file system level.


=====================================
debian/changelog
=====================================
@@ -1,11 +1,12 @@
-mapproxy (1.13.2-2) UNRELEASED; urgency=medium
+mapproxy (1.14.0-1) unstable; urgency=medium
 
+  * New upstream release.
   * Bump Standards-Version to 4.6.0, no changes.
   * Bump debhelper compat to 12, changes:
     - Drop --list-missing from dh_install
   * Update lintian overrides.
 
- -- Bas Couwenberg <sebastic at debian.org>  Wed, 08 Sep 2021 16:40:30 +0200
+ -- Bas Couwenberg <sebastic at debian.org>  Wed, 24 Nov 2021 15:06:43 +0100
 
 mapproxy (1.13.2-1) unstable; urgency=medium
 


=====================================
debian/python3-mapproxy.lintian-overrides deleted
=====================================
@@ -1,3 +0,0 @@
-# Not a problem
-breakout-link *
-


=====================================
doc/conf.py
=====================================
@@ -49,9 +49,9 @@ copyright = u'Oliver Tonnhofer, Omniscale'
 # built documents.
 #
 # The short X.Y version.
-version = '1.13'
+version = '1.14'
 # The full version, including alpha/beta/rc tags.
-release = '1.13.2'
+release = '1.14.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.


=====================================
doc/configuration.rst
=====================================
@@ -513,6 +513,43 @@ To trigger the rescaling behaviour, a tile needs to be missing in the cache and
 
 Tiles created by the ``upscale_tiles`` or ``downscale_tiles`` option are only stored in the cache if this option is set to true.
 
+``refresh_before``
+"""""""""""""""""""
+
+Here you can force MapProxy to refresh tiles from the source while serving if they are found to be expired.
+The validity conditions are the same as for seeding:
+
+Explanation::
+
+  # absolute as ISO time
+  refresh_before:
+    time: 2010-10-21T12:35:00
+
+  # relative from the time of the tile request
+  refresh_before:
+    weeks: 1
+    days: 7
+    hours: 4
+    minutes: 15
+
+  # modification time of a given file
+  refresh_before:
+    mtime: path/to/file
+
+Example
+~~~~~~~~
+
+::
+
+   caches:
+     osm_cache:
+     grids: ['osm_grid']
+     sources: [OSM]
+     disable_storage: false
+     refresh_before:
+       days: 1
+
+
 ``disable_storage``
 """"""""""""""""""""
 


=====================================
doc/mapproxy_util.rst
=====================================
@@ -73,7 +73,7 @@ base-config:
 
 
 log-ini:
-  Creates an example logging configuration. You need to pass the target filename to the command.
+  Creates an example logging configuration. You need to pass the target filename to the command (i.e. `my-app/log.ini`).
 
 wsgi-app:
   Creates an example server script for the given MapProxy configuration (:option:`--f/--mapproxy-conf<mapproxy-util create -f>`) . You need to pass the target filename to the command.
@@ -105,6 +105,10 @@ You need to pass the MapProxy configuration as an argument. The server will auto
 
   The server address where the HTTP server should listen for incomming connections. Can be a port (``:8080``), a host (``localhost``) or both (``localhost:8081``). The default is ``localhost:8080``. You need to use ``0.0.0.0`` to be able to connect to the server from external clients.
 
+.. cmdoption:: --debug
+
+  The server outputs debug logging information to the console.
+
 
 Example
 -------


=====================================
doc/seed.rst
=====================================
@@ -69,6 +69,16 @@ Options
 
   This will simulate the seed/cleanup process without requesting, creating or removing any tiles.
 
+.. option:: -l N, --skip-geoms-for-last-levels N
+
+  This will skip checking the intersections between tiles and seed geometries on the last N levels.
+
+.. option:: --skip-uncached
+
+  This will seed only tiles which are already in the cache. This option is interesting in combination with
+  the configuration entry `refresh_before`_ to refresh regularly the existing tiles and to avoid loading
+  all available tiles.
+
 .. option:: --summary
 
   Print a summary of all seeding and cleanup tasks and exit.


=====================================
doc/sources.rst
=====================================
@@ -242,6 +242,12 @@ Each status code takes the following options:
 
 You need to enable ``transparent`` for your source, if you use ``on_error`` responses with transparency.
 
+``authorize_stale``
+
+  Set this to ``True`` if MapProxy should serve in priority stale tiles present in cache. If the specified source error occurs, MapProxy will serve a stale tile which is still in cache instead of the error reponse, even if the tile in cache should be refreshed according to refresh_before date. Otherwise (``False``) MapProxy will serve the unicolor error response defined by the error handler if the source is faulty and the tile is not in cache, or is stale.
+
+You need to enable ``transparent`` for your source, if you use ``on_error`` responses with transparency.
+
 ::
 
   my_tile_source:
@@ -250,6 +256,10 @@ You need to enable ``transparent`` for your source, if you use ``on_error`` resp
       url: http://localhost:8080/service?
       layers: base
     on_error:
+      404:
+        response: 'transparent'
+        cache: False
+        authorize_stale: True
       500:
         response: '#ede9e3'
         cache: False


=====================================
mapproxy/cache/tile.py
=====================================
@@ -44,8 +44,10 @@ from mapproxy.image.opts import ImageOptions
 from mapproxy.image.merge import merge_images
 from mapproxy.image.tile import TileSplitter, TiledImage
 from mapproxy.layer import MapQuery, BlankImage
+from mapproxy.source import SourceError
 from mapproxy.util import async_
-from mapproxy.util.py import reraise
+from mapproxy.util.py import reraise, reraise_exception
+import sys
 
 
 class TileManager(object):
@@ -76,6 +78,7 @@ class TileManager(object):
         self.sources = sources
         self.minimize_meta_requests = minimize_meta_requests
         self._expire_timestamp = None
+        self._refresh_before = {}
         self.pre_store_filter = pre_store_filter or []
         self.concurrent_tile_creators = concurrent_tile_creators
         self.tile_creator_class = tile_creator_class or TileCreator
@@ -202,7 +205,9 @@ class TileManager(object):
         max_mtime = self.expire_timestamp(tile)
         if cached and max_mtime is not None:
             self.cache.load_tile_metadata(tile)
-            stale = tile.timestamp < max_mtime
+            # file time stamp must be rounded to integer since time conversion functions
+            # mktime and timetuple strip decimals from seconds
+            stale = int(tile.timestamp) <= max_mtime
             if stale:
                 cached = False
         return cached
@@ -228,6 +233,9 @@ class TileManager(object):
 
         :note: Returns _expire_timestamp by default.
         """
+        if self._refresh_before:
+            from mapproxy.seed.config import before_timestamp_from_options
+            return before_timestamp_from_options(self._refresh_before)
         return self._expire_timestamp
 
     def apply_tile_filter(self, tile):
@@ -318,6 +326,12 @@ class TileCreator(object):
         """
         return self.tile_mgr.is_cached(tile)
 
+    def is_stale(self, tile):
+        """
+        Return True if the tile exists in cache and is expired.
+        """
+        return self.tile_mgr.is_stale(tile)
+
     def create_tiles(self, tiles):
         if not self.sources:
             return []
@@ -362,8 +376,20 @@ class TileCreator(object):
                          self.tile_mgr.request_format, dimensions=self.dimensions)
         with self.tile_mgr.lock(tile):
             if not self.is_cached(tile):
-                source = self._query_sources(query)
+                try:
+                    source = self._query_sources(query)
+                # if source is not available, try to serve tile in cache
+                except SourceError as e:
+                    if self.is_stale(tile):
+                        self.cache.load_tile(tile)
+                    else:
+                        reraise_exception(e, sys.exc_info())
                 if not source: return []
+                if source.authorize_stale and self.is_stale(tile):
+                    # The configuration authorises blank tiles generated by the error_handler
+                    # to be replaced by stale tiles from cache.
+                    self.cache.load_tile(tile)
+                    return [tile]
                 if self.tile_mgr.image_opts != source.image_opts:
                     # call as_buffer to force conversion into cache format
                     source.as_buffer(self.tile_mgr.image_opts)


=====================================
mapproxy/config/loader.py
=====================================
@@ -611,11 +611,12 @@ class SourceConfiguration(ConfigurationBase):
                 raise ConfigurationError("invalid error code %r in on_error", status_code)
             cacheable = response_conf.get('cache', False)
             color = response_conf.get('response', 'transparent')
+            authorize_stale = response_conf.get('authorize_stale', False)
             if color == 'transparent':
                 color = (255, 255, 255, 0)
             else:
                 color = parse_color(color)
-            error_handler.add_handler(status_code, color, cacheable)
+            error_handler.add_handler(status_code, color, cacheable, authorize_stale)
 
         return error_handler
 
@@ -1574,6 +1575,8 @@ class CacheConfiguration(ConfigurationBase):
                 cache_rescaled_tiles=cache_rescaled_tiles,
                 rescale_tiles=rescale_tiles,
             )
+            if self.conf['name'] in self.context.caches:
+                mgr._refresh_before = self.context.caches[self.conf['name']].conf.get('refresh_before', {})
             extent = merge_layer_extents(sources)
             if extent.is_default:
                 extent = map_extent_from_grid(tile_grid)


=====================================
mapproxy/config/spec.py
=====================================
@@ -36,6 +36,16 @@ def validate_options(conf_dict):
     else:
         return [], True
 
+time_spec = {
+    'seconds': number(),
+    'minutes': number(),
+    'hours': number(),
+    'days': number(),
+    'weeks': number(),
+    'time': anything(),
+    'mtime': str(),
+}
+
 coverage = recursive({
     'polygons': str(),
     'polygons_srs': str(),
@@ -179,6 +189,7 @@ on_error = {
     anything(): {
         required('response'): one_of([int], str),
         'cache': bool,
+        'authorize_stale': bool
     }
 }
 
@@ -419,6 +430,7 @@ mapproxy_yaml_spec = {
             'cache_rescaled_tiles': bool(),
             'upscale_tiles': int(),
             'downscale_tiles': int(),
+            'refresh_before': time_spec,
             'watermark': {
                 'text': string_type,
                 'font_size': number(),
@@ -588,8 +600,7 @@ mapproxy_yaml_spec = {
             }
         })])
     ),
-     # `parts` can be used for partial configurations that are referenced
-     # from other sections (e.g. coverages, dimensions, etc.)
+    # `parts` can be used for partial configurations that are referenced
+    # from other sections (e.g. coverages, dimensions, etc.)
     'parts': anything(),
 }
-


=====================================
mapproxy/image/__init__.py
=====================================
@@ -89,7 +89,24 @@ class GeoReference(object):
         return tags
 
 
-class ImageSource(object):
+class BaseImageSource(object):
+    """
+    Virtual parent class for ImageSource and BlankImageSource
+    """
+    def __init__(self):
+        raise Exception("Virtual class BaseImageSource, cannot be instanciated.")
+
+    def as_image(self):
+        raise Exception("Virtual class BaseImageSource, method as_image cannot be called.")
+
+    def as_buffer(self, image_opts=None, format=None, seekable=False):
+        raise Exception("Virtual class BaseImageSource, method as_buffer cannot be called.")
+
+    def close_buffers(self):
+        pass
+
+
+class ImageSource(BaseImageSource):
     """
     This class wraps either a PIL image, a file-like object, or a file name.
     You can access the result as an image (`as_image` ) or a file-like buffer
@@ -111,6 +128,7 @@ class ImageSource(object):
         self._size = size
         self.cacheable = cacheable
         self.georef = georef
+        self.authorize_stale = False
 
     @property
     def source(self):
@@ -238,7 +256,7 @@ def SubImageSource(source, size, offset, image_opts, cacheable=True):
     img.paste(subimg, offset)
     return ImageSource(img, size=size, image_opts=new_image_opts, cacheable=cacheable)
 
-class BlankImageSource(object):
+class BlankImageSource(BaseImageSource):
     """
     ImageSource for transparent or solid-color images.
     Implements optimized as_buffer() method.
@@ -249,6 +267,7 @@ class BlankImageSource(object):
         self._buf = None
         self._img = None
         self.cacheable = cacheable
+        self.authorize_stale = False
 
     def as_image(self):
         if not self._img:


=====================================
mapproxy/response.py
=====================================
@@ -42,6 +42,12 @@ class Response(object):
             content_type = self.default_content_type
         self.headers['Content-type'] = content_type
 
+        if content_type.startswith(('text/', 'application/')):
+            # Capability documents can be dependent on the value of a few X-headers.
+            # Tell this caching proxies via the Vary HTTP header. This also prevents
+            # malicious cache poisoning.
+            self.headers['Vary'] = 'X-Script-Name, X-Forwarded-Host, X-Forwarded-Proto'
+
     def _status_set(self, status):
         if isinstance(status, int):
             status = status_code(status)


=====================================
mapproxy/script/conf/app.py
=====================================
@@ -71,7 +71,10 @@ def write_header(f, capabilities):
 @contextmanager
 def file_or_stdout(name):
     if name == '-':
-        yield codecs.getwriter('utf-8')(sys.stdout)
+        if hasattr(sys.stdout, 'buffer'):
+            yield codecs.getwriter('utf-8')(sys.stdout.buffer)
+        else:
+            yield codecs.getwriter('utf-8')(sys.stdout)
     else:
         with open(name, 'wb') as f:
             yield codecs.getwriter('utf-8')(f)


=====================================
mapproxy/seed/script.py
=====================================
@@ -105,6 +105,11 @@ class SeedScript(object):
                       metavar="N",
                       help="do not check for intersections between tiles"
                            " and seed geometries on the last N levels")
+    parser.add_option("--skip-uncached",
+                      action="store_true", dest="skip_uncached", default=False,
+                      help="only treat tiles which are already present in the cache."
+                           " This can be used with the configuration entry `refresh_before`"
+                           " to refresh only the existing cache.")
     parser.add_option("--summary",
                       action="store_true", dest="summary", default=False,
                       help="print summary with all seeding tasks and exit."
@@ -243,7 +248,8 @@ class SeedScript(object):
                         progress_store=progress)
                     seed(seed_tasks, progress_logger=logger, dry_run=options.dry_run,
                          concurrency=options.concurrency, cache_locker=cache_locker,
-                         skip_geoms_for_last_levels=options.geom_levels)
+                         skip_geoms_for_last_levels=options.geom_levels,
+                         skip_uncached=options.skip_uncached)
                 if cleanup_tasks:
                     print('========== Cleanup tasks ==========')
                     print('Start cleanup process (%d task%s)' % (


=====================================
mapproxy/seed/seeder.py
=====================================
@@ -477,7 +477,7 @@ class CleanupTask(object):
         return NONE
 
 def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
-    progress_logger=None, cache_locker=None):
+    progress_logger=None, cache_locker=None, skip_uncached=False):
     if cache_locker is None:
         cache_locker = DummyCacheLocker()
 
@@ -496,7 +496,7 @@ def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
                     start_progress = None
                 seed_progress = SeedProgress(old_progress_identifier=start_progress)
                 seed_task(task, concurrency, dry_run, skip_geoms_for_last_levels, progress_logger,
-                    seed_progress=seed_progress)
+                    seed_progress=seed_progress, skip_uncached=skip_uncached)
         except CacheLockedError:
             print('    ...cache is locked, skipping')
             active_tasks = [task] + active_tasks[:-1]
@@ -505,7 +505,7 @@ def seed(tasks, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
 
 
 def seed_task(task, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
-    progress_logger=None, seed_progress=None):
+    progress_logger=None, seed_progress=None, skip_uncached=False):
     if task.coverage is False:
         return
     if task.refresh_timestamp is not None:
@@ -518,7 +518,11 @@ def seed_task(task, concurrency=2, dry_run=False, skip_geoms_for_last_levels=0,
 
     tile_worker_pool = TileWorkerPool(task, TileSeedWorker, dry_run=dry_run,
         size=concurrency, progress_logger=progress_logger)
-    tile_walker = TileWalker(task, tile_worker_pool, handle_uncached=True,
+    # If the configuration requests to only refresh tiles which are already in cache,
+    # tile walker parameters shall be adapted
+    handle_stale = skip_uncached
+    handle_uncached = not skip_uncached
+    tile_walker = TileWalker(task, tile_worker_pool, handle_uncached=handle_uncached, handle_stale=handle_stale,
         skip_geoms_for_last_levels=skip_geoms_for_last_levels, progress_logger=progress_logger,
         seed_progress=seed_progress,
         work_on_metatiles=work_on_metatiles,


=====================================
mapproxy/seed/spec.py
=====================================
@@ -17,7 +17,7 @@ from mapproxy.util.ext.dictspec.validator import validate, ValidationError
 from mapproxy.util.ext.dictspec.spec import one_off, anything, number
 from mapproxy.util.ext.dictspec.spec import required
 
-from mapproxy.config.spec import coverage
+from mapproxy.config.spec import coverage, time_spec
 
 def validate_seed_conf(conf_dict):
     """
@@ -31,16 +31,6 @@ def validate_seed_conf(conf_dict):
     else:
         return [], True
 
-time_spec = {
-    'seconds': number(),
-    'minutes': number(),
-    'hours': number(),
-    'days': number(),
-    'weeks': number(),
-    'time': anything(),
-    'mtime': str(),
-}
-
 from_to_spec = {
     'from': number(),
     'to': number(),


=====================================
mapproxy/source/error.py
=====================================
@@ -20,19 +20,20 @@ class HTTPSourceErrorHandler(object):
 	def __init__(self):
 		self.response_error_codes = {}
 	
-	def add_handler(self, http_code, color, cacheable=False):
-		self.response_error_codes[http_code] = (color, cacheable)
+	def add_handler(self, http_code, color, cacheable=False, authorize_stale=False):
+		self.response_error_codes[http_code] = (color, cacheable, authorize_stale)
 
 	def handle(self, status_code, query):
 		color = cacheable = None
 		if status_code in self.response_error_codes:
-			color, cacheable = self.response_error_codes[status_code]
+			color, cacheable, authorize_stale = self.response_error_codes[status_code]
 		elif 'other' in self.response_error_codes:
-			color, cacheable = self.response_error_codes['other']
+			color, cacheable, authorize_stale = self.response_error_codes['other']
 		else:
 			return None
 
 		transparent = len(color) == 4
 		image_opts = ImageOptions(bgcolor=color, transparent=transparent)
 		img_source = BlankImageSource(query.size, image_opts, cacheable=cacheable)
-		return img_source
\ No newline at end of file
+		img_source.authorize_stale = authorize_stale
+		return img_source


=====================================
mapproxy/test/system/fixture/tileservice_refresh.yaml
=====================================
@@ -0,0 +1,59 @@
+globals:
+  cache:
+    base_dir: cache_data/
+    meta_size: [1, 1]
+    meta_buffer: 0
+
+services:
+  tms:
+    # origin: 'nw'
+
+layers:
+  - name: wms_cache
+    title: Direct Layer
+    sources: [wms_cache]
+
+  - name: wms_cache_isotime
+    title: Direct Layer
+    sources: [wms_cache_isotime]
+
+  - name: wms_cache_png
+    title: Direct Layer
+    sources: [wms_cache_png]
+
+caches:
+  wms_cache:
+    format: image/jpeg
+    sources: [wms_source]
+    refresh_before:
+      seconds: 1
+
+  wms_cache_isotime:
+    format: image/jpeg
+    sources: [wms_source]
+    refresh_before:
+      time: "2009-02-15T23:31:30"
+
+  wms_cache_png:
+    format: image/png
+    sources: [wms_source]
+    refresh_before:
+      seconds: 1
+
+sources:
+  wms_source:
+    type: wms
+    req:
+      url: http://localhost:42423/service
+      layers: bar
+    on_error:
+      404:
+        response: 'transparent'
+        cache: False
+      405:
+        response: '#ff0000'
+        cache: False
+      406:
+        response: 'transparent'
+        cache: False
+        authorize_stale: True


=====================================
mapproxy/test/system/test_refresh.py
=====================================
@@ -0,0 +1,207 @@
+# This file is part of the MapProxy project.
+# Copyright (C) 2010-2012 Omniscale <http://omniscale.de>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import time
+import hashlib
+
+from io import BytesIO
+import pytest
+
+from mapproxy.compat.image import Image
+from mapproxy.test.image import is_jpeg, tmp_image
+from mapproxy.test.http import mock_httpd
+from mapproxy.test.system import SysTest
+from mapproxy.util.times import timestamp_from_isodate
+
+ at pytest.fixture(scope="module")
+def config_file():
+    return "tileservice_refresh.yaml"
+
+
+class TestRefresh(SysTest):
+
+    def test_refresh_tile_1s(self, app, cache_dir):
+        with tmp_image((256, 256), format="jpeg") as img:
+            expected_req = (
+                {
+                    "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+                    "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+                    "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+                    "&WIDTH=256"
+                },
+                {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+            )
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                file_path = cache_dir.join(
+                    "wms_cache_EPSG900913/01/000/000/000/000/000/000.jpeg"
+                )
+                assert file_path.check()
+                t1 = file_path.mtime()
+                # file_path.remove()
+                # assert not file_path.check()
+            resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+            assert resp.content_type == "image/jpeg"
+            assert file_path.check()
+            t2 = file_path.mtime()
+            # tile is expired after 1 sec, so it will be fetched again from mock server
+            time.sleep(1.2)
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                assert file_path.check()
+                t3 = file_path.mtime()
+        assert t2 == t1
+        assert t3 > t2
+
+    def test_refresh_tile_mtime(self, app, cache_dir):
+        with tmp_image((256, 256), format="jpeg") as img:
+            expected_req = (
+                {
+                    "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+                    "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+                    "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+                    "&WIDTH=256"
+                },
+                {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+            )
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                file_path = cache_dir.join(
+                    "wms_cache_isotime_EPSG900913/01/000/000/000/000/000/000.jpeg"
+                )
+                assert file_path.check()
+                timestamp = timestamp_from_isodate("2009-02-15T23:31:30")
+                file_path.setmtime(timestamp + 1.2)
+                t1 = file_path.mtime()
+            resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+            assert resp.content_type == "image/jpeg"
+            t2 = file_path.mtime()
+            file_path.setmtime(timestamp - 1.2)
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache_isotime/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                assert file_path.check()
+                t3 = file_path.mtime()
+            assert t2 == t1
+            assert t3 > t2
+
+    def test_refresh_tile_source_error_no_stale(self, app, cache_dir):
+        source_request = {
+            "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fpng"
+            "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+            "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+            "&WIDTH=256"
+        }
+        with tmp_image((256, 256), format="png") as img:
+            expected_req = (
+                source_request,
+                {"body": img.read(), "headers": {"content-type": "image/png"}},
+            )
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+                assert resp.content_type == "image/png"
+                img.seek(0)
+                assert resp.body == img.read()
+            resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+            assert resp.content_type == "image/png"
+            img.seek(0)
+            assert resp.body == img.read()
+            # tile is expired after 1 sec, so it will be requested again from mock server
+            time.sleep(1.2)
+            expected_req = (
+                source_request,
+                {"body": "", "status": 404},
+            )
+            with mock_httpd(
+                    ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+                assert resp.content_type == "image/png"
+                # error handler for 404 does not authorise stale tiles, so transparent tile will be rendered
+                resp_img = Image.open(BytesIO(resp.body))
+                # check response transparency
+                assert resp_img.getbands() == ('R', 'G', 'B', 'A')
+                assert resp_img.getextrema()[3] == (0, 0)
+
+            expected_req = (
+                source_request,
+                {"body": "", "status": 405},
+            )
+            with mock_httpd(
+                    ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache_png/1/0/0.png")
+                assert resp.content_type == "image/png"
+                # error handler for 405 does not authorise stale tiles, so red tile will be rendered
+                resp_img = Image.open(BytesIO(resp.body))
+                # check response red color
+                assert resp_img.getbands() == ('R', 'G', 'B')
+                assert resp_img.getextrema() == ((255, 255), (0, 0), (0, 0))
+
+    def test_refresh_tile_source_error_stale(self, app, cache_dir):
+        with tmp_image((256, 256), format="jpeg") as img:
+            expected_req = (
+                {
+                    "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+                    "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+                    "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+                    "&WIDTH=256"
+                },
+                {"body": img.read(), "headers": {"content-type": "image/jpeg"}},
+            )
+            with mock_httpd(
+                ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                img.seek(0)
+                assert resp.body == img.read()
+            resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+            assert resp.content_type == "image/jpeg"
+            img.seek(0)
+            assert resp.body == img.read()
+            # tile is expired after 1 sec, so it will be fetched again from mock server
+            time.sleep(1.2)
+            expected_req = (
+                {
+                    "path": r"/service?LAYERs=bar&SERVICE=WMS&FORMAT=image%2Fjpeg"
+                    "&REQUEST=GetMap&HEIGHT=256&SRS=EPSG%3A900913&styles="
+                    "&VERSION=1.1.1&BBOX=-20037508.3428,-20037508.3428,0.0,0.0"
+                    "&WIDTH=256"
+                },
+                {"body": "", "status": 406},
+            )
+            with mock_httpd(
+                    ("localhost", 42423), [expected_req], bbox_aware_query_comparator=True
+            ):
+                resp = app.get("/tiles/wms_cache/1/0/0.jpeg")
+                assert resp.content_type == "image/jpeg"
+                # Check that initial non empty img is served as a stale tile
+                img.seek(0)
+                assert resp.body == img.read()


=====================================
mapproxy/test/system/test_response_headers.py
=====================================
@@ -0,0 +1,54 @@
+# This file is part of the MapProxy project.
+# Copyright (C) 2011 Omniscale <http://omniscale.de>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+
+import pytest
+
+from mapproxy.test.system import SysTest
+
+
+class TestResponseHeaders(SysTest):
+    """
+    Check if the vary header is set for text / xml content like capabilities 
+    """
+    @pytest.fixture(scope='class')
+    def config_file(self):
+        return 'auth.yaml'
+
+    def test_tms(self, app):
+        resp = app.get('http://localhost/tms')
+        assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+    def test_wms(self, app):
+        resp = app.get('http://localhost/service?SERVICE=WMS&REQUEST=GetCapabilities'
+                            '&VERSION=1.1.0')
+        assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+    def test_wmts(self, app):
+        resp = app.get('http://localhost/service?SERVICE=WMTS&REQUEST=GetCapabilities')
+        assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+    def test_restful_wmts(self, app):
+        resp = app.get('http://localhost/wmts/1.0.0/WMTSCapabilities.xml')
+        assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+    def test_no_endpoint(self, app):
+        resp = app.get('http://localhost/service?')
+        assert resp.vary == ('X-Script-Name', 'X-Forwarded-Host', 'X-Forwarded-Proto')
+
+    def test_image_response(self, app):
+        resp = app.get('http://localhost/tms/1.0.0/layer1a/EPSG900913/0/0/0.png')
+        assert resp.vary == None


=====================================
mapproxy/test/system/test_seed.py
=====================================
@@ -86,11 +86,42 @@ class SeedTestBase(SeedTestEnvironment):
                             {'body': img_data, 'headers': {'content-type': 'image/png'}})
             with mock_httpd(('localhost', 42423), [expected_req]):
                 with local_base_config(self.mapproxy_conf.base_config):
-                    seed_conf  = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
+                    seed_conf = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
                     tasks, cleanup_tasks = seed_conf.seeds(['one']), seed_conf.cleanups()
                     seed(tasks, dry_run=False)
                     cleanup(cleanup_tasks, verbose=False, dry_run=False)
 
+    def test_seed_skip_uncached(self):
+        with tmp_image((256, 256), format='png') as img:
+            img_data = img.read()
+            with local_base_config(self.mapproxy_conf.base_config):
+                expected_req = ({'path': r'/service?LAYERS=foo&SERVICE=WMS&FORMAT=image%2Fpng'
+                                 '&REQUEST=GetMap&VERSION=1.1.1&bbox=-180.0,-90.0,180.0,90.0'
+                                 '&width=256&height=128&srs=EPSG:4326'},
+                                {'body': img_data, 'headers': {'content-type': 'image/png'}})
+                seed_conf = load_seed_tasks_conf(self.seed_conf_file, self.mapproxy_conf)
+                tasks, cleanup_tasks = seed_conf.seeds(['one']), seed_conf.cleanups()
+
+                # tile not in cache => skipped by seeder
+                seed(tasks, dry_run=False, skip_uncached=True)
+                assert not self.tile_exists((0, 0, 0))
+
+                with mock_httpd(('localhost', 42423), [expected_req]):
+                    # force tile generation in cache (via skip_uncached=False)
+                    seed(tasks, dry_run=False, skip_uncached=False)
+                assert self.tile_exists((0, 0, 0))
+
+                # no refresh since tile is not older than 1 day (cf. config seed.yaml)
+                seed(tasks, dry_run=False, skip_uncached=True)
+
+                # create stale tile (older than 1 day)
+                self.make_tile((0, 0, 0), timestamp=time.time() - (60*60*25))
+                with mock_httpd(('localhost', 42423), [expected_req]):
+                    # check that old tile in cache is refreshed
+                    seed(tasks, dry_run=False, skip_uncached=True)
+                assert self.tile_exists((0, 0, 0))
+                cleanup(cleanup_tasks, verbose=False, dry_run=False)
+
     def test_reseed_uptodate(self):
         # tile already there.
         self.make_tile((0, 0, 0))


=====================================
mapproxy/util/times.py
=====================================
@@ -72,4 +72,4 @@ def timestamp_from_isodate(isodate):
         date = isodate
     else:
         date = datetime.datetime.strptime(isodate, "%Y-%m-%dT%H:%M:%S")
-    return mktime(date.timetuple())
\ No newline at end of file
+    return mktime(date.timetuple())


=====================================
requirements-tests.txt
=====================================
@@ -1,8 +1,8 @@
-Jinja2==2.11.2
+Jinja2==2.11.3
 MarkupSafe==1.1.1
 Pillow==6.2.2;python_version<"3.0"
-Pillow==7.2.0;python_version>="3.0"
-PyYAML==5.3.1
+Pillow==8.2.0;python_version>="3.0"
+PyYAML==5.4
 Shapely==1.7.0
 WebOb==1.8.6
 WebTest==2.0.35
@@ -17,7 +17,7 @@ certifi==2020.6.20
 cffi==1.14.2
 cfn-lint==0.35.0
 chardet==3.0.4
-cryptography==3.0
+cryptography==3.3.2
 decorator==4.4.2
 docker==4.3.0
 docutils==0.15.2
@@ -33,7 +33,7 @@ jsonpickle==1.4.1
 jsonpointer==2.0
 jsonschema==3.2.0
 junit-xml==1.9
-lxml==4.5.2
+lxml==4.6.3
 mock==3.0.5;python_version<"3.6"
 mock==4.0.2;python_version>="3.6"
 more-itertools==5.0.0;python_version<"3.0"
@@ -43,7 +43,7 @@ networkx==2.2;python_version<"3.0"
 networkx==2.4;python_version>="3.0"
 packaging==20.4
 pluggy==0.13.1
-py==1.9.0
+py==1.10.0
 pyasn1==0.4.8
 pycparser==2.20
 pyparsing==2.2.2;python_version<"3.0"
@@ -61,7 +61,7 @@ requests==2.24.0
 responses==0.10.16
 riak==2.7.0
 rsa==4.5;python_version<"3.0"
-rsa==4.6;python_version>="3.0"
+rsa==4.7;python_version>="3.0"
 s3transfer==0.3.3
 six==1.15.0
 soupsieve==1.9.6;python_version<"3.0"


=====================================
setup.py
=====================================
@@ -21,11 +21,11 @@ def package_installed(pkg):
 # depend on PIL if it is installed, otherwise
 # require Pillow
 if package_installed('Pillow'):
-    install_requires.append('Pillow !=2.4.0')
+    install_requires.append('Pillow !=2.4.0,!=8.3.0,!=8.3.1')
 elif package_installed('PIL'):
     install_requires.append('PIL>=1.1.6,<1.2.99')
 else:
-    install_requires.append('Pillow !=2.4.0')
+    install_requires.append('Pillow !=2.4.0,!=8.3.0,!=8.3.1')
 
 if platform.python_version_tuple() < ('2', '6'):
     # for mapproxy-seed
@@ -54,7 +54,7 @@ def long_description(changelog_releases=10):
 
 setup(
     name='MapProxy',
-    version="1.13.2",
+    version="1.14.0",
     description='An accelerating proxy for tile and web map services',
     long_description=long_description(7),
     author='Oliver Tonnhofer',



View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/compare/2057d5232a8c5f922b98e413719f69ae1830c524...668f1c627e7afe956786583ddd278d08f4fbfa69

-- 
View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/compare/2057d5232a8c5f922b98e413719f69ae1830c524...668f1c627e7afe956786583ddd278d08f4fbfa69
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20211124/dd5fe809/attachment-0001.htm>


More information about the Pkg-grass-devel mailing list