[Git][debian-gis-team/mapproxy][upstream] New upstream version 1.13.1

Bas Couwenberg (@sebastic) gitlab at salsa.debian.org
Tue Jul 13 12:41:19 BST 2021



Bas Couwenberg pushed to branch upstream at Debian GIS Project / mapproxy


Commits:
3d3f3804 by Bas Couwenberg at 2021-07-13T13:28:13+02:00
New upstream version 1.13.1
- - - - -


17 changed files:

- .travis.yml
- CHANGES.txt
- doc/conf.py
- doc/configuration.rst
- doc/configuration_examples.rst
- doc/sources.rst
- mapproxy/client/http.py
- mapproxy/config/defaults.py
- mapproxy/config/loader.py
- mapproxy/config/spec.py
- mapproxy/config_template/base_config/full_example.yaml
- mapproxy/config_template/base_config/mapproxy.yaml
- mapproxy/service/demo.py
- mapproxy/test/system/fixture/layer.yaml
- mapproxy/test/system/test_wms.py
- mapproxy/test/unit/test_client.py
- setup.py


Changes:

=====================================
.travis.yml
=====================================
@@ -10,6 +10,7 @@ python:
 services:
   - couchdb
   - redis-server
+  - docker
 
 addons:
   apt:
@@ -31,6 +32,8 @@ env:
   global:
     - MAPPROXY_TEST_COUCHDB=http://127.0.0.1:5984
     - MAPPROXY_TEST_REDIS=127.0.0.1:6379
+    - MAPPROXY_TEST_RIAK_HTTP=http://localhost:8098
+    - MAPPROXY_TEST_RIAK_PBC=pbc://localhost:8087
 
     # do not load /etc/boto.cfg with Python 3 incompatible plugin
     # https://github.com/travis-ci/travis-ci/issues/5246#issuecomment-166460882
@@ -48,6 +51,9 @@ cache:
   directories:
     - $HOME/.cache/pip
 
+before_install:
+    - docker run --detach --rm --publish 8087:8087 --publish 8098:8098 basho/riak-kv
+
 install:
     - "pip install -r requirements-tests.txt"
     - "if [[ $USE_LATEST_PILLOW = '1' ]]; then pip install -U Pillow; fi"


=====================================
CHANGES.txt
=====================================
@@ -1,3 +1,18 @@
+Nightly
+~~~~~~~~~~~~~~~~~
+
+
+1.13.1 2021-07-13
+~~~~~~~~~~~~~~~~~
+
+Improvements:
+
+- Support cookie management for HTTP sources.
+
+Fixes:
+
+- Security fix for local file disclosure (#526).
+
 1.13.0 2020-11-18
 ~~~~~~~~~~~~~~~~~
 


=====================================
doc/conf.py
=====================================
@@ -51,7 +51,7 @@ copyright = u'Oliver Tonnhofer, Omniscale'
 # The short X.Y version.
 version = '1.13'
 # The full version, including alpha/beta/rc tags.
-release = '1.13.0'
+release = '1.13.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.


=====================================
doc/configuration.rst
=====================================
@@ -1024,6 +1024,15 @@ Add additional HTTP headers to all requests to your sources.
 
 Sets the ``Access-control-allow-origin`` header to HTTP responses for `Cross-origin resource sharing <http://en.wikipedia.org/wiki/Cross-origin_resource_sharing>`_. This header is required for WebGL or Canvas web clients. Defaults to `*`. Leave empty to disable the header. This option is only available in `globals`.
 
+``manage_cookies``
+^^^^^^^^^^^^^^^^^^
+
+.. versionadded:: 1.14.0
+
+Enables MapProxy cookie management for HTTP sources. When enabled MapProxy will accept and store server cookies. Accepted cookies will be passed
+back to the source on subsequent requests. Usefull for sources which require to maintain an HTTP session to work efficiently, maybe in combination
+with basic authentication. Depending on your deployment MapProxy will still start multiple sessions (e.g. one per MapProxy process).
+Cookie handling is based on Python `CookieJar <https://docs.python.org/3/library/http.cookiejar.html>`_. Disabled by default.
 
 ``hide_error_details``
 ^^^^^^^^^^^^^^^^^^^^^^


=====================================
doc/configuration_examples.rst
=====================================
@@ -667,7 +667,7 @@ Then we can add a layer with all available dimensions::
               - 3000
             default: 0
 
-You can know access this layer with the elevation and time dimensions via the WMTS KVP service.
+You can now access this layer with the elevation and time dimensions via the WMTS KVP service.
 The RESTful service requires a custom URL template that contains the dimensions. For example::
 
     services:
@@ -708,6 +708,8 @@ You can disable the certificate verification if you you don't need it.
       url: https://username:mypassword@example.org/service?
       layers: securelayer
 
+.. note:: If the source requires session handling through cookies, have a look at the ``manage_cookies`` configuration option.
+
 .. _http_proxy:
 
 Access sources through HTTP proxy
@@ -734,11 +736,28 @@ You can also set this in your :ref:`server script <server_script>`::
 
 Add a username and password to the URL if your HTTP proxy requires authentication. For example ``http://username:password@example.com:3128``.
 
+.. note:: If the source requires session handling through cookies, have a look at the ``manage_cookies`` configuration option.
+
 You can use the ``no_proxy`` environment variable if you need to bypass the proxy for some hosts::
 
   $ export no_proxy="localhost,127.0.0.1,196.168.1.99"
 
 
+Cookie Management
+=================
+
+MapProxy can handle server cookies of HTTP sources, like browsers do. That is, MapProxy accepts cookies and passes them back
+on subsequent calls. This is useful for sources that use cookie for session management or rate-limiting for example::
+
+  sources:
+    wms_with_session_management:
+      type: wms
+      http:
+        manage_cookies: True
+      req:
+        url: http://example.org/service?
+        layers: layer0
+
 .. _paster_urlmap:
 
 Serve multiple MapProxy instances


=====================================
doc/sources.rst
=====================================
@@ -184,6 +184,7 @@ You can configure the following HTTP related options for this source:
 - ``client_timeout``
 - ``ssl_ca_certs``
 - ``ssl_no_cert_checks``
+- ``manage_cookies``
 
 See :ref:`HTTP Options <http_ssl>` for detailed documentation.
 
@@ -432,6 +433,7 @@ You can configure the following HTTP related options for this source:
 - ``client_timeout``
 - ``ssl_ca_certs``
 - ``ssl_no_cert_checks``
+- ``manage_cookies``
 
 See :ref:`HTTP Options <http_ssl>` for detailed documentation.
 


=====================================
mapproxy/client/http.py
=====================================
@@ -29,12 +29,15 @@ from mapproxy.compat.modules import urlparse
 
 if PY2:
     import urllib2
-    from urllib2 import URLError, HTTPError
+    from urllib2 import URLError, HTTPError, HTTPCookieProcessor
     import httplib
+    from cookielib import CookieJar
 else:
     from urllib import request as urllib2
     from urllib.error import URLError, HTTPError
+    from urllib.request import HTTPCookieProcessor
     from http import client as httplib
+    from http.cookiejar import CookieJar
 
 import socket
 import ssl
@@ -131,8 +134,8 @@ class _URLOpenerCache(object):
     def __init__(self):
         self._opener = {}
 
-    def __call__(self, ssl_ca_certs, url, username, password, insecure=False):
-        cache_key = (ssl_ca_certs, insecure)
+    def __call__(self, ssl_ca_certs, url, username, password, insecure=False, manage_cookies=False):
+        cache_key = (ssl_ca_certs, insecure, manage_cookies)
         if cache_key not in self._opener:
             handlers = []
             https_handler = build_https_handler(ssl_ca_certs, insecure)
@@ -143,6 +146,9 @@ class _URLOpenerCache(object):
             handlers.append(authhandler)
             authhandler = urllib2.HTTPDigestAuthHandler(passman)
             handlers.append(authhandler)
+            if manage_cookies:
+                cj = CookieJar()
+                handlers.append(HTTPCookieProcessor(cj))
 
             opener = urllib2.build_opener(*handlers)
 
@@ -161,7 +167,8 @@ create_url_opener = _URLOpenerCache()
 
 class HTTPClient(object):
     def __init__(self, url=None, username=None, password=None, insecure=False,
-                 ssl_ca_certs=None, timeout=None, headers=None, hide_error_details=False):
+                 ssl_ca_certs=None, timeout=None, headers=None, hide_error_details=False,
+                 manage_cookies=False):
         self._timeout = timeout
         if url and url.startswith('https'):
             if insecure:
@@ -170,7 +177,7 @@ class HTTPClient(object):
                     raise HTTPClientError('No ca_certs file set (http.ssl_ca_certs). '
                         'Set file or disable verification with http.ssl_no_cert_checks option.')
 
-        self.opener = create_url_opener(ssl_ca_certs, url, username, password, insecure=insecure)
+        self.opener = create_url_opener(ssl_ca_certs, url, username, password, insecure=insecure, manage_cookies=manage_cookies)
         self.header_list = headers.items() if headers else []
         self.hide_error_details = hide_error_details
 


=====================================
mapproxy/config/defaults.py
=====================================
@@ -94,4 +94,5 @@ http = dict(
     method = 'AUTO',
     access_control_allow_origin = '*',
     hide_error_details = True,
+    manage_cookies = False,
 )


=====================================
mapproxy/config/loader.py
=====================================
@@ -592,10 +592,12 @@ class SourceConfiguration(ConfigurationBase):
         timeout = self.context.globals.get_value('http.client_timeout', self.conf)
         headers = self.context.globals.get_value('http.headers', self.conf)
         hide_error_details = self.context.globals.get_value('http.hide_error_details', self.conf)
+        manage_cookies = self.context.globals.get_value('http.manage_cookies', self.conf)
 
         http_client = HTTPClient(url, username, password, insecure=insecure,
                                  ssl_ca_certs=ssl_ca_certs, timeout=timeout,
-                                 headers=headers, hide_error_details=hide_error_details)
+                                 headers=headers, hide_error_details=hide_error_details,
+                                 manage_cookies=manage_cookies)
         return http_client, url
 
     @memoize


=====================================
mapproxy/config/spec.py
=====================================
@@ -75,6 +75,7 @@ http_opts = {
     'headers': {
         anything(): str()
     },
+    'manage_cookies': bool(),
 }
 
 mapserver_opts = {


=====================================
mapproxy/config_template/base_config/full_example.yaml
=====================================
@@ -429,6 +429,21 @@ sources:
       transparent: true
       layers: securelayer
 
+  # WMS source that requires authentication and session management
+  # through HTTP cookies
+  session_source:
+    type: wms
+    http:
+      # Accept session cookies and forward on subsequent requests
+      manage_cookies: true
+      # Use basic auth header directly
+      headers:
+        Authorization: Basic YWRtaW46Z2Vvc2VydmVy
+    req:
+      url: https://my-service.com/service?
+      transparent: true
+      layers: securelayer
+
   feature_info_source:
     type: wms
     wms_opts:


=====================================
mapproxy/config_template/base_config/mapproxy.yaml
=====================================
@@ -18,7 +18,7 @@
 #     first tile: http://localhost:8080/tiles/osm/webmercator/0/0/0.png
 # TMS:
 #     note: TMS is not compatible with OSM/Google Maps/etc.
-#     fist tile: http://localhost:8080/tms/1.0.0/osm/webmercator/0/0/0.png
+#     first tile: http://localhost:8080/tms/1.0.0/osm/webmercator/0/0/0.png
 # KML:
 #     initial doc: http://localhost:8080/kml/osm/webmercator
 


=====================================
mapproxy/service/demo.py
=====================================
@@ -68,6 +68,8 @@ class DemoServer(Server):
 
     def handle(self, req):
         if req.path.startswith('/demo/static/'):
+            if '..' in req:
+                return Response('file not found', content_type='text/plain', status=404)
             filename = req.path.lstrip('/')
             filename = static_filename(filename)
             if not os.path.isfile(filename):


=====================================
mapproxy/test/system/fixture/layer.yaml
=====================================
@@ -126,6 +126,9 @@ layers:
   - name: watermark_cache
     title: TMS Cache + watermark
     sources: [watermark_cache]
+  - name: wms_managed_cookies_cache
+    title: WMS with cookies management
+    sources: [wms_managed_cookies]
 
 caches:
   wms_cache:
@@ -164,6 +167,8 @@ caches:
     disable_storage: true
     watermark:
       text: '@ Omniscale'
+  wms_managed_cookies_cache:
+    sources: [wms_managed_cookies]
 
 sources:
   direct:
@@ -244,3 +249,12 @@ sources:
     coverage:
       bbox: [-180,-90,170,80]
       srs: 'EPSG:4326'
+  wms_managed_cookies:
+    type: wms
+    wms_opts:
+      featureinfo: True
+    req:
+      url: http://localhost:42423/service
+      layers: layer1
+    http:
+      manage_cookies: True


=====================================
mapproxy/test/system/test_wms.py
=====================================
@@ -195,6 +195,7 @@ class TestWMS111(SysTest):
                 "wms_cache_link_single",
                 "wms_cache_110",
                 "watermark_cache",
+                "wms_managed_cookies_cache",
             ]
         )
         assert layer_names == expected_names
@@ -902,6 +903,7 @@ class TestWMS110(SysTest):
                 "wms_cache_link_single",
                 "wms_cache_110",
                 "watermark_cache",
+                "wms_managed_cookies_cache",
             ]
         )
         assert layer_names == expected_names
@@ -1056,6 +1058,41 @@ class TestWMS110(SysTest):
         assert "tms_cache is not queryable" in xml.xpath("//ServiceException/text()")[0]
         assert validate_with_dtd(xml, "wms/1.1.0/exception_1_1_0.dtd")
 
+    def test_managed_cookies(self, app):
+        def assert_no_cookie(req_handler):
+            return 'Cookie' not in req_handler.headers
+
+        def assert_cookie(req_handler):
+            assert 'Cookie' in req_handler.headers
+            cookie_name, cookie_val = req_handler.headers['Cookie'].split(';')[0].split('=')
+            assert cookie_name == 'testcookie'
+            assert cookie_val == '42'
+            return True
+
+        url = (r"/service?LAYERs=layer1&SERVICE=WMS&FORMAT=image%2Fpng"
+                "&REQUEST=GetFeatureInfo&HEIGHT=200&SRS=EPSG%3A900913"
+                "&VERSION=1.1.1&BBOX=1000.0,400.0,2000.0,1400.0&styles="
+                "&WIDTH=200&QUERY_LAYERS=layer1&X=10&Y=20")
+        # First response has a Set-Cookie => with managed_cookies=True, mapproxy should send the
+        # cookie in the second request
+        expected_requests = [
+            (
+                {'path': url, 'req_assert_function': assert_no_cookie},
+                {'body': b'nothing', 'headers': {'Set-Cookie': "testcookie=42"}}
+            ),
+            (
+                {'path': url, 'req_assert_function': assert_cookie},
+                {'body': b'nothing'}
+            )
+        ]
+        with mock_httpd(("localhost", 42423), expected_requests):
+            self.common_fi_req.params["layers"] = "wms_managed_cookies_cache"
+            self.common_fi_req.params["query_layers"] = "wms_managed_cookies_cache"
+            resp = app.get(self.common_fi_req)
+            assert resp.body == b"nothing"
+            resp = app.get(self.common_fi_req)
+            assert resp.body == b"nothing"
+
 
 class TestWMS100(SysTest):
     config_file = "layer.yaml"
@@ -1119,6 +1156,7 @@ class TestWMS100(SysTest):
                 "wms_cache_link_single",
                 "wms_cache_110",
                 "watermark_cache",
+                "wms_managed_cookies_cache",
             ]
         )
         assert layer_names == expected_names
@@ -1334,6 +1372,7 @@ class TestWMS130(SysTest):
                 "wms_cache_link_single",
                 "wms_cache_110",
                 "watermark_cache",
+                "wms_managed_cookies_cache",
             ]
         )
         assert layer_names == expected_names


=====================================
mapproxy/test/unit/test_client.py
=====================================
@@ -185,6 +185,60 @@ class TestHTTPClient(object):
         assert 0.1 <= duration1 < 0.5, duration1
         assert 0.5 <= duration2 < 0.9, duration2
 
+    def test_manage_cookies_off(self):
+        """
+        Test the behavior when manage_cookies is off (the default). Cookies shouldn't be sent
+        """
+        self.client = HTTPClient()
+
+        def assert_no_cookie(req_handler):
+            return 'Cookie' not in req_handler.headers
+
+        test_requests = [
+            (
+                {'path': '/', 'req_assert_function': assert_no_cookie},
+                {'body': b'nothing', 'headers': {'Set-Cookie': "testcookie=42"}}
+            ),
+            (
+                {'path': '/', 'req_assert_function': assert_no_cookie},
+                {'body': b'nothing'}
+            )
+        ]
+        with mock_httpd(TESTSERVER_ADDRESS, test_requests):
+            self.client.open(TESTSERVER_URL + '/')
+            self.client.open(TESTSERVER_URL + '/')
+
+    def test_manage_cookies_on(self):
+        """
+        Test behavior of manage_cookies=True. Once the remote server sends a cookie back, it should
+        be included in future requests
+        """
+        self.client = HTTPClient(manage_cookies=True)
+
+        def assert_no_cookie(req_handler):
+            return 'Cookie' not in req_handler.headers
+
+        def assert_cookie(req_handler):
+            assert 'Cookie' in req_handler.headers
+            cookie_name, cookie_val = req_handler.headers['Cookie'].split(';')[0].split('=')
+            assert cookie_name == 'testcookie'
+            assert cookie_val == '42'
+            return True
+
+        test_requests = [
+            (
+                {'path': '/', 'req_assert_function': assert_no_cookie},
+                {'body': b'nothing', 'headers': {'Set-Cookie': "testcookie=42"}}
+            ),
+            (
+                {'path': '/', 'req_assert_function': assert_cookie},
+                {'body': b'nothing'}
+            )
+        ]
+        with mock_httpd(TESTSERVER_ADDRESS, test_requests):
+            self.client.open(TESTSERVER_URL + '/')
+            self.client.open(TESTSERVER_URL + '/')
+
 
 # root certificates for google.com, if no ca-certificates.cert
 # file is found


=====================================
setup.py
=====================================
@@ -54,7 +54,7 @@ def long_description(changelog_releases=10):
 
 setup(
     name='MapProxy',
-    version="1.13.0",
+    version="1.13.1",
     description='An accelerating proxy for tile and web map services',
     long_description=long_description(7),
     author='Oliver Tonnhofer',



View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/commit/3d3f3804d7b222a70ec5420f723c45e000e6e5c1

-- 
View it on GitLab: https://salsa.debian.org/debian-gis-team/mapproxy/-/commit/3d3f3804d7b222a70ec5420f723c45e000e6e5c1
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20210713/2036d558/attachment-0001.htm>


More information about the Pkg-grass-devel mailing list