[Git][debian-gis-team/pyosmium][upstream] New upstream version 3.7.0
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Mon Nov 20 05:06:47 GMT 2023
Bas Couwenberg pushed to branch upstream at Debian GIS Project / pyosmium
Commits:
46e86a62 by Bas Couwenberg at 2023-11-20T05:34:32+01:00
New upstream version 3.7.0
- - - - -
13 changed files:
- .github/actions/run-tests/action.yml
- .github/workflows/build_wheels.yml
- .github/workflows/ci.yml
- CHANGELOG.md
- README.md
- README.rst
- lib/osm.cc
- + pyproject.toml
- setup.py
- src/osmium/replication/server.py
- src/osmium/version.py
- test/test_pyosmium_get_changes.py
- test/test_replication.py
Changes:
=====================================
.github/actions/run-tests/action.yml
=====================================
@@ -4,7 +4,7 @@ runs:
using: "composite"
steps:
- name: Install test requirements
- run: pip install pytest shapely
+ run: pip install pytest pytest-httpserver shapely
shell: bash
- name: Run tests
=====================================
.github/workflows/build_wheels.yml
=====================================
@@ -19,7 +19,7 @@ jobs:
- uses: actions/checkout at v3
with:
repository: pybind/pybind11
- ref: v2.10.3
+ ref: v2.11.1
path: contrib/pybind11
- uses: actions/checkout at v3
@@ -31,16 +31,17 @@ jobs:
- uses: actions/checkout at v3
with:
repository: osmcode/libosmium
- ref: v2.19.0
+ ref: v2.20.0
path: contrib/libosmium
- name: Build wheels
- uses: pypa/cibuildwheel at v2.11.3
+ uses: pypa/cibuildwheel at v2.16.2
env:
CIBW_ARCHS: native
CIBW_SKIP: "pp* *musllinux*"
- CIBW_TEST_REQUIRES: pytest shapely
+ CIBW_TEST_REQUIRES: pytest pytest-httpserver shapely
+ CIBW_TEST_REQUIRES_LINUX: urllib3<2.0 pytest pytest-httpserver shapely
CIBW_TEST_COMMAND: pytest {project}/test
CIBW_BUILD_FRONTEND: build
CIBW_BEFORE_BUILD_LINUX: yum install -y sparsehash-devel expat-devel boost-devel zlib-devel bzip2-devel lz4-devel
=====================================
.github/workflows/ci.yml
=====================================
@@ -78,6 +78,17 @@ jobs:
python -m build
shell: bash
+ - name: Set up Python 3.12
+ uses: actions/setup-python at v4
+ with:
+ python-version: "3.12"
+
+ - name: Build package 3.12
+ run: |
+ pip install build wheel
+ python -m build
+ shell: bash
+
- name: Upload Artifact
uses: actions/upload-artifact at v3
with:
@@ -91,7 +102,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- python-version: [3.6, 3.7, 3.8, 3.9, "3.10", "3.11"]
+ python-version: [3.6, 3.7, 3.8, 3.9, "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout at v3
@@ -123,20 +134,20 @@ jobs:
compiler: [gcc-old, clang-old, gcc, clang, macos]
include:
- compiler: gcc-old
- cc: gcc-7
- cxx: g++-7
- platform: ubuntu-18.04
+ cc: gcc-10
+ cxx: g++-10
+ platform: ubuntu-20.04
python: 3.6
deps: release
- compiler: clang-old
- cc: clang-6.0
- cxx: clang++-6.0
- platform: ubuntu-18.04
+ cc: clang-10
+ cxx: clang++-10
+ platform: ubuntu-20.04
python: 3.6
deps: release
- compiler: gcc
- cc: gcc-11
- cxx: g++-11
+ cc: gcc-12
+ cxx: g++-12
platform: ubuntu-22.04
python: "3.11"
deps: develop
@@ -182,7 +193,7 @@ jobs:
- name: Install prerequisites
run: |
python -m pip install --upgrade pip
- pip install pytest shapely setuptools requests
+ pip install pytest pytest-httpserver shapely setuptools requests
shell: bash
- name: Build package
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,22 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.7.0] - 2023-11-19
+
+### Added
+
+- transparently retry download on transient HTTP errors
+
+### Fixed
+
+- catch non-200 status for HTTP responses
+
+### Changed
+
+- update to pybind 2.11.1
+- update to libosmium 2.20.0
+
+
## [3.6.0] - 2023-01-20
### Changed
@@ -12,6 +28,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- update to libosmium 2.19.0
- change minimum required version of Cmake to 3.0
+
## [3.6.0rc1] - 2022-12-13
### Changed
@@ -20,6 +37,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- invalid buffers are now checked on access time, no more reference count
checks on leaving the handler callback
+
## [3.5.0] - 2022-11-09
### Added
=====================================
README.md
=====================================
@@ -74,13 +74,15 @@ They are mostly ports of the examples in Libosmium and osmium-contrib.
## Testing
-There is a small test suite in the test directory. This provides regression
+There is a small test suite in the test directory. This provides unit
test for the python bindings, it is not meant to be a test suite for Libosmium.
-You'll need the Python `pytest` module. On Debian/Ubuntu install the package
-`python3-pytest`.
+Testing requires `pytest` and `pytest-httpserver`. On Debian/Ubuntu install
+the dependencies with:
-The suite can be run with:
+ sudo apt-get install python3-pytest python3-pytest-httpserver
+
+The test suite can be run with:
pytest test
=====================================
README.rst
=====================================
@@ -24,7 +24,7 @@ development packages for these libraries. On Debian/Ubuntu do::
libexpat1-dev zlib1g-dev libbz2-dev
-Python >= 3.4 is supported.
+Python >= 3.6 is supported. Pypy is known not to work.
Documentation
=============
=====================================
lib/osm.cc
=====================================
@@ -18,6 +18,38 @@ namespace py = pybind11;
using TagIterator = osmium::TagList::const_iterator;
using MemberIterator = osmium::RelationMemberList::const_iterator;
+#if PYBIND11_VERSION_MINOR >= 11 || PYBIND11_VERSION_MAJOR > 2
+/*
+Work-around false positive added by pybind/pybind11 at f701654 change:
+ItemIterator/CollectionIterator ARE copy/move constructible, even if their template
+parameter is not. Indeed, those iterators iterate over low-level memory representation
+of the objects, without relying on their constructors.
+
+For eg.
+// static_assert(std::is_move_constructible<osmium::memory::CollectionIterator<osmium::RelationMember const>>::value);
+// static_assert(!std::is_copy_constructible<osmium::RelationMember>::value);
+
+The work-around relies on officially exposed pybind11::detail::is_copy_constructible/is_copy_constructible:
+https://github.com/pybind/pybind11/pull/4631
+*/
+namespace pybind11 {
+namespace detail {
+template <typename T>
+struct is_copy_constructible<osmium::memory::CollectionIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_move_constructible<osmium::memory::CollectionIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_copy_constructible<osmium::memory::ItemIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_move_constructible<osmium::memory::ItemIterator<T>>
+ : std::true_type {};
+} // namespace detail
+} // namespace pybind11
+#endif
+
static py::object tag_iterator_next(TagIterator &it, TagIterator const &cend)
{
if (it == cend)
=====================================
pyproject.toml
=====================================
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["packaging", "setuptools"]
+build-backend = "setuptools.build_meta"
=====================================
setup.py
=====================================
@@ -10,7 +10,7 @@ from pathlib import Path
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
from setuptools.command.sdist import sdist as orig_sdist
-from distutils.version import LooseVersion
+from packaging.version import Version
BASEDIR = os.path.split(os.path.abspath(__file__))[0]
@@ -75,8 +75,8 @@ class CMakeBuild(build_ext):
", ".join(e.name for e in self.extensions))
if platform.system() == "Windows":
- cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
- if cmake_version < '3.1.0':
+ cmake_version = Version(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
+ if cmake_version < Version('3.1.0'):
raise RuntimeError("CMake >= 3.1.0 is required on Windows")
for ext in self.extensions:
=====================================
src/osmium/replication/server.py
=====================================
@@ -7,7 +7,6 @@
""" Helper functions to communicate with replication servers.
"""
from typing import NamedTuple, Optional, Any, Iterator, cast, Dict, Mapping, Tuple
-import requests
import urllib.request as urlrequest
from urllib.error import URLError
import datetime as dt
@@ -15,6 +14,10 @@ from collections import namedtuple
from contextlib import contextmanager
from math import ceil
+import requests
+from requests.adapters import HTTPAdapter
+from urllib3.util import Retry
+
from osmium import MergeInputReader, BaseHandler
from osmium import io as oio
from osmium import version
@@ -52,6 +55,8 @@ class ReplicationServer:
self.diff_type = diff_type
self.extra_request_params: dict[str, Any] = dict(timeout=60, stream=True)
self.session: Optional[requests.Session] = None
+ self.retry = Retry(total=3, backoff_factor=0.5, allowed_methods={'GET'},
+ status_forcelist=[408, 429, 500, 502, 503, 504])
def close(self) -> None:
""" Close any open connection to the replication server.
@@ -62,6 +67,8 @@ class ReplicationServer:
def __enter__(self) -> 'ReplicationServer':
self.session = requests.Session()
+ self.session.mount('http://', HTTPAdapter(max_retries=self.retry))
+ self.session.mount('https://', HTTPAdapter(max_retries=self.retry))
return self
def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
@@ -97,6 +104,8 @@ class ReplicationServer:
@contextmanager
def _get_url_with_session() -> Iterator[requests.Response]:
with requests.Session() as session:
+ session.mount('http://', HTTPAdapter(max_retries=self.retry))
+ session.mount('https://', HTTPAdapter(max_retries=self.retry))
request = session.get(url.get_full_url(), **get_params)
yield request
@@ -133,7 +142,7 @@ class ReplicationServer:
try:
diffdata = self.get_diff_block(current_id)
except:
- LOG.debug("Error during diff download. Bailing out.")
+ LOG.error("Error during diff download. Bailing out.")
diffdata = ''
if len(diffdata) == 0:
if start_id == current_id:
@@ -348,6 +357,7 @@ class ReplicationServer:
with self.open_url(self.make_request(self.get_state_url(seq))) as response:
if hasattr(response, 'iter_lines'):
# generated by requests
+ response.raise_for_status()
lines = response.iter_lines()
else:
lines = response
@@ -372,7 +382,7 @@ class ReplicationServer:
ts = ts.replace(tzinfo=dt.timezone.utc)
except (URLError, IOError) as err:
- LOG.debug("Loading state info %s failed with: %s", seq, str(err))
+ LOG.debug("Loading state info failed with: %s", str(err))
return None
if ts is not None and next_seq is not None:
@@ -382,12 +392,13 @@ class ReplicationServer:
def get_diff_block(self, seq: int) -> str:
""" Downloads the diff with the given sequence number and returns
- it as a byte sequence. Throws a :code:`urllib.error.HTTPError`
+ it as a byte sequence. Throws an :code:`requests.HTTPError`
if the file cannot be downloaded.
"""
with self.open_url(self.make_request(self.get_diff_url(seq))) as resp:
if hasattr(resp, 'content'):
# generated by requests
+ resp.raise_for_status()
return cast(str, resp.content)
# generated by urllib.request
=====================================
src/osmium/version.py
=====================================
@@ -9,13 +9,13 @@ Version information.
"""
# the major version
-pyosmium_major = '3.6'
+pyosmium_major = '3.7'
# current release (Pip version)
-pyosmium_release = '3.6.0'
+pyosmium_release = '3.7.0'
# libosmium version shipped with the Pip release
-libosmium_version = '2.19.0'
+libosmium_version = '2.20.0'
# protozero version shipped with the Pip release
protozero_version = '1.7.1'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.10.3'
+pybind11_version = '2.11.1'
=====================================
test/test_pyosmium_get_changes.py
=====================================
@@ -2,10 +2,9 @@
#
# This file is part of Pyosmium.
#
-# Copyright (C) 2022 Sarah Hoffmann.
+# Copyright (C) 2023 Sarah Hoffmann.
""" Tests for the pyosmium-get-changes script.
"""
-from io import BytesIO
from pathlib import Path
from textwrap import dedent
@@ -18,101 +17,76 @@ except ImportError:
import cookielib as cookiejarlib
-class RequestsResponses(BytesIO):
-
- def __init__(self, bytes):
- super(RequestsResponses, self).__init__(bytes)
- self.content = bytes
-
- def iter_lines(self):
- return self.readlines()
-
-
class TestPyosmiumGetChanges:
@pytest.fixture(autouse=True)
- def setUp(self, monkeypatch):
+ def setup(self):
self.script = dict()
filename = (Path(__file__) / ".." / ".." / "tools"/ "pyosmium-get-changes").resolve()
with filename.open("rb") as f:
exec(compile(f.read(), str(filename), 'exec'), self.script)
- self.urls = dict()
-
-
- @pytest.fixture
- def mock_requests(self, monkeypatch):
- def mock_get(session, url, **kwargs):
- return RequestsResponses(self.urls[url])
- monkeypatch.setattr(osmium.replication.server.requests.Session, "get", mock_get)
-
-
- def url(self, url, result):
- self.urls[url] = dedent(result).encode()
- def main(self, *args):
- return self.script['main'](args)
+ def main(self, httpserver, *args):
+ return self.script['main'](['--server', httpserver.url_for('')] + list(args))
- def test_init_id(self, capsys):
- assert 0 == self.main('-I', '453')
+ def test_init_id(self, capsys, httpserver):
+ assert 0 == self.main(httpserver, '-I', '453')
output = capsys.readouterr().out.strip()
assert output == '453'
- def test_init_date(self, capsys, mock_requests):
- self.url('https://planet.osm.org/replication/minute//state.txt',
- """\
+ def test_init_date(self, capsys, httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data(dedent("""\
sequenceNumber=100
timestamp=2017-08-26T11\\:04\\:02Z
- """)
- self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
- """\
+ """))
+ httpserver.expect_request('/000/000/000.state.txt').respond_with_data(dedent("""\
sequenceNumber=0
timestamp=2016-08-26T11\\:04\\:02Z
- """)
- assert 0 == self.main('-D', '2015-12-24T08:08:08Z')
+ """))
+ assert 0 == self.main(httpserver, '-D', '2015-12-24T08:08:08Z')
output = capsys.readouterr().out.strip()
assert output == '-1'
- def test_init_to_file(self, tmp_path):
+ def test_init_to_file(self, tmp_path, httpserver):
fname = tmp_path / 'db.seq'
- assert 0 == self.main('-I', '453', '-f', str(fname))
+ assert 0 == self.main(httpserver, '-I', '453', '-f', str(fname))
assert fname.read_text() == '453'
- def test_init_from_seq_file(self, tmp_path):
+ def test_init_from_seq_file(self, tmp_path, httpserver):
fname = tmp_path / 'db.seq'
fname.write_text('453')
- assert 0 == self.main('-f', str(fname))
+ assert 0 == self.main(httpserver, '-f', str(fname))
assert fname.read_text() == '453'
- def test_init_date_with_cookie(self, capsys, tmp_path, mock_requests):
- self.url('https://planet.osm.org/replication/minute//state.txt',
- """\
+ def test_init_date_with_cookie(self, capsys, tmp_path, httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data(dedent("""\
sequenceNumber=100
timestamp=2017-08-26T11\\:04\\:02Z
- """)
- self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
- """\
+ """))
+ httpserver.expect_request('/000/000/000.state.txt').respond_with_data(dedent("""\
sequenceNumber=0
timestamp=2016-08-26T11\\:04\\:02Z
- """)
+ """))
fname = tmp_path / 'my.cookie'
cookie_jar = cookiejarlib.MozillaCookieJar(str(fname))
cookie_jar.save()
- assert 0 == self.main('--cookie', str(fname), '-D', '2015-12-24T08:08:08Z')
+ assert 0 == self.main(httpserver, '--cookie', str(fname),
+ '-D', '2015-12-24T08:08:08Z')
output = capsys.readouterr().out.strip()
=====================================
test/test_replication.py
=====================================
@@ -2,13 +2,14 @@
#
# This file is part of Pyosmium.
#
-# Copyright (C) 2022 Sarah Hoffmann.
-from io import BytesIO
+# Copyright (C) 2023 Sarah Hoffmann.
+import logging
+import time
from textwrap import dedent
-from urllib.error import URLError
import pytest
-import requests.exceptions
+
+from werkzeug.wrappers import Response
from helpers import mkdate, CountingHandler
@@ -16,15 +17,6 @@ import osmium as o
import osmium.replication.server as rserv
import osmium.replication
-class RequestsResponses(BytesIO):
-
- def __init__(self, bytes):
- super(RequestsResponses, self).__init__(bytes)
- self.content = bytes
-
- def iter_lines(self):
- return self.readlines()
-
@pytest.mark.parametrize("inp,outp", [
(None, 'https://text.org/state.txt'),
@@ -60,297 +52,391 @@ def test_get_newest_change_from_file(tmp_path):
assert val == mkdate(2018, 10, 29, 3, 56, 7)
-class TestReplication:
-
- @pytest.fixture(params=["requests", "urllib"], autouse=True)
- def setup_mocks(self, request, monkeypatch):
- self.url_requests = []
- self.url_exception = None
- if request.param == "requests":
- # Default use of the requests library. Simply patch the get method.
- def mock_get(*args, **kwargs):
- if self.url_exception is not None:
- raise self.url_exception
- assert self.url_requests
- return RequestsResponses(self.url_requests.pop(0))
- monkeypatch.setattr(osmium.replication.server.requests.Session, "get", mock_get)
-
- def mk_server(*args, **kwargs):
- return rserv.ReplicationServer(*args, **kwargs)
- self.mk_replication_server = mk_server
-
- elif request.param == "urllib":
- def mock_get(*args, **kwargs):
- if self.url_exception is not None:
- raise self.url_exception
- assert self.url_requests
- return BytesIO(self.url_requests.pop(0))
- def mk_server(*args, **kwargs):
- server = rserv.ReplicationServer(*args, **kwargs)
- server.open_url = mock_get
- return server
- self.mk_replication_server = mk_server
-
-
- def set_result(self, content):
- self.url_requests = [dedent(content).encode()]
-
-
- def set_script(self, files):
- self.url_requests = [dedent(s).encode() for s in files]
-
-
- def test_get_state_valid(self):
- self.set_result("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z
- txnReadyList=
- txnMax=1219304113
- txnActiveList=1219303583,1219304054,1219304104""")
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_sequence_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=259""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_date_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-2""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_timestamp_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_too_many_retries(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is None
-
-
- @pytest.mark.parametrize("error", [URLError(reason='Mock'),
- requests.exceptions.ConnectTimeout])
- def test_get_state_server_timeout(self, error):
- self.url_exception = error
-
- svr = self.mk_replication_server("https://test.io")
- assert svr.get_state_info() is None
-
-
- def test_apply_diffs_count(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
+def test_get_state_valid(httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_sequence_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=259""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_date_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-2""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_timestamp_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_permanent_error(httpserver, caplog):
+ httpserver.expect_request('/state.txt').respond_with_data('stuff', status=404)
+
+ with caplog.at_level(logging.DEBUG):
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is None
+ assert "Loading state info failed" in caplog.text
+
+
+def test_get_state_transient_error(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data('stuff', status=500)
+ httpserver.expect_ordered_request('/state.txt').respond_with_data('stuff', status=500)
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_too_many_retries(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is None
+
+
+def test_get_state_server_timeout(httpserver):
+ def sleeping(request):
+ time.sleep(2)
+ return Response("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ httpserver.expect_request("/state.txt").respond_with_handler(sleeping)
+
+ with rserv.ReplicationServer(httpserver.url_for('')) as svr:
+ svr.set_request_parameter('timeout', 1)
+
+ res = svr.get_state_info()
+
+ assert res is None
+
+
+def test_apply_diffs_count(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1
+ w1
+ r1
+ """))
+
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000)
-
assert h.counts == [1, 1, 1, 0]
- def test_apply_diffs_without_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_diffs_without_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000, simplify=False)
assert [2, 1, 1, 0] == h.counts
- def test_apply_diffs_with_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_diffs_with_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000, simplify=True)
assert [1, 1, 1, 0] == h.counts
- def test_apply_with_location(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 x10.0 y23.0
- w1 Nn1,n2
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
- class Handler(CountingHandler):
- def way(self, w):
- self.counts[1] += 1
- assert 2 == len(w.nodes)
- assert 1 == w.nodes[0].ref
- assert 10 == w.nodes[0].location.lon
- assert 23 == w.nodes[0].location.lat
- assert 2 == w.nodes[1].ref
- assert not w.nodes[1].location.valid()
-
- h = Handler()
+def test_apply_with_location(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+
+ class Handler(CountingHandler):
+ def way(self, w):
+ self.counts[1] += 1
+ assert 2 == len(w.nodes)
+ assert 1 == w.nodes[0].ref
+ assert 10 == w.nodes[0].location.lon
+ assert 23 == w.nodes[0].location.lat
+ assert 2 == w.nodes[1].ref
+ assert not w.nodes[1].location.valid()
+
+ h = Handler()
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
assert 100 == svr.apply_diffs(h, 100, 10000, idx="flex_mem")
-
assert h.counts == [1, 1, 0, 0]
- def test_apply_reader_without_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_reader_without_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
-
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
- diffs.reader.apply(h, simplify=False)
- assert [2, 1, 1, 0] == h.counts
+ assert diffs is not None
+ diffs.reader.apply(h, simplify=False)
+ assert [2, 1, 1, 0] == h.counts
- def test_apply_reader_with_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_reader_with_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
-
- diffs.reader.apply(h, simplify=True)
- assert [1, 1, 1, 0] == h.counts
-
-
- def test_apply_reader_with_location(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 x10.0 y23.0
- w1 Nn1,n2
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
- class Handler(CountingHandler):
- def way(self, w):
- self.counts[1] += 1
- assert 2 == len(w.nodes)
- assert 1 == w.nodes[0].ref
- assert 10 == w.nodes[0].location.lon
- assert 23 == w.nodes[0].location.lat
- assert 2 == w.nodes[1].ref
- assert not w.nodes[1].location.valid()
+ assert diffs is not None
+ diffs.reader.apply(h, simplify=True)
+ assert [1, 1, 1, 0] == h.counts
+
+
+def test_apply_reader_with_location(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+
+ class Handler(CountingHandler):
+ def way(self, w):
+ self.counts[1] += 1
+ assert 2 == len(w.nodes)
+ assert 1 == w.nodes[0].ref
+ assert 10 == w.nodes[0].location.lon
+ assert 23 == w.nodes[0].location.lat
+ assert 2 == w.nodes[1].ref
+ assert not w.nodes[1].location.valid()
+
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = Handler()
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
- diffs.reader.apply(h, idx="flex_mem")
-
- assert h.counts == [1, 1, 0, 0]
+ assert diffs is not None
+ diffs.reader.apply(h, idx="flex_mem")
+ assert h.counts == [1, 1, 0, 0]
+
+
+def test_apply_diffs_permanent_error(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl')\
+ .respond_with_data('not a file', status=404)
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert None == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [0, 0, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
+
+
+def test_apply_diffs_permanent_error_later_diff(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=404)
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 100 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [1, 1, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
+
+
+def test_apply_diffs_transient_error(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=503)
+ httpserver.expect_ordered_request('/000/000/101.opl').respond_with_data(dedent("""\
+ n2 x10.0 y23.0
+ """))
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 101 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [2, 1, 0, 0]
+
+ assert 'Error during diff download' not in caplog.text
+
+
+
+
+def test_apply_diffs_transient_error_permanent(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ for _ in range(4):
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=503)
+ httpserver.expect_ordered_request('/000/000/101.opl').respond_with_data(dedent("""\
+ n2 x10.0 y23.0
+ """))
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 100 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [1, 1, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/46e86a62251aaf08116023d76f3149866e44b0a5
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/46e86a62251aaf08116023d76f3149866e44b0a5
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20231120/196afb2d/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list