[Git][debian-gis-team/pyosmium][master] 6 commits: New upstream version 3.7.0
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Mon Nov 20 05:06:20 GMT 2023
Bas Couwenberg pushed to branch master at Debian GIS Project / pyosmium
Commits:
46e86a62 by Bas Couwenberg at 2023-11-20T05:34:32+01:00
New upstream version 3.7.0
- - - - -
ba2120fa by Bas Couwenberg at 2023-11-20T05:34:35+01:00
Update upstream source from tag 'upstream/3.7.0'
Update to upstream version '3.7.0'
with Debian dir 570fe8e4a6d60b80633835f94c26961a2e71e282
- - - - -
e54075aa by Bas Couwenberg at 2023-11-20T05:34:49+01:00
New upstream release.
- - - - -
fc259a5b by Bas Couwenberg at 2023-11-20T05:36:45+01:00
Drop patches, applied/included upstream.
- - - - -
e8979193 by Bas Couwenberg at 2023-11-20T05:59:30+01:00
Add python3-werkzeug & python3-pytest-httpserver to build dependencies.
- - - - -
0c3b14b0 by Bas Couwenberg at 2023-11-20T05:59:30+01:00
Set distribution to unstable.
- - - - -
18 changed files:
- .github/actions/run-tests/action.yml
- .github/workflows/build_wheels.yml
- .github/workflows/ci.yml
- CHANGELOG.md
- README.md
- README.rst
- debian/changelog
- debian/control
- − debian/patches/no-distutils.patch
- − debian/patches/pybind-2.11.patch
- − debian/patches/series
- lib/osm.cc
- + pyproject.toml
- setup.py
- src/osmium/replication/server.py
- src/osmium/version.py
- test/test_pyosmium_get_changes.py
- test/test_replication.py
Changes:
=====================================
.github/actions/run-tests/action.yml
=====================================
@@ -4,7 +4,7 @@ runs:
using: "composite"
steps:
- name: Install test requirements
- run: pip install pytest shapely
+ run: pip install pytest pytest-httpserver shapely
shell: bash
- name: Run tests
=====================================
.github/workflows/build_wheels.yml
=====================================
@@ -19,7 +19,7 @@ jobs:
- uses: actions/checkout at v3
with:
repository: pybind/pybind11
- ref: v2.10.3
+ ref: v2.11.1
path: contrib/pybind11
- uses: actions/checkout at v3
@@ -31,16 +31,17 @@ jobs:
- uses: actions/checkout at v3
with:
repository: osmcode/libosmium
- ref: v2.19.0
+ ref: v2.20.0
path: contrib/libosmium
- name: Build wheels
- uses: pypa/cibuildwheel at v2.11.3
+ uses: pypa/cibuildwheel at v2.16.2
env:
CIBW_ARCHS: native
CIBW_SKIP: "pp* *musllinux*"
- CIBW_TEST_REQUIRES: pytest shapely
+ CIBW_TEST_REQUIRES: pytest pytest-httpserver shapely
+ CIBW_TEST_REQUIRES_LINUX: urllib3<2.0 pytest pytest-httpserver shapely
CIBW_TEST_COMMAND: pytest {project}/test
CIBW_BUILD_FRONTEND: build
CIBW_BEFORE_BUILD_LINUX: yum install -y sparsehash-devel expat-devel boost-devel zlib-devel bzip2-devel lz4-devel
=====================================
.github/workflows/ci.yml
=====================================
@@ -78,6 +78,17 @@ jobs:
python -m build
shell: bash
+ - name: Set up Python 3.12
+ uses: actions/setup-python at v4
+ with:
+ python-version: "3.12"
+
+ - name: Build package 3.12
+ run: |
+ pip install build wheel
+ python -m build
+ shell: bash
+
- name: Upload Artifact
uses: actions/upload-artifact at v3
with:
@@ -91,7 +102,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- python-version: [3.6, 3.7, 3.8, 3.9, "3.10", "3.11"]
+ python-version: [3.6, 3.7, 3.8, 3.9, "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout at v3
@@ -123,20 +134,20 @@ jobs:
compiler: [gcc-old, clang-old, gcc, clang, macos]
include:
- compiler: gcc-old
- cc: gcc-7
- cxx: g++-7
- platform: ubuntu-18.04
+ cc: gcc-10
+ cxx: g++-10
+ platform: ubuntu-20.04
python: 3.6
deps: release
- compiler: clang-old
- cc: clang-6.0
- cxx: clang++-6.0
- platform: ubuntu-18.04
+ cc: clang-10
+ cxx: clang++-10
+ platform: ubuntu-20.04
python: 3.6
deps: release
- compiler: gcc
- cc: gcc-11
- cxx: g++-11
+ cc: gcc-12
+ cxx: g++-12
platform: ubuntu-22.04
python: "3.11"
deps: develop
@@ -182,7 +193,7 @@ jobs:
- name: Install prerequisites
run: |
python -m pip install --upgrade pip
- pip install pytest shapely setuptools requests
+ pip install pytest pytest-httpserver shapely setuptools requests
shell: bash
- name: Build package
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,22 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.7.0] - 2023-11-19
+
+### Added
+
+- transparently retry download on transient HTTP errors
+
+### Fixed
+
+- catch non-200 status for HTTP responses
+
+### Changed
+
+- update to pybind 2.11.1
+- update to libosmium 2.20.0
+
+
## [3.6.0] - 2023-01-20
### Changed
@@ -12,6 +28,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- update to libosmium 2.19.0
- change minimum required version of Cmake to 3.0
+
## [3.6.0rc1] - 2022-12-13
### Changed
@@ -20,6 +37,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- invalid buffers are now checked on access time, no more reference count
checks on leaving the handler callback
+
## [3.5.0] - 2022-11-09
### Added
=====================================
README.md
=====================================
@@ -74,13 +74,15 @@ They are mostly ports of the examples in Libosmium and osmium-contrib.
## Testing
-There is a small test suite in the test directory. This provides regression
+There is a small test suite in the test directory. This provides unit
test for the python bindings, it is not meant to be a test suite for Libosmium.
-You'll need the Python `pytest` module. On Debian/Ubuntu install the package
-`python3-pytest`.
+Testing requires `pytest` and `pytest-httpserver`. On Debian/Ubuntu install
+the dependencies with:
-The suite can be run with:
+ sudo apt-get install python3-pytest python3-pytest-httpserver
+
+The test suite can be run with:
pytest test
=====================================
README.rst
=====================================
@@ -24,7 +24,7 @@ development packages for these libraries. On Debian/Ubuntu do::
libexpat1-dev zlib1g-dev libbz2-dev
-Python >= 3.4 is supported.
+Python >= 3.6 is supported. Pypy is known not to work.
Documentation
=============
=====================================
debian/changelog
=====================================
@@ -1,12 +1,15 @@
-pyosmium (3.6.0-3) UNRELEASED; urgency=medium
+pyosmium (3.7.0-1) unstable; urgency=medium
+ * New upstream release.
* Remove generated files in clean target.
(closes: #1045177)
* Update lintian overrides.
* Use execute_after instead of override in rules file.
* Switch to dh-sequence-*.
+ * Drop patches, applied/included upstream.
+ * Add python3-werkzeug & python3-pytest-httpserver to build dependencies.
- -- Bas Couwenberg <sebastic at debian.org> Fri, 11 Aug 2023 13:08:37 +0200
+ -- Bas Couwenberg <sebastic at debian.org> Mon, 20 Nov 2023 05:37:30 +0100
pyosmium (3.6.0-2) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -17,11 +17,13 @@ Build-Depends: cmake (>= 3.0),
python3-all-dev,
python3-packaging,
python3-pytest <!nocheck>,
+ python3-pytest-httpserver <!nocheck>,
python3-requests,
python3-setuptools,
python3-shapely <!nocheck>,
python3-sphinx,
- python3-sphinxcontrib.autoprogram
+ python3-sphinxcontrib.autoprogram,
+ python3-werkzeug <!nocheck>
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/debian-gis-team/pyosmium/
Vcs-Git: https://salsa.debian.org/debian-gis-team/pyosmium.git
=====================================
debian/patches/no-distutils.patch deleted
=====================================
@@ -1,33 +0,0 @@
-Description: Don't use deprecated distutils module.
-Author: Bas Couwenberg <sebastic at debian.org>
-Forwarded: https://github.com/osmcode/pyosmium/pull/226
-Applied-Upstream: https://github.com/osmcode/pyosmium/commit/1f3e9d19a475610687a9202767ae9b838b445efe
-
---- a/setup.py
-+++ b/setup.py
-@@ -10,7 +10,7 @@ from pathlib import Path
- from setuptools import setup, Extension
- from setuptools.command.build_ext import build_ext
- from setuptools.command.sdist import sdist as orig_sdist
--from distutils.version import LooseVersion
-+from packaging.version import Version
-
- BASEDIR = os.path.split(os.path.abspath(__file__))[0]
-
-@@ -75,8 +75,8 @@ class CMakeBuild(build_ext):
- ", ".join(e.name for e in self.extensions))
-
- if platform.system() == "Windows":
-- cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
-- if cmake_version < '3.1.0':
-+ cmake_version = Version(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
-+ if cmake_version < Version('3.1.0'):
- raise RuntimeError("CMake >= 3.1.0 is required on Windows")
-
- for ext in self.extensions:
---- /dev/null
-+++ b/pyproject.toml
-@@ -0,0 +1,3 @@
-+[build-system]
-+requires = ["packaging", "setuptools"]
-+build-backend = "setuptools.build_meta"
=====================================
debian/patches/pybind-2.11.patch deleted
=====================================
@@ -1,51 +0,0 @@
-Description: Compatibility with recent pybind versions
- Work-around false positive added by pybind/pybind11 at f701654 change:
- ItemIterator/CollectionIterator ARE copy/move constructible, even if their template
- parameter is not.
- .
- Enabled for pybind version >= 2.11.
-From: Aurélien Grenotton <agrenott at gmail.com>
-Origin: https://github.com/osmcode/pyosmium/commit/31b1363389b423f49e14140ce868ecac83e92f69
-Forwarded: not-needed
-
---- a/lib/osm.cc
-+++ b/lib/osm.cc
-@@ -18,6 +18,38 @@ namespace py = pybind11;
- using TagIterator = osmium::TagList::const_iterator;
- using MemberIterator = osmium::RelationMemberList::const_iterator;
-
-+#if PYBIND11_VERSION_MINOR >= 11 || PYBIND11_VERSION_MAJOR > 2
-+/*
-+Work-around false positive added by pybind/pybind11 at f701654 change:
-+ItemIterator/CollectionIterator ARE copy/move constructible, even if their template
-+parameter is not. Indeed, those iterators iterate over low-level memory representation
-+of the objects, without relying on their constructors.
-+
-+For eg.
-+// static_assert(std::is_move_constructible<osmium::memory::CollectionIterator<osmium::RelationMember const>>::value);
-+// static_assert(!std::is_copy_constructible<osmium::RelationMember>::value);
-+
-+The work-around relies on officially exposed pybind11::detail::is_copy_constructible/is_copy_constructible:
-+https://github.com/pybind/pybind11/pull/4631
-+*/
-+namespace pybind11 {
-+namespace detail {
-+template <typename T>
-+struct is_copy_constructible<osmium::memory::CollectionIterator<T>>
-+ : std::true_type {};
-+template <typename T>
-+struct is_move_constructible<osmium::memory::CollectionIterator<T>>
-+ : std::true_type {};
-+template <typename T>
-+struct is_copy_constructible<osmium::memory::ItemIterator<T>>
-+ : std::true_type {};
-+template <typename T>
-+struct is_move_constructible<osmium::memory::ItemIterator<T>>
-+ : std::true_type {};
-+} // namespace detail
-+} // namespace pybind11
-+#endif
-+
- static py::object tag_iterator_next(TagIterator &it, TagIterator const &cend)
- {
- if (it == cend)
=====================================
debian/patches/series deleted
=====================================
@@ -1,2 +0,0 @@
-no-distutils.patch
-pybind-2.11.patch
=====================================
lib/osm.cc
=====================================
@@ -18,6 +18,38 @@ namespace py = pybind11;
using TagIterator = osmium::TagList::const_iterator;
using MemberIterator = osmium::RelationMemberList::const_iterator;
+#if PYBIND11_VERSION_MINOR >= 11 || PYBIND11_VERSION_MAJOR > 2
+/*
+Work-around false positive added by pybind/pybind11 at f701654 change:
+ItemIterator/CollectionIterator ARE copy/move constructible, even if their template
+parameter is not. Indeed, those iterators iterate over low-level memory representation
+of the objects, without relying on their constructors.
+
+For eg.
+// static_assert(std::is_move_constructible<osmium::memory::CollectionIterator<osmium::RelationMember const>>::value);
+// static_assert(!std::is_copy_constructible<osmium::RelationMember>::value);
+
+The work-around relies on officially exposed pybind11::detail::is_copy_constructible/is_copy_constructible:
+https://github.com/pybind/pybind11/pull/4631
+*/
+namespace pybind11 {
+namespace detail {
+template <typename T>
+struct is_copy_constructible<osmium::memory::CollectionIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_move_constructible<osmium::memory::CollectionIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_copy_constructible<osmium::memory::ItemIterator<T>>
+ : std::true_type {};
+template <typename T>
+struct is_move_constructible<osmium::memory::ItemIterator<T>>
+ : std::true_type {};
+} // namespace detail
+} // namespace pybind11
+#endif
+
static py::object tag_iterator_next(TagIterator &it, TagIterator const &cend)
{
if (it == cend)
=====================================
pyproject.toml
=====================================
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["packaging", "setuptools"]
+build-backend = "setuptools.build_meta"
=====================================
setup.py
=====================================
@@ -10,7 +10,7 @@ from pathlib import Path
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
from setuptools.command.sdist import sdist as orig_sdist
-from distutils.version import LooseVersion
+from packaging.version import Version
BASEDIR = os.path.split(os.path.abspath(__file__))[0]
@@ -75,8 +75,8 @@ class CMakeBuild(build_ext):
", ".join(e.name for e in self.extensions))
if platform.system() == "Windows":
- cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
- if cmake_version < '3.1.0':
+ cmake_version = Version(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
+ if cmake_version < Version('3.1.0'):
raise RuntimeError("CMake >= 3.1.0 is required on Windows")
for ext in self.extensions:
=====================================
src/osmium/replication/server.py
=====================================
@@ -7,7 +7,6 @@
""" Helper functions to communicate with replication servers.
"""
from typing import NamedTuple, Optional, Any, Iterator, cast, Dict, Mapping, Tuple
-import requests
import urllib.request as urlrequest
from urllib.error import URLError
import datetime as dt
@@ -15,6 +14,10 @@ from collections import namedtuple
from contextlib import contextmanager
from math import ceil
+import requests
+from requests.adapters import HTTPAdapter
+from urllib3.util import Retry
+
from osmium import MergeInputReader, BaseHandler
from osmium import io as oio
from osmium import version
@@ -52,6 +55,8 @@ class ReplicationServer:
self.diff_type = diff_type
self.extra_request_params: dict[str, Any] = dict(timeout=60, stream=True)
self.session: Optional[requests.Session] = None
+ self.retry = Retry(total=3, backoff_factor=0.5, allowed_methods={'GET'},
+ status_forcelist=[408, 429, 500, 502, 503, 504])
def close(self) -> None:
""" Close any open connection to the replication server.
@@ -62,6 +67,8 @@ class ReplicationServer:
def __enter__(self) -> 'ReplicationServer':
self.session = requests.Session()
+ self.session.mount('http://', HTTPAdapter(max_retries=self.retry))
+ self.session.mount('https://', HTTPAdapter(max_retries=self.retry))
return self
def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
@@ -97,6 +104,8 @@ class ReplicationServer:
@contextmanager
def _get_url_with_session() -> Iterator[requests.Response]:
with requests.Session() as session:
+ session.mount('http://', HTTPAdapter(max_retries=self.retry))
+ session.mount('https://', HTTPAdapter(max_retries=self.retry))
request = session.get(url.get_full_url(), **get_params)
yield request
@@ -133,7 +142,7 @@ class ReplicationServer:
try:
diffdata = self.get_diff_block(current_id)
except:
- LOG.debug("Error during diff download. Bailing out.")
+ LOG.error("Error during diff download. Bailing out.")
diffdata = ''
if len(diffdata) == 0:
if start_id == current_id:
@@ -348,6 +357,7 @@ class ReplicationServer:
with self.open_url(self.make_request(self.get_state_url(seq))) as response:
if hasattr(response, 'iter_lines'):
# generated by requests
+ response.raise_for_status()
lines = response.iter_lines()
else:
lines = response
@@ -372,7 +382,7 @@ class ReplicationServer:
ts = ts.replace(tzinfo=dt.timezone.utc)
except (URLError, IOError) as err:
- LOG.debug("Loading state info %s failed with: %s", seq, str(err))
+ LOG.debug("Loading state info failed with: %s", str(err))
return None
if ts is not None and next_seq is not None:
@@ -382,12 +392,13 @@ class ReplicationServer:
def get_diff_block(self, seq: int) -> str:
""" Downloads the diff with the given sequence number and returns
- it as a byte sequence. Throws a :code:`urllib.error.HTTPError`
+ it as a byte sequence. Throws an :code:`requests.HTTPError`
if the file cannot be downloaded.
"""
with self.open_url(self.make_request(self.get_diff_url(seq))) as resp:
if hasattr(resp, 'content'):
# generated by requests
+ resp.raise_for_status()
return cast(str, resp.content)
# generated by urllib.request
=====================================
src/osmium/version.py
=====================================
@@ -9,13 +9,13 @@ Version information.
"""
# the major version
-pyosmium_major = '3.6'
+pyosmium_major = '3.7'
# current release (Pip version)
-pyosmium_release = '3.6.0'
+pyosmium_release = '3.7.0'
# libosmium version shipped with the Pip release
-libosmium_version = '2.19.0'
+libosmium_version = '2.20.0'
# protozero version shipped with the Pip release
protozero_version = '1.7.1'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.10.3'
+pybind11_version = '2.11.1'
=====================================
test/test_pyosmium_get_changes.py
=====================================
@@ -2,10 +2,9 @@
#
# This file is part of Pyosmium.
#
-# Copyright (C) 2022 Sarah Hoffmann.
+# Copyright (C) 2023 Sarah Hoffmann.
""" Tests for the pyosmium-get-changes script.
"""
-from io import BytesIO
from pathlib import Path
from textwrap import dedent
@@ -18,101 +17,76 @@ except ImportError:
import cookielib as cookiejarlib
-class RequestsResponses(BytesIO):
-
- def __init__(self, bytes):
- super(RequestsResponses, self).__init__(bytes)
- self.content = bytes
-
- def iter_lines(self):
- return self.readlines()
-
-
class TestPyosmiumGetChanges:
@pytest.fixture(autouse=True)
- def setUp(self, monkeypatch):
+ def setup(self):
self.script = dict()
filename = (Path(__file__) / ".." / ".." / "tools"/ "pyosmium-get-changes").resolve()
with filename.open("rb") as f:
exec(compile(f.read(), str(filename), 'exec'), self.script)
- self.urls = dict()
-
-
- @pytest.fixture
- def mock_requests(self, monkeypatch):
- def mock_get(session, url, **kwargs):
- return RequestsResponses(self.urls[url])
- monkeypatch.setattr(osmium.replication.server.requests.Session, "get", mock_get)
-
-
- def url(self, url, result):
- self.urls[url] = dedent(result).encode()
- def main(self, *args):
- return self.script['main'](args)
+ def main(self, httpserver, *args):
+ return self.script['main'](['--server', httpserver.url_for('')] + list(args))
- def test_init_id(self, capsys):
- assert 0 == self.main('-I', '453')
+ def test_init_id(self, capsys, httpserver):
+ assert 0 == self.main(httpserver, '-I', '453')
output = capsys.readouterr().out.strip()
assert output == '453'
- def test_init_date(self, capsys, mock_requests):
- self.url('https://planet.osm.org/replication/minute//state.txt',
- """\
+ def test_init_date(self, capsys, httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data(dedent("""\
sequenceNumber=100
timestamp=2017-08-26T11\\:04\\:02Z
- """)
- self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
- """\
+ """))
+ httpserver.expect_request('/000/000/000.state.txt').respond_with_data(dedent("""\
sequenceNumber=0
timestamp=2016-08-26T11\\:04\\:02Z
- """)
- assert 0 == self.main('-D', '2015-12-24T08:08:08Z')
+ """))
+ assert 0 == self.main(httpserver, '-D', '2015-12-24T08:08:08Z')
output = capsys.readouterr().out.strip()
assert output == '-1'
- def test_init_to_file(self, tmp_path):
+ def test_init_to_file(self, tmp_path, httpserver):
fname = tmp_path / 'db.seq'
- assert 0 == self.main('-I', '453', '-f', str(fname))
+ assert 0 == self.main(httpserver, '-I', '453', '-f', str(fname))
assert fname.read_text() == '453'
- def test_init_from_seq_file(self, tmp_path):
+ def test_init_from_seq_file(self, tmp_path, httpserver):
fname = tmp_path / 'db.seq'
fname.write_text('453')
- assert 0 == self.main('-f', str(fname))
+ assert 0 == self.main(httpserver, '-f', str(fname))
assert fname.read_text() == '453'
- def test_init_date_with_cookie(self, capsys, tmp_path, mock_requests):
- self.url('https://planet.osm.org/replication/minute//state.txt',
- """\
+ def test_init_date_with_cookie(self, capsys, tmp_path, httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data(dedent("""\
sequenceNumber=100
timestamp=2017-08-26T11\\:04\\:02Z
- """)
- self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
- """\
+ """))
+ httpserver.expect_request('/000/000/000.state.txt').respond_with_data(dedent("""\
sequenceNumber=0
timestamp=2016-08-26T11\\:04\\:02Z
- """)
+ """))
fname = tmp_path / 'my.cookie'
cookie_jar = cookiejarlib.MozillaCookieJar(str(fname))
cookie_jar.save()
- assert 0 == self.main('--cookie', str(fname), '-D', '2015-12-24T08:08:08Z')
+ assert 0 == self.main(httpserver, '--cookie', str(fname),
+ '-D', '2015-12-24T08:08:08Z')
output = capsys.readouterr().out.strip()
=====================================
test/test_replication.py
=====================================
@@ -2,13 +2,14 @@
#
# This file is part of Pyosmium.
#
-# Copyright (C) 2022 Sarah Hoffmann.
-from io import BytesIO
+# Copyright (C) 2023 Sarah Hoffmann.
+import logging
+import time
from textwrap import dedent
-from urllib.error import URLError
import pytest
-import requests.exceptions
+
+from werkzeug.wrappers import Response
from helpers import mkdate, CountingHandler
@@ -16,15 +17,6 @@ import osmium as o
import osmium.replication.server as rserv
import osmium.replication
-class RequestsResponses(BytesIO):
-
- def __init__(self, bytes):
- super(RequestsResponses, self).__init__(bytes)
- self.content = bytes
-
- def iter_lines(self):
- return self.readlines()
-
@pytest.mark.parametrize("inp,outp", [
(None, 'https://text.org/state.txt'),
@@ -60,297 +52,391 @@ def test_get_newest_change_from_file(tmp_path):
assert val == mkdate(2018, 10, 29, 3, 56, 7)
-class TestReplication:
-
- @pytest.fixture(params=["requests", "urllib"], autouse=True)
- def setup_mocks(self, request, monkeypatch):
- self.url_requests = []
- self.url_exception = None
- if request.param == "requests":
- # Default use of the requests library. Simply patch the get method.
- def mock_get(*args, **kwargs):
- if self.url_exception is not None:
- raise self.url_exception
- assert self.url_requests
- return RequestsResponses(self.url_requests.pop(0))
- monkeypatch.setattr(osmium.replication.server.requests.Session, "get", mock_get)
-
- def mk_server(*args, **kwargs):
- return rserv.ReplicationServer(*args, **kwargs)
- self.mk_replication_server = mk_server
-
- elif request.param == "urllib":
- def mock_get(*args, **kwargs):
- if self.url_exception is not None:
- raise self.url_exception
- assert self.url_requests
- return BytesIO(self.url_requests.pop(0))
- def mk_server(*args, **kwargs):
- server = rserv.ReplicationServer(*args, **kwargs)
- server.open_url = mock_get
- return server
- self.mk_replication_server = mk_server
-
-
- def set_result(self, content):
- self.url_requests = [dedent(content).encode()]
-
-
- def set_script(self, files):
- self.url_requests = [dedent(s).encode() for s in files]
-
-
- def test_get_state_valid(self):
- self.set_result("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z
- txnReadyList=
- txnMax=1219304113
- txnActiveList=1219303583,1219304054,1219304104""")
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_sequence_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=259""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_date_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-2""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_timestamp_cut(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is not None
- assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
- assert res.sequence == 2594669
-
-
- def test_get_state_too_many_retries(self):
- self.set_script(("""\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=""",
- """\
- #Sat Aug 26 11:04:04 UTC 2017
- txnMaxQueried=1219304113
- sequenceNumber=2594669
- timestamp=2017-08-26T11\\:04\\:02Z"""))
-
- res = self.mk_replication_server("https://test.io").get_state_info()
-
- assert res is None
-
-
- @pytest.mark.parametrize("error", [URLError(reason='Mock'),
- requests.exceptions.ConnectTimeout])
- def test_get_state_server_timeout(self, error):
- self.url_exception = error
-
- svr = self.mk_replication_server("https://test.io")
- assert svr.get_state_info() is None
-
-
- def test_apply_diffs_count(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
+def test_get_state_valid(httpserver):
+ httpserver.expect_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_sequence_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=259""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_date_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-2""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_timestamp_cut(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_permanent_error(httpserver, caplog):
+ httpserver.expect_request('/state.txt').respond_with_data('stuff', status=404)
+
+ with caplog.at_level(logging.DEBUG):
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is None
+ assert "Loading state info failed" in caplog.text
+
+
+def test_get_state_transient_error(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data('stuff', status=500)
+ httpserver.expect_ordered_request('/state.txt').respond_with_data('stuff', status=500)
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is not None
+ assert res.timestamp == mkdate(2017, 8, 26, 11, 4, 2)
+ assert res.sequence == 2594669
+
+
+def test_get_state_too_many_retries(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=""")
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z""")
+
+ res = rserv.ReplicationServer(httpserver.url_for('')).get_state_info()
+
+ assert res is None
+
+
+def test_get_state_server_timeout(httpserver):
+ def sleeping(request):
+ time.sleep(2)
+ return Response("""\
+ #Sat Aug 26 11:04:04 UTC 2017
+ txnMaxQueried=1219304113
+ sequenceNumber=2594669
+ timestamp=2017-08-26T11\\:04\\:02Z
+ txnReadyList=
+ txnMax=1219304113
+ txnActiveList=1219303583,1219304054,1219304104""")
+
+ httpserver.expect_request("/state.txt").respond_with_handler(sleeping)
+
+ with rserv.ReplicationServer(httpserver.url_for('')) as svr:
+ svr.set_request_parameter('timeout', 1)
+
+ res = svr.get_state_info()
+
+ assert res is None
+
+
+def test_apply_diffs_count(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1
+ w1
+ r1
+ """))
+
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000)
-
assert h.counts == [1, 1, 1, 0]
- def test_apply_diffs_without_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_diffs_without_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000, simplify=False)
assert [2, 1, 1, 0] == h.counts
- def test_apply_diffs_with_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_diffs_with_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
assert 100 == svr.apply_diffs(h, 100, 10000, simplify=True)
assert [1, 1, 1, 0] == h.counts
- def test_apply_with_location(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 x10.0 y23.0
- w1 Nn1,n2
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
- class Handler(CountingHandler):
- def way(self, w):
- self.counts[1] += 1
- assert 2 == len(w.nodes)
- assert 1 == w.nodes[0].ref
- assert 10 == w.nodes[0].location.lon
- assert 23 == w.nodes[0].location.lat
- assert 2 == w.nodes[1].ref
- assert not w.nodes[1].location.valid()
-
- h = Handler()
+def test_apply_with_location(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+
+ class Handler(CountingHandler):
+ def way(self, w):
+ self.counts[1] += 1
+ assert 2 == len(w.nodes)
+ assert 1 == w.nodes[0].ref
+ assert 10 == w.nodes[0].location.lon
+ assert 23 == w.nodes[0].location.lat
+ assert 2 == w.nodes[1].ref
+ assert not w.nodes[1].location.valid()
+
+ h = Handler()
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
assert 100 == svr.apply_diffs(h, 100, 10000, idx="flex_mem")
-
assert h.counts == [1, 1, 0, 0]
- def test_apply_reader_without_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_reader_without_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
-
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
- diffs.reader.apply(h, simplify=False)
- assert [2, 1, 1, 0] == h.counts
+ assert diffs is not None
+ diffs.reader.apply(h, simplify=False)
+ assert [2, 1, 1, 0] == h.counts
- def test_apply_reader_with_simplify(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 v23
- n1 v24
- w1
- r1
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
+def test_apply_reader_with_simplify(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 v23
+ n1 v24
+ w1
+ r1
+ """))
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = CountingHandler()
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
-
- diffs.reader.apply(h, simplify=True)
- assert [1, 1, 1, 0] == h.counts
-
-
- def test_apply_reader_with_location(self):
- self.set_script(("""\
- sequenceNumber=100
- timestamp=2017-08-26T11\\:04\\:02Z
- """, """
- n1 x10.0 y23.0
- w1 Nn1,n2
- """))
- svr = self.mk_replication_server("https://test.io", "opl")
-
- class Handler(CountingHandler):
- def way(self, w):
- self.counts[1] += 1
- assert 2 == len(w.nodes)
- assert 1 == w.nodes[0].ref
- assert 10 == w.nodes[0].location.lon
- assert 23 == w.nodes[0].location.lat
- assert 2 == w.nodes[1].ref
- assert not w.nodes[1].location.valid()
+ assert diffs is not None
+ diffs.reader.apply(h, simplify=True)
+ assert [1, 1, 1, 0] == h.counts
+
+
+def test_apply_reader_with_location(httpserver):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+
+ class Handler(CountingHandler):
+ def way(self, w):
+ self.counts[1] += 1
+ assert 2 == len(w.nodes)
+ assert 1 == w.nodes[0].ref
+ assert 10 == w.nodes[0].location.lon
+ assert 23 == w.nodes[0].location.lat
+ assert 2 == w.nodes[1].ref
+ assert not w.nodes[1].location.valid()
+
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
h = Handler()
diffs = svr.collect_diffs(100, 100000)
- assert diffs is not None
- diffs.reader.apply(h, idx="flex_mem")
-
- assert h.counts == [1, 1, 0, 0]
+ assert diffs is not None
+ diffs.reader.apply(h, idx="flex_mem")
+ assert h.counts == [1, 1, 0, 0]
+
+
+def test_apply_diffs_permanent_error(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl')\
+ .respond_with_data('not a file', status=404)
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert None == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [0, 0, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
+
+
+def test_apply_diffs_permanent_error_later_diff(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=404)
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 100 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [1, 1, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
+
+
+def test_apply_diffs_transient_error(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=503)
+ httpserver.expect_ordered_request('/000/000/101.opl').respond_with_data(dedent("""\
+ n2 x10.0 y23.0
+ """))
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 101 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [2, 1, 0, 0]
+
+ assert 'Error during diff download' not in caplog.text
+
+
+
+
+def test_apply_diffs_transient_error_permanent(httpserver, caplog):
+ httpserver.expect_ordered_request('/state.txt').respond_with_data("""\
+ sequenceNumber=101
+ timestamp=2017-08-26T11\\:04\\:02Z
+ """)
+ httpserver.expect_ordered_request('/000/000/100.opl').respond_with_data(dedent("""\
+ n1 x10.0 y23.0
+ w1 Nn1,n2
+ """))
+ for _ in range(4):
+ httpserver.expect_ordered_request('/000/000/101.opl')\
+ .respond_with_data('not a file', status=503)
+ httpserver.expect_ordered_request('/000/000/101.opl').respond_with_data(dedent("""\
+ n2 x10.0 y23.0
+ """))
+
+ with caplog.at_level(logging.ERROR):
+ with rserv.ReplicationServer(httpserver.url_for(''), "opl") as svr:
+ h = CountingHandler()
+ assert 100 == svr.apply_diffs(h, 100, 10000)
+ assert h.counts == [1, 1, 0, 0]
+
+ assert 'Error during diff download' in caplog.text
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/206e77ac4fc8a447df5504eba34713f79fd8035e...0c3b14b05e6492923bc01ccd107b48e569f837e6
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/206e77ac4fc8a447df5504eba34713f79fd8035e...0c3b14b05e6492923bc01ccd107b48e569f837e6
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20231120/a22dbf32/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list