[Git][debian-gis-team/pyosmium][upstream] New upstream version 3.2.0
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Wed Aug 11 05:10:55 BST 2021
Bas Couwenberg pushed to branch upstream at Debian GIS Project / pyosmium
Commits:
57f84a49 by Bas Couwenberg at 2021-08-11T05:53:50+02:00
New upstream version 3.2.0
- - - - -
12 changed files:
- .github/actions/install-dependencies/action.yml
- .github/workflows/ci.yml
- CHANGELOG.md
- README.md
- README.rst
- doc/updating_osm_data.rst
- lib/merge_input_reader.cc
- setup.py
- src/osmium/replication/server.py
- src/osmium/version.py
- test/test_pyosmium_get_changes.py
- test/test_replication.py
Changes:
=====================================
.github/actions/install-dependencies/action.yml
=====================================
@@ -7,7 +7,7 @@ runs:
- name: Install pip dependencies
run: |
python -m pip install --upgrade pip
- pip install nose shapely setuptools
+ pip install nose shapely setuptools requests
shell: bash
- name: Install package dependencies
=====================================
.github/workflows/ci.yml
=====================================
@@ -33,8 +33,8 @@ jobs:
working-directory: test
- build-ubuntu-1604:
- runs-on: ubuntu-16.04
+ build-ubuntu-1804:
+ runs-on: ubuntu-18.04
strategy:
matrix:
@@ -50,14 +50,18 @@ jobs:
- name: Install packages
run: sudo apt-get install -y -qq libboost-dev libexpat1-dev zlib1g-dev libbz2-dev libproj-dev libgeos-dev liblz4-dev
+ - name: Install clang package
+ run: sudo apt-get install -y -qq clang-6.0
+ if: ${{ matrix.compiler == 'clang' }}
+
- uses: ./.github/actions/install-dependencies
- name: Build package
run: python setup.py build
shell: bash
env:
- CC: gcc-5
- CXX: g++-5
+ CC: gcc-7
+ CXX: g++-7
if: ${{ matrix.compiler == 'gcc' }}
- name: Build package
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,18 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.2.0] - 2021-08-09
+
+### Fixed
+
+- merge change files correctly so that latest applied object comes first
+
+### Changed
+
+- switch to requests library for downloading files (thanks @jocelynj)
+- update to libosmium 2.17.0
+- update to pybind11 2.7.1
+
## [3.1.3] - 2021-02-05
### Fixed
=====================================
README.md
=====================================
@@ -21,7 +21,7 @@ We provide binary wheels for Linux and Windows 64 for all actively
maintained Python versions.
For other versions, a source wheel is provided. Make sure to install all
-external dependencies first. On Deian/Ubuntu-like systems, the following
+external dependencies first. On Debian/Ubuntu-like systems, the following
command installs all required packages:
sudo apt-get install build-essential cmake libboost-dev \
@@ -42,6 +42,7 @@ pyosmium has the following dependencies:
* [libz](https://www.zlib.net/)
* [libbz2](https://www.sourceware.org/bzip2/)
* [Boost](https://www.boost.org/) variant and iterator >= 1.41
+ * [Python Requests](https://docs.python-requests.org/en/master/)
* Python setuptools
* a recent C++ compiler (Clang 3.4+, GCC 4.8+)
=====================================
README.rst
=====================================
@@ -24,7 +24,7 @@ development packages for these libraries. On Debian/Ubuntu do::
libexpat1-dev zlib1g-dev libbz2-dev
-Python >= 2.7 is supported but a version >= 3.3 is strongly recommended.
+Python >= 3.4 is supported.
Documentation
=============
=====================================
doc/updating_osm_data.rst
=====================================
@@ -154,12 +154,14 @@ Creating a change file
Now you can create change files using the state::
- pyosmium-get-changes --server <replication server> -f sequence.state -o newchange.osm.gz
+ pyosmium-get-changes --server <replication server> -f sequence.state -o newchange.osc.gz
This downloads the latest changes from the server, saves them in the file
-`newchange.osm.gz` and updates your state file. `<replication server>` is the
+`newchange.osc.gz` and updates your state file. `<replication server>` is the
URL that was printed, when you set up the state file. The parameter can be
omitted when you use minutely change files from openstreetmap.org.
+This simplifies multiple edits of the same element into the final change. If you want to
+retain the full version history specify `--no-deduplicate`.
`pyosmium-get-changes` loads only about 100MB worth of updates at once (about
8 hours of planet updates). If you want more, then add a `--size` parameter.
@@ -179,13 +181,15 @@ All other error codes indicate fatal errors.
A simple shell script can look like this::
while true; do
+ # pyosmium-get-changes would not overwrite an existing changes file
+ rm -f newchange.osc.gz
# get the next batch of changes
- pyosmium-get-changes -f sequence.state -o newchange.osm.gz
+ pyosmium-get-changes -f sequence.state -o newchange.osc.gz
# save the return code
status=$?
if [ $status -eq 0 ]; then
- # apply newchange.osm.gz here
+ # apply newchange.osc.gz here
....
elif [ $status -eq 3 ]; then
# No new data, so sleep for a bit
=====================================
lib/merge_input_reader.cc
=====================================
@@ -80,6 +80,13 @@ public:
// For normal data files we sort with the largest version of each
// object first and then only copy this last version of any object
// to the output.
+ // Caution: When change files have been
+ // created from extracts it is possible that they contain objects
+ // with the same type, id, version, and timestamp. In that case we
+ // still want to get the last object available. So we have to make
+ // sure it appears first in the objects vector before doing the
+ // stable sort.
+ std::reverse(objects.ptr_begin(), objects.ptr_end());
objects.sort(osmium::object_order_type_id_reverse_version());
auto output_it = boost::make_function_output_iterator(
=====================================
setup.py
=====================================
@@ -154,6 +154,7 @@ setup(
package_dir = {'' : 'src'},
package_data = { 'osmium' : [ '*.dll' ] },
python_requires = ">=3.4",
+ install_requires = ['requests'],
cmdclass=dict(build_ext=CMakeBuild, sdist=Pyosmium_sdist),
zip_safe=False,
)
=====================================
src/osmium/replication/server.py
=====================================
@@ -1,6 +1,7 @@
""" Helper functions to communicate with replication servers.
"""
+import requests
import urllib.request as urlrequest
import datetime as dt
from collections import namedtuple
@@ -26,6 +27,7 @@ class ReplicationServer:
def __init__(self, url, diff_type='osc.gz'):
self.baseurl = url
self.diff_type = diff_type
+ self.session = requests.Session()
def make_request(self, url):
headers = {"User-Agent" : "pyosmium/{}".format(version.pyosmium_release)}
@@ -48,7 +50,10 @@ class ReplicationServer:
svr = ReplicationServer()
svr.open_url = my_open_url
"""
- return urlrequest.urlopen(url)
+ headers = dict()
+ for h in url.header_items():
+ headers[h[0]] = h[1]
+ return self.session.get(url.get_full_url(), headers=headers, stream=True)
def collect_diffs(self, start_id, max_size=1024):
""" Create a MergeInputReader and download diffs starting with sequence
@@ -292,8 +297,14 @@ class ReplicationServer:
ts = None
seq = None
- line = response.readline()
- while line:
+ if hasattr(response, 'iter_lines'):
+ # generated by requests
+ get_line = response.iter_lines
+ else:
+ # generated by urllib.request
+ get_line = response.readline
+
+ for line in get_line():
line = line.decode('utf-8')
if '#' in line:
line = line[0:line.index('#')]
@@ -311,7 +322,6 @@ class ReplicationServer:
except ValueError:
break
ts = ts.replace(tzinfo=dt.timezone.utc)
- line = response.readline()
if ts is not None and seq is not None:
return OsmosisState(sequence=seq, timestamp=ts)
@@ -324,7 +334,13 @@ class ReplicationServer:
(or :code:`urllib2.HTTPError` in python2)
if the file cannot be downloaded.
"""
- return self.open_url(self.make_request(self.get_diff_url(seq))).read()
+ resp = self.open_url(self.make_request(self.get_diff_url(seq)))
+ if hasattr(resp, 'content'):
+ # generated by requests
+ return resp.content
+ else:
+ # generated by urllib.request
+ return resp.read()
def get_state_url(self, seq):
=====================================
src/osmium/version.py
=====================================
@@ -3,13 +3,13 @@ Version information.
"""
# the major version
-pyosmium_major = '3.1'
+pyosmium_major = '3.2'
# current release (Pip version)
-pyosmium_release = '3.1.3'
+pyosmium_release = '3.2.0'
# libosmium version shipped with the Pip release
-libosmium_version = '2.16.0'
+libosmium_version = '2.17.0'
# protozero version shipped with the Pip release
protozero_version = '1.7.0'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.6.2'
+pybind11_version = '2.7.1'
=====================================
test/test_pyosmium_get_changes.py
=====================================
@@ -16,6 +16,13 @@ try:
except:
from io import StringIO
+from requests import Session
+
+try:
+ import http.cookiejar as cookiejarlib
+except ImportError:
+ import cookielib as cookiejarlib
+
try:
from urllib.error import URLError
except ImportError:
@@ -36,6 +43,14 @@ class Capturing(list):
del self._stringio # free up some memory
sys.stdout = self._stdout
+class RequestsResponses(BytesIO):
+
+ def __init__(self, bytes):
+ super(RequestsResponses, self).__init__(bytes)
+ self.content = bytes
+
+ def iter_lines(self):
+ return self.readlines()
class TestPyosmiumGetChanges(unittest.TestCase):
@@ -44,11 +59,15 @@ class TestPyosmiumGetChanges(unittest.TestCase):
"../../tools/pyosmium-get-changes"))
self.url_mock = MagicMock()
self.urls = dict()
- self.url_mock.side_effect = lambda url : self.urls[url.get_full_url()]
- self.script['rserv'].urlrequest.urlopen = self.url_mock
+ self.url_mock.side_effect = lambda url,data=None,timeout=None : self.urls[url.get_full_url()]
+ self.script['rserv'].urlrequest.OpenerDirector.open = self.url_mock
+
+ self.urlreq_mock = MagicMock()
+ self.urlreq_mock.side_effect = lambda url,**kw : self.urls[url]
+ self.script['rserv'].requests.Session.get = self.urlreq_mock
def url(self, url, result):
- self.urls[url] = BytesIO(dedent(result).encode())
+ self.urls[url] = RequestsResponses(dedent(result).encode())
def main(self, *args):
with Capturing() as output:
@@ -86,8 +105,8 @@ class TestPyosmiumGetChanges(unittest.TestCase):
content = fd.read()
try:
assert_equals('454', content)
- fd.close()
finally:
+ fd.close()
unlink(fname)
def test_init_from_seq_file(self):
@@ -98,5 +117,32 @@ class TestPyosmiumGetChanges(unittest.TestCase):
assert_equals(0, self.main('-f', fname))
fd = open(fname, 'r')
content = fd.read()
- assert_equals('454', content)
+ try:
+ assert_equals('454', content)
+ finally:
+ fd.close()
+ unlink(fname)
+
+ def test_init_date_with_cookie(self):
+ self.url('https://planet.osm.org/replication/minute//state.txt',
+ """\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\:04\:02Z
+ """)
+ self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
+ """\
+ sequenceNumber=0
+ timestamp=2016-08-26T11\:04\:02Z
+ """)
+
+ with tempfile.NamedTemporaryFile(dir=tempfile.gettempdir(), suffix='.cookie', delete=False) as fd:
+ fname = fd.name
+ cookie_jar = cookiejarlib.MozillaCookieJar(fname)
+ cookie_jar.save()
+ try:
+ assert_equals(0, self.main('--cookie', fname, '-D', '2015-12-24T08:08:08Z'))
+ assert_equals(1, len(self.stdout))
+ assert_equals('1', self.stdout[0])
+ finally:
+ unlink(fname)
=====================================
test/test_replication.py
=====================================
@@ -22,13 +22,22 @@ import osmium.replication
import tempfile
import datetime
+class RequestsResponses(BytesIO):
+
+ def __init__(self, bytes):
+ super(RequestsResponses, self).__init__(bytes)
+ self.content = bytes
+
+ def iter_lines(self):
+ return self.readlines()
+
class UrllibMock(MagicMock):
def set_result(self, s):
- self.return_value = BytesIO(dedent(s).encode())
+ self.return_value = RequestsResponses(dedent(s).encode())
def set_script(self, files):
- self.side_effect = [BytesIO(dedent(s).encode()) for s in files]
+ self.side_effect = [RequestsResponses(dedent(s).encode()) for s in files]
def test_get_state_url():
svr = rserv.ReplicationServer("https://text.org")
@@ -58,7 +67,7 @@ def test_get_diff_url():
for i, o in data:
assert_equals(o, svr.get_diff_url(i))
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_valid(mock):
mock.set_result("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -77,7 +86,7 @@ def test_get_state_valid(mock):
assert_equal(mock.call_count, 1)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_sequence_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -97,7 +106,7 @@ def test_get_state_sequence_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_date_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -118,7 +127,7 @@ def test_get_state_date_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_timestamp_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -139,7 +148,7 @@ def test_get_state_timestamp_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_too_many_retries(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -170,14 +179,14 @@ def test_get_state_too_many_retries(mock):
- at patch('osmium.replication.server.urlrequest.urlopen')
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_server_timeout(mock):
mock.side_effect = URLError(reason='Mock')
svr = rserv.ReplicationServer("https://test.io")
assert_is_none(svr.get_state_info())
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_count(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -194,7 +203,7 @@ def test_apply_diffs_count(mock):
assert_equals(h.counts, [1, 1, 1, 0])
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_without_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -211,7 +220,7 @@ def test_apply_diffs_without_simplify(mock):
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=False))
assert_equals([2, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_with_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -228,7 +237,7 @@ def test_apply_diffs_with_simplify(mock):
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=True))
assert_equals([1, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_with_location(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -256,7 +265,7 @@ def test_apply_with_location(mock):
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_without_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -277,7 +286,7 @@ def test_apply_reader_without_simplify(mock):
diffs.reader.apply(h, simplify=False)
assert_equals([2, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_with_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -297,7 +306,7 @@ def test_apply_reader_with_simplify(mock):
diffs.reader.apply(h, simplify=True)
assert_equals([1, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_with_location(mock):
mock.set_script(("""\
sequenceNumber=100
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/57f84a493ac7084bccb64374a0f89971fcd76e8e
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/57f84a493ac7084bccb64374a0f89971fcd76e8e
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20210811/75f26faf/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list