[Git][debian-gis-team/pyosmium][master] 9 commits: Update branch in gbp.conf & Vcs-Git URL.
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Sun Aug 15 10:24:20 BST 2021
Bas Couwenberg pushed to branch master at Debian GIS Project / pyosmium
Commits:
33867762 by Bas Couwenberg at 2021-08-11T05:53:10+02:00
Update branch in gbp.conf & Vcs-Git URL.
- - - - -
57f84a49 by Bas Couwenberg at 2021-08-11T05:53:50+02:00
New upstream version 3.2.0
- - - - -
06543933 by Bas Couwenberg at 2021-08-11T05:53:54+02:00
Update upstream source from tag 'upstream/3.2.0'
Update to upstream version '3.2.0'
with Debian dir 10ea170bdd36e0b6b68a072e0b5669d1b6164aff
- - - - -
3833420a by Bas Couwenberg at 2021-08-11T05:54:17+02:00
New upstream release.
- - - - -
e1285070 by Bas Couwenberg at 2021-08-11T05:57:42+02:00
Require at least libosmium2-dev 2.17.0.
- - - - -
66e11082 by Bas Couwenberg at 2021-08-11T05:59:02+02:00
Add python3-requests to build dependencies.
- - - - -
292bc379 by Bas Couwenberg at 2021-08-11T05:59:42+02:00
Set distribution to experimental.
- - - - -
25d5e0c5 by Bas Couwenberg at 2021-08-15T11:15:21+02:00
Revert "Update branch in gbp.conf & Vcs-Git URL."
This reverts commit 3386776298aeb51b77d6f688aa154cf48b609094.
- - - - -
f620d281 by Bas Couwenberg at 2021-08-15T11:15:48+02:00
Move from experimental to unstable.
- - - - -
14 changed files:
- .github/actions/install-dependencies/action.yml
- .github/workflows/ci.yml
- CHANGELOG.md
- README.md
- README.rst
- debian/changelog
- debian/control
- doc/updating_osm_data.rst
- lib/merge_input_reader.cc
- setup.py
- src/osmium/replication/server.py
- src/osmium/version.py
- test/test_pyosmium_get_changes.py
- test/test_replication.py
Changes:
=====================================
.github/actions/install-dependencies/action.yml
=====================================
@@ -7,7 +7,7 @@ runs:
- name: Install pip dependencies
run: |
python -m pip install --upgrade pip
- pip install nose shapely setuptools
+ pip install nose shapely setuptools requests
shell: bash
- name: Install package dependencies
=====================================
.github/workflows/ci.yml
=====================================
@@ -33,8 +33,8 @@ jobs:
working-directory: test
- build-ubuntu-1604:
- runs-on: ubuntu-16.04
+ build-ubuntu-1804:
+ runs-on: ubuntu-18.04
strategy:
matrix:
@@ -50,14 +50,18 @@ jobs:
- name: Install packages
run: sudo apt-get install -y -qq libboost-dev libexpat1-dev zlib1g-dev libbz2-dev libproj-dev libgeos-dev liblz4-dev
+ - name: Install clang package
+ run: sudo apt-get install -y -qq clang-6.0
+ if: ${{ matrix.compiler == 'clang' }}
+
- uses: ./.github/actions/install-dependencies
- name: Build package
run: python setup.py build
shell: bash
env:
- CC: gcc-5
- CXX: g++-5
+ CC: gcc-7
+ CXX: g++-7
if: ${{ matrix.compiler == 'gcc' }}
- name: Build package
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,18 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.2.0] - 2021-08-09
+
+### Fixed
+
+- merge change files correctly so that latest applied object comes first
+
+### Changed
+
+- switch to requests library for downloading files (thanks @jocelynj)
+- update to libosmium 2.17.0
+- update to pybind11 2.7.1
+
## [3.1.3] - 2021-02-05
### Fixed
=====================================
README.md
=====================================
@@ -21,7 +21,7 @@ We provide binary wheels for Linux and Windows 64 for all actively
maintained Python versions.
For other versions, a source wheel is provided. Make sure to install all
-external dependencies first. On Deian/Ubuntu-like systems, the following
+external dependencies first. On Debian/Ubuntu-like systems, the following
command installs all required packages:
sudo apt-get install build-essential cmake libboost-dev \
@@ -42,6 +42,7 @@ pyosmium has the following dependencies:
* [libz](https://www.zlib.net/)
* [libbz2](https://www.sourceware.org/bzip2/)
* [Boost](https://www.boost.org/) variant and iterator >= 1.41
+ * [Python Requests](https://docs.python-requests.org/en/master/)
* Python setuptools
* a recent C++ compiler (Clang 3.4+, GCC 4.8+)
=====================================
README.rst
=====================================
@@ -24,7 +24,7 @@ development packages for these libraries. On Debian/Ubuntu do::
libexpat1-dev zlib1g-dev libbz2-dev
-Python >= 2.7 is supported but a version >= 3.3 is strongly recommended.
+Python >= 3.4 is supported.
Documentation
=============
=====================================
debian/changelog
=====================================
@@ -1,8 +1,17 @@
-pyosmium (3.1.3-2) UNRELEASED; urgency=medium
+pyosmium (3.2.0-1) unstable; urgency=medium
+ * Move from experimental to unstable.
+
+ -- Bas Couwenberg <sebastic at debian.org> Sun, 15 Aug 2021 11:15:39 +0200
+
+pyosmium (3.2.0-1~exp1) experimental; urgency=medium
+
+ * New upstream release.
* Update watch file for GitHub URL changes.
+ * Require at least libosmium2-dev 2.17.0.
+ * Add python3-requests to build dependencies.
- -- Bas Couwenberg <sebastic at debian.org> Sat, 20 Mar 2021 09:04:01 +0100
+ -- Bas Couwenberg <sebastic at debian.org> Wed, 11 Aug 2021 05:59:28 +0200
pyosmium (3.1.3-1) unstable; urgency=medium
=====================================
debian/control
=====================================
@@ -9,9 +9,10 @@ Build-Depends: cmake (>= 2.8.12),
libboost-dev,
libbz2-dev,
libexpat1-dev,
- libosmium2-dev (>= 2.16.0),
+ libosmium2-dev (>= 2.17.0),
pybind11-dev,
python3-all-dev,
+ python3-requests,
python3-setuptools,
python3-mock <!nocheck>,
python3-nose <!nocheck>,
=====================================
doc/updating_osm_data.rst
=====================================
@@ -154,12 +154,14 @@ Creating a change file
Now you can create change files using the state::
- pyosmium-get-changes --server <replication server> -f sequence.state -o newchange.osm.gz
+ pyosmium-get-changes --server <replication server> -f sequence.state -o newchange.osc.gz
This downloads the latest changes from the server, saves them in the file
-`newchange.osm.gz` and updates your state file. `<replication server>` is the
+`newchange.osc.gz` and updates your state file. `<replication server>` is the
URL that was printed, when you set up the state file. The parameter can be
omitted when you use minutely change files from openstreetmap.org.
+This simplifies multiple edits of the same element into the final change. If you want to
+retain the full version history specify `--no-deduplicate`.
`pyosmium-get-changes` loads only about 100MB worth of updates at once (about
8 hours of planet updates). If you want more, then add a `--size` parameter.
@@ -179,13 +181,15 @@ All other error codes indicate fatal errors.
A simple shell script can look like this::
while true; do
+ # pyosmium-get-changes would not overwrite an existing changes file
+ rm -f newchange.osc.gz
# get the next batch of changes
- pyosmium-get-changes -f sequence.state -o newchange.osm.gz
+ pyosmium-get-changes -f sequence.state -o newchange.osc.gz
# save the return code
status=$?
if [ $status -eq 0 ]; then
- # apply newchange.osm.gz here
+ # apply newchange.osc.gz here
....
elif [ $status -eq 3 ]; then
# No new data, so sleep for a bit
=====================================
lib/merge_input_reader.cc
=====================================
@@ -80,6 +80,13 @@ public:
// For normal data files we sort with the largest version of each
// object first and then only copy this last version of any object
// to the output.
+ // Caution: When change files have been
+ // created from extracts it is possible that they contain objects
+ // with the same type, id, version, and timestamp. In that case we
+ // still want to get the last object available. So we have to make
+ // sure it appears first in the objects vector before doing the
+ // stable sort.
+ std::reverse(objects.ptr_begin(), objects.ptr_end());
objects.sort(osmium::object_order_type_id_reverse_version());
auto output_it = boost::make_function_output_iterator(
=====================================
setup.py
=====================================
@@ -154,6 +154,7 @@ setup(
package_dir = {'' : 'src'},
package_data = { 'osmium' : [ '*.dll' ] },
python_requires = ">=3.4",
+ install_requires = ['requests'],
cmdclass=dict(build_ext=CMakeBuild, sdist=Pyosmium_sdist),
zip_safe=False,
)
=====================================
src/osmium/replication/server.py
=====================================
@@ -1,6 +1,7 @@
""" Helper functions to communicate with replication servers.
"""
+import requests
import urllib.request as urlrequest
import datetime as dt
from collections import namedtuple
@@ -26,6 +27,7 @@ class ReplicationServer:
def __init__(self, url, diff_type='osc.gz'):
self.baseurl = url
self.diff_type = diff_type
+ self.session = requests.Session()
def make_request(self, url):
headers = {"User-Agent" : "pyosmium/{}".format(version.pyosmium_release)}
@@ -48,7 +50,10 @@ class ReplicationServer:
svr = ReplicationServer()
svr.open_url = my_open_url
"""
- return urlrequest.urlopen(url)
+ headers = dict()
+ for h in url.header_items():
+ headers[h[0]] = h[1]
+ return self.session.get(url.get_full_url(), headers=headers, stream=True)
def collect_diffs(self, start_id, max_size=1024):
""" Create a MergeInputReader and download diffs starting with sequence
@@ -292,8 +297,14 @@ class ReplicationServer:
ts = None
seq = None
- line = response.readline()
- while line:
+ if hasattr(response, 'iter_lines'):
+ # generated by requests
+ get_line = response.iter_lines
+ else:
+ # generated by urllib.request
+ get_line = response.readline
+
+ for line in get_line():
line = line.decode('utf-8')
if '#' in line:
line = line[0:line.index('#')]
@@ -311,7 +322,6 @@ class ReplicationServer:
except ValueError:
break
ts = ts.replace(tzinfo=dt.timezone.utc)
- line = response.readline()
if ts is not None and seq is not None:
return OsmosisState(sequence=seq, timestamp=ts)
@@ -324,7 +334,13 @@ class ReplicationServer:
(or :code:`urllib2.HTTPError` in python2)
if the file cannot be downloaded.
"""
- return self.open_url(self.make_request(self.get_diff_url(seq))).read()
+ resp = self.open_url(self.make_request(self.get_diff_url(seq)))
+ if hasattr(resp, 'content'):
+ # generated by requests
+ return resp.content
+ else:
+ # generated by urllib.request
+ return resp.read()
def get_state_url(self, seq):
=====================================
src/osmium/version.py
=====================================
@@ -3,13 +3,13 @@ Version information.
"""
# the major version
-pyosmium_major = '3.1'
+pyosmium_major = '3.2'
# current release (Pip version)
-pyosmium_release = '3.1.3'
+pyosmium_release = '3.2.0'
# libosmium version shipped with the Pip release
-libosmium_version = '2.16.0'
+libosmium_version = '2.17.0'
# protozero version shipped with the Pip release
protozero_version = '1.7.0'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.6.2'
+pybind11_version = '2.7.1'
=====================================
test/test_pyosmium_get_changes.py
=====================================
@@ -16,6 +16,13 @@ try:
except:
from io import StringIO
+from requests import Session
+
+try:
+ import http.cookiejar as cookiejarlib
+except ImportError:
+ import cookielib as cookiejarlib
+
try:
from urllib.error import URLError
except ImportError:
@@ -36,6 +43,14 @@ class Capturing(list):
del self._stringio # free up some memory
sys.stdout = self._stdout
+class RequestsResponses(BytesIO):
+
+ def __init__(self, bytes):
+ super(RequestsResponses, self).__init__(bytes)
+ self.content = bytes
+
+ def iter_lines(self):
+ return self.readlines()
class TestPyosmiumGetChanges(unittest.TestCase):
@@ -44,11 +59,15 @@ class TestPyosmiumGetChanges(unittest.TestCase):
"../../tools/pyosmium-get-changes"))
self.url_mock = MagicMock()
self.urls = dict()
- self.url_mock.side_effect = lambda url : self.urls[url.get_full_url()]
- self.script['rserv'].urlrequest.urlopen = self.url_mock
+ self.url_mock.side_effect = lambda url,data=None,timeout=None : self.urls[url.get_full_url()]
+ self.script['rserv'].urlrequest.OpenerDirector.open = self.url_mock
+
+ self.urlreq_mock = MagicMock()
+ self.urlreq_mock.side_effect = lambda url,**kw : self.urls[url]
+ self.script['rserv'].requests.Session.get = self.urlreq_mock
def url(self, url, result):
- self.urls[url] = BytesIO(dedent(result).encode())
+ self.urls[url] = RequestsResponses(dedent(result).encode())
def main(self, *args):
with Capturing() as output:
@@ -86,8 +105,8 @@ class TestPyosmiumGetChanges(unittest.TestCase):
content = fd.read()
try:
assert_equals('454', content)
- fd.close()
finally:
+ fd.close()
unlink(fname)
def test_init_from_seq_file(self):
@@ -98,5 +117,32 @@ class TestPyosmiumGetChanges(unittest.TestCase):
assert_equals(0, self.main('-f', fname))
fd = open(fname, 'r')
content = fd.read()
- assert_equals('454', content)
+ try:
+ assert_equals('454', content)
+ finally:
+ fd.close()
+ unlink(fname)
+
+ def test_init_date_with_cookie(self):
+ self.url('https://planet.osm.org/replication/minute//state.txt',
+ """\
+ sequenceNumber=100
+ timestamp=2017-08-26T11\:04\:02Z
+ """)
+ self.url('https://planet.osm.org/replication/minute//000/000/000.state.txt',
+ """\
+ sequenceNumber=0
+ timestamp=2016-08-26T11\:04\:02Z
+ """)
+
+ with tempfile.NamedTemporaryFile(dir=tempfile.gettempdir(), suffix='.cookie', delete=False) as fd:
+ fname = fd.name
+ cookie_jar = cookiejarlib.MozillaCookieJar(fname)
+ cookie_jar.save()
+ try:
+ assert_equals(0, self.main('--cookie', fname, '-D', '2015-12-24T08:08:08Z'))
+ assert_equals(1, len(self.stdout))
+ assert_equals('1', self.stdout[0])
+ finally:
+ unlink(fname)
=====================================
test/test_replication.py
=====================================
@@ -22,13 +22,22 @@ import osmium.replication
import tempfile
import datetime
+class RequestsResponses(BytesIO):
+
+ def __init__(self, bytes):
+ super(RequestsResponses, self).__init__(bytes)
+ self.content = bytes
+
+ def iter_lines(self):
+ return self.readlines()
+
class UrllibMock(MagicMock):
def set_result(self, s):
- self.return_value = BytesIO(dedent(s).encode())
+ self.return_value = RequestsResponses(dedent(s).encode())
def set_script(self, files):
- self.side_effect = [BytesIO(dedent(s).encode()) for s in files]
+ self.side_effect = [RequestsResponses(dedent(s).encode()) for s in files]
def test_get_state_url():
svr = rserv.ReplicationServer("https://text.org")
@@ -58,7 +67,7 @@ def test_get_diff_url():
for i, o in data:
assert_equals(o, svr.get_diff_url(i))
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_valid(mock):
mock.set_result("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -77,7 +86,7 @@ def test_get_state_valid(mock):
assert_equal(mock.call_count, 1)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_sequence_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -97,7 +106,7 @@ def test_get_state_sequence_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_date_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -118,7 +127,7 @@ def test_get_state_date_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_timestamp_cut(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -139,7 +148,7 @@ def test_get_state_timestamp_cut(mock):
assert_equal(mock.call_count, 2)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_too_many_retries(mock):
mock.set_script(("""\
#Sat Aug 26 11:04:04 UTC 2017
@@ -170,14 +179,14 @@ def test_get_state_too_many_retries(mock):
- at patch('osmium.replication.server.urlrequest.urlopen')
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_get_state_server_timeout(mock):
mock.side_effect = URLError(reason='Mock')
svr = rserv.ReplicationServer("https://test.io")
assert_is_none(svr.get_state_info())
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_count(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -194,7 +203,7 @@ def test_apply_diffs_count(mock):
assert_equals(h.counts, [1, 1, 1, 0])
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_without_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -211,7 +220,7 @@ def test_apply_diffs_without_simplify(mock):
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=False))
assert_equals([2, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_diffs_with_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -228,7 +237,7 @@ def test_apply_diffs_with_simplify(mock):
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=True))
assert_equals([1, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_with_location(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -256,7 +265,7 @@ def test_apply_with_location(mock):
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_without_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -277,7 +286,7 @@ def test_apply_reader_without_simplify(mock):
diffs.reader.apply(h, simplify=False)
assert_equals([2, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_with_simplify(mock):
mock.set_script(("""\
sequenceNumber=100
@@ -297,7 +306,7 @@ def test_apply_reader_with_simplify(mock):
diffs.reader.apply(h, simplify=True)
assert_equals([1, 1, 1, 0], h.counts)
- at patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
+ at patch('osmium.replication.server.requests.Session.get', new_callable=UrllibMock)
def test_apply_reader_with_location(mock):
mock.set_script(("""\
sequenceNumber=100
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/34ddb199958a887e9f4f35daaa772ba4f9655c8e...f620d281b22e9799c76b58186bcd677b68203095
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/34ddb199958a887e9f4f35daaa772ba4f9655c8e...f620d281b22e9799c76b58186bcd677b68203095
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20210815/5b235dc1/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list