[Git][debian-gis-team/pyosmium][master] 5 commits: New upstream version 3.1.0
Bas Couwenberg
gitlab at salsa.debian.org
Thu Nov 12 03:29:15 GMT 2020
Bas Couwenberg pushed to branch master at Debian GIS Project / pyosmium
Commits:
c105af43 by Bas Couwenberg at 2020-11-12T04:07:30+01:00
New upstream version 3.1.0
- - - - -
78fdb689 by Bas Couwenberg at 2020-11-12T04:07:34+01:00
Update upstream source from tag 'upstream/3.1.0'
Update to upstream version '3.1.0'
with Debian dir 852a8e104050ab76aeae604bcbe202d22a27c693
- - - - -
cb887db3 by Bas Couwenberg at 2020-11-12T04:09:45+01:00
New upstream release.
- - - - -
bb2270fe by Bas Couwenberg at 2020-11-12T04:11:50+01:00
Drop pybind11-2.6.0.patch, fixed upstream.
- - - - -
91619a0b by Bas Couwenberg at 2020-11-12T04:11:59+01:00
Set distribution to unstable.
- - - - -
25 changed files:
- + .github/actions/install-dependencies/action.yml
- + .github/workflows/ci.yml
- − .travis.yml
- CHANGELOG.md
- CMakeLists.txt
- README.md
- debian/changelog
- − debian/patches/pybind11-2.6.0.patch
- − debian/patches/series
- doc/updating_osm_data.rst
- examples/amenity_list.py
- examples/pub_names.py
- examples/use_nodecache.py
- lib/io.cc
- lib/merge_input_reader.cc
- setup.py
- src/osmium/helper.py
- src/osmium/osm/__init__.py
- src/osmium/osm/mutable.py
- src/osmium/replication/server.py
- src/osmium/replication/utils.py
- src/osmium/version.py
- test/test_replication.py
- tools/pyosmium-get-changes
- tools/pyosmium-up-to-date
Changes:
=====================================
.github/actions/install-dependencies/action.yml
=====================================
@@ -0,0 +1,18 @@
+name: 'Install pyosmium dependencies'
+
+runs:
+ using: "composite"
+
+ steps:
+ - name: Install pip dependencies
+ run: |
+ python -m pip install --upgrade pip
+ pip install nose shapely setuptools
+ shell: bash
+
+ - name: Install package dependencies
+ run: |
+ git clone --quiet --depth 1 https://github.com/osmcode/libosmium.git contrib/libosmium
+ git clone --quiet --depth 1 https://github.com/mapbox/protozero.git contrib/protozero
+ git clone --quiet --depth 1 https://github.com/pybind/pybind11.git contrib/pybind11
+ shell: bash
=====================================
.github/workflows/ci.yml
=====================================
@@ -0,0 +1,141 @@
+name: CI
+
+on: [ push, pull_request ]
+
+jobs:
+ build-default:
+ runs-on: ubuntu-20.04
+
+ strategy:
+ matrix:
+ python-version: [3.6, 3.7, 3.8]
+
+ steps:
+ - uses: actions/checkout at v2
+
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python at v2
+ with:
+ python-version: ${{ matrix.python-version }}
+
+ - name: Install packages
+ run: sudo apt-get install -y -qq libboost-dev libexpat1-dev zlib1g-dev libbz2-dev libproj-dev libgeos-dev
+
+ - uses: ./.github/actions/install-dependencies
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+
+ - name: Run tests
+ run: python run_tests.py
+ shell: bash
+ working-directory: test
+
+
+ build-ubuntu-1604:
+ runs-on: ubuntu-16.04
+
+ strategy:
+ matrix:
+ compiler: [gcc, clang]
+
+ steps:
+ - uses: actions/checkout at v2
+
+ - uses: actions/setup-python at v2
+ with:
+ python-version: 3.5
+
+ - name: Install packages
+ run: sudo apt-get install -y -qq libboost-dev libexpat1-dev zlib1g-dev libbz2-dev libproj-dev libgeos-dev
+
+ - uses: ./.github/actions/install-dependencies
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+ env:
+ CC: gcc-5
+ CXX: g++-5
+ if: ${{ matrix.compiler == 'gcc' }}
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+ env:
+ CC: clang-6.0
+ CXX: clang++-6.0
+ if: ${{ matrix.compiler == 'clang' }}
+
+ - name: Run tests
+ run: python run_tests.py
+ shell: bash
+ working-directory: test
+
+ build-ubuntu-2004:
+ runs-on: ubuntu-20.04
+
+ strategy:
+ matrix:
+ compiler: [gcc, clang]
+
+ steps:
+ - uses: actions/checkout at v2
+
+ - uses: actions/setup-python at v2
+ with:
+ python-version: 3.9
+
+ - name: Install packages
+ run: sudo apt-get install -y -qq libboost-dev libexpat1-dev zlib1g-dev libbz2-dev libproj-dev libgeos-dev
+
+ - uses: ./.github/actions/install-dependencies
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+ env:
+ CC: gcc-10
+ CXX: g++-10
+ if: ${{ matrix.compiler == 'gcc' }}
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+ env:
+ CC: clang-10
+ CXX: clang++-10
+ if: ${{ matrix.compiler == 'clang' }}
+
+ - name: Run tests
+ run: python run_tests.py
+ shell: bash
+ working-directory: test
+
+ build-macos:
+ runs-on: macos-latest
+
+ steps:
+ - uses: actions/checkout at v2
+
+ - uses: actions/setup-python at v2
+ with:
+ python-version: 3
+
+ - name: Install packages
+ run: brew install boost geos
+ shell: bash
+
+ - uses: ./.github/actions/install-dependencies
+
+ - name: Build package
+ run: python setup.py build
+ shell: bash
+
+ - name: Run tests
+ run: python run_tests.py
+ shell: bash
+ working-directory: test
+
+
=====================================
.travis.yml deleted
=====================================
@@ -1,84 +0,0 @@
-#-----------------------------------------------------------------------------
-#
-# Configuration for continuous integration service at travis-ci.org
-#
-#-----------------------------------------------------------------------------
-
-
-matrix:
- include:
- - os: linux
- compiler: "clang-3.4"
- dist: xenial
- language: cpp
- - os: linux
- compiler: "clang-7"
- dist: bionic
- language: cpp
- - os: linux
- compiler: "gcc-5"
- dist: xenial
- language: cpp
- - os: linux
- compiler: "gcc-8"
- dist: bionic
- language: cpp
- - os: linux
- compiler: gcc
- python: 3.4
- dist: xenial
- language: python
- - os: linux
- compiler: gcc
- python: 3.5
- dist: bionic
- language: python
- - os: linux
- compiler: gcc
- python: 3.6
- dist: bionic
- language: python
- - os: linux
- compiler: gcc
- python: 3.7
- dist: bionic
- language: python
- - os: linux
- compiler: gcc
- python: 3.8
- dist: bionic
- language: python
- - os: osx
- osx_image: xcode7
- compiler: clang
- language: cpp
- - os: osx
- osx_image: xcode10.1
- compiler: clang
- language: cpp
-
-# http://docs.travis-ci.com/user/apt/
-addons:
- apt:
- packages:
- - libboost-dev
- - python3
- - python3-dev
- - python3-pip
-
-install:
- - git clone --quiet --depth 1 https://github.com/osmcode/libosmium.git contrib/libosmium
- - git clone --quiet --depth 1 https://github.com/mapbox/protozero.git contrib/protozero
- - git clone --quiet --depth 1 https://github.com/pybind/pybind11.git contrib/pybind11
- - if [ "$TRAVIS_OS_NAME" = 'osx' ]; then
- pip3 install --user -q nose mock shapely setuptools;
- else
- pip3 install -q nose mock shapely setuptools;
- fi
-
-script:
- - python3 --version
- - python3 setup.py build
- - cd test
- - python3 run_tests.py
-
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,20 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.1.0] - 2020-11-03
+
+### Added
+
+### Changed
+
+- improved help messages for update tools
+- update to pybind11 2.6
+
+### Fixed
+
+- pyosmium-up-to-date: fix missing flag when applying changes to history files
+- smaller linting and documentation issues
+
## [3.0.1] - 2020-07-25
### Added
=====================================
CMakeLists.txt
=====================================
@@ -1,5 +1,6 @@
cmake_minimum_required(VERSION 2.8.12)
project(pyosmium)
+set(PACKAGE_VERSION 3.0.1)
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
@@ -7,12 +8,23 @@ find_package(Osmium 2.15 REQUIRED COMPONENTS io pbf xml)
include_directories(SYSTEM ${OSMIUM_INCLUDE_DIRS} ${PROTOZERO_INCLUDE_DIR})
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/lib)
+if(NOT "${CMAKE_CXX_STANDARD}")
+ if(MSVC)
+ set(CMAKE_CXX_STANDARD 14)
+ else()
+ set(CMAKE_CXX_STANDARD 11)
+ endif()
+endif()
+
+# required for pybind11 < 2.6
if(MSVC)
- set(PYBIND11_CPP_STANDARD /std:c++14)
+ set(PYBIND11_CPP_STANDARD /std=c++${CMAKE_CXX_STANDARD})
else()
- set(PYBIND11_CPP_STANDARD -std=c++11)
+ set(PYBIND11_CPP_STANDARD -std=c++${CMAKE_CXX_STANDARD})
endif()
+message(STATUS "Building in C++${CMAKE_CXX_STANDARD} mode")
+
if(PYBIND11_PREFIX)
add_subdirectory(${PYBIND11_PREFIX} contrib/pybind11)
else()
=====================================
README.md
=====================================
@@ -4,7 +4,7 @@ Provides Python bindings for the [Libosmium](https://github.com/osmcode/libosmiu
library, a library for working with OpenStreetMap data in a fast and flexible
manner.
-[](http://travis-ci.org/osmcode/pyosmium)
+[](https://github.com/osmcode/pyosmium/actions?query=workflow%3ACI)
[](https://ci.appveyor.com/project/lonvia/pyosmium/branch/master)
## Installation
=====================================
debian/changelog
=====================================
@@ -1,8 +1,10 @@
-pyosmium (3.0.1-3) UNRELEASED; urgency=medium
+pyosmium (3.1.0-1) unstable; urgency=medium
+ * New upstream release.
* Bump watch file version to 4.
+ * Drop pybind11-2.6.0.patch, fixed upstream.
- -- Bas Couwenberg <sebastic at debian.org> Fri, 06 Nov 2020 19:49:38 +0100
+ -- Bas Couwenberg <sebastic at debian.org> Thu, 12 Nov 2020 04:11:51 +0100
pyosmium (3.0.1-2) unstable; urgency=medium
=====================================
debian/patches/pybind11-2.6.0.patch deleted
=====================================
@@ -1,19 +0,0 @@
-Description: Use CMAKE_CXX_STANDARD instead of PYBIND11_CPP_STANDARD.
- See: https://github.com/pybind/pybind11/blob/v2.6.0/docs/upgrade.rst#cmake-support
-Author: Bas Couwenberg <sebastic at debian.org>
-Bug: https://github.com/osmcode/pyosmium/issues/147
-
---- a/CMakeLists.txt
-+++ b/CMakeLists.txt
-@@ -8,9 +8,9 @@ include_directories(SYSTEM ${OSMIUM_INCL
- include_directories(${CMAKE_CURRENT_SOURCE_DIR}/lib)
-
- if(MSVC)
-- set(PYBIND11_CPP_STANDARD /std:c++14)
-+ set(CMAKE_CXX_STANDARD 14)
- else()
-- set(PYBIND11_CPP_STANDARD -std=c++11)
-+ set(CMAKE_CXX_STANDARD 11)
- endif()
-
- if(PYBIND11_PREFIX)
=====================================
debian/patches/series deleted
=====================================
@@ -1 +0,0 @@
-pybind11-2.6.0.patch
=====================================
doc/updating_osm_data.rst
=====================================
@@ -33,14 +33,15 @@ There are multiple sources for OSM change files available:
for planet-wide updates. There are change files for
minutely, hourly and daily intervals available.
- * `Geofabrik <http://download.geofabrik.de>`_ offers daily change files
+ * `Geofabrik <https://download.geofabrik.de>`_ offers daily change files
for all its updates. See the extract page for a link to the replication URL.
Note that change files go only about 3 months back. Older files are deleted.
- * `openstreetmap.fr <http://download.geofabrik.de>`_ offers minutely change
- files for all its extracts.
+ * download.openstreetmap.fr offers
+ `minutely change files <https://download.openstreetmap.fr/replication/>`_
+ for all its `extracts <https://download.openstreetmap.fr/extracts/>`_..
-For other services also check out the list of services on the
+For other services also check out the list of providers on the
`OSM wiki <https://wiki.openstreetmap.org/wiki/Planet.osm>`_.
Updating a planet or extract
=====================================
examples/amenity_list.py
=====================================
@@ -13,7 +13,7 @@ wkbfab = o.geom.WKBFactory()
class AmenityListHandler(o.SimpleHandler):
- def print_amenity(amenity, tags, lon, lat):
+ def print_amenity(self, tags, lon, lat):
name = tags.get('name', '')
print("%f %f %-15s %s" % (lon, lat, tags['amenity'], name))
=====================================
examples/pub_names.py
=====================================
@@ -7,9 +7,8 @@ import sys
class NamesHandler(osmium.SimpleHandler):
def output_pubs(self, tags):
- if tags.get('amenity') == 'pub':
- if 'name' in tags:
- print(tags['name'])
+ if tags.get('amenity') == 'pub' and 'name' in tags:
+ print(tags['name'])
def node(self, n):
self.output_pubs(n.tags)
=====================================
examples/use_nodecache.py
=====================================
@@ -9,9 +9,8 @@ class WayHandler(o.SimpleHandler):
def way(self, w):
for n in w.nodes:
- n.lat, n.lon # throws an exception if the coordinates are missing
loc = idx.get(n.ref)
- print("%d %s" %(w.id, len(w.nodes)))
+ print("%d %s" % (w.id, len(w.nodes)))
if len(sys.argv) != 3:
print("Usage: python create_nodecache.py <osm file> <node cache>")
=====================================
lib/io.cc
=====================================
@@ -59,7 +59,7 @@ PYBIND11_MODULE(io, m)
py::class_<osmium::io::Writer>(m, "Writer",
"Class for writing OSM data to a file. This class just encapsulates an "
- "OSM file,. Have a look `osmium.SimpleWriter` for a high-level interface "
+ "OSM file. Have a look `osmium.SimpleWriter` for a high-level interface "
"for writing out data.")
.def(py::init<std::string>())
.def(py::init<osmium::io::File>())
=====================================
lib/merge_input_reader.cc
=====================================
@@ -53,8 +53,7 @@ namespace {
class MergeInputReader
{
public:
- void apply(BaseHandler& handler, std::string const &idx = "",
- bool simplify = true)
+ void apply(BaseHandler& handler, std::string const &idx, bool simplify)
{
if (idx.empty())
apply_without_location(handler, simplify);
@@ -63,7 +62,7 @@ public:
}
void apply_to_reader(osmium::io::Reader &reader, osmium::io::Writer &writer,
- bool with_history = true)
+ bool with_history)
{
auto input = osmium::io::make_input_iterator_range<osmium::OSMObject>(reader);
if (with_history) {
@@ -110,7 +109,7 @@ public:
}
private:
- void apply_without_location(BaseHandler& handler, bool simplify = true)
+ void apply_without_location(BaseHandler& handler, bool simplify)
{
if (simplify) {
objects.sort(osmium::object_order_type_id_reverse_version());
@@ -133,7 +132,7 @@ private:
}
void apply_with_location(BaseHandler& handler, std::string const &idx,
- bool simplify = true)
+ bool simplify)
{
using Index_fab =
osmium::index::MapFactory<osmium::unsigned_object_id_type, osmium::Location>;
=====================================
setup.py
=====================================
@@ -105,6 +105,9 @@ class CMakeBuild(build_ext):
if 'BOOST_PREFIX' in env:
cmake_args += ['-DBOOST_ROOT={}'.format(env['BOOST_PREFIX'])]
+ if 'CMAKE_CXX_STANDARD' in env:
+ cmake_args += ['-DCMAKE_CXX_STANDARD={}'.format(env['CMAKE_CXX_STANDARD'])]
+
if not os.path.exists(self.build_temp):
os.makedirs(self.build_temp)
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
@@ -112,8 +115,8 @@ class CMakeBuild(build_ext):
versions = get_versions()
-if sys.version_info < (3,0):
- raise RuntimeError("Python 3.3 or larger required.")
+if sys.version_info < (3,4):
+ raise RuntimeError("Python 3.4 or larger required.")
with open('README.rst', 'r') as descfile:
long_description = descfile.read()
@@ -128,7 +131,7 @@ setup(
maintainer='Sarah Hoffmann',
maintainer_email='lonvia at denofr.de',
download_url='https://github.com/osmcode/pyosmium',
- url='http://osmcode.org/pyosmium',
+ url='https://osmcode.org/pyosmium',
keywords=["OSM", "OpenStreetMap", "Osmium"],
license='BSD',
scripts=['tools/pyosmium-get-changes', 'tools/pyosmium-up-to-date'],
=====================================
src/osmium/helper.py
=====================================
@@ -19,4 +19,3 @@ def make_simple_handler(node=None, way=None, relation=None, area=None):
__HandlerWithCallbacks.area = staticmethod(area)
return __HandlerWithCallbacks()
-
=====================================
src/osmium/osm/__init__.py
=====================================
@@ -1,5 +1,5 @@
-from ._osm import *
import osmium.osm.mutable
+from ._osm import *
def create_mutable_node(node, **args):
""" Create a mutable node replacing the properties given in the
@@ -26,59 +26,72 @@ Node.replace = create_mutable_node
Way.replace = create_mutable_way
Relation.replace = create_mutable_relation
-Location.__repr__ = lambda l : 'osmium.osm.Location(x=%r, y=%r)' \
- % (l.x, l.y) \
- if l.valid() else 'osmium.osm.Location()'
-Location.__str__ = lambda l : '%f/%f' % (l.lon_without_check(), l.lat_without_check()) \
- if l.valid() else 'invalid'
+def _make_repr(attr_list):
+ fmt_string = 'osmium.osm.{0}('\
+ + ', '.join(['{0}={{1.{0}!r}}'.format(x) for x in attr_list])\
+ + ')'
-Box.__repr__ = lambda b : 'osmium.osm.Box(bottom_left=%r, top_right=%r)' \
- % (b.bottom_left, b.top_right)
-Box.__str__ = lambda b : '(%s %s)' % (b.bottom_left, b.top_right)
+ return lambda o: fmt_string.format(o.__class__.__name__, o)
-Tag.__repr__ = lambda t : 'osmium.osm.Tag(k=%r, v=%r)' % (t.k, t.v)
+def _list_repr(obj):
+ return 'osmium.osm.{}([{}])'.format(obj.__class__.__name__,
+ ', '.join(map(repr, obj)))
+
+def _list_elipse(obj):
+ objects = ','.join(map(str, obj))
+ if len(objects) > 50:
+ objects = objects[:47] + '...'
+ return objects
+
+Location.__repr__ = lambda l: 'osmium.osm.Location(x={0.x!r}, y={0.y!r})'.format(l) \
+ if l.valid() else 'osmium.osm.Location()'
+Location.__str__ = lambda l: '{:f}/{:f}'.format(l.lon_without_check(),
+ l.lat_without_check()) \
+ if l.valid() else 'invalid'
-Tag.__str__ = lambda t : '%s=%s' % (t.k, t.v)
+Box.__repr__ = _make_repr(['bottom_left', 'top_right'])
+Box.__str__ = lambda b: '({0.bottom_left!s} {0.top_right!s})'.format(b)
-TagList.__repr__ = lambda t : "osmium.osm.TagList({%s})" \
- % ",".join(["%r=%r" % (i.k, i.v) for i in t])
-TagList.__str__ = lambda t : "{%s}" % ",".join([str(i) for i in t])
+Tag.__repr__ = _make_repr(['k', 'v'])
+Tag.__str__ = lambda t: '{0.k}={0.v}'.format(t)
-NodeRef.__repr__ = lambda n : 'osmium.osm.NodeRef(ref=%r, location=%r)' % (n.ref, n.location)
-NodeRef.__str__ = lambda n : '%s@%s' % (n.ref, n.location) if n.location.valid() \
- else str(n.ref)
+TagList.__repr__ = lambda t: "osmium.osm.TagList({%s})" \
+ % ', '.join(["%r: %r" % (i.k, i.v) for i in t])
+TagList.__str__ = lambda t: '{' + _list_elipse(t) + '}'
-NodeRefList.__repr__ = lambda t : "%s([%s])" % (t.__class__.__name__,
- ",".join([repr(i) for i in t]))
-NodeRefList.__str__ = lambda t : "[%s]" % ",".join([str(i) for i in t])
+NodeRef.__repr__ = _make_repr(['ref', 'location'])
+NodeRef.__str__ = lambda n: '{0.ref:d}@{0.location!s}'.format(n) \
+ if n.location.valid() else str(n.ref)
-RelationMember.__repr__ = lambda r : 'osmium.osm.RelationMember(ref=%r, type=%r, role=%r)' \
- % (r.ref, r.type, r.role)
-RelationMember.__str__ = lambda r : '%s%d@%s' % (r.type, r.ref, r.role) \
- if r.role else '%s%d' % (r.type, r.ref)
+NodeRefList.__repr__ = _list_repr
+NodeRefList.__str__ = lambda o: '[' + _list_elipse(o) + ']'
-RelationMemberList.__repr__ = lambda t : "osmium.osm.RelationMemberList([%s])" \
- % ",".join([repr(i) for i in t])
-RelationMemberList.__str__ = lambda t : "[%s]" % ",".join([str(i) for i in t])
+RelationMember.__repr__ = _make_repr(['ref', 'type', 'role'])
+RelationMember.__str__ = lambda r: ('{0.type}{0.ref:d}@{0.role}' \
+ if r.role else '{0.type}{0.ref:d}').format(r)
-OSMObject.__repr__ = lambda o : '%s(id=%r, deleted=%r, visible=%r, version=%r, changeset=%r, uid=%r, timestamp=%r, user=%r, tags=%r)'% (o.__class__.__name__, o.id, o.deleted, o.visible, o.version, o.changeset, o.uid, o.timestamp, o.user, o.tags)
+RelationMemberList.__repr__ = _list_repr
+RelationMemberList.__str__ = lambda o: '[' + _list_elipse(o) + ']'
-def _str_ellipse(s, length=50):
- s = str(s)
- return s if len(s) <= length else (s[:length - 4] + '...' + s[-1])
+OSMObject.__repr__ = _make_repr(['id', 'deleted', 'visible', 'version', 'changeset',
+ 'uid', 'timestamp', 'user', 'tags'])
-Node.__repr__ = lambda o : '%s(id=%r, deleted=%r, visible=%r, version=%r, changeset=%r, uid=%r, timestamp=%r, user=%r, tags=%r, location=%r)'% (o.__class__.__name__, o.id, o.deleted, o.visible, o.version, o.changeset, o.uid, o.timestamp, o.user, o.tags, o.location)
-Node.__str__ = lambda n : 'n%d: location=%s tags=%s' \
- % (n.id, n.location, _str_ellipse(n.tags))
+Node.__repr__ = _make_repr(['id', 'deleted', 'visible', 'version', 'changeset',
+ 'uid', 'timestamp', 'user', 'tags', 'location'])
+Node.__str__ = lambda n: 'n{0.id:d}: location={0.location!s} tags={0.tags!s}'\
+ .format(n)
-Way.__repr__ = lambda o : '%s(id=%r, deleted=%r, visible=%r, version=%r, changeset=%r, uid=%r, timestamp=%r, user=%r, tags=%r, nodes=%r)'% (o.__class__.__name__, o.id, o.deleted, o.visible, o.version, o.changeset, o.uid, o.timestamp, o.user, o.tags, o.nodes)
-Way.__str__ = lambda o : 'w%d: nodes=%s tags=%s' \
- % (o.id, _str_ellipse(o.nodes), _str_ellipse(o.tags))
+Way.__repr__ = _make_repr(['id', 'deleted', 'visible', 'version', 'changeset',
+ 'uid', 'timestamp', 'user', 'tags', 'nodes'])
+Way.__str__ = lambda o: 'w{0.id:d}: nodes={0.nodes!s} tags={0.tags!s}' \
+ .format(o)
-Relation.__repr__ = lambda o : '%s(id=%r, deleted=%r, visible=%r, version=%r, changeset=%r, uid=%r, timestamp=%r, user=%r, tags=%r, members=%r)'% (o.__class__.__name__, o.id, o.deleted, o.visible, o.version, o.changeset, o.uid, o.timestamp, o.user, o.tags, o.members)
-Relation.__str__ = lambda o : 'r%d: members=%s, tags=%s' \
- % (o.id, _str_ellipse(o.members), _str_ellipse(o.tags))
+Relation.__repr__ = _make_repr(['id', 'deleted', 'visible', 'version', 'changeset',
+ 'uid', 'timestamp', 'user', 'tags', 'members'])
+Relation.__str__ = lambda o: 'r{0.id:d}: members={0.members!s}, tags={0.tags!s}' \
+ .format(o)
-Changeset.__repr__ = lambda o : '%s(id=%r, uid=%r, created_at=%r, closed_at=%r, open=%r, num_changes=%r, bounds=%r, user=%r, tags=%s)' %(o.__class__.__name__, o.id, o.uid, o.created_at, o.closed_at, o.open, o.num_changes, o.bounds, o.user, o.tags)
-Changeset.__str__ = lambda o : 'c%d: closed_at=%s, bounds=%s, tags=%s' \
- % (o.id, o.closed_at, o.bounds, _str_ellipse(o.tags))
+Changeset.__repr__ = _make_repr(['id', 'uid', 'created_at', 'closed_at', 'open',
+ 'num_changes', 'bounds', 'user', 'tags'])
+Changeset.__str__ = lambda o: 'c{0.id:d}: closed_at={0.closed_at!s}, bounds={0.bounds!s}, tags={0.tags!s}' \
+ .format(o)
=====================================
src/osmium/osm/mutable.py
=====================================
@@ -10,7 +10,7 @@ class OSMObject(object):
"""
def __init__(self, base=None, id=None, version=None, visible=None, changeset=None,
- timestamp=None, uid=None, tags=None, user=None):
+ timestamp=None, uid=None, tags=None, user=None):
if base is None:
self.id = id
self.version = version
@@ -73,5 +73,3 @@ class Relation(OSMObject):
self.members = members
else:
self.members = members if members is not None else base.members
-
-
=====================================
src/osmium/replication/server.py
=====================================
@@ -1,14 +1,7 @@
""" Helper functions to communicate with replication servers.
"""
-try:
- import urllib.request as urlrequest
-except ImportError:
- import urllib2 as urlrequest
-try:
- import urllib.error as urlerror
-except ImportError:
- import urllib2 as urlerror
+import urllib.request as urlrequest
import datetime as dt
from collections import namedtuple
from math import ceil
@@ -18,13 +11,13 @@ from osmium import version
import logging
-log = logging.getLogger('pyosmium')
-log.addHandler(logging.NullHandler())
+LOG = logging.getLogger('pyosmium')
+LOG.addHandler(logging.NullHandler())
OsmosisState = namedtuple('OsmosisState', ['sequence', 'timestamp'])
DownloadResult = namedtuple('DownloadResult', ['id', 'reader', 'newest'])
-class ReplicationServer(object):
+class ReplicationServer:
""" Represents a server that publishes replication data. Replication
change files allow to keep local OSM data up-to-date without downloading
the full dataset again.
@@ -88,7 +81,7 @@ class ReplicationServer(object):
try:
diffdata = self.get_diff_block(current_id)
except:
- log.debug("Error during diff download. Bailing out.")
+ LOG.debug("Error during diff download. Bailing out.")
diffdata = ''
if len(diffdata) == 0:
if start_id == current_id:
@@ -96,8 +89,8 @@ class ReplicationServer(object):
break
left_size -= rd.add_buffer(diffdata, self.diff_type)
- log.debug("Downloaded change %d. (%d kB available in download buffer)"
- % (current_id, left_size / 1024))
+ LOG.debug("Downloaded change %d. (%d kB available in download buffer)",
+ current_id, left_size / 1024)
current_id += 1
return DownloadResult(current_id - 1, rd, newest.sequence)
@@ -137,7 +130,7 @@ class ReplicationServer(object):
return diffs.id
def apply_diffs_to_file(self, infile, outfile, start_id, max_size=1024,
- set_replication_header=True, extra_headers={},
+ set_replication_header=True, extra_headers=None,
outformat=None):
""" Download diffs starting with sequence id `start_id`, merge them
with the data from the OSM file named `infile` and write the result
@@ -178,17 +171,19 @@ class ReplicationServer(object):
h.set("osmosis_replication_sequence_number", str(diffs.id))
info = self.get_state_info(diffs.id)
h.set("osmosis_replication_timestamp", info.timestamp.strftime("%Y-%m-%dT%H:%M:%SZ"))
- for k,v in extra_headers.items():
- h.set(k, v)
+ if extra_headers is not None:
+ for k, v in extra_headers.items():
+ h.set(k, v)
if outformat is None:
of = oio.File(outfile)
else:
of = oio.File(outfile, outformat)
+ of.has_multiple_object_versions = has_history
writer = oio.Writer(of, h)
- log.debug("Merging changes into OSM file.")
+ LOG.debug("Merging changes into OSM file.")
diffs.reader.apply_to_reader(reader, writer, has_history)
@@ -220,7 +215,7 @@ class ReplicationServer(object):
lower = None
lowerid = 0
while lower is None:
- log.debug("Trying with Id %s" % lowerid)
+ LOG.debug("Trying with Id %s", lowerid)
lower = self.get_state_info(lowerid)
if lower is not None and lower.timestamp >= timestamp:
@@ -287,11 +282,11 @@ class ReplicationServer(object):
returns `None`. `retries` sets the number of times the download
is retried when pyosmium detects a truncated state file.
"""
- for retry in range(retries + 1):
+ for _ in range(retries + 1):
try:
response = self.open_url(self.make_request(self.get_state_url(seq)))
except Exception as err:
- log.debug("Loading state info {} failed with: {}".format(seq, str(err)))
+ LOG.debug("Loading state info %s failed with: %s", seq, str(err))
return None
ts = None
@@ -341,13 +336,14 @@ class ReplicationServer(object):
if seq is None:
return self.baseurl + '/state.txt'
- return '%s/%03i/%03i/%03i.state.txt' % (self.baseurl,
- seq / 1000000, (seq % 1000000) / 1000, seq % 1000)
+ return '%s/%03i/%03i/%03i.state.txt' % \
+ (self.baseurl, seq / 1000000, (seq % 1000000) / 1000, seq % 1000)
def get_diff_url(self, seq):
""" Returns the URL to the diff file for the given sequence id.
"""
- return '%s/%03i/%03i/%03i.%s' % (self.baseurl,
- seq / 1000000, (seq % 1000000) / 1000, seq % 1000,
- self.diff_type)
+ return '%s/%03i/%03i/%03i.%s' % \
+ (self.baseurl,
+ seq / 1000000, (seq % 1000000) / 1000, seq % 1000,
+ self.diff_type)
=====================================
src/osmium/replication/utils.py
=====================================
@@ -6,10 +6,10 @@ from collections import namedtuple
from osmium.io import Reader as oreader
from osmium.osm import NOTHING
-log = logging.getLogger('pyosmium')
+LOG = logging.getLogger('pyosmium')
ReplicationHeader = namedtuple('ReplicationHeader',
- ['url', 'sequence', 'timestamp'])
+ ['url', 'sequence', 'timestamp'])
def get_replication_header(fname):
""" Scans the given file for an Osmosis replication header. It returns
@@ -28,21 +28,21 @@ def get_replication_header(fname):
url = h.get("osmosis_replication_base_url")
if url or ts:
- log.debug("Replication information found in OSM file header.")
+ LOG.debug("Replication information found in OSM file header.")
if url:
- log.debug("Replication URL: %s" % url)
+ LOG.debug("Replication URL: %s", url)
# the sequence ID is only considered valid, if an URL is given
seq = h.get("osmosis_replication_sequence_number")
if seq:
- log.debug("Replication sequence: %s" % seq)
+ LOG.debug("Replication sequence: %s", seq)
try:
seq = int(seq)
if seq < 0:
- log.warning("Sequence id '%d' in OSM file header is negative. Ignored." % seq)
+ LOG.warning("Sequence id '%d' in OSM file header is negative. Ignored.", seq)
seq = None
except ValueError:
- log.warning("Sequence id '%s' in OSM file header is not a number.Ignored" % seq)
+ LOG.warning("Sequence id '%s' in OSM file header is not a number. Ignored.", seq)
seq = None
else:
seq = None
@@ -51,13 +51,13 @@ def get_replication_header(fname):
seq = None
if ts:
- log.debug("Replication timestamp: %s" % ts)
+ LOG.debug("Replication timestamp: %s", ts)
try:
ts = dt.datetime.strptime(ts, "%Y-%m-%dT%H:%M:%SZ")
ts = ts.replace(tzinfo=dt.timezone.utc)
except ValueError:
- log.warning("Date in OSM file header is not in ISO8601 format (e.g. 2015-12-24T08:08Z). Ignored")
+ LOG.warning("Date in OSM file header is not in ISO8601 format (e.g. 2015-12-24T08:08Z). Ignored.")
ts = None
else:
ts = None
=====================================
src/osmium/version.py
=====================================
@@ -3,13 +3,13 @@ Version information.
"""
# the major version
-pyosmium_major = '3.0'
+pyosmium_major = '3.1'
# current release (Pip version)
-pyosmium_release = '3.0.1'
+pyosmium_release = '3.1.0'
# libosmium version shipped with the Pip release
libosmium_version = '2.15.6'
# protozero version shipped with the Pip release
protozero_version = '1.7.0'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.5.0'
+pybind11_version = '2.6.0'
=====================================
test/test_replication.py
=====================================
@@ -31,15 +31,15 @@ class UrllibMock(MagicMock):
self.side_effect = [BytesIO(dedent(s).encode()) for s in files]
def test_get_state_url():
- svr = rserv.ReplicationServer("http://text.org")
+ svr = rserv.ReplicationServer("https://text.org")
data = [
- (None, 'http://text.org/state.txt'),
- (1, 'http://text.org/000/000/001.state.txt'),
- (999, 'http://text.org/000/000/999.state.txt'),
- (1000, 'http://text.org/000/001/000.state.txt'),
- (573923, 'http://text.org/000/573/923.state.txt'),
- (3290012, 'http://text.org/003/290/012.state.txt'),
+ (None, 'https://text.org/state.txt'),
+ (1, 'https://text.org/000/000/001.state.txt'),
+ (999, 'https://text.org/000/000/999.state.txt'),
+ (1000, 'https://text.org/000/001/000.state.txt'),
+ (573923, 'https://text.org/000/573/923.state.txt'),
+ (3290012, 'https://text.org/003/290/012.state.txt'),
]
for i, o in data:
@@ -69,7 +69,7 @@ def test_get_state_valid(mock):
txnMax=1219304113
txnActiveList=1219303583,1219304054,1219304104""")
- res = rserv.ReplicationServer("http://test.io").get_state_info()
+ res = rserv.ReplicationServer("https://test.io").get_state_info()
assert_is_not_none(res)
assert_equals(res.timestamp, mkdate(2017, 8, 26, 11, 4, 2))
@@ -89,7 +89,7 @@ def test_get_state_sequence_cut(mock):
sequenceNumber=2594669
timestamp=2017-08-26T11\:04\:02Z"""))
- res = rserv.ReplicationServer("http://test.io").get_state_info()
+ res = rserv.ReplicationServer("https://test.io").get_state_info()
assert_is_not_none(res)
assert_equals(res.timestamp, mkdate(2017, 8, 26, 11, 4, 2))
@@ -110,7 +110,7 @@ def test_get_state_date_cut(mock):
sequenceNumber=2594669
timestamp=2017-08-26T11\:04\:02Z"""))
- res = rserv.ReplicationServer("http://test.io").get_state_info()
+ res = rserv.ReplicationServer("https://test.io").get_state_info()
assert_is_not_none(res)
assert_equals(res.timestamp, mkdate(2017, 8, 26, 11, 4, 2))
@@ -131,7 +131,7 @@ def test_get_state_timestamp_cut(mock):
sequenceNumber=2594669
timestamp=2017-08-26T11\:04\:02Z"""))
- res = rserv.ReplicationServer("http://test.io").get_state_info()
+ res = rserv.ReplicationServer("https://test.io").get_state_info()
assert_is_not_none(res)
assert_equals(res.timestamp, mkdate(2017, 8, 26, 11, 4, 2))
@@ -162,7 +162,7 @@ def test_get_state_too_many_retries(mock):
sequenceNumber=2594669
timestamp=2017-08-26T11\:04\:02Z"""))
- res = rserv.ReplicationServer("http://test.io").get_state_info()
+ res = rserv.ReplicationServer("https://test.io").get_state_info()
assert_is_none(res)
@@ -174,7 +174,7 @@ def test_get_state_too_many_retries(mock):
def test_get_state_server_timeout(mock):
mock.side_effect = URLError(reason='Mock')
- svr = rserv.ReplicationServer("http://test.io")
+ svr = rserv.ReplicationServer("https://test.io")
assert_is_none(svr.get_state_info())
@patch('osmium.replication.server.urlrequest.urlopen', new_callable=UrllibMock)
@@ -187,7 +187,7 @@ def test_apply_diffs_count(mock):
w1
r1
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
h = CountingHandler()
assert_equals(100, svr.apply_diffs(h, 100, 10000))
@@ -205,7 +205,7 @@ def test_apply_diffs_without_simplify(mock):
w1
r1
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
h = CountingHandler()
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=False))
@@ -222,7 +222,7 @@ def test_apply_diffs_with_simplify(mock):
w1
r1
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
h = CountingHandler()
assert_equals(100, svr.apply_diffs(h, 100, 10000, simplify=True))
@@ -237,7 +237,7 @@ def test_apply_with_location(mock):
n1 x10.0 y23.0
w1 Nn1,n2
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
class Handler(CountingHandler):
def way(self, w):
@@ -267,7 +267,7 @@ def test_apply_reader_without_simplify(mock):
w1
r1
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
h = CountingHandler()
@@ -288,7 +288,7 @@ def test_apply_reader_with_simplify(mock):
w1
r1
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
h = CountingHandler()
diffs = svr.collect_diffs(100, 100000)
@@ -306,7 +306,7 @@ def test_apply_reader_with_location(mock):
n1 x10.0 y23.0
w1 Nn1,n2
"""))
- svr = rserv.ReplicationServer("http://test.io", "opl")
+ svr = rserv.ReplicationServer("https://test.io", "opl")
class Handler(CountingHandler):
def way(self, w):
=====================================
tools/pyosmium-get-changes
=====================================
@@ -27,11 +27,11 @@ cookies to the server and will save received cookies to the jar file.
from argparse import ArgumentParser, RawDescriptionHelpFormatter, ArgumentTypeError
import datetime as dt
-from sys import version_info as python_version
import socket
from osmium.replication import server as rserv
from osmium.replication import newest_change_from_file
from osmium.replication.utils import get_replication_header
+from osmium.version import pyosmium_release
from osmium import SimpleHandler, WriteHandler
try:
@@ -85,8 +85,7 @@ class ReplicationStart(object):
def from_date(datestr):
try:
date = dt.datetime.strptime(datestr, "%Y-%m-%dT%H:%M:%SZ")
- if python_version >= (3,0):
- date = date.replace(tzinfo=dt.timezone.utc)
+ date = date.replace(tzinfo=dt.timezone.utc)
except ValueError:
raise ArgumentTypeError("Date needs to be in ISO8601 format (e.g. 2015-12-24T08:08:08Z).")
@@ -133,7 +132,7 @@ def get_arg_parser(from_main=False):
usage=None if from_main else 'pyosmium-get-changes [options]',
formatter_class=RawDescriptionHelpFormatter)
parser.add_argument('-v', dest='loglevel', action='count', default=0,
- help='Increase verbosity')
+ help='Increase verbosity (can be used multiple times)')
parser.add_argument('-o', '--outfile', dest='outfile',
help=h("""Name of diff output file. If omitted, only the
sequence ID will be printed where updates would start."""))
@@ -166,16 +165,18 @@ def get_arg_parser(from_main=False):
ignore potential replication information in the
header and search for the newest OSM object."""))
parser.add_argument('-d', '--no-deduplicate', action='store_false', dest='simplify',
- help='Do not deduplicate and sort diffs.')
+ help='Do not deduplicate diffs.')
parser.add_argument('--socket-timeout', dest='socket_timeout', type=int, default=60,
help='Set timeout for file downloads.')
+ parser.add_argument('--version', action='version', version='pyosmium ' + pyosmium_release)
return parser
def main(args):
logging.basicConfig(stream=sys.stderr,
- format='%(levelname)s: %(message)s')
+ format='%(asctime)s %(levelname)s: %(message)s',
+ datefmt='%Y-%m-%d %H:%M:%S')
options = get_arg_parser(from_main=True).parse_args(args)
=====================================
tools/pyosmium-up-to-date
=====================================
@@ -15,6 +15,9 @@ information from the osmosis headers. That means that after the first update,
subsequent calls to pyosmium-up-to-date will continue the updates from the same
server exactly where they have left of.
+This program can update normal OSM data files as well as OSM history files.
+It detects automatically on what type of file it is called.
+
The program returns 0, if updates have been successfully applied up to
the newest data. It returns 1, if some updates have been applied but
there is still data available on the server (either because the size
@@ -35,7 +38,6 @@ import socket
from argparse import ArgumentParser, RawDescriptionHelpFormatter
import datetime as dt
-from sys import version_info as python_version
from osmium.replication import server as rserv
from osmium.replication.utils import get_replication_header
from osmium.replication import newest_change_from_file
@@ -94,8 +96,7 @@ def update_from_custom_server(url, seq, ts, options):
if not options.force_update:
cmpdate = dt.datetime.utcnow() - dt.timedelta(days=90)
- if python_version >= (3,0):
- cmpdate = cmpdate.replace(tzinfo=dt.timezone.utc)
+ cmpdate = cmpdate.replace(tzinfo=dt.timezone.utc)
if ts < cmpdate:
log.error(
"""The OSM file is more than 3 months old. You should download a
@@ -116,18 +117,25 @@ def update_from_custom_server(url, seq, ts, options):
else:
ofname = outfile
- extra_headers = { 'generator' : 'pyosmium-up-to-date/' + pyosmium_release }
- outseqs = svr.apply_diffs_to_file(infile, ofname, startseq,
- max_size=options.outsize*1024,
- extra_headers=extra_headers,
- outformat=options.outformat)
-
- if outseqs is None:
- log.info("No new updates found.")
- return 3
+ try:
+ extra_headers = { 'generator' : 'pyosmium-up-to-date/' + pyosmium_release }
+ outseqs = svr.apply_diffs_to_file(infile, ofname, startseq,
+ max_size=options.outsize*1024,
+ extra_headers=extra_headers,
+ outformat=options.outformat)
+
+ if outseqs is None:
+ log.info("No new updates found.")
+ return 3
- if outfile is None:
- os.rename(ofname, infile)
+ if outfile is None:
+ os.rename(ofname, infile)
+ finally:
+ if outfile is None:
+ try:
+ os.remove(ofname)
+ except FileNotFoundError:
+ pass
log.info("Downloaded until %d. Server has data available until %d." % outseqs)
@@ -177,7 +185,7 @@ def get_arg_parser(from_main=False):
usage=None if from_main else 'pyosmium-up-to-date [options] <osm file>',
formatter_class=RawDescriptionHelpFormatter)
parser.add_argument('-v', dest='loglevel', action='count', default=0,
- help='Increase verbosity')
+ help='Increase verbosity (can be used multiple times)')
parser.add_argument('infile', metavar='<osm file>', help="OSM file to update")
parser.add_argument('-o', '--outfile', dest='outfile',
help=h("""Name output of file. If missing, the input file
@@ -209,6 +217,7 @@ def get_arg_parser(from_main=False):
received cookies will be written to.""")
parser.add_argument('--socket-timeout', dest='socket_timeout', type=int, default=60,
help='Set timeout for file downloads.')
+ parser.add_argument('--version', action='version', version='pyosmium ' + pyosmium_release)
return parser
@@ -217,7 +226,8 @@ def open_with_cookie(url):
if __name__ == '__main__':
logging.basicConfig(stream=sys.stderr,
- format='%(asctime)s %(levelname)s: %(message)s')
+ format='%(asctime)s %(levelname)s: %(message)s',
+ datefmt='%Y-%m-%d %H:%M:%S')
options = get_arg_parser(from_main=True).parse_args()
log.setLevel(max(3 - options.loglevel, 0) * 10)
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/6bf05da7b18818683fa5c7ffcf2a3f739c3663de...91619a0b3acb1a0c9013eca66cd8cd5ff439d474
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/compare/6bf05da7b18818683fa5c7ffcf2a3f739c3663de...91619a0b3acb1a0c9013eca66cd8cd5ff439d474
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20201112/cbbfeb36/attachment-0001.html>
More information about the Pkg-grass-devel
mailing list