[Git][debian-gis-team/pyosmium][upstream] New upstream version 3.0.0
Bas Couwenberg
gitlab at salsa.debian.org
Fri May 8 05:31:50 BST 2020
Bas Couwenberg pushed to branch upstream at Debian GIS Project / pyosmium
Commits:
7d8323eb by Bas Couwenberg at 2020-05-08T06:18:16+02:00
New upstream version 3.0.0
- - - - -
14 changed files:
- .travis.yml
- CHANGELOG.md
- README.md
- appveyor.yml
- doc/updating_osm_data.rst
- lib/cast.h
- lib/io.cc
- setup.py
- src/osmium/osm/__init__.py
- src/osmium/replication/server.py
- src/osmium/replication/utils.py
- src/osmium/version.py
- tools/pyosmium-get-changes
- tools/pyosmium-up-to-date
Changes:
=====================================
.travis.yml
=====================================
@@ -4,70 +4,81 @@
#
#-----------------------------------------------------------------------------
-language: cpp
-
-dist: xenial
matrix:
include:
- os: linux
- compiler: clang
- env: USE_PYTHON_VERSION=2
+ compiler: "clang-3.4"
+ dist: xenial
+ language: cpp
- os: linux
- compiler: clang
- env: USE_PYTHON_VERSION=3
+ compiler: "clang-7"
+ dist: bionic
+ language: cpp
+ - os: linux
+ compiler: "gcc-5"
+ dist: xenial
+ language: cpp
+ - os: linux
+ compiler: "gcc-8"
+ dist: bionic
+ language: cpp
+ - os: linux
+ compiler: gcc
+ python: 3.4
+ dist: xenial
+ language: python
+ - os: linux
+ compiler: gcc
+ python: 3.5
+ dist: bionic
+ language: python
- os: linux
compiler: gcc
- env: USE_PYTHON_VERSION=2
+ python: 3.6
+ dist: bionic
+ language: python
- os: linux
compiler: gcc
- env: USE_PYTHON_VERSION=3
+ python: 3.7
+ dist: bionic
+ language: python
+ - os: linux
+ compiler: gcc
+ python: 3.8
+ dist: bionic
+ language: python
- os: osx
osx_image: xcode7
compiler: clang
- env: USE_PYTHON_VERSION=
- - os: osx
- osx_image: xcode10.1
- compiler: clang
- env: USE_PYTHON_VERSION=
+ language: cpp
- os: osx
osx_image: xcode10.1
compiler: clang
- env: USE_PYTHON_VERSION=3
+ language: cpp
# http://docs.travis-ci.com/user/apt/
addons:
apt:
- sources:
- - boost-latest
packages:
- - libboost1.58-dev
- - python-dev
- - python-nose
- - python-mock
- - python-shapely
+ - libboost-dev
- python3
- python3-dev
- - python3-nose
- - python3-setuptools
- - python3-shapely
+ - python3-pip
install:
- git clone --quiet --depth 1 https://github.com/osmcode/libosmium.git contrib/libosmium
- git clone --quiet --depth 1 https://github.com/mapbox/protozero.git contrib/protozero
- git clone --quiet --depth 1 https://github.com/pybind/pybind11.git contrib/pybind11
- if [ "$TRAVIS_OS_NAME" = 'osx' ]; then
- pip${USE_PYTHON_VERSION} install --user -q nose mock shapely;
+ pip3 install --user -q nose mock shapely setuptools;
+ else
+ pip3 install -q nose mock shapely setuptools;
fi
script:
- - if [ "$TRAVIS_OS_NAME" = 'osx' ]; then
- PYTHON=python${USE_PYTHON_VERSION};
- else
- PYTHON=/usr/bin/python${USE_PYTHON_VERSION};
- fi
- - $PYTHON --version
- - $PYTHON setup.py build
+ - python3 --version
+ - python3 setup.py build
- cd test
- - $PYTHON run_tests.py
+ - python3 run_tests.py
=====================================
CHANGELOG.md
=====================================
@@ -4,6 +4,17 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
+## [3.0.0] - 2020-05-03
+
+### Added
+
+- socket timeouts for pyosmium-get-changes and pyosmium-up-to-date
+- pyosmium-up-to-date: allow writing the diff to stdout (requires format option)
+
+### Changed
+
+- remove support for Python2 and Python 3.3
+
## [2.15.4] - 2020-02-29
### Added
=====================================
README.md
=====================================
@@ -7,41 +7,43 @@ manner.
[](http://travis-ci.org/osmcode/pyosmium)
[](https://ci.appveyor.com/project/lonvia/pyosmium/branch/master)
+## Installation
+Pyosmium works with Python >= 3.4. Pypy is known to not work.
-## Dependencies
-
-Python >= 2.7 is supported but a version >= 3.3 is strongly recommended.
-Pypy is known to not work.
-
-Other requirements are:
+### Using Pip
- * Python setuptools
- * [cmake](https://cmake.org/)
- * [Pybind11](https://github.com/pybind/pybind11) >= 2.2
- * libosmium, protozero
- * expat, libz, libbz2 and Boost variant and iterator
- * a recent C++ compiler (Clang 3.4+, GCC 4.8+)
+The recommended way to install pyosmium is via pip:
-## Installation
+ pip install osmium
-### Installing prerequisites
+We provide binary wheels for Linux and Windows 64 for all actively
+maintained Python versions.
-On Debian/Ubuntu-like systems the following should install all
-dependencies required for installing via pip:
+For other versions, a source wheel is provided. Make sure to install all
+external dependencies first. On Deian/Ubuntu-like systems, the following
+command installs all required packages:
sudo apt-get install build-essential cmake libboost-dev \
libexpat1-dev zlib1g-dev libbz2-dev
-### Using Pip
-The recommended way to install pyosmium is via pip:
+### Installing from source
- pip install osmium
+#### Prerequisites
+
+pyosmium has the following dependencies:
-You need to install development packages first for the dependencies
-mentioned above. For Windows, there are also experimental binary wheels
-available with all required dependencies packed already.
+ * [libosmium](https://github.com/osmcode/libosmium)
+ * [protozero](https://github.com/mapbox/protozero)
+ * [cmake](https://cmake.org/)
+ * [Pybind11](https://github.com/pybind/pybind11) >= 2.2
+ * [expat](https://libexpat.github.io/)
+ * [libz](https://www.zlib.net/)
+ * [libbz2](https://www.sourceware.org/bzip2/)
+ * [Boost](https://www.boost.org/) variant and iterator >= 1.41
+ * Python setuptools
+ * a recent C++ compiler (Clang 3.4+, GCC 4.8+)
### Compiling from Source
=====================================
appveyor.yml
=====================================
@@ -4,22 +4,20 @@ environment:
EXPAT_VER: 2.2.5
ZLIB_VER: 1.2.11
SHAPELY_VER: 1.6.4
- BOOST_PREFIX: C:\Libraries\boost_1_67_0
+ BOOST_PREFIX: C:\Libraries\boost_1_71_0
+ PIPINSTALLS: nose wheel
matrix:
- - PYTHON_VER: 2.7
- PIPINSTALLS: nose wheel mock
- MINICONDA: "C:\\Miniconda-x64"
- arch: x64
- PYTHON_VER: 3.6
- PIPINSTALLS: nose wheel
MINICONDA: "C:\\Miniconda36-x64"
arch: x64
- PYTHON_VER: 3.7
- PIPINSTALLS: nose wheel
MINICONDA: "C:\\Miniconda37-x64"
arch: x64
+ - PYTHON_VER: 3.8
+ MINICONDA: "C:\\Miniconda38-x64"
+ arch: x64
-os: Visual Studio 2017
+os: Visual Studio 2019
clone_depth: 1
=====================================
doc/updating_osm_data.rst
=====================================
@@ -127,7 +127,7 @@ Method 1: Starting from the import file
If you still have the OSM file you used to set up your database, then
create a state file as follows::
- pyosmium-get-changes -O <osmfile.osm.obf> -f sequence.state -v
+ pyosmium-get-changes -O <osmfile.osm.pbf> -f sequence.state -v
Note that there is no output file yet. This creates a new file `sequence.state`
with the sequence ID where updates should start and prints the URL of the
=====================================
lib/cast.h
=====================================
@@ -27,18 +27,8 @@ namespace pybind11 { namespace detail {
return false;
}
-#if PY_VERSION_HEX >= 0x03040000
auto ts = src.attr("timestamp")();
value = (unsigned) ts.cast<double>();
-#else
- // XXX terribly inefficient because of the double string conversion
- // but the only painless method for converting a datetime
- // in python < 3.3.
- auto ts = src.attr("strftime")("%Y-%m-%dT%H:%M:%SZ");
-
- value = osmium::Timestamp(ts.cast<std::string>());
-
-#endif
return true;
}
@@ -60,13 +50,9 @@ namespace pybind11 { namespace detail {
localtime.tm_sec,
0);
-#if PY_VERSION_HEX < 0x03000000
- return pydate;
-#else
auto utc = pybind11::module::import("datetime").attr("timezone").attr("utc");
using namespace pybind11::literals;
return pydate.attr("replace")("tzinfo"_a=utc).inc_ref();
-#endif
}
PYBIND11_TYPE_CASTER(type, _("datetime.datetime"));
=====================================
lib/io.cc
=====================================
@@ -7,6 +7,20 @@ namespace py = pybind11;
PYBIND11_MODULE(io, m)
{
+ py::class_<osmium::io::File>(m, "File",
+ "A data file.")
+ .def(py::init<std::string>())
+ .def(py::init<std::string, std::string>())
+ .def_property("has_multiple_object_versions",
+ &osmium::io::File::has_multiple_object_versions,
+ &osmium::io::File::set_has_multiple_object_versions,
+ "True if there may be more than one version of the same "
+ "object in the file. This happens normally only in history "
+ "files.")
+ .def("parse_format", &osmium::io::File::parse_format,
+ "Set the format of the file from a format string.")
+ ;
+
py::class_<osmium::io::Header>(m, "Header",
"File header with global information about the file.")
.def(py::init<>())
@@ -48,7 +62,9 @@ PYBIND11_MODULE(io, m)
"OSM file,. Have a look `osmium.SimpleWriter` for a high-level interface "
"for writing out data.")
.def(py::init<std::string>())
+ .def(py::init<osmium::io::File>())
.def(py::init<std::string, osmium::io::Header>())
+ .def(py::init<osmium::io::File, osmium::io::Header>())
.def("close", &osmium::io::Writer::close,
"Close any open file handles. The writer is unusable afterwards.")
;
=====================================
setup.py
=====================================
@@ -112,6 +112,9 @@ class CMakeBuild(build_ext):
versions = get_versions()
+if sys.version_info < (3,0):
+ raise RuntimeError("Python 3.3 or larger required.")
+
with open('README.rst', 'r') as descfile:
long_description = descfile.read()
@@ -133,12 +136,11 @@ setup(
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: BSD License",
- "Programming Language :: Python :: 2.7",
- "Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
+ "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: C++",
],
@@ -147,6 +149,7 @@ setup(
packages = ['osmium', 'osmium/osm', 'osmium/replication'],
package_dir = {'' : 'src'},
package_data = { 'osmium' : [ '*.dll' ] },
+ python_requires = ">=3.4",
cmdclass=dict(build_ext=CMakeBuild, sdist=Pyosmium_sdist),
zip_safe=False,
)
=====================================
src/osmium/osm/__init__.py
=====================================
@@ -1,6 +1,5 @@
from ._osm import *
import osmium.osm.mutable
-from sys import version_info as python_version
def create_mutable_node(node, **args):
""" Create a mutable node replacing the properties given in the
@@ -39,10 +38,7 @@ Box.__str__ = lambda b : '(%s %s)' % (b.bottom_left, b.top_right)
Tag.__repr__ = lambda t : 'osmium.osm.Tag(k=%r, v=%r)' % (t.k, t.v)
-if python_version >= (3,0):
- Tag.__str__ = lambda t : '%s=%s' % (t.k, t.v)
-else:
- Tag.__str__ = lambda t : '%s=%s' % (t.k.encode('utf-8'), t.v.encode('utf-8'))
+Tag.__str__ = lambda t : '%s=%s' % (t.k, t.v)
TagList.__repr__ = lambda t : "osmium.osm.TagList({%s})" \
% ",".join(["%r=%r" % (i.k, i.v) for i in t])
=====================================
src/osmium/replication/server.py
=====================================
@@ -1,7 +1,6 @@
""" Helper functions to communicate with replication servers.
"""
-import sys
try:
import urllib.request as urlrequest
except ImportError:
@@ -89,6 +88,7 @@ class ReplicationServer(object):
try:
diffdata = self.get_diff_block(current_id)
except:
+ log.debug("Error during diff download. Bailing out.")
diffdata = ''
if len(diffdata) == 0:
if start_id == current_id:
@@ -137,7 +137,8 @@ class ReplicationServer(object):
return diffs.id
def apply_diffs_to_file(self, infile, outfile, start_id, max_size=1024,
- set_replication_header=True, extra_headers={}):
+ set_replication_header=True, extra_headers={},
+ outformat=None):
""" Download diffs starting with sequence id `start_id`, merge them
with the data from the OSM file named `infile` and write the result
into a file with the name `outfile`. The output file must not yet
@@ -155,6 +156,9 @@ class ReplicationServer(object):
`extra_headers` is a dict with additional header fields to be set.
Most notably, the 'generator' can be set this way.
+ `outformat` sets the format of the output file. If None, the format
+ is determined from the file name.
+
The function returns a tuple of last downloaded sequence id and
newest available sequence id if new data has been written or None
if no data was available or the download failed completely.
@@ -177,7 +181,12 @@ class ReplicationServer(object):
for k,v in extra_headers.items():
h.set(k, v)
- writer = oio.Writer(outfile, h)
+ if outformat is None:
+ of = oio.File(outfile)
+ else:
+ of = oio.File(outfile, outformat)
+
+ writer = oio.Writer(of, h)
log.debug("Merging changes into OSM file.")
@@ -305,8 +314,7 @@ class ReplicationServer(object):
ts = dt.datetime.strptime(kv[1], "%Y-%m-%dT%H\\:%M\\:%SZ")
except ValueError:
break
- if sys.version_info >= (3,0):
- ts = ts.replace(tzinfo=dt.timezone.utc)
+ ts = ts.replace(tzinfo=dt.timezone.utc)
line = response.readline()
if ts is not None and seq is not None:
=====================================
src/osmium/replication/utils.py
=====================================
@@ -5,7 +5,6 @@ import datetime as dt
from collections import namedtuple
from osmium.io import Reader as oreader
from osmium.osm import NOTHING
-from sys import version_info as python_version
log = logging.getLogger('pyosmium')
@@ -55,8 +54,7 @@ def get_replication_header(fname):
log.debug("Replication timestamp: %s" % ts)
try:
ts = dt.datetime.strptime(ts, "%Y-%m-%dT%H:%M:%SZ")
- if python_version >= (3,0):
- ts = ts.replace(tzinfo=dt.timezone.utc)
+ ts = ts.replace(tzinfo=dt.timezone.utc)
except ValueError:
log.warning("Date in OSM file header is not in ISO8601 format (e.g. 2015-12-24T08:08Z). Ignored")
=====================================
src/osmium/version.py
=====================================
@@ -3,13 +3,13 @@ Version information.
"""
# the major version
-pyosmium_major = '2.15'
+pyosmium_major = '3.0'
# current release (Pip version)
-pyosmium_release = '2.15.4'
+pyosmium_release = '3.0.0'
# libosmium version shipped with the Pip release
-libosmium_version = '2.15.4'
+libosmium_version = '2.15.5'
# protozero version shipped with the Pip release
protozero_version = '1.6.8'
# pybind11 version shipped with the Pip release
-pybind11_version = '2.4.3'
+pybind11_version = '2.5.0'
=====================================
tools/pyosmium-get-changes
=====================================
@@ -28,6 +28,7 @@ cookies to the server and will save received cookies to the jar file.
from argparse import ArgumentParser, RawDescriptionHelpFormatter, ArgumentTypeError
import datetime as dt
from sys import version_info as python_version
+import socket
from osmium.replication import server as rserv
from osmium.replication import newest_change_from_file
from osmium.replication.utils import get_replication_header
@@ -166,6 +167,8 @@ def get_arg_parser(from_main=False):
header and search for the newest OSM object."""))
parser.add_argument('-d', '--no-deduplicate', action='store_false', dest='simplify',
help='Do not deduplicate and sort diffs.')
+ parser.add_argument('--socket-timeout', dest='socket_timeout', type=int, default=60,
+ help='Set timeout for file downloads.')
return parser
@@ -212,6 +215,7 @@ def main(args):
or 'https://planet.osm.org/replication/minute/'
logging.info("Using replication server at %s" % url)
+ socket.setdefaulttimeout(options.socket_timeout)
svr = rserv.ReplicationServer(url)
if options.cookie is not None:
=====================================
tools/pyosmium-up-to-date
=====================================
@@ -31,6 +31,7 @@ cookies to the server and will save received cookies to the jar file.
import re
import sys
import logging
+import socket
from argparse import ArgumentParser, RawDescriptionHelpFormatter
import datetime as dt
@@ -118,7 +119,8 @@ def update_from_custom_server(url, seq, ts, options):
extra_headers = { 'generator' : 'pyosmium-up-to-date/' + pyosmium_release }
outseqs = svr.apply_diffs_to_file(infile, ofname, startseq,
max_size=options.outsize*1024,
- extra_headers=extra_headers)
+ extra_headers=extra_headers,
+ outformat=options.outformat)
if outseqs is None:
log.info("No new updates found.")
@@ -180,6 +182,8 @@ def get_arg_parser(from_main=False):
parser.add_argument('-o', '--outfile', dest='outfile',
help=h("""Name output of file. If missing, the input file
will be overwritten."""))
+ parser.add_argument('--format', dest='outformat', metavar='FORMAT',
+ help="Format the data should be saved in.")
parser.add_argument('--server', action='store', dest='server_url',
help=h("""Base URL of the replication server. Default:
'https://planet.osm.org/replication/hour/'
@@ -203,6 +207,8 @@ def get_arg_parser(from_main=False):
parser.add_argument('--cookie', dest='cookie',
help="""Netscape-style cookie jar file to read cookies from and where
received cookies will be written to.""")
+ parser.add_argument('--socket-timeout', dest='socket_timeout', type=int, default=60,
+ help='Set timeout for file downloads.')
return parser
@@ -216,6 +222,8 @@ if __name__ == '__main__':
options = get_arg_parser(from_main=True).parse_args()
log.setLevel(max(3 - options.loglevel, 0) * 10)
+ socket.setdefaulttimeout(options.socket_timeout)
+
try:
url, seq, ts = compute_start_point(options)
except RuntimeError as e:
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/7d8323eba35cc69665c6f19b9cb756bafc54ffee
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/pyosmium/-/commit/7d8323eba35cc69665c6f19b9cb756bafc54ffee
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20200508/7c6b315c/attachment-0001.html>
More information about the Pkg-grass-devel
mailing list