[Git][debian-gis-team/osm2pgsql][upstream] New upstream version 1.5.1+ds
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Fri Jul 30 05:45:21 BST 2021
Bas Couwenberg pushed to branch upstream at Debian GIS Project / osm2pgsql
Commits:
b2608bdf by Bas Couwenberg at 2021-07-30T06:27:09+02:00
New upstream version 1.5.1+ds
- - - - -
17 changed files:
- + .github/ISSUE_TEMPLATE/bug-report.md
- + .github/ISSUE_TEMPLATE/config.yml
- CMakeLists.txt
- README.md
- docs/osm2pgsql.1
- docs/osm2pgsql.md
- flex-config/README.md
- src/flex-table.cpp
- src/input.cpp
- src/input.hpp
- src/options.cpp
- src/ordered-index.cpp
- src/ordered-index.hpp
- src/output-flex.cpp
- src/pgsql-helper.cpp
- tests/regression.py
- tests/test-check-input.cpp
Changes:
=====================================
.github/ISSUE_TEMPLATE/bug-report.md
=====================================
@@ -0,0 +1,48 @@
+---
+name: Report problems with the software
+about: You found a (possible) bug in osm2pgsql
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+## What version of osm2pgsql are you using?
+
+<!-- Paste output of `osm2pgsql --version` here. Please use the [latest
+release](https://osm2pgsql.org/releases/) of osm2pgsql if at all
+possible. -->
+
+
+## What operating system and PostgreSQL/PostGIS version are you using?
+
+<!-- Also what Linux distribution if applicable, OS version? -->
+
+
+## Tell us something about your system
+
+<!-- How much RAM do you have, how many CPUs, bare metal or cloud setup? -->
+
+
+## What did you do exactly?
+
+<!-- Please provide the command(s) you used including all options etc. Include
+links to input files. -->
+
+
+## What did you expect to happen?
+
+<!-- Describe in detail what you expected the above would do. -->
+
+
+## What did happen instead?
+
+<!-- Please describe what happened and why you think this is wrong. Please include
+(or link to, if it is too verbose) the log output. -->
+
+
+## What did you do to try analyzing the problem?
+
+<!-- Describe what steps you already did to try analyzing the problem before
+reporting. -->
+
=====================================
.github/ISSUE_TEMPLATE/config.yml
=====================================
@@ -0,0 +1,4 @@
+contact_links:
+ - name: osm2pgsql Discussions
+ url: https://github.com/openstreetmap/osm2pgsql/discussions
+ about: Ask questions, get support, share ideas and discuss with community members.
=====================================
CMakeLists.txt
=====================================
@@ -1,6 +1,6 @@
set(PACKAGE osm2pgsql)
set(PACKAGE_NAME osm2pgsql)
-set(PACKAGE_VERSION 1.5.0)
+set(PACKAGE_VERSION 1.5.1)
cmake_minimum_required(VERSION 2.8.12)
@@ -190,7 +190,7 @@ endif()
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src ${CMAKE_CURRENT_BINARY_DIR})
-find_package(Osmium 2.15.6 REQUIRED COMPONENTS io)
+find_package(Osmium 2.17.0 REQUIRED COMPONENTS io)
include_directories(SYSTEM ${OSMIUM_INCLUDE_DIRS} ${PROTOZERO_INCLUDE_DIR} ${FMT_INCLUDE_DIR} ${RAPIDJSON_INCLUDE_DIR})
if (WITH_LUA)
=====================================
README.md
=====================================
@@ -56,6 +56,15 @@ Required libraries are
* [Python](https://python.org/) (only for running tests)
* [Psycopg](http://initd.org/psycopg/) (only for running tests)
+The following libraries are included in the `contrib` directory. You can build
+with other versions of those libraries (set the `EXTERNAL_*libname*` option to
+`ON`) but make sure you are using a compatible version:
+
+* [fmt](https://fmt.dev/) (>= 7.1.3)
+* [libosmium](https://osmcode.org/libosmium/) (>= 2.17.0)
+* [protozero](https://github.com/mapbox/protozero) (>= 1.6.3)
+* [rapidjson](https://rapidjson.org/) (>= 1.1.0)
+
It also requires access to a database server running
[PostgreSQL](https://www.postgresql.org/) 9.5+ and
[PostGIS](http://www.postgis.net/) 2.2+.
=====================================
docs/osm2pgsql.1
=====================================
@@ -1,4 +1,4 @@
-.TH "OSM2PGSQL" "1" "1.5.0" "" ""
+.TH "OSM2PGSQL" "1" "1.5.1" "" ""
.SH NAME
.PP
osm2pgsql \- Openstreetmap data to PostgreSQL converter
@@ -251,16 +251,17 @@ See documentation for details.
.SH OUTPUT OPTIONS
.TP
.B \-O, \-\-output=OUTPUT
-Specifies the output back\-end to use.
+Specifies the output to use.
Currently osm2pgsql supports \f[B]pgsql\f[], \f[B]flex\f[],
\f[B]gazetteer\f[] and \f[B]null\f[].
-\f[B]pgsql\f[] is the default output back\-end and is optimized for
-rendering with Mapnik.
-\f[B]gazetteer\f[] is intended for geocoding with Nominatim.
-The experimental \f[B]flex\f[] backend allows more flexible
-configuration.
-\f[B]null\f[] does not write any output and is only useful for testing
-or with \f[B]\-\-slim\f[] for creating slim tables.
+\f[B]pgsql\f[] is the default output still available for backwards
+compatibility.
+New setups should use the \f[B]flex\f[] output which allows for a much
+more flexible configuration.
+The \f[B]gazetteer\f[] output is intended for geocoding with Nominatim
+only.
+The \f[B]null\f[] output does not write anything and is only useful for
+testing or with \f[B]\-\-slim\f[] for creating slim tables.
.RS
.RE
.TP
=====================================
docs/osm2pgsql.md
=====================================
@@ -174,13 +174,13 @@ mandatory for short options too.
# OUTPUT OPTIONS
-O, \--output=OUTPUT
-: Specifies the output back-end to use. Currently osm2pgsql
- supports **pgsql**, **flex**, **gazetteer** and **null**. **pgsql** is
- the default output back-end and is optimized for rendering with Mapnik.
- **gazetteer** is intended for geocoding with Nominatim.
- The experimental **flex** backend allows more flexible configuration.
- **null** does not write any output and is only useful for testing or with
- **\--slim** for creating slim tables.
+: Specifies the output to use. Currently osm2pgsql supports **pgsql**,
+ **flex**, **gazetteer** and **null**. **pgsql** is
+ the default output still available for backwards compatibility. New
+ setups should use the **flex** output which allows for a much more flexible
+ configuration. The **gazetteer** output is intended for geocoding with
+ Nominatim only. The **null** output does not write anything and is only
+ useful for testing or with **\--slim** for creating slim tables.
-S, \--style=FILE
: The style file. This specifies how the data is imported into the database,
=====================================
flex-config/README.md
=====================================
@@ -1,7 +1,5 @@
# Flex Output Configuration
-**The flex output is experimental. Everything in here is subject to change.**
-
See the [Flex Output](https://osm2pgsql.org/doc/manual.html#the-flex-output)
chapter in the manual for all the details.
=====================================
src/flex-table.cpp
=====================================
@@ -108,7 +108,11 @@ flex_table_t::build_sql_create_table(table_type ttype,
ttype == table_type::interim ? "UNLOGGED" : "", table_name);
for (auto const &column : m_columns) {
- sql += column.sql_create();
+ // create_only columns are only created in permanent, not in the
+ // interim tables
+ if (ttype == table_type::permanent || !column.create_only()) {
+ sql += column.sql_create();
+ }
}
assert(sql.back() == ',');
@@ -173,7 +177,7 @@ void table_connection_t::start(bool append)
if (!append) {
m_db_connection->exec(table().build_sql_create_table(
- table().has_geom_column() ? flex_table_t::table_type::interim
+ table().cluster_by_geom() ? flex_table_t::table_type::interim
: flex_table_t::table_type::permanent,
table().full_name()));
@@ -214,8 +218,9 @@ void table_connection_t::stop(bool updateable, bool append)
m_db_connection->exec(table().build_sql_create_table(
flex_table_t::table_type::permanent, table().full_tmp_name()));
- std::string sql = "INSERT INTO {} SELECT * FROM {}"_format(
- table().full_tmp_name(), table().full_name());
+ std::string const columns = table().build_sql_column_list();
+ std::string sql = "INSERT INTO {} ({}) SELECT {} FROM {}"_format(
+ table().full_tmp_name(), columns, columns, table().full_name());
auto const postgis_version = get_postgis_version(*m_db_connection);
=====================================
src/input.cpp
=====================================
@@ -21,7 +21,7 @@
#include "osmdata.hpp"
#include "progress-display.hpp"
-type_id_version check_input(type_id_version const &last, type_id_version curr)
+type_id check_input(type_id const &last, type_id curr)
{
if (curr.id < 0) {
throw std::runtime_error{
@@ -40,14 +40,10 @@ type_id_version check_input(type_id_version const &last, type_id_version curr)
osmium::item_type_to_name(last.type), curr.id, last.id)};
}
- if (last.version < curr.version) {
- return curr;
- }
-
throw std::runtime_error{
- "Input data is not ordered: {} id {} version {} after {}."_format(
- osmium::item_type_to_name(last.type), curr.id, curr.version,
- last.version)};
+ "Input data is not ordered:"
+ " {} id {} appears more than once."_format(
+ osmium::item_type_to_name(last.type), curr.id)};
}
if (item_type_to_nwr_index(last.type) <=
@@ -60,10 +56,9 @@ type_id_version check_input(type_id_version const &last, type_id_version curr)
osmium::item_type_to_name(last.type))};
}
-type_id_version check_input(type_id_version const &last,
- osmium::OSMObject const &object)
+type_id check_input(type_id const &last, osmium::OSMObject const &object)
{
- return check_input(last, {object.type(), object.id(), object.version()});
+ return check_input(last, {object.type(), object.id()});
}
/**
@@ -130,7 +125,7 @@ private:
osmium::memory::Buffer m_buffer{};
iterator m_it{};
iterator m_end{};
- type_id_version m_last = {osmium::item_type::node, 0, 0};
+ type_id m_last = {osmium::item_type::node, 0};
}; // class data_source_t
@@ -274,7 +269,7 @@ static void process_single_file(osmium::io::File const &file,
progress_display_t *progress, bool append)
{
osmium::io::Reader reader{file};
- type_id_version last{osmium::item_type::node, 0, 0};
+ type_id last{osmium::item_type::node, 0};
input_context_t ctx{osmdata, progress, append};
while (osmium::memory::Buffer buffer = reader.read()) {
=====================================
src/input.hpp
=====================================
@@ -28,21 +28,19 @@
class osmdata_t;
-struct type_id_version
+struct type_id
{
osmium::item_type type;
osmid_t id;
- osmium::object_version_type version;
};
/**
- * Compare two tuples (type, id, version) throw a descriptive error if either
- * the curr id is negative or if the data is not ordered.
+ * Compare two tuples (type, id). Throw a descriptive error if either the
+ * curr id is negative or if the data is not ordered.
*/
-type_id_version check_input(type_id_version const &last, type_id_version curr);
+type_id check_input(type_id const &last, type_id curr);
-type_id_version check_input(type_id_version const &last,
- osmium::OSMObject const &object);
+type_id check_input(type_id const &last, osmium::OSMObject const &object);
/**
* Prepare input file(s). Does format checks as far as this is possible
=====================================
src/options.cpp
=====================================
@@ -682,7 +682,7 @@ void options_t::check_options()
if (cache == 0) {
if (!slim) {
throw std::runtime_error{
- "RAM node cache can only be disable in slim mode."};
+ "RAM node cache can only be disabled in slim mode."};
}
if (flat_node_file.empty()) {
log_warn("RAM cache is disabled. This will likely slow down "
=====================================
src/ordered-index.cpp
=====================================
@@ -19,7 +19,8 @@ void ordered_index_t::add(osmid_t id, std::size_t offset)
(last().offset_from + last().index.back().offset) < offset));
if (need_new_2nd_level() ||
- (id - last().from) > std::numeric_limits<uint32_t>::max()) {
+ (id - last().from) > std::numeric_limits<uint32_t>::max() ||
+ (offset - last().offset_from) >= std::numeric_limits<uint32_t>::max()) {
if (!m_ranges.empty()) {
m_ranges.back().to = id - 1;
}
=====================================
src/ordered-index.hpp
=====================================
@@ -31,7 +31,7 @@
* An index that is never used doesn't need more memory than
* sizeof(ordered_index_t).
*
- * All allocated memory can be freed by calling clear(). Afer that the index
+ * All allocated memory can be freed by calling clear(). After that the index
* can NOT be reused.
*
* There are two ways of accessing the data through the index:
@@ -185,7 +185,7 @@ private:
std::pair<osmid_t, std::size_t> get_internal(osmid_t id) const noexcept;
- static constexpr std::size_t const max_block_size = 16 * 1024 * 1204;
+ static constexpr std::size_t const max_block_size = 16 * 1024 * 1024;
std::vector<range_entry> m_ranges;
std::size_t m_block_size;
=====================================
src/output-flex.cpp
=====================================
@@ -36,6 +36,8 @@ extern "C"
#include <rapidjson/stringbuffer.h>
#include <rapidjson/writer.h>
+#include <boost/filesystem.hpp>
+
#include <cassert>
#include <cstdlib>
#include <cstring>
@@ -1630,6 +1632,10 @@ void output_flex_t::init_lua(std::string const &filename)
m_options.append ? "append" : "create");
luaX_add_table_int(lua_state(), "stage", 1);
+ std::string const dir_path =
+ boost::filesystem::path{filename}.parent_path().string();
+ luaX_add_table_str(lua_state(), "config_dir", dir_path.c_str());
+
luaX_add_table_func(lua_state(), "define_table",
lua_trampoline_app_define_table);
=====================================
src/pgsql-helper.cpp
=====================================
@@ -48,7 +48,7 @@ void create_geom_check_trigger(pg_conn_t *db_connection,
" END IF;\n"
" RETURN NULL;\n"
"END;"
- "$$ LANGUAGE plpgsql IMMUTABLE;"_format(func_name, geom_column));
+ "$$ LANGUAGE plpgsql;"_format(func_name, geom_column));
db_connection->exec(
"CREATE TRIGGER \"{}\""
=====================================
tests/regression.py
=====================================
@@ -50,14 +50,18 @@ def setUpModule():
logging.info("Setting up test database")
dbname = CONFIG['test_database']
- with psycopg2.connect("dbname='template1'") as conn:
+ conn = psycopg2.connect("dbname='template1'")
+ try:
conn.autocommit = True
with conn.cursor() as cur:
cur.execute('DROP DATABASE IF EXISTS "{}"'.format(dbname))
cur.execute("CREATE DATABASE \"{}\" WITH ENCODING 'UTF8'"
.format(dbname))
+ finally:
+ conn.close()
- with psycopg2.connect("dbname='{}'".format(dbname)) as conn:
+ conn = psycopg2.connect("dbname='{}'".format(dbname))
+ try:
conn.autocommit = True
with conn.cursor() as cur:
# Check if there is a dataspace, we will skip tests otherwise.
@@ -68,17 +72,22 @@ def setUpModule():
cur.execute('CREATE EXTENSION postgis')
cur.execute('CREATE EXTENSION hstore')
+ finally:
+ conn.close()
def tearDownModule():
""" Destroy the global database.
"""
logging.info("Cleaning up test database")
- with psycopg2.connect("dbname='template1'") as conn:
+ conn = psycopg2.connect("dbname='template1'")
+ try:
+ conn.autocommit = True
with conn.cursor() as cur:
- conn.autocommit = True
cur.execute('DROP DATABASE IF EXISTS "{}"'
.format(CONFIG['test_database']))
+ finally:
+ conn.close()
########################################################################
#
@@ -111,13 +120,17 @@ class BaseRunner(object):
cls.skipTest(None, "No Lua configured.")
if 'tablespacetest' in cls.extra_params and not CONFIG['have_lua']:
cls.skipTest(None, "Test tablespace 'tablespacetest' not configured.")
- with psycopg2.connect("dbname='{}'".format(CONFIG['test_database'])) as conn:
+ conn = psycopg2.connect("dbname='{}'".format(CONFIG['test_database']))
+ try:
with conn.cursor() as cur:
for t in ('nodes', 'ways', 'rels', 'point', 'line', 'roads', 'polygon'):
cur.execute("DROP TABLE IF EXISTS planet_osm_" + t)
cur.execute("DROP SCHEMA IF EXISTS osm CASCADE")
if cls.schema:
cur.execute("CREATE SCHEMA " + cls.schema)
+ conn.commit()
+ finally:
+ conn.close()
if cls.import_file:
cls.run_import(cls.get_def_params() + cls.extra_params,
@@ -134,8 +147,11 @@ class BaseRunner(object):
BaseRunner.conn.close()
BaseRunner.conn = None
if cls.schema:
- with psycopg2.connect("dbname='{}'".format(CONFIG['test_database'])) as conn:
+ conn = psycopg2.connect("dbname='{}'".format(CONFIG['test_database']))
+ try:
conn.cursor().execute("DROP SCHEMA IF EXISTS {} CASCADE".format(cls.schema))
+ finally:
+ conn.close()
try:
os.remove('flat.nodes')
except (FileNotFoundError):
=====================================
tests/test-check-input.cpp
=====================================
@@ -13,27 +13,25 @@
TEST_CASE("it's good if input data is ordered", "[NoDB]")
{
- type_id_version const tiv1{osmium::item_type::node, 1, 1};
- type_id_version const tiv2{osmium::item_type::node, 1, 2};
- type_id_version const tiv3{osmium::item_type::node, 2, 1};
- type_id_version const tiv4{osmium::item_type::way, 1, 1};
- type_id_version const tiv5{osmium::item_type::way, 2, 1};
- type_id_version const tiv6{osmium::item_type::relation, 1, 1};
- type_id_version const tiv7{osmium::item_type::relation, 1, 2};
+ type_id const tiv1{osmium::item_type::node, 1};
+ type_id const tiv2{osmium::item_type::node, 2};
+ type_id const tiv3{osmium::item_type::way, 1};
+ type_id const tiv4{osmium::item_type::way, 2};
+ type_id const tiv5{osmium::item_type::relation, 1};
+ type_id const tiv6{osmium::item_type::relation, 2};
REQUIRE_NOTHROW(check_input(tiv1, tiv2));
REQUIRE_NOTHROW(check_input(tiv2, tiv3));
REQUIRE_NOTHROW(check_input(tiv3, tiv4));
REQUIRE_NOTHROW(check_input(tiv4, tiv5));
REQUIRE_NOTHROW(check_input(tiv5, tiv6));
- REQUIRE_NOTHROW(check_input(tiv6, tiv7));
}
TEST_CASE("negative OSM object ids are not allowed", "[NoDB]")
{
- type_id_version const tivn{osmium::item_type::node, -17, 1};
- type_id_version const tivw{osmium::item_type::way, -1, 1};
- type_id_version const tivr{osmium::item_type::relation, -999, 17};
+ type_id const tivn{osmium::item_type::node, -17};
+ type_id const tivw{osmium::item_type::way, -1};
+ type_id const tivr{osmium::item_type::relation, -999};
REQUIRE_THROWS_WITH(
check_input(tivn, tivn),
@@ -47,8 +45,8 @@ TEST_CASE("negative OSM object ids are not allowed", "[NoDB]")
TEST_CASE("objects of the same type must be ordered", "[NoDB]")
{
- type_id_version const tiv1{osmium::item_type::node, 42, 1};
- type_id_version const tiv2{osmium::item_type::node, 3, 1};
+ type_id const tiv1{osmium::item_type::node, 42};
+ type_id const tiv2{osmium::item_type::node, 3};
REQUIRE_THROWS_WITH(check_input(tiv1, tiv2),
"Input data is not ordered: node id 3 after 42.");
@@ -56,9 +54,9 @@ TEST_CASE("objects of the same type must be ordered", "[NoDB]")
TEST_CASE("a node after a way or relation is not allowed", "[NoDB]")
{
- type_id_version const tiv1w{osmium::item_type::way, 42, 1};
- type_id_version const tiv1r{osmium::item_type::relation, 42, 1};
- type_id_version const tiv2{osmium::item_type::node, 100, 1};
+ type_id const tiv1w{osmium::item_type::way, 42};
+ type_id const tiv1r{osmium::item_type::relation, 42};
+ type_id const tiv2{osmium::item_type::node, 100};
REQUIRE_THROWS_WITH(check_input(tiv1w, tiv2),
"Input data is not ordered: node after way.");
@@ -68,20 +66,20 @@ TEST_CASE("a node after a way or relation is not allowed", "[NoDB]")
TEST_CASE("a way after a relation is not allowed", "[NoDB]")
{
- type_id_version const tiv1{osmium::item_type::relation, 42, 1};
- type_id_version const tiv2{osmium::item_type::way, 100, 1};
+ type_id const tiv1{osmium::item_type::relation, 42};
+ type_id const tiv2{osmium::item_type::way, 100};
REQUIRE_THROWS_WITH(check_input(tiv1, tiv2),
"Input data is not ordered: way after relation.");
}
-TEST_CASE("versions must be ordered", "[NoDB]")
+TEST_CASE("no object may appear twice", "[NoDB]")
{
- type_id_version const tiv1{osmium::item_type::way, 42, 2};
- type_id_version const tiv2{osmium::item_type::way, 42, 1};
+ type_id const tiv1{osmium::item_type::way, 42};
+ type_id const tiv2{osmium::item_type::way, 42};
REQUIRE_THROWS_WITH(
check_input(tiv1, tiv2),
- "Input data is not ordered: way id 42 version 1 after 2.");
+ "Input data is not ordered: way id 42 appears more than once.");
}
View it on GitLab: https://salsa.debian.org/debian-gis-team/osm2pgsql/-/commit/b2608bdf54732638bd6abd93c0b530e3af3099fb
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/osm2pgsql/-/commit/b2608bdf54732638bd6abd93c0b530e3af3099fb
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20210730/6b0fc78a/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list