[Git][debian-gis-team/osm2pgsql][experimental] 4 commits: New upstream version 1.5.1+ds

Bas Couwenberg (@sebastic) gitlab at salsa.debian.org
Fri Jul 30 05:45:16 BST 2021



Bas Couwenberg pushed to branch experimental at Debian GIS Project / osm2pgsql


Commits:
b2608bdf by Bas Couwenberg at 2021-07-30T06:27:09+02:00
New upstream version 1.5.1+ds
- - - - -
95258a54 by Bas Couwenberg at 2021-07-30T06:27:12+02:00
Update upstream source from tag 'upstream/1.5.1+ds'

Update to upstream version '1.5.1+ds'
with Debian dir 17ab9f44ff883c7820679423559ee881533717a4
- - - - -
feafe1a4 by Bas Couwenberg at 2021-07-30T06:29:02+02:00
New upstream release.

- - - - -
f4282b59 by Bas Couwenberg at 2021-07-30T06:31:37+02:00
Set distribution to experimental.

- - - - -


18 changed files:

- + .github/ISSUE_TEMPLATE/bug-report.md
- + .github/ISSUE_TEMPLATE/config.yml
- CMakeLists.txt
- README.md
- debian/changelog
- docs/osm2pgsql.1
- docs/osm2pgsql.md
- flex-config/README.md
- src/flex-table.cpp
- src/input.cpp
- src/input.hpp
- src/options.cpp
- src/ordered-index.cpp
- src/ordered-index.hpp
- src/output-flex.cpp
- src/pgsql-helper.cpp
- tests/regression.py
- tests/test-check-input.cpp


Changes:

=====================================
.github/ISSUE_TEMPLATE/bug-report.md
=====================================
@@ -0,0 +1,48 @@
+---
+name: Report problems with the software
+about: You found a (possible) bug in osm2pgsql
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+## What version of osm2pgsql are you using?
+
+<!-- Paste output of `osm2pgsql --version` here. Please use the [latest
+release](https://osm2pgsql.org/releases/) of osm2pgsql if at all
+possible. -->
+
+
+## What operating system and PostgreSQL/PostGIS version are you using?
+
+<!-- Also what Linux distribution if applicable, OS version? -->
+
+
+## Tell us something about your system
+
+<!-- How much RAM do you have, how many CPUs, bare metal or cloud setup? -->
+
+
+## What did you do exactly?
+
+<!-- Please provide the command(s) you used including all options etc. Include
+links to input files. -->
+
+
+## What did you expect to happen?
+
+<!-- Describe in detail what you expected the above would do. -->
+
+
+## What did happen instead?
+
+<!-- Please describe what happened and why you think this is wrong. Please include
+(or link to, if it is too verbose) the log output. -->
+
+
+## What did you do to try analyzing the problem?
+
+<!-- Describe what steps you already did to try analyzing the problem before
+reporting. -->
+


=====================================
.github/ISSUE_TEMPLATE/config.yml
=====================================
@@ -0,0 +1,4 @@
+contact_links:
+  - name: osm2pgsql Discussions
+    url: https://github.com/openstreetmap/osm2pgsql/discussions
+    about: Ask questions, get support, share ideas and discuss with community members.


=====================================
CMakeLists.txt
=====================================
@@ -1,6 +1,6 @@
 set(PACKAGE osm2pgsql)
 set(PACKAGE_NAME osm2pgsql)
-set(PACKAGE_VERSION 1.5.0)
+set(PACKAGE_VERSION 1.5.1)
 
 cmake_minimum_required(VERSION 2.8.12)
 
@@ -190,7 +190,7 @@ endif()
 
 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src ${CMAKE_CURRENT_BINARY_DIR})
 
-find_package(Osmium 2.15.6 REQUIRED COMPONENTS io)
+find_package(Osmium 2.17.0 REQUIRED COMPONENTS io)
 include_directories(SYSTEM ${OSMIUM_INCLUDE_DIRS} ${PROTOZERO_INCLUDE_DIR} ${FMT_INCLUDE_DIR} ${RAPIDJSON_INCLUDE_DIR})
 
 if (WITH_LUA)


=====================================
README.md
=====================================
@@ -56,6 +56,15 @@ Required libraries are
 * [Python](https://python.org/) (only for running tests)
 * [Psycopg](http://initd.org/psycopg/) (only for running tests)
 
+The following libraries are included in the `contrib` directory. You can build
+with other versions of those libraries (set the `EXTERNAL_*libname*` option to
+`ON`) but make sure you are using a compatible version:
+
+* [fmt](https://fmt.dev/) (>= 7.1.3)
+* [libosmium](https://osmcode.org/libosmium/) (>= 2.17.0)
+* [protozero](https://github.com/mapbox/protozero) (>= 1.6.3)
+* [rapidjson](https://rapidjson.org/) (>= 1.1.0)
+
 It also requires access to a database server running
 [PostgreSQL](https://www.postgresql.org/) 9.5+ and
 [PostGIS](http://www.postgis.net/) 2.2+.


=====================================
debian/changelog
=====================================
@@ -1,3 +1,9 @@
+osm2pgsql (1.5.1+ds-1~exp1) experimental; urgency=medium
+
+  * New upstream release.
+
+ -- Bas Couwenberg <sebastic at debian.org>  Fri, 30 Jul 2021 06:31:09 +0200
+
 osm2pgsql (1.5.0+ds-1~exp1) experimental; urgency=medium
 
   * New upstream release.


=====================================
docs/osm2pgsql.1
=====================================
@@ -1,4 +1,4 @@
-.TH "OSM2PGSQL" "1" "1.5.0" "" ""
+.TH "OSM2PGSQL" "1" "1.5.1" "" ""
 .SH NAME
 .PP
 osm2pgsql \- Openstreetmap data to PostgreSQL converter
@@ -251,16 +251,17 @@ See documentation for details.
 .SH OUTPUT OPTIONS
 .TP
 .B \-O, \-\-output=OUTPUT
-Specifies the output back\-end to use.
+Specifies the output to use.
 Currently osm2pgsql supports \f[B]pgsql\f[], \f[B]flex\f[],
 \f[B]gazetteer\f[] and \f[B]null\f[].
-\f[B]pgsql\f[] is the default output back\-end and is optimized for
-rendering with Mapnik.
-\f[B]gazetteer\f[] is intended for geocoding with Nominatim.
-The experimental \f[B]flex\f[] backend allows more flexible
-configuration.
-\f[B]null\f[] does not write any output and is only useful for testing
-or with \f[B]\-\-slim\f[] for creating slim tables.
+\f[B]pgsql\f[] is the default output still available for backwards
+compatibility.
+New setups should use the \f[B]flex\f[] output which allows for a much
+more flexible configuration.
+The \f[B]gazetteer\f[] output is intended for geocoding with Nominatim
+only.
+The \f[B]null\f[] output does not write anything and is only useful for
+testing or with \f[B]\-\-slim\f[] for creating slim tables.
 .RS
 .RE
 .TP


=====================================
docs/osm2pgsql.md
=====================================
@@ -174,13 +174,13 @@ mandatory for short options too.
 # OUTPUT OPTIONS
 
 -O, \--output=OUTPUT
-:   Specifies the output back-end to use. Currently osm2pgsql
-    supports **pgsql**, **flex**, **gazetteer** and **null**. **pgsql** is
-    the default output back-end and is optimized for rendering with Mapnik.
-    **gazetteer** is intended for geocoding with Nominatim.
-    The experimental **flex** backend allows more flexible configuration.
-    **null** does not write any output and is only useful for testing or with
-    **\--slim** for creating slim tables.
+:   Specifies the output to use. Currently osm2pgsql supports **pgsql**,
+    **flex**, **gazetteer** and **null**. **pgsql** is
+    the default output still available for backwards compatibility. New
+    setups should use the **flex** output which allows for a much more flexible
+    configuration. The **gazetteer** output is intended for geocoding with
+    Nominatim only. The **null** output does not write anything and is only
+    useful for testing or with **\--slim** for creating slim tables.
 
 -S, \--style=FILE
 :   The style file. This specifies how the data is imported into the database,


=====================================
flex-config/README.md
=====================================
@@ -1,7 +1,5 @@
 # Flex Output Configuration
 
-**The flex output is experimental. Everything in here is subject to change.**
-
 See the [Flex Output](https://osm2pgsql.org/doc/manual.html#the-flex-output)
 chapter in the manual for all the details.
 


=====================================
src/flex-table.cpp
=====================================
@@ -108,7 +108,11 @@ flex_table_t::build_sql_create_table(table_type ttype,
         ttype == table_type::interim ? "UNLOGGED" : "", table_name);
 
     for (auto const &column : m_columns) {
-        sql += column.sql_create();
+        // create_only columns are only created in permanent, not in the
+        // interim tables
+        if (ttype == table_type::permanent || !column.create_only()) {
+            sql += column.sql_create();
+        }
     }
 
     assert(sql.back() == ',');
@@ -173,7 +177,7 @@ void table_connection_t::start(bool append)
 
     if (!append) {
         m_db_connection->exec(table().build_sql_create_table(
-            table().has_geom_column() ? flex_table_t::table_type::interim
+            table().cluster_by_geom() ? flex_table_t::table_type::interim
                                       : flex_table_t::table_type::permanent,
             table().full_name()));
 
@@ -214,8 +218,9 @@ void table_connection_t::stop(bool updateable, bool append)
         m_db_connection->exec(table().build_sql_create_table(
             flex_table_t::table_type::permanent, table().full_tmp_name()));
 
-        std::string sql = "INSERT INTO {} SELECT * FROM {}"_format(
-            table().full_tmp_name(), table().full_name());
+        std::string const columns = table().build_sql_column_list();
+        std::string sql = "INSERT INTO {} ({}) SELECT {} FROM {}"_format(
+            table().full_tmp_name(), columns, columns, table().full_name());
 
         auto const postgis_version = get_postgis_version(*m_db_connection);
 


=====================================
src/input.cpp
=====================================
@@ -21,7 +21,7 @@
 #include "osmdata.hpp"
 #include "progress-display.hpp"
 
-type_id_version check_input(type_id_version const &last, type_id_version curr)
+type_id check_input(type_id const &last, type_id curr)
 {
     if (curr.id < 0) {
         throw std::runtime_error{
@@ -40,14 +40,10 @@ type_id_version check_input(type_id_version const &last, type_id_version curr)
                     osmium::item_type_to_name(last.type), curr.id, last.id)};
         }
 
-        if (last.version < curr.version) {
-            return curr;
-        }
-
         throw std::runtime_error{
-            "Input data is not ordered: {} id {} version {} after {}."_format(
-                osmium::item_type_to_name(last.type), curr.id, curr.version,
-                last.version)};
+            "Input data is not ordered:"
+            " {} id {} appears more than once."_format(
+                osmium::item_type_to_name(last.type), curr.id)};
     }
 
     if (item_type_to_nwr_index(last.type) <=
@@ -60,10 +56,9 @@ type_id_version check_input(type_id_version const &last, type_id_version curr)
         osmium::item_type_to_name(last.type))};
 }
 
-type_id_version check_input(type_id_version const &last,
-                            osmium::OSMObject const &object)
+type_id check_input(type_id const &last, osmium::OSMObject const &object)
 {
-    return check_input(last, {object.type(), object.id(), object.version()});
+    return check_input(last, {object.type(), object.id()});
 }
 
 /**
@@ -130,7 +125,7 @@ private:
     osmium::memory::Buffer m_buffer{};
     iterator m_it{};
     iterator m_end{};
-    type_id_version m_last = {osmium::item_type::node, 0, 0};
+    type_id m_last = {osmium::item_type::node, 0};
 
 }; // class data_source_t
 
@@ -274,7 +269,7 @@ static void process_single_file(osmium::io::File const &file,
                                 progress_display_t *progress, bool append)
 {
     osmium::io::Reader reader{file};
-    type_id_version last{osmium::item_type::node, 0, 0};
+    type_id last{osmium::item_type::node, 0};
 
     input_context_t ctx{osmdata, progress, append};
     while (osmium::memory::Buffer buffer = reader.read()) {


=====================================
src/input.hpp
=====================================
@@ -28,21 +28,19 @@
 
 class osmdata_t;
 
-struct type_id_version
+struct type_id
 {
     osmium::item_type type;
     osmid_t id;
-    osmium::object_version_type version;
 };
 
 /**
- * Compare two tuples (type, id, version) throw a descriptive error if either
- * the curr id is negative or if the data is not ordered.
+ * Compare two tuples (type, id). Throw a descriptive error if either the
+ * curr id is negative or if the data is not ordered.
  */
-type_id_version check_input(type_id_version const &last, type_id_version curr);
+type_id check_input(type_id const &last, type_id curr);
 
-type_id_version check_input(type_id_version const &last,
-                            osmium::OSMObject const &object);
+type_id check_input(type_id const &last, osmium::OSMObject const &object);
 
 /**
  * Prepare input file(s). Does format checks as far as this is possible


=====================================
src/options.cpp
=====================================
@@ -682,7 +682,7 @@ void options_t::check_options()
     if (cache == 0) {
         if (!slim) {
             throw std::runtime_error{
-                "RAM node cache can only be disable in slim mode."};
+                "RAM node cache can only be disabled in slim mode."};
         }
         if (flat_node_file.empty()) {
             log_warn("RAM cache is disabled. This will likely slow down "


=====================================
src/ordered-index.cpp
=====================================
@@ -19,7 +19,8 @@ void ordered_index_t::add(osmid_t id, std::size_t offset)
             (last().offset_from + last().index.back().offset) < offset));
 
     if (need_new_2nd_level() ||
-        (id - last().from) > std::numeric_limits<uint32_t>::max()) {
+        (id - last().from) > std::numeric_limits<uint32_t>::max() ||
+        (offset - last().offset_from) >= std::numeric_limits<uint32_t>::max()) {
         if (!m_ranges.empty()) {
             m_ranges.back().to = id - 1;
         }


=====================================
src/ordered-index.hpp
=====================================
@@ -31,7 +31,7 @@
  * An index that is never used doesn't need more memory than
  * sizeof(ordered_index_t).
  *
- * All allocated memory can be freed by calling clear(). Afer that the index
+ * All allocated memory can be freed by calling clear(). After that the index
  * can NOT be reused.
  *
  * There are two ways of accessing the data through the index:
@@ -185,7 +185,7 @@ private:
 
     std::pair<osmid_t, std::size_t> get_internal(osmid_t id) const noexcept;
 
-    static constexpr std::size_t const max_block_size = 16 * 1024 * 1204;
+    static constexpr std::size_t const max_block_size = 16 * 1024 * 1024;
 
     std::vector<range_entry> m_ranges;
     std::size_t m_block_size;


=====================================
src/output-flex.cpp
=====================================
@@ -36,6 +36,8 @@ extern "C"
 #include <rapidjson/stringbuffer.h>
 #include <rapidjson/writer.h>
 
+#include <boost/filesystem.hpp>
+
 #include <cassert>
 #include <cstdlib>
 #include <cstring>
@@ -1630,6 +1632,10 @@ void output_flex_t::init_lua(std::string const &filename)
                        m_options.append ? "append" : "create");
     luaX_add_table_int(lua_state(), "stage", 1);
 
+    std::string const dir_path =
+        boost::filesystem::path{filename}.parent_path().string();
+    luaX_add_table_str(lua_state(), "config_dir", dir_path.c_str());
+
     luaX_add_table_func(lua_state(), "define_table",
                         lua_trampoline_app_define_table);
 


=====================================
src/pgsql-helper.cpp
=====================================
@@ -48,7 +48,7 @@ void create_geom_check_trigger(pg_conn_t *db_connection,
         "  END IF;\n"
         "  RETURN NULL;\n"
         "END;"
-        "$$ LANGUAGE plpgsql IMMUTABLE;"_format(func_name, geom_column));
+        "$$ LANGUAGE plpgsql;"_format(func_name, geom_column));
 
     db_connection->exec(
         "CREATE TRIGGER \"{}\""


=====================================
tests/regression.py
=====================================
@@ -50,14 +50,18 @@ def setUpModule():
     logging.info("Setting up test database")
     dbname = CONFIG['test_database']
 
-    with psycopg2.connect("dbname='template1'") as conn:
+    conn = psycopg2.connect("dbname='template1'")
+    try:
         conn.autocommit = True
         with conn.cursor() as cur:
             cur.execute('DROP DATABASE IF EXISTS "{}"'.format(dbname))
             cur.execute("CREATE DATABASE \"{}\" WITH ENCODING 'UTF8'"
                         .format(dbname))
+    finally:
+        conn.close()
 
-    with psycopg2.connect("dbname='{}'".format(dbname)) as conn:
+    conn = psycopg2.connect("dbname='{}'".format(dbname))
+    try:
         conn.autocommit = True
         with conn.cursor() as cur:
             # Check if there is a dataspace, we will skip tests otherwise.
@@ -68,17 +72,22 @@ def setUpModule():
 
             cur.execute('CREATE EXTENSION postgis')
             cur.execute('CREATE EXTENSION hstore')
+    finally:
+        conn.close()
 
 def tearDownModule():
     """ Destroy the global database.
     """
     logging.info("Cleaning up test database")
 
-    with psycopg2.connect("dbname='template1'") as conn:
+    conn = psycopg2.connect("dbname='template1'")
+    try:
+        conn.autocommit = True
         with conn.cursor() as cur:
-            conn.autocommit = True
             cur.execute('DROP DATABASE IF EXISTS "{}"'
                         .format(CONFIG['test_database']))
+    finally:
+        conn.close()
 
 ########################################################################
 #
@@ -111,13 +120,17 @@ class BaseRunner(object):
             cls.skipTest(None, "No Lua configured.")
         if 'tablespacetest' in cls.extra_params and not CONFIG['have_lua']:
             cls.skipTest(None, "Test tablespace 'tablespacetest' not configured.")
-        with psycopg2.connect("dbname='{}'".format(CONFIG['test_database'])) as conn:
+        conn = psycopg2.connect("dbname='{}'".format(CONFIG['test_database']))
+        try:
             with conn.cursor() as cur:
                 for t in ('nodes', 'ways', 'rels', 'point', 'line', 'roads', 'polygon'):
                     cur.execute("DROP TABLE IF EXISTS planet_osm_" + t)
                 cur.execute("DROP SCHEMA IF EXISTS osm CASCADE")
                 if cls.schema:
                     cur.execute("CREATE SCHEMA " + cls.schema)
+            conn.commit()
+        finally:
+            conn.close()
 
         if cls.import_file:
             cls.run_import(cls.get_def_params() + cls.extra_params,
@@ -134,8 +147,11 @@ class BaseRunner(object):
             BaseRunner.conn.close()
             BaseRunner.conn = None
         if cls.schema:
-            with psycopg2.connect("dbname='{}'".format(CONFIG['test_database'])) as conn:
+            conn = psycopg2.connect("dbname='{}'".format(CONFIG['test_database']))
+            try:
                 conn.cursor().execute("DROP SCHEMA IF EXISTS {} CASCADE".format(cls.schema))
+            finally:
+                conn.close()
         try:
             os.remove('flat.nodes')
         except (FileNotFoundError):


=====================================
tests/test-check-input.cpp
=====================================
@@ -13,27 +13,25 @@
 
 TEST_CASE("it's good if input data is ordered", "[NoDB]")
 {
-    type_id_version const tiv1{osmium::item_type::node, 1, 1};
-    type_id_version const tiv2{osmium::item_type::node, 1, 2};
-    type_id_version const tiv3{osmium::item_type::node, 2, 1};
-    type_id_version const tiv4{osmium::item_type::way, 1, 1};
-    type_id_version const tiv5{osmium::item_type::way, 2, 1};
-    type_id_version const tiv6{osmium::item_type::relation, 1, 1};
-    type_id_version const tiv7{osmium::item_type::relation, 1, 2};
+    type_id const tiv1{osmium::item_type::node, 1};
+    type_id const tiv2{osmium::item_type::node, 2};
+    type_id const tiv3{osmium::item_type::way, 1};
+    type_id const tiv4{osmium::item_type::way, 2};
+    type_id const tiv5{osmium::item_type::relation, 1};
+    type_id const tiv6{osmium::item_type::relation, 2};
 
     REQUIRE_NOTHROW(check_input(tiv1, tiv2));
     REQUIRE_NOTHROW(check_input(tiv2, tiv3));
     REQUIRE_NOTHROW(check_input(tiv3, tiv4));
     REQUIRE_NOTHROW(check_input(tiv4, tiv5));
     REQUIRE_NOTHROW(check_input(tiv5, tiv6));
-    REQUIRE_NOTHROW(check_input(tiv6, tiv7));
 }
 
 TEST_CASE("negative OSM object ids are not allowed", "[NoDB]")
 {
-    type_id_version const tivn{osmium::item_type::node, -17, 1};
-    type_id_version const tivw{osmium::item_type::way, -1, 1};
-    type_id_version const tivr{osmium::item_type::relation, -999, 17};
+    type_id const tivn{osmium::item_type::node, -17};
+    type_id const tivw{osmium::item_type::way, -1};
+    type_id const tivr{osmium::item_type::relation, -999};
 
     REQUIRE_THROWS_WITH(
         check_input(tivn, tivn),
@@ -47,8 +45,8 @@ TEST_CASE("negative OSM object ids are not allowed", "[NoDB]")
 
 TEST_CASE("objects of the same type must be ordered", "[NoDB]")
 {
-    type_id_version const tiv1{osmium::item_type::node, 42, 1};
-    type_id_version const tiv2{osmium::item_type::node, 3, 1};
+    type_id const tiv1{osmium::item_type::node, 42};
+    type_id const tiv2{osmium::item_type::node, 3};
 
     REQUIRE_THROWS_WITH(check_input(tiv1, tiv2),
                         "Input data is not ordered: node id 3 after 42.");
@@ -56,9 +54,9 @@ TEST_CASE("objects of the same type must be ordered", "[NoDB]")
 
 TEST_CASE("a node after a way or relation is not allowed", "[NoDB]")
 {
-    type_id_version const tiv1w{osmium::item_type::way, 42, 1};
-    type_id_version const tiv1r{osmium::item_type::relation, 42, 1};
-    type_id_version const tiv2{osmium::item_type::node, 100, 1};
+    type_id const tiv1w{osmium::item_type::way, 42};
+    type_id const tiv1r{osmium::item_type::relation, 42};
+    type_id const tiv2{osmium::item_type::node, 100};
 
     REQUIRE_THROWS_WITH(check_input(tiv1w, tiv2),
                         "Input data is not ordered: node after way.");
@@ -68,20 +66,20 @@ TEST_CASE("a node after a way or relation is not allowed", "[NoDB]")
 
 TEST_CASE("a way after a relation is not allowed", "[NoDB]")
 {
-    type_id_version const tiv1{osmium::item_type::relation, 42, 1};
-    type_id_version const tiv2{osmium::item_type::way, 100, 1};
+    type_id const tiv1{osmium::item_type::relation, 42};
+    type_id const tiv2{osmium::item_type::way, 100};
 
     REQUIRE_THROWS_WITH(check_input(tiv1, tiv2),
                         "Input data is not ordered: way after relation.");
 }
 
-TEST_CASE("versions must be ordered", "[NoDB]")
+TEST_CASE("no object may appear twice", "[NoDB]")
 {
-    type_id_version const tiv1{osmium::item_type::way, 42, 2};
-    type_id_version const tiv2{osmium::item_type::way, 42, 1};
+    type_id const tiv1{osmium::item_type::way, 42};
+    type_id const tiv2{osmium::item_type::way, 42};
 
     REQUIRE_THROWS_WITH(
         check_input(tiv1, tiv2),
-        "Input data is not ordered: way id 42 version 1 after 2.");
+        "Input data is not ordered: way id 42 appears more than once.");
 }
 



View it on GitLab: https://salsa.debian.org/debian-gis-team/osm2pgsql/-/compare/064a9b45fcd0fa9af71e52d53d52e9554fbb2767...f4282b5918329f4eb9ed8838fa883bfd028d917e

-- 
View it on GitLab: https://salsa.debian.org/debian-gis-team/osm2pgsql/-/compare/064a9b45fcd0fa9af71e52d53d52e9554fbb2767...f4282b5918329f4eb9ed8838fa883bfd028d917e
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20210730/c2fe8797/attachment-0001.htm>


More information about the Pkg-grass-devel mailing list