[osm2pgsql] 01/01: Imported Upstream version 0.88.0~rc1

Sebastiaan Couwenberg sebastic at moszumanska.debian.org
Sat Jul 11 09:02:52 UTC 2015


This is an automated email from the git hooks/post-receive script.

sebastic pushed a commit to branch upstream
in repository osm2pgsql.

commit 0a8b63fb27b8084adec80eb729bbb98ad7628a21
Author: Bas Couwenberg <sebastic at xs4all.nl>
Date:   Sat Jul 11 10:20:57 2015 +0200

    Imported Upstream version 0.88.0~rc1
---
 .gitignore                                    |   5 +-
 AUTHORS                                       |   4 +-
 CONTRIBUTING.md                               | 102 ++++++++++++++
 Makefile.am                                   |  15 ++-
 README.md                                     |  25 +---
 configure.ac                                  |   2 +-
 docs/migrations.md                            |  15 +++
 geometry-builder.cpp                          |  57 ++++----
 node-persistent-cache.cpp                     |  89 ++++++------
 node-ram-cache.cpp                            |  53 +++++---
 node-ram-cache.hpp                            |  21 ++-
 options.cpp                                   |  14 +-
 pgsql-id-tracker.cpp                          | 153 ---------------------
 pgsql-id-tracker.hpp                          |  31 -----
 tagtransform.cpp                              |  52 ++++---
 tests/common-pg.cpp                           |  35 ++---
 tests/common-pg.hpp                           |   2 +
 tests/regression-test.py                      |   4 +-
 tests/test-middle-flat.cpp                    | 112 ++++++++++++++++
 tests/test-output-multi-line.cpp              |   2 -
 tests/test-output-multi-point-multi-table.cpp |   2 -
 tests/test-output-multi-point.cpp             |   2 -
 tests/test-output-multi-polygon.cpp           |   2 -
 tests/test-output-pgsql-tablespace.cpp        | 146 ++++++++++++++++++++
 tests/test-output-pgsql-z_order.cpp           | 186 ++++++++++++++++++++++++++
 tests/test-output-pgsql.cpp                   | 103 +++++++++++++-
 tests/test_output_pgsql_z_order.osm           |  75 +++++++++++
 27 files changed, 938 insertions(+), 371 deletions(-)

diff --git a/.gitignore b/.gitignore
index 6ca8a4d..a4cd9b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -45,6 +45,7 @@ tests/.dirstamp
 tests/test-parse-xml2
 tests/test-middle-ram
 tests/test-middle-pgsql
+tests/test-middle-flat
 tests/test-pgsql-escape
 tests/test-parse-options
 tests/test-output-multi-tags
@@ -55,10 +56,12 @@ tests/test-output-multi-point-multi-table
 tests/test-output-multi-polygon
 tests/test-output-multi-poly-trivial
 tests/test-output-pgsql
+tests/test-output-pgsql-tablespace
+tests/test-output-pgsql-z_order
 tests/test-expire-tiles
 tests/*.log
 tests/*.trs
-tests/test_output_pgsql_area_way.flat.nodes.bin
+tests/*.flat.nodes.bin
 
 .libs/
 *.lo
diff --git a/AUTHORS b/AUTHORS
index af8498b..0f6bbfa 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -1,3 +1,3 @@
 osm2pgsql was written by Jon Burgess, Artem Pavlenko,  Martijn van Oosterhout
-Sarah Hoffman, Kai Krueger, Frederik Ramm, Brian Quinion and other
-OpenStreetMap project members.
\ No newline at end of file
+Sarah Hoffmann, Kai Krueger, Frederik Ramm, Brian Quinion, Matt Amos,
+Kevin Kreiser, Paul Norman and other OpenStreetMap project members.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..27bb5ca
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,102 @@
+# Osm2pgsql contribution guidelines
+
+## Workflow
+
+We operate the "Fork & Pull" model explained at
+
+https://help.github.com/articles/using-pull-requests
+
+You should fork the project into your own repo, create a topic branch
+there and then make one or more pull requests back to the openstreetmap repository.
+Your pull requests will then be reviewed and discussed.
+
+## History
+
+To understand the osm2pgsql code, it helps to know some history on it. Osm2pgsql
+was written in C in 2007 as a port of an older Python utility. In 2014 it was 
+ported to C++ by MapQuest and the last C version was released as 0.86.0. In it's
+time, it has had varying contribution activity, including times with no 
+maintainer or active developers.
+
+Parts of the codebase still clearly show their C origin and could use rewriting
+in modern C++, making use of data structures in the standard library.
+
+## Versioning
+
+Osm2pgsql uses a X.Y.Z version number, where Y tells you if you are on a stable
+or development series. Like the Linux Kernel, even numbers are stable and 
+development versions are odd.
+
+Old branches and versions are not generally maintained.
+
+## Code style
+
+The current codebase is a mix of styles, but new code should be written in the 
+[K&R 1TBS style](https://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS) with
+4 spaces indentation. Tabs should never be used in the C++ code.
+
+e.g.
+
+```
+int main(int argc, char *argv[])
+{
+    ...
+    while (x == y) {
+        something();
+        somethingelse();
+
+        if (some_error) {
+            do_correct();
+        } else {
+            continue_as_usual();
+        }
+    }
+
+    finalthing();
+    ...
+}
+```
+
+Names should use underscores, not camel case, with class/struct names ending in `_t`.
+
+## Platforms targeted
+
+Ideally osm2pgsql should compile on Linux, OS X, FreeBSD and Windows. It is
+actively tested on Debian, Ubuntu and FreeBSD by the maintainers.
+
+## Testing
+
+The code also comes with a suite of tests which can be run by
+executing ``make check``.
+
+Most of these tests depend on being able to set up a database and run osm2pgsql
+against it. You need to ensure that PostgreSQL is running and that your user is
+a superuser of that system. To do that, run:
+
+```sh
+sudo -u postgres createuser -s $USER
+sudo mkdir -p /tmp/psql-tablespace
+sudo chown postgres.postgres /tmp/psql-tablespace
+psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres
+```
+
+Once this is all set up, all the tests should run (no SKIPs), and pass
+(no FAILs). If you encounter a failure, you can find more information
+by looking in the `test-suite.log`. If you find something which seems
+to be a bug, please check to see if it is a known issue at
+https://github.com/openstreetmap/osm2pgsql/issues and, if it's not
+already known, report it there.
+
+If running the tests in a virtual machine, allocate sufficient disk space for a
+20GB flat nodes file.
+
+### Performance Testing
+
+If performance testing with a full planet import is required, indicate what
+needs testing in a pull request.
+
+## Maintainers
+
+The current maintainers of osm2pgsql are [Sarah Hoffmann](https://github.com/lonvia/)
+and [Paul Norman](https://github.com/pnorman/). Sarah has more experience with 
+the gazetteer backend and Paul with the pgsql and multi backends.
diff --git a/Makefile.am b/Makefile.am
index 8ce3f4f..c91d408 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -32,7 +32,6 @@ libosm2pgsql_la_SOURCES = \
 	parse-pbf.cpp \
 	parse-xml2.cpp \
 	pgsql.cpp \
-	pgsql-id-tracker.cpp \
 	processor-line.cpp \
 	processor-point.cpp \
 	processor-polygon.cpp \
@@ -51,6 +50,7 @@ check_PROGRAMS = \
 	tests/test-parse-xml2 \
 	tests/test-middle-ram \
 	tests/test-middle-pgsql \
+	tests/test-middle-flat \
 	tests/test-output-multi-tags \
 	tests/test-output-multi-line \
 	tests/test-output-multi-line-storage \
@@ -59,6 +59,8 @@ check_PROGRAMS = \
 	tests/test-output-multi-polygon \
 	tests/test-output-multi-poly-trivial \
 	tests/test-output-pgsql \
+	tests/test-output-pgsql-z_order \
+	tests/test-output-pgsql-tablespace \
 	tests/test-pgsql-escape \
 	tests/test-parse-options \
 	tests/test-expire-tiles
@@ -69,6 +71,8 @@ tests_test_middle_ram_SOURCES = tests/test-middle-ram.cpp tests/middle-tests.cpp
 tests_test_middle_ram_LDADD = libosm2pgsql.la
 tests_test_middle_pgsql_SOURCES = tests/test-middle-pgsql.cpp tests/middle-tests.cpp tests/common-pg.cpp
 tests_test_middle_pgsql_LDADD = libosm2pgsql.la
+tests_test_middle_flat_SOURCES = tests/test-middle-flat.cpp tests/middle-tests.cpp tests/common-pg.cpp
+tests_test_middle_flat_LDADD = libosm2pgsql.la
 tests_test_output_multi_tags_SOURCES = tests/test-output-multi-tags.cpp tests/common-pg.cpp
 tests_test_output_multi_tags_LDADD = libosm2pgsql.la
 tests_test_output_multi_line_SOURCES = tests/test-output-multi-line.cpp tests/common-pg.cpp
@@ -85,6 +89,10 @@ tests_test_output_multi_poly_trivial_SOURCES = tests/test-output-multi-poly-triv
 tests_test_output_multi_poly_trivial_LDADD = libosm2pgsql.la
 tests_test_output_pgsql_SOURCES = tests/test-output-pgsql.cpp tests/common-pg.cpp
 tests_test_output_pgsql_LDADD = libosm2pgsql.la
+tests_test_output_pgsql_tablespace_SOURCES = tests/test-output-pgsql-tablespace.cpp tests/common-pg.cpp
+tests_test_output_pgsql_tablespace_LDADD = libosm2pgsql.la
+tests_test_output_pgsql_z_order_SOURCES = tests/test-output-pgsql-z_order.cpp tests/common-pg.cpp
+tests_test_output_pgsql_z_order_LDADD = libosm2pgsql.la
 tests_test_pgsql_escape_SOURCES = tests/test-pgsql-escape.cpp
 tests_test_pgsql_escape_LDADD = libosm2pgsql.la
 tests_test_parse_options_SOURCES = tests/test-parse-options.cpp
@@ -92,6 +100,8 @@ tests_test_parse_options_LDADD = libosm2pgsql.la
 tests_test_expire_tiles_SOURCES = tests/test-expire-tiles.cpp
 tests_test_expire_tiles_LDADD = libosm2pgsql.la
 
+MOSTLYCLEANFILES = tests/test_middle_flat.flat.nodes.bin tests/test_output_pgsql_area_way.flat.nodes.bin
+
 TESTS = $(check_PROGRAMS) tests/regression-test.sh
 TEST_EXTENSIONS = .sh
 SH_LOG_COMPILER = sh
@@ -138,6 +148,7 @@ osm2pgsql_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_parse_xml2_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_middle_ram_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_middle_pgsql_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_middle_flat_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_multi_tags_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_multi_line_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_multi_line_storage_LDADD += $(GLOBAL_LDFLAGS)
@@ -146,6 +157,8 @@ tests_test_output_multi_point_multi_table_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_multi_polygon_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_multi_poly_trivial_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_output_pgsql_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_output_pgsql_tablespace_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_output_pgsql_z_order_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_pgsql_escape_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_parse_options_LDADD += $(GLOBAL_LDFLAGS)
 tests_test_expire_tiles_LDADD += $(GLOBAL_LDFLAGS)
diff --git a/README.md b/README.md
index f6bdd68..b601b8c 100644
--- a/README.md
+++ b/README.md
@@ -143,33 +143,12 @@ postgres tables instead of those provided in the pgsql backend.
 Any questions should be directed at the osm dev list
 http://wiki.openstreetmap.org/index.php/Mailing_lists
 
-## Testing ##
-
-The code also comes with a suite of tests which can be run by
-executing ``make check``.
-
-Some of these tests depend on being able to set up a database and run
-osm2pgsql against it. You need to ensure that PostgreSQL is running
-and that your user is a superuser of that system. To do that, run:
-
-```sh
-sudo -u postgres createuser -s $USER
-sudo mkdir -p /tmp/psql-tablespace
-sudo chown postgres.postgres /tmp/psql-tablespace
-psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres
-```
-
-Once this is all set up, all the tests should run (no SKIPs), and pass
-(no FAILs). If you encounter a failure, you can find more information
-by looking in the `test-suite.log`. If you find something which seems
-to be a bug, please check to see if it is a known issue at
-https://github.com/openstreetmap/osm2pgsql/issues and, if it's not
-already known, report it there.
-
 ## Contributing ##
 
 We welcome contributions to osm2pgsql. If you would like to report an issue,
 please use the [issue tracker on GitHub](https://github.com/openstreetmap/osm2pgsql/issues).
 
+More information can be found in [CONTRIBUTING.md](CONTRIBUTING.md).
+
 General queries can be sent to the tile-serving@ or dev@
 [mailing lists](http://wiki.openstreetmap.org/wiki/Mailing_lists).
diff --git a/configure.ac b/configure.ac
index 278de33..a545f9d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,5 +1,5 @@
 dnl Process this file with autoconf to produce a configure script.
-AC_INIT(osm2pgsql, 0.87.4)
+AC_INIT(osm2pgsql, 0.88.0-RC1)
 
 dnl Required autoconf version
 AC_PREREQ(2.61)
diff --git a/docs/migrations.md b/docs/migrations.md
index 10457c8..af1c70a 100644
--- a/docs/migrations.md
+++ b/docs/migrations.md
@@ -4,6 +4,21 @@ Some osm2pgsql changes have slightly changed the database schema it expects. If
 updating an old database, a migration may be needed. The migrations here assume
 the default `planet_osm` prefix.
 
+## 0.88.0 z_order changes ##
+
+0.88.0 z_order logic was changed, requuiring an increase in z_order values. To
+migrate to the new range of values, run
+
+```sql
+UPDATE planet_osm_line SET z_order = z_order * 10;
+UPDATE planet_osm_roads SET z_order = z_order * 10;
+```
+
+This will not apply the new logic, but will get the existing z_orders in the right
+group of 100 for the new logic.
+
+If not using osm2pgsql z_orders, this change may be ignored.
+
 ## 0.87.0 pending removal ##
 
 0.87.0 moved the in-database tracking of pending ways and relations to
diff --git a/geometry-builder.cpp b/geometry-builder.cpp
index 143a1cc..5c4a6b5 100644
--- a/geometry-builder.cpp
+++ b/geometry-builder.cpp
@@ -81,9 +81,9 @@ struct polygondata
 };
 
 struct polygondata_comparearea {
-  bool operator()(const polygondata& lhs, const polygondata& rhs) {
-    return lhs.area > rhs.area;
-  }
+    bool operator()(const polygondata& lhs, const polygondata& rhs) {
+        return lhs.area > rhs.area;
+    }
 };
 
 } // anonymous namespace
@@ -190,25 +190,25 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t
                 // length of the line in `segment` over the `split_at` distance.
 
                 if (distance + delta > split_at) {
-                  const size_t splits = std::floor((distance + delta) / split_at);
-                  // use the splitting distance to split the current segment up
-                  // into as many parts as necessary to keep each part below
-                  // the `split_at` distance.
-                  for (size_t i = 0; i < splits; ++i) {
-                    double frac = (double(i + 1) * split_at - distance) / delta;
-                    const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x,
-                                                  frac * (this_pt.y - prev_pt.y) + prev_pt.y);
-                    segment->add(interpolated);
-                    geom_ptr geom = geom_ptr(gf.createLineString(segment.release()));
+                    const size_t splits = std::floor((distance + delta) / split_at);
+                    // use the splitting distance to split the current segment up
+                    // into as many parts as necessary to keep each part below
+                    // the `split_at` distance.
+                    for (size_t i = 0; i < splits; ++i) {
+                        double frac = (double(i + 1) * split_at - distance) / delta;
+                        const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x,
+                                                      frac * (this_pt.y - prev_pt.y) + prev_pt.y);
+                        segment->add(interpolated);
+                        geom_ptr geom = geom_ptr(gf.createLineString(segment.release()));
 
-                    //copy of an empty one should be cheapest
-                    wkts->push_back(geometry_builder::wkt_t());
-                    //then we set on the one we already have
-                    wkts->back().geom = writer.write(geom.get());
-                    wkts->back().area = 0;
+                        //copy of an empty one should be cheapest
+                        wkts->push_back(geometry_builder::wkt_t());
+                        //then we set on the one we already have
+                        wkts->back().geom = writer.write(geom.get());
+                        wkts->back().area = 0;
 
-                    segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
-                    segment->add(interpolated);
+                        segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
+                        segment->add(interpolated);
                   }
                   // reset the distance based on the final splitting point for
                   // the next iteration.
@@ -255,12 +255,12 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t
 }
 
 int geometry_builder::parse_wkt(const char * wkt, multinodelist_t &nodes, int *polygon) {
-    GeometryFactory		gf;
-    WKTReader		reader(&gf);
-    std::string		wkt_string(wkt);
-    GeometryCollection *	gc;
-    CoordinateSequence *	coords;
-    size_t			num_geometries;
+    GeometryFactory gf;
+    WKTReader reader(&gf);
+    std::string wkt_string(wkt);
+    GeometryCollection * gc;
+    CoordinateSequence * coords;
+    size_t num_geometries;
 
     *polygon = 0;
     try {
@@ -320,7 +320,6 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel
 
     maybe_wkts_t wkts(new std::vector<geometry_builder::wkt_t>);
 
-
     try
     {
         for (multinodelist_t::const_iterator it = xnodes.begin(); it != xnodes.end(); ++it) {
@@ -422,7 +421,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel
                         }
                     }
                 }
-              pgf.destroy(preparedtoplevelpolygon);
+                pgf.destroy(preparedtoplevelpolygon);
             }
             // polys now is a list of polygons tagged with which ones are inside each other
 
@@ -608,7 +607,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_both(const multinodelist_
             }
             else
             {
-                        //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl;
+                //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl;
                 double distance = 0;
                 std::auto_ptr<CoordinateSequence> segment;
                 segment = std::auto_ptr<CoordinateSequence>(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
diff --git a/node-persistent-cache.cpp b/node-persistent-cache.cpp
index 61a4804..3689ce7 100644
--- a/node-persistent-cache.cpp
+++ b/node-persistent-cache.cpp
@@ -49,10 +49,10 @@ void node_persistent_cache::writeout_dirty_nodes()
 {
     for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
     {
-        if (readNodeBlockCache[i].dirty)
+        if (readNodeBlockCache[i].dirty())
         {
             if (lseek64(node_cache_fd,
-                    (readNodeBlockCache[i].block_offset
+                    ((osmid_t) readNodeBlockCache[i].block_offset
                             << READ_NODE_BLOCK_SHIFT)
                             * sizeof(ramNode)
                             + sizeof(persistentCacheHeader),
@@ -70,7 +70,7 @@ void node_persistent_cache::writeout_dirty_nodes()
                 util::exit_nicely();
             }
         }
-        readNodeBlockCache[i].dirty = 0;
+        readNodeBlockCache[i].reset_used();
     }
 }
 
@@ -85,9 +85,9 @@ int node_persistent_cache::replace_block()
 
     for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
     {
-        if (readNodeBlockCache[i].used < min_used)
+        if (readNodeBlockCache[i].used() < min_used)
         {
-            min_used = readNodeBlockCache[i].used;
+            min_used = readNodeBlockCache[i].used();
             block_id = i;
         }
     }
@@ -95,9 +95,9 @@ int node_persistent_cache::replace_block()
     {
         for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
         {
-            if (readNodeBlockCache[i].used > 1)
+            if (readNodeBlockCache[i].used() > 1)
             {
-                readNodeBlockCache[i].used--;
+                readNodeBlockCache[i].dec_used();
             }
         }
     }
@@ -214,10 +214,10 @@ int node_persistent_cache::load_block(osmid_t block_offset)
 {
     const int block_id = replace_block();
 
-    if (readNodeBlockCache[block_id].dirty)
+    if (readNodeBlockCache[block_id].dirty())
     {
         if (lseek64(node_cache_fd,
-                (readNodeBlockCache[block_id].block_offset
+                ((osmid_t) readNodeBlockCache[block_id].block_offset
                         << READ_NODE_BLOCK_SHIFT) * sizeof(ramNode)
                     + sizeof(struct persistentCacheHeader), SEEK_SET) < 0) {
             fprintf(stderr, "Failed to seek to correct position in node cache: %s\n",
@@ -232,13 +232,21 @@ int node_persistent_cache::load_block(osmid_t block_offset)
                     strerror(errno));
             util::exit_nicely();
         }
-        readNodeBlockCache[block_id].dirty = 0;
+        readNodeBlockCache[block_id].reset_used();
     }
 
-    remove_from_cache_idx(readNodeBlockCache[block_id].block_offset);
-    new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE];
+    if (readNodeBlockCache[block_id].nodes) {
+        remove_from_cache_idx((osmid_t) readNodeBlockCache[block_id].block_offset);
+        new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE];
+    } else {
+        readNodeBlockCache[block_id].nodes = new ramNode[READ_NODE_BLOCK_SIZE];
+        if (!readNodeBlockCache[block_id].nodes) {
+            fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
+            util::exit_nicely();
+        }
+    }
     readNodeBlockCache[block_id].block_offset = block_offset;
-    readNodeBlockCache[block_id].used = READ_NODE_CACHE_SIZE;
+    readNodeBlockCache[block_id].set_used(READ_NODE_CACHE_SIZE);
 
     /* Make sure the node cache is correctly initialised for the block that will be read */
     if (cacheHeader.max_initialised_id
@@ -289,7 +297,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
      * node cache file in buffer cache therefore duplicates the data wasting 16GB of ram.
      * Therefore tell the OS not to cache the node-persistent-cache during initial import.
      * */
-    if (sync_file_range(node_cache_fd, writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+    if (sync_file_range(node_cache_fd, (osmid_t) writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
                         sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode),
                         SYNC_FILE_RANGE_WRITE) < 0) {
         fprintf(stderr, "Info: Sync_file_range writeout has an issue. This shouldn't be anything to worry about.: %s\n",
@@ -297,7 +305,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
     };
 
     if (writeNodeBlock.block_offset > 16) {
-        if(sync_file_range(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+        if(sync_file_range(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
                            sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode),
                             SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER) < 0) {
             fprintf(stderr, "Info: Sync_file_range block has an issue. This shouldn't be anything to worry about.: %s\n",
@@ -305,7 +313,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
 
         }
 #ifdef HAVE_POSIX_FADVISE
-        if (posix_fadvise(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+        if (posix_fadvise(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
                           sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode), POSIX_FADV_DONTNEED) !=0 ) {
             fprintf(stderr, "Info: Posix_fadvise failed. This shouldn't be anything to worry about.: %s\n",
                 strerror(errno));
@@ -320,23 +328,22 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon)
     assert(!append_mode);
     assert(!read_mode);
 
-    osmid_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT;
+    int32_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT;
 
     if (writeNodeBlock.block_offset != block_offset)
     {
-        if (writeNodeBlock.dirty)
+        if (writeNodeBlock.dirty())
         {
             nodes_set_create_writeout_block();
-            writeNodeBlock.dirty = 0;
             /* After writing out the node block, the file pointer is at the next block level */
             writeNodeBlock.block_offset++;
-            cacheHeader.max_initialised_id = (writeNodeBlock.block_offset
+            cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset
                     << WRITE_NODE_BLOCK_SHIFT) - 1;
         }
         if (writeNodeBlock.block_offset > block_offset)
         {
             fprintf(stderr,
-                    "ERROR: Block_offset not in sequential order: %" PRIdOSMID "%" PRIdOSMID "\n",
+                    "ERROR: Block_offset not in sequential order: %d %d\n",
                     writeNodeBlock.block_offset, block_offset);
             util::exit_nicely();
         }
@@ -353,7 +360,7 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon)
     }
 
     writeNodeBlock.nodes[id & WRITE_NODE_BLOCK_MASK] = ramNode(lon, lat);
-    writeNodeBlock.dirty = 1;
+    writeNodeBlock.set_dirty();
 
     return 0;
 }
@@ -373,8 +380,8 @@ int node_persistent_cache::set_append(osmid_t id, double lat, double lon)
         readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode();
     else
         readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode(lon, lat);
-    readNodeBlockCache[block_id].used++;
-    readNodeBlockCache[block_id].dirty = 1;
+    readNodeBlockCache[block_id].inc_used();
+    readNodeBlockCache[block_id].set_dirty();
 
     return 1;
 }
@@ -399,7 +406,7 @@ int node_persistent_cache::get(osmNode *out, osmid_t id)
         block_id = load_block(block_offset);
     }
 
-    readNodeBlockCache[block_id].used++;
+    readNodeBlockCache[block_id].inc_used();
 
     if (!readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK].is_valid())
         return 1;
@@ -451,13 +458,12 @@ void node_persistent_cache::set_read_mode()
     if (read_mode)
         return;
 
-    if (writeNodeBlock.dirty > 0) {
+    if (writeNodeBlock.dirty()) {
         assert(!append_mode);
-        fprintf(stderr, "Switching to read mode\n");
         nodes_set_create_writeout_block();
-        writeNodeBlock.dirty = 0;
+        writeNodeBlock.reset_used();
         writeNodeBlock.block_offset++;
-        cacheHeader.max_initialised_id = (writeNodeBlock.block_offset
+        cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset
                 << WRITE_NODE_BLOCK_SHIFT) - 1;
 
         /* write out the header */
@@ -475,8 +481,6 @@ void node_persistent_cache::set_read_mode()
     }
 
     read_mode = true;
-
-    fprintf(stderr, "Switching to read mode done\n");
 }
 
 node_persistent_cache::node_persistent_cache(const options_t *options, int append,
@@ -484,7 +488,6 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
     : node_cache_fd(0), node_cache_fname(NULL), append_mode(0), cacheHeader(),
       writeNodeBlock(), readNodeBlockCache(NULL), read_mode(ro), ram_cache(ptr)
 {
-    int i, err;
     append_mode = append;
     if (options->flat_node_file) {
         node_cache_fname = options->flat_node_file->c_str();
@@ -533,12 +536,10 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
         };
 
         writeNodeBlock.block_offset = 0;
-        writeNodeBlock.dirty = 0;
-        writeNodeBlock.nodes = 0;
 
         if (!read_mode)
         {
-
+            int err;
             #ifdef HAVE_POSIX_FALLOCATE
             if ((err = posix_fallocate(node_cache_fd, 0,
                     sizeof(ramNode) * MAXIMUM_INITIAL_ID)) != 0)
@@ -606,27 +607,16 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
 
     fprintf(stderr,"Maximum node in persistent node cache: %" PRIdOSMID "\n", cacheHeader.max_initialised_id);
 
-    readNodeBlockCache = new ramNodeBlock [READ_NODE_CACHE_SIZE];
+    readNodeBlockCache = new ramNodeBlock[READ_NODE_CACHE_SIZE];
     if (!readNodeBlockCache) {
         fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
         util::exit_nicely();
     }
-    for (i = 0; i < READ_NODE_CACHE_SIZE; i++)
-    {
-        readNodeBlockCache[i].nodes = new ramNode[READ_NODE_BLOCK_SIZE];
-        if (!readNodeBlockCache[i].nodes) {
-            fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
-            util::exit_nicely();
-        }
-        readNodeBlockCache[i].block_offset = -1;
-        readNodeBlockCache[i].used = 0;
-        readNodeBlockCache[i].dirty = 0;
-    }
 }
 
 node_persistent_cache::~node_persistent_cache()
 {
-    if (writeNodeBlock.dirty > 0)
+    if (writeNodeBlock.dirty())
         nodes_set_create_writeout_block();
 
     writeout_dirty_nodes();
@@ -659,7 +649,8 @@ node_persistent_cache::~node_persistent_cache()
     if (readNodeBlockCache) {
         for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
         {
-            delete[] readNodeBlockCache[i].nodes;
+            if (readNodeBlockCache[i].nodes)
+                delete[] readNodeBlockCache[i].nodes;
         }
         delete[] readNodeBlockCache;
     }
diff --git a/node-ram-cache.cpp b/node-ram-cache.cpp
index 167f6b7..23f8d24 100644
--- a/node-ram-cache.cpp
+++ b/node-ram-cache.cpp
@@ -60,7 +60,7 @@
 
 int ramNode::scale;
 
-static int id2block(osmid_t id)
+static int32_t id2block(osmid_t id)
 {
     /* + NUM_BLOCKS/2 allows for negative IDs */
     return (id >> BLOCK_SHIFT) + NUM_BLOCKS/2;
@@ -71,7 +71,7 @@ static int id2offset(osmid_t id)
     return id & (PER_BLOCK-1);
 }
 
-static osmid_t block2id(int block, int offset)
+static osmid_t block2id(int32_t block, int offset)
 {
     return (((osmid_t) block - NUM_BLOCKS/2) << BLOCK_SHIFT) + (osmid_t) offset;
 }
@@ -84,7 +84,7 @@ void node_ram_cache::percolate_up( int pos )
     while( i > 0 )
     {
       int parent = (i-1)>>1;
-      if( queue[i]->used < queue[parent]->used )
+      if( queue[i]->used() < queue[parent]->used() )
       {
         Swap( queue[i], queue[parent] )
         i = parent;
@@ -108,7 +108,11 @@ ramNode *node_ram_cache::next_chunk() {
 
 
 int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
-    if ((sizeSparseTuples > maxSparseTuples) || ( cacheUsed > cacheSize)) {
+    // Sparse cache depends on ordered nodes, reject out-of-order ids.
+    // Also check that there is still space.
+    if ((maxSparseId && id < maxSparseId)
+        || (sizeSparseTuples > maxSparseTuples)
+        || ( cacheUsed > cacheSize)) {
         if ((allocStrategy & ALLOC_LOSSY) > 0)
             return 1;
         else {
@@ -116,6 +120,7 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
             util::exit_nicely();
         }
     }
+    maxSparseId = id;
     sparseBlock[sizeSparseTuples].id = id;
     sparseBlock[sizeSparseTuples].coord = coord;
 
@@ -126,9 +131,8 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
 }
 
 int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
-    int block  = id2block(id);
-    int offset = id2offset(id);
-    int i = 0;
+    int32_t const block  = id2block(id);
+    int const offset = id2offset(id);
 
     if (maxBlocks == 0) return 1;
 
@@ -143,7 +147,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
                  * to store it in dense representation. If not, push all elements of the block
                  * to the sparse node cache and reuse memory of the previous block for the current block */
                 if ( ((allocStrategy & ALLOC_SPARSE) == 0) ||
-                     ((queue[usedBlocks - 1]->used / (double)(1<< BLOCK_SHIFT)) >
+                     ((queue[usedBlocks - 1]->used() / (double)(1<< BLOCK_SHIFT)) >
                       (sizeof(ramNode) / (double)sizeof(ramNodeID)))) {
                     /* Block has reached the level to keep it in dense representation */
                     /* We've just finished with the previous block, so we need to percolate it up the queue to its correct position */
@@ -152,7 +156,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
                     blocks[block].nodes = next_chunk();
                 } else {
                     /* previous block was not dense enough, so push it into the sparse node cache instead */
-                    for (i = 0; i < (1 << BLOCK_SHIFT); i++) {
+                    for (int i = 0; i < (1 << BLOCK_SHIFT); i++) {
                         if (queue[usedBlocks -1]->nodes[i].is_valid()) {
                             set_sparse(block2id(queue[usedBlocks - 1]->block_offset, i),
                                        queue[usedBlocks -1]->nodes[i]);
@@ -160,7 +164,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
                         }
                     }
                     /* reuse previous block, as its content is now in the sparse representation */
-                    storedNodes -= queue[usedBlocks - 1]->used;
+                    storedNodes -= queue[usedBlocks - 1]->used();
                     blocks[block].nodes = queue[usedBlocks - 1]->nodes;
                     blocks[queue[usedBlocks - 1]->block_offset].nodes = NULL;
                     usedBlocks--;
@@ -170,7 +174,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
                 blocks[block].nodes = next_chunk();
             }
 
-            blocks[block].used = 0;
+            blocks[block].reset_used();
             blocks[block].block_offset = block;
             if (!blocks[block].nodes) {
                 fprintf(stderr, "Error allocating nodes\n");
@@ -194,17 +198,17 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
              * current head of the tree down to the right level to restore the
              * priority queue invariant. Upto log(maxBlocks) iterations */
 
-            i=0;
+            int i = 0;
             while( 2*i+1 < usedBlocks - 1 ) {
-                if( queue[2*i+1]->used <= queue[2*i+2]->used ) {
-                    if( queue[i]->used > queue[2*i+1]->used ) {
+                if( queue[2*i+1]->used() <= queue[2*i+2]->used() ) {
+                    if( queue[i]->used() > queue[2*i+1]->used() ) {
                         Swap( queue[i], queue[2*i+1] );
                         i = 2*i+1;
                     }
                     else
                         break;
                 } else {
-                    if( queue[i]->used > queue[2*i+2]->used ) {
+                    if( queue[i]->used() > queue[2*i+2]->used() ) {
                         Swap( queue[i], queue[2*i+2] );
                         i = 2*i+2;
                     } else
@@ -213,13 +217,13 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
             }
             /* Now the head of the queue is the smallest, so it becomes our replacement candidate */
             blocks[block].nodes = queue[0]->nodes;
-            blocks[block].used = 0;
+            blocks[block].reset_used();
             new(blocks[block].nodes) ramNode[PER_BLOCK];
 
             /* Clear old head block and point to new block */
-            storedNodes -= queue[0]->used;
+            storedNodes -= queue[0]->used();
             queue[0]->nodes = NULL;
-            queue[0]->used = 0;
+            queue[0]->reset_used();
             queue[0] = &blocks[block];
         }
     } else {
@@ -243,7 +247,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
     }
 
     blocks[block].nodes[offset] = coord;
-    blocks[block].used++;
+    blocks[block].inc_used();
     storedNodes++;
     return 0;
 }
@@ -275,8 +279,8 @@ int node_ram_cache::get_sparse(osmNode *out, osmid_t id) {
 }
 
 int node_ram_cache::get_dense(osmNode *out, osmid_t id) {
-    int block  = id2block(id);
-    int offset = id2offset(id);
+    int32_t const block  = id2block(id);
+    int const offset = id2offset(id);
 
     if (!blocks[block].nodes)
         return 1;
@@ -294,7 +298,7 @@ int node_ram_cache::get_dense(osmNode *out, osmid_t id) {
 node_ram_cache::node_ram_cache( int strategy, int cacheSizeMB, int fixpointscale )
     : allocStrategy(ALLOC_DENSE), blocks(NULL), usedBlocks(0),
       maxBlocks(0), blockCache(NULL), queue(NULL), sparseBlock(NULL),
-      maxSparseTuples(0), sizeSparseTuples(0), cacheUsed(0),
+      maxSparseTuples(0), sizeSparseTuples(0), maxSparseId(0), cacheUsed(0),
       cacheSize(0), storedNodes(0), totalNodes(0), nodesCacheHits(0),
       nodesCacheLookups(0), warn_node_order(0) {
 
@@ -390,6 +394,11 @@ node_ram_cache::~node_ram_cache() {
 }
 
 int node_ram_cache::set(osmid_t id, double lat, double lon, const taglist_t &) {
+    if ((id > 0 && id >> BLOCK_SHIFT >> 32) || (id < 0 && ~id >> BLOCK_SHIFT >> 32 )) {
+        fprintf(stderr, "\nAbsolute node IDs must not be larger than %lld (got %lld)\n",
+                1ULL << 42, (long long) id);
+        util::exit_nicely();
+    }
     totalNodes++;
     /* if ALLOC_DENSE and ALLOC_SPARSE are set, send it through
      * ram_nodes_set_dense. If a block is non dense, it will automatically
diff --git a/node-ram-cache.hpp b/node-ram-cache.hpp
index 0404904..c292c2a 100644
--- a/node-ram-cache.hpp
+++ b/node-ram-cache.hpp
@@ -83,11 +83,23 @@ struct ramNodeID {
     ramNode coord;
 };
 
-struct ramNodeBlock {
+class ramNodeBlock {
+public:
+    ramNodeBlock() : nodes(NULL), block_offset(-1), _used(0) {}
+
+    void set_dirty() { _used |= 1; }
+    bool dirty() const { return _used & 1; }
+
+    void reset_used() { _used = 0; }
+    void inc_used() { _used += 2; }
+    void dec_used() { _used -= 2; }
+    void set_used(int used) { _used = (used << 1) || (_used & 1); }
+    int used() const { return _used >> 1; }
+
     ramNode *nodes;
-    osmid_t block_offset;
-    int used;
-    int dirty;
+    int32_t block_offset;
+private:
+    int32_t _used; // 0-bit indicates dirty
 };
 
 struct node_ram_cache : public boost::noncopyable
@@ -120,6 +132,7 @@ private:
     ramNodeID *sparseBlock;
     int64_t maxSparseTuples;
     int64_t sizeSparseTuples;
+    osmid_t maxSparseId;
 
     int64_t cacheUsed, cacheSize;
     osmid_t storedNodes, totalNodes;
diff --git a/options.cpp b/options.cpp
index 19462b4..8c2ed03 100644
--- a/options.cpp
+++ b/options.cpp
@@ -385,7 +385,7 @@ options_t options_t::parse(int argc, char *argv[])
             break;
         case 'k':
             if (options.hstore_mode != HSTORE_NONE) {
-                throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n");
+                throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n");
             }
             options.hstore_mode = HSTORE_NORM;
             break;
@@ -394,7 +394,7 @@ options_t options_t::parse(int argc, char *argv[])
             break;
         case 'j':
             if (options.hstore_mode != HSTORE_NONE) {
-                throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n");
+                throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n");
             }
             options.hstore_mode = HSTORE_ALL;
             break;
@@ -425,7 +425,7 @@ options_t options_t::parse(int argc, char *argv[])
             else if (strcmp(optarg, "optimized") == 0)
                 options.alloc_chunkwise = ALLOC_DENSE | ALLOC_SPARSE;
             else {
-                throw std::runtime_error((boost::format("ERROR: Unrecognized cache strategy %1%.\n") % optarg).str());
+                throw std::runtime_error((boost::format("Unrecognized cache strategy %1%.\n") % optarg).str());
             }
             break;
         case 205:
@@ -481,11 +481,15 @@ options_t options_t::parse(int argc, char *argv[])
     }
 
     if (options.append && options.create) {
-        throw std::runtime_error("Error: --append and --create options can not be used at the same time!\n");
+        throw std::runtime_error("--append and --create options can not be used at the same time!\n");
+    }
+
+    if (options.append && !options.slim) {
+        throw std::runtime_error("--append can only be used with slim mode!\n");
     }
 
     if (options.droptemp && !options.slim) {
-        throw std::runtime_error("Error: --drop only makes sense with --slim.\n");
+        throw std::runtime_error("--drop only makes sense with --slim.\n");
     }
 
     if (options.unlogged && !options.create) {
diff --git a/pgsql-id-tracker.cpp b/pgsql-id-tracker.cpp
deleted file mode 100644
index bcd54af..0000000
--- a/pgsql-id-tracker.cpp
+++ /dev/null
@@ -1,153 +0,0 @@
-#include "pgsql-id-tracker.hpp"
-
-#include <cassert>
-#include <cstdio>
-#include <limits>
-#include <libpq-fe.h>
-#include <boost/format.hpp>
-
-#include "osmtypes.hpp"
-#include "pgsql.hpp"
-#include "util.hpp"
-
-struct pgsql_id_tracker::pimpl {
-    pimpl(const std::string &conninfo,
-          const std::string &prefix,
-          const std::string &type,
-          bool owns_table);
-    ~pimpl();
-
-    PGconn *conn;
-    std::string table_name;
-    bool owns_table;
-    osmid_t old_id;
-};
-
-pgsql_id_tracker::pimpl::pimpl(const std::string &conninfo,
-                               const std::string &prefix,
-                               const std::string &type,
-                               bool owns_table_)
-    : conn(PQconnectdb(conninfo.c_str())),
-      table_name((boost::format("%1%_%2%") % prefix % type).str()),
-      owns_table(owns_table_),
-      old_id(0) {
-    if (PQstatus(conn) != CONNECTION_OK) {
-        fprintf(stderr, "Connection to database failed: %s\n", PQerrorMessage(conn));
-        util::exit_nicely();
-    }
-    if (owns_table) {
-        pgsql_exec(conn, PGRES_COMMAND_OK,
-                   "DROP TABLE IF EXISTS \"%s\"",
-                   table_name.c_str());
-        pgsql_exec(conn, PGRES_COMMAND_OK,
-                   "CREATE TABLE \"%s\" (id " POSTGRES_OSMID_TYPE ")",
-                   table_name.c_str());
-    }
-    pgsql_exec(conn, PGRES_COMMAND_OK,
-               "PREPARE set_mark(" POSTGRES_OSMID_TYPE ") AS INSERT INTO \"%s\" (id) "
-               "SELECT $1 WHERE NOT EXISTS (SELECT id FROM \"%s\" WHERE id = $1)",
-               table_name.c_str(), table_name.c_str());
-    pgsql_exec(conn, PGRES_COMMAND_OK,
-               "PREPARE get_mark(" POSTGRES_OSMID_TYPE ") AS SELECT id FROM \"%s\" "
-               "WHERE id = $1",
-               table_name.c_str());
-    pgsql_exec(conn, PGRES_COMMAND_OK,
-               "PREPARE get_min AS SELECT min(id) AS id FROM \"%s\"",
-               table_name.c_str());
-    pgsql_exec(conn, PGRES_COMMAND_OK,
-               "PREPARE drop_mark(" POSTGRES_OSMID_TYPE ") AS DELETE FROM \"%s\" "
-               "WHERE id = $1",
-               table_name.c_str());
-    pgsql_exec(conn, PGRES_COMMAND_OK, "BEGIN");
-}
-
-pgsql_id_tracker::pimpl::~pimpl() {
-    if (conn) {
-        pgsql_exec(conn, PGRES_COMMAND_OK, "COMMIT");
-        if (owns_table) {
-            pgsql_exec(conn, PGRES_COMMAND_OK, "DROP TABLE \"%s\"", table_name.c_str());
-        }
-        PQfinish(conn);
-    }
-    conn = NULL;
-}
-
-pgsql_id_tracker::pgsql_id_tracker(const std::string &conninfo,
-                                   const std::string &prefix,
-                                   const std::string &type,
-                                   bool owns_table)
-    : impl() {
-    impl.reset(new pimpl(conninfo, prefix, type, owns_table));
-}
-
-pgsql_id_tracker::~pgsql_id_tracker() {
-}
-
-void pgsql_id_tracker::mark(osmid_t id) {
-    char tmp[16];
-    char const *paramValues[1];
-
-    snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
-    paramValues[0] = tmp;
-
-    pgsql_execPrepared(impl->conn, "set_mark", 1, paramValues, PGRES_COMMAND_OK);
-}
-
-bool pgsql_id_tracker::is_marked(osmid_t id) {
-    char tmp[16];
-    char const *paramValues[1] = {NULL};
-    PGresult *result = NULL;
-
-    snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
-    paramValues[0] = tmp;
-
-    result = pgsql_execPrepared(impl->conn, "get_mark", 1, paramValues, PGRES_TUPLES_OK);
-    bool done = PQntuples(result) > 0;
-    PQclear(result);
-    return done;
-}
-
-osmid_t pgsql_id_tracker::pop_mark() {
-    osmid_t id = std::numeric_limits<osmid_t>::max();
-    PGresult *result = NULL;
-
-    result = pgsql_execPrepared(impl->conn, "get_min", 0, NULL, PGRES_TUPLES_OK);
-    if ((PQntuples(result) == 1) &&
-        (PQgetisnull(result, 0, 0) == 0)) {
-        id = strtoosmid(PQgetvalue(result, 0, 0), NULL, 10);
-    }
-
-    PQclear(result);
-
-    if (id != std::numeric_limits<osmid_t>::max()) {
-        unmark(id);
-    }
-
-    assert((id > impl->old_id) || (id == std::numeric_limits<osmid_t>::max()));
-    impl->old_id = id;
-
-    return id;
-}
-
-void pgsql_id_tracker::unmark(osmid_t id) {
-    char tmp[16];
-    char const *paramValues[1] = {NULL};
-
-    snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
-    paramValues[0] = tmp;
-
-    pgsql_execPrepared(impl->conn, "drop_mark", 1, paramValues, PGRES_COMMAND_OK);
-}
-
-void pgsql_id_tracker::commit() {
-    if (impl->owns_table) {
-        pgsql_exec(impl->conn, PGRES_COMMAND_OK, "CREATE INDEX ON \"%s\" (id)", impl->table_name.c_str());
-    }
-    pgsql_exec(impl->conn, PGRES_COMMAND_OK, "COMMIT");
-    pgsql_exec(impl->conn, PGRES_COMMAND_OK, "BEGIN");
-}
-
-void pgsql_id_tracker::force_release() {
-    impl->owns_table = false;
-    impl->conn = NULL;
-}
diff --git a/pgsql-id-tracker.hpp b/pgsql-id-tracker.hpp
deleted file mode 100644
index 4a140ee..0000000
--- a/pgsql-id-tracker.hpp
+++ /dev/null
@@ -1,31 +0,0 @@
-#ifndef PGSQL_ID_TRACKER_HPP
-#define PGSQL_ID_TRACKER_HPP
-
-#include "id-tracker.hpp"
-
-#include <string>
-#include <boost/smart_ptr/scoped_ptr.hpp>
-
-struct pgsql_id_tracker : public id_tracker {
-    pgsql_id_tracker(const std::string &conninfo,
-                     const std::string &prefix,
-                     const std::string &type,
-                     bool owns_table);
-    ~pgsql_id_tracker();
-
-    void mark(osmid_t id);
-    bool is_marked(osmid_t id);
-
-    osmid_t pop_mark();
-
-    void commit();
-    void force_release(); // to avoid brain-damages with fork()
-
-private:
-    void unmark(osmid_t id);
-
-    struct pimpl;
-    boost::scoped_ptr<pimpl> impl;
-};
-
-#endif /* PGSQL_ID_TRACKER_HPP */
diff --git a/tagtransform.cpp b/tagtransform.cpp
index 208b33c..dd7d9fc 100644
--- a/tagtransform.cpp
+++ b/tagtransform.cpp
@@ -25,20 +25,33 @@ static const struct {
     const char *highway;
     int roads;
 } layers[] = {
-    { 3, "minor",         0 },
-    { 3, "road",          0 },
-    { 3, "unclassified",  0 },
-    { 3, "residential",   0 },
-    { 4, "tertiary_link", 0 },
-    { 4, "tertiary",      0 },
-    { 6, "secondary_link",1 },
-    { 6, "secondary",     1 },
-    { 7, "primary_link",  1 },
-    { 7, "primary",       1 },
-    { 8, "trunk_link",    1 },
-    { 8, "trunk",         1 },
-    { 9, "motorway_link", 1 },
-    { 9, "motorway",      1 }
+    { 1, "proposed", 0 },
+    { 2, "construction", 0 },
+    { 10, "steps", 0 },
+    { 10, "cycleway", 0 },
+    { 10, "bridleway", 0 },
+    { 10, "footway", 0 },
+    { 10, "path", 0 },
+    { 11, "track", 0 },
+    { 15, "service", 0 },
+
+    { 24, "tertiary_link", 0 },
+    { 25, "secondary_link",1 },
+    { 27, "primary_link",  1 },
+    { 28, "trunk_link",    1 },
+    { 29, "motorway_link", 1 },
+
+    { 30, "raceway",       0 },
+    { 31, "pedestrian",    0 },
+    { 32, "living_street", 0 },
+    { 33, "road",          0 },
+    { 33, "unclassified",  0 },
+    { 33, "residential",   0 },
+    { 34, "tertiary",      0 },
+    { 36, "secondary",     1 },
+    { 37, "primary",       1 },
+    { 38, "trunk",         1 },
+    { 39, "motorway",      1 }
 };
 
 static const unsigned int nLayers = (sizeof(layers)/sizeof(*layers));
@@ -56,12 +69,11 @@ void add_z_order(taglist_t &tags, int *roads)
     int z_order = 0;
 
     int l = layer ? strtol(layer->c_str(), NULL, 10) : 0;
-    z_order = 10 * l;
+    z_order = 100 * l;
     *roads = 0;
 
     if (highway) {
         for (unsigned i = 0; i < nLayers; i++) {
-            //if (layers[i].highway == *highway) {
             if (!strcmp(layers[i].highway, highway->c_str())) {
                 z_order += layers[i].offset;
                 *roads = layers[i].roads;
@@ -71,7 +83,7 @@ void add_z_order(taglist_t &tags, int *roads)
     }
 
     if (railway && !railway->empty()) {
-        z_order += 5;
+        z_order += 35;
         *roads = 1;
     }
     /* Administrative boundaries are rendered at low zooms so we prefer to use the roads table */
@@ -79,10 +91,10 @@ void add_z_order(taglist_t &tags, int *roads)
         *roads = 1;
 
     if (bridge)
-        z_order += 10;
+        z_order += 100;
 
     if (tunnel)
-        z_order -= 10;
+        z_order -= 100;
 
     char z[13];
     snprintf(z, sizeof(z), "%d", z_order);
@@ -118,7 +130,7 @@ unsigned int c_filter_rel_member_tags(const taglist_t &rel_tags,
         if (is_route && (it->key == "name"))
             out_tags.push_dedupe(tag("route_name", it->value));
         //copy all other tags except for "type"
-        else if (it->key != "type")
+        if (it->key != "type")
             out_tags.push_dedupe(*it);
     }
 
diff --git a/tests/common-pg.cpp b/tests/common-pg.cpp
index 48f5633..19d7e81 100644
--- a/tests/common-pg.cpp
+++ b/tests/common-pg.cpp
@@ -75,22 +75,7 @@ result::~result() {
 
 tempdb::tempdb()
     : m_conn(conn::connect("dbname=postgres")) {
-    result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE "
-                                  "spcname = 'tablespacetest'");
-
-    if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) ||
-        (PQntuples(res->get()) != 1)) {
-        std::ostringstream out;
-        out << "The test needs a temporary tablespace to run in, but it does not "
-            << "exist. Please create the temporary tablespace. On Linux, you can "
-            << "do this by running:\n"
-            << "  sudo mkdir -p /tmp/psql-tablespace\n"
-            << "  sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n"
-            << "  psql -c \"CREATE TABLESPACE tablespacetest LOCATION "
-            << "'/tmp/psql-tablespace'\" postgres\n";
-        throw std::runtime_error(out.str());
-    }
-
+    result_ptr res = NULL;
     m_db_name = (boost::format("osm2pgsql-test-%1%-%2%") % getpid() % time(NULL)).str();
     m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name);
     //tests can be run concurrently which means that this query can collide with other similar ones
@@ -116,6 +101,24 @@ tempdb::tempdb()
     setup_extension(db, "hstore", NULL);
 }
 
+void tempdb::check_tblspc() {
+  result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE "
+                                "spcname = 'tablespacetest'");
+  if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) ||
+      (PQntuples(res->get()) != 1)) {
+      std::ostringstream out;
+      out << "The test needs a temporary tablespace to run in, but it does not "
+          << "exist. Please create the temporary tablespace. On Linux, you can "
+          << "do this by running:\n"
+          << "  sudo mkdir -p /tmp/psql-tablespace\n"
+          << "  sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n"
+          << "  psql -c \"CREATE TABLESPACE tablespacetest LOCATION "
+          << "'/tmp/psql-tablespace'\" postgres\n";
+      throw std::runtime_error(out.str());
+  }
+
+}
+
 tempdb::~tempdb() {
     if (m_conn) {
         m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name);
diff --git a/tests/common-pg.hpp b/tests/common-pg.hpp
index bb3cb08..faa2888 100644
--- a/tests/common-pg.hpp
+++ b/tests/common-pg.hpp
@@ -56,6 +56,8 @@ struct tempdb
 
     const std::string &conninfo() const;
 
+    void check_tblspc();
+
 private:
     void setup_extension(conn_ptr db, const std::string &extension, ...);
 
diff --git a/tests/regression-test.py b/tests/regression-test.py
index 617fd50..4ebcef6 100755
--- a/tests/regression-test.py
+++ b/tests/regression-test.py
@@ -168,11 +168,11 @@ sql_test_statements=[
     ( 92, 'Basic line length', 'SELECT round(sum(ST_Length(way))) FROM planet_osm_roads;', 2032023),
     ( 93, 'Basic number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4228),
     ( 94, 'Basic number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2316),
-    ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 10897),
+    ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11131),
     ( 96, 'Basic number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9540),
     ( 97, 'Diff import number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4352),
     ( 98, 'Diff import number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2340),
-    ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11020),
+    ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11254),
     ( 100, 'Diff import number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9834),
     #**** Tests to check if inner polygon appears when outer tags change after initially identicall inner and outer way tags in a multi-polygon ****
     #**** These tests are currently broken and noted in trac ticket #2853 ****
diff --git a/tests/test-middle-flat.cpp b/tests/test-middle-flat.cpp
new file mode 100644
index 0000000..bb9e45d
--- /dev/null
+++ b/tests/test-middle-flat.cpp
@@ -0,0 +1,112 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "output-null.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+/* This is basically the same as test-middle-pgsql, but with flat nodes. */
+
+void run_tests(options_t options, const std::string cache_type) {
+  options.append = 0;
+  options.create = 1;
+  options.flat_node_cache_enabled = true;
+  // flat nodes truncates the file each time it's started, so we can reuse the same file
+  options.flat_node_file = boost::optional<std::string>("tests/test_middle_flat.flat.nodes.bin");
+
+  {
+    middle_pgsql_t mid_pgsql;
+    output_null_t out_test(&mid_pgsql, options);
+
+    mid_pgsql.start(&options);
+
+    if (test_node_set(&mid_pgsql) != 0) { throw std::runtime_error("test_node_set failed."); }
+
+    mid_pgsql.commit();
+    mid_pgsql.stop();
+  }
+  {
+    middle_pgsql_t mid_pgsql;
+    output_null_t out_test(&mid_pgsql, options);
+
+    mid_pgsql.start(&options);
+
+    if (test_nodes_comprehensive_set(&mid_pgsql) != 0) { throw std::runtime_error("test_nodes_comprehensive_set failed."); }
+
+    mid_pgsql.commit();
+    mid_pgsql.stop();
+  }
+  /* This should work, but doesn't. More tests are needed that look at updates
+     without the complication of ways.
+  */
+/*  {
+    middle_pgsql_t mid_pgsql;
+    output_null_t out_test(&mid_pgsql, options);
+
+    mid_pgsql.start(&options);
+    mid_pgsql.commit();
+    mid_pgsql.stop();
+    // Switch to append mode because this tests updates
+    options.append = 1;
+    options.create = 0;
+    mid_pgsql.start(&options);
+    if (test_way_set(&mid_pgsql) != 0) { throw std::runtime_error("test_way_set failed."); }
+
+    mid_pgsql.commit();
+    mid_pgsql.stop();
+  }*/
+}
+int main(int argc, char *argv[]) {
+  boost::scoped_ptr<pg::tempdb> db;
+
+  try {
+    db.reset(new pg::tempdb);
+  } catch (const std::exception &e) {
+    std::cerr << "Unable to setup database: " << e.what() << "\n";
+    return 77; // <-- code to skip this test.
+  }
+
+  try {
+    options_t options;
+    options.conninfo = db->conninfo().c_str();
+    options.scale = 10000000;
+    options.cache = 1;
+    options.num_procs = 1;
+    options.prefix = "osm2pgsql_test";
+    options.slim = 1;
+
+    options.alloc_chunkwise = ALLOC_SPARSE | ALLOC_DENSE; // what you get with optimized
+    run_tests(options, "optimized");
+    options.alloc_chunkwise = ALLOC_SPARSE;
+    run_tests(options, "sparse");
+
+    options.alloc_chunkwise = ALLOC_DENSE;
+    run_tests(options, "dense");
+
+    options.alloc_chunkwise = ALLOC_DENSE | ALLOC_DENSE_CHUNK; // what you get with chunk
+    run_tests(options, "chunk");
+  } catch (const std::exception &e) {
+    std::cerr << "ERROR: " << e.what() << std::endl;
+    return 1;
+  } catch (...) {
+    std::cerr << "UNKNOWN ERROR" << std::endl;
+    return 1;
+  }
+  return 0;
+}
diff --git a/tests/test-output-multi-line.cpp b/tests/test-output-multi-line.cpp
index cf53067..fe6b60b 100644
--- a/tests/test-output-multi-line.cpp
+++ b/tests/test-output-multi-line.cpp
@@ -61,8 +61,6 @@ int main(int argc, char *argv[]) {
         options_t options;
         options.conninfo = db->conninfo().c_str();
         options.num_procs = 1;
-        options.tblsslim_index = "tablespacetest";
-        options.tblsslim_data = "tablespacetest";
         options.slim = 1;
 
         boost::shared_ptr<geometry_processor> processor =
diff --git a/tests/test-output-multi-point-multi-table.cpp b/tests/test-output-multi-point-multi-table.cpp
index d35f5bb..f76c147 100644
--- a/tests/test-output-multi-point-multi-table.cpp
+++ b/tests/test-output-multi-point-multi-table.cpp
@@ -67,8 +67,6 @@ int main(int argc, char *argv[]) {
         options.conninfo = db->conninfo().c_str();
         options.num_procs = 1;
         options.prefix = "osm2pgsql_test";
-        options.tblsslim_index = "tablespacetest";
-        options.tblsslim_data = "tablespacetest";
         options.slim = 1;
 
         export_list columns;
diff --git a/tests/test-output-multi-point.cpp b/tests/test-output-multi-point.cpp
index 36e102a..fe29481 100644
--- a/tests/test-output-multi-point.cpp
+++ b/tests/test-output-multi-point.cpp
@@ -62,8 +62,6 @@ int main(int argc, char *argv[]) {
         options.conninfo = db->conninfo().c_str();
         options.num_procs = 1;
         options.prefix = "osm2pgsql_test";
-        options.tblsslim_index = "tablespacetest";
-        options.tblsslim_data = "tablespacetest";
         options.slim = 1;
 
         boost::shared_ptr<geometry_processor> processor =
diff --git a/tests/test-output-multi-polygon.cpp b/tests/test-output-multi-polygon.cpp
index 38be794..98370c6 100644
--- a/tests/test-output-multi-polygon.cpp
+++ b/tests/test-output-multi-polygon.cpp
@@ -62,8 +62,6 @@ int main(int argc, char *argv[]) {
         options.conninfo = db->conninfo().c_str();
         options.num_procs = 1;
         options.prefix = "osm2pgsql_test";
-        options.tblsslim_index = "tablespacetest";
-        options.tblsslim_data = "tablespacetest";
         options.slim = 1;
 
         boost::shared_ptr<geometry_processor> processor = geometry_processor::create("polygon", &options);
diff --git a/tests/test-output-pgsql-tablespace.cpp b/tests/test-output-pgsql-tablespace.cpp
new file mode 100755
index 0000000..3f75299
--- /dev/null
+++ b/tests/test-output-pgsql-tablespace.cpp
@@ -0,0 +1,146 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "osmdata.hpp"
+#include "output-pgsql.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+#include "middle-ram.hpp"
+#include "taginfo_impl.hpp"
+#include "parse.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+#include <boost/lexical_cast.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+namespace {
+
+struct skip_test : public std::exception {
+    const char *what() { return "Test skipped."; }
+};
+
+void run_test(const char* test_name, void (*testfunc)()) {
+    try {
+        fprintf(stderr, "%s\n", test_name);
+        testfunc();
+
+    } catch (const skip_test &) {
+        exit(77); // <-- code to skip this test.
+
+    } catch (const std::exception& e) {
+        fprintf(stderr, "%s\n", e.what());
+        fprintf(stderr, "FAIL\n");
+        exit(EXIT_FAILURE);
+    }
+
+    fprintf(stderr, "PASS\n");
+}
+#define RUN_TEST(x) run_test(#x, &(x))
+
+void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
+    pg::result_ptr res = conn->exec(query);
+
+    int ntuples = PQntuples(res->get());
+    if (ntuples != 1) {
+        throw std::runtime_error((boost::format("Expected only one tuple from a query "
+                                                "to check COUNT(*), but got %1%. Query "
+                                                "was: %2%.")
+                                  % ntuples % query).str());
+    }
+
+    std::string numstr = PQgetvalue(res->get(), 0, 0);
+    int count = boost::lexical_cast<int>(numstr);
+
+    if (count != expected) {
+        throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+                                                "query: %3%.")
+                                  % expected % count % query).str());
+    }
+}
+
+void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
+    std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
+                                       "where relname = '%1%'")
+                         % table_name).str();
+
+    check_count(test_conn, 1, query);
+}
+
+// "simple" test modeled on the basic regression test from
+// the python script. this is just to check everything is
+// working as expected before we start the complex stuff.
+void test_regression_simple() {
+    boost::scoped_ptr<pg::tempdb> db;
+
+    try {
+        db.reset(new pg::tempdb);
+        db->check_tblspc(); // Unlike others, these tests require a test tablespace
+    } catch (const std::exception &e) {
+        std::cerr << "Unable to setup database: " << e.what() << "\n";
+        throw skip_test();
+    }
+
+    std::string proc_name("test-output-pgsql"), input_file("-");
+    char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+    boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+    options_t options = options_t::parse(2, argv);
+    options.conninfo = db->conninfo().c_str();
+    options.num_procs = 1;
+    options.prefix = "osm2pgsql_test";
+    options.slim = 1;
+    options.style = "default.style";
+
+    options.tblsslim_index = "tablespacetest";
+    options.tblsslim_data = "tablespacetest";
+
+    boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+    osmdata_t osmdata(mid_pgsql, out_test);
+
+    boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+    osmdata.start();
+
+    if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) {
+        throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'.");
+    }
+
+    parser.reset(NULL);
+
+    osmdata.stop();
+
+    // start a new connection to run tests on
+    pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+    assert_has_table(test_conn, "osm2pgsql_test_point");
+    assert_has_table(test_conn, "osm2pgsql_test_line");
+    assert_has_table(test_conn, "osm2pgsql_test_polygon");
+    assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+    check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point");
+    check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line");
+    check_count(test_conn,  375, "SELECT count(*) FROM osm2pgsql_test_roads");
+    check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+}
+
+} // anonymous namespace
+
+int main(int argc, char *argv[]) {
+    RUN_TEST(test_regression_simple);
+
+    return 0;
+}
diff --git a/tests/test-output-pgsql-z_order.cpp b/tests/test-output-pgsql-z_order.cpp
new file mode 100644
index 0000000..c940f44
--- /dev/null
+++ b/tests/test-output-pgsql-z_order.cpp
@@ -0,0 +1,186 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "osmdata.hpp"
+#include "output-pgsql.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+#include "middle-ram.hpp"
+#include "taginfo_impl.hpp"
+#include "parse.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+#include <boost/lexical_cast.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+namespace {
+
+struct skip_test : public std::exception {
+    const char *what() { return "Test skipped."; }
+};
+
+void run_test(const char* test_name, void (*testfunc)()) {
+    try {
+        fprintf(stderr, "%s\n", test_name);
+        testfunc();
+
+    } catch (const skip_test &) {
+        exit(77); // <-- code to skip this test.
+
+    } catch (const std::exception& e) {
+        fprintf(stderr, "%s\n", e.what());
+        fprintf(stderr, "FAIL\n");
+        exit(EXIT_FAILURE);
+    }
+
+    fprintf(stderr, "PASS\n");
+}
+#define RUN_TEST(x) run_test(#x, &(x))
+
+void check_string(pg::conn_ptr &conn, std::string expected, const std::string &query) {
+    pg::result_ptr res = conn->exec(query);
+
+    int ntuples = PQntuples(res->get());
+    if (ntuples != 1) {
+        throw std::runtime_error((boost::format("Expected only one tuple from a query "
+                                                "to check a string, but got %1%. Query "
+                                                "was: %2%.")
+                                  % ntuples % query).str());
+    }
+
+    std::string actual = PQgetvalue(res->get(), 0, 0);
+
+    if (actual != expected) {
+        throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+                                                "query: %3%.")
+                                  % expected % actual % query).str());
+    }
+}
+
+
+void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
+    pg::result_ptr res = conn->exec(query);
+
+    int ntuples = PQntuples(res->get());
+    if (ntuples != 1) {
+        throw std::runtime_error((boost::format("Expected only one tuple from a query "
+                                                "to check COUNT(*), but got %1%. Query "
+                                                "was: %2%.")
+                                  % ntuples % query).str());
+    }
+
+    std::string numstr = PQgetvalue(res->get(), 0, 0);
+    int count = boost::lexical_cast<int>(numstr);
+
+    if (count != expected) {
+        throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+                                                "query: %3%.")
+                                  % expected % count % query).str());
+    }
+}
+
+void check_number(pg::conn_ptr &conn, double expected, const std::string &query) {
+    pg::result_ptr res = conn->exec(query);
+
+    int ntuples = PQntuples(res->get());
+    if (ntuples != 1) {
+        throw std::runtime_error((boost::format("Expected only one tuple from a query, "
+                                                " but got %1%. Query was: %2%.")
+                                  % ntuples % query).str());
+    }
+
+    std::string numstr = PQgetvalue(res->get(), 0, 0);
+    double num = boost::lexical_cast<double>(numstr);
+
+    // floating point isn't exact, so allow a 0.01% difference
+    if ((num > 1.0001*expected) || (num < 0.9999*expected)) {
+        throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+                                                "query: %3%.")
+                                  % expected % num % query).str());
+    }
+}
+
+void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
+    std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
+                                       "where relname = '%1%'")
+                         % table_name).str();
+
+    check_count(test_conn, 1, query);
+}
+
+// "simple" test modeled on the basic regression test from
+// the python script. this is just to check everything is
+// working as expected before we start the complex stuff.
+void test_z_order() {
+    boost::scoped_ptr<pg::tempdb> db;
+
+    try {
+        db.reset(new pg::tempdb);
+    } catch (const std::exception &e) {
+        std::cerr << "Unable to setup database: " << e.what() << "\n";
+        throw skip_test();
+    }
+
+    std::string proc_name("test-output-pgsql"), input_file("-");
+    char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+    boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+    options_t options = options_t::parse(2, argv);
+    options.conninfo = db->conninfo().c_str();
+    options.num_procs = 1;
+    options.prefix = "osm2pgsql_test";
+    options.style = "default.style";
+
+    boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+    osmdata_t osmdata(mid_pgsql, out_test);
+
+    boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+    osmdata.start();
+
+    if (parser->streamFile("libxml2", "tests/test_output_pgsql_z_order.osm", options.sanitize, &osmdata) != 0) {
+        throw std::runtime_error("Unable to read input file `tests/test_output_pgsql_z_order.osm'.");
+    }
+
+    parser.reset(NULL);
+
+    osmdata.stop();
+
+    // start a new connection to run tests on
+    pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+    assert_has_table(test_conn, "osm2pgsql_test_point");
+    assert_has_table(test_conn, "osm2pgsql_test_line");
+    assert_has_table(test_conn, "osm2pgsql_test_polygon");
+    assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+    check_string(test_conn, "motorway", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 0");
+    check_string(test_conn, "trunk", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 1");
+    check_string(test_conn, "primary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 2");
+    check_string(test_conn, "secondary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 3");
+    check_string(test_conn, "tertiary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 4");
+
+    check_string(test_conn, "residential", "SELECT highway FROM osm2pgsql_test_line ORDER BY z_order DESC LIMIT 1 OFFSET 0");
+}
+
+} // anonymous namespace
+
+int main(int argc, char *argv[]) {
+    RUN_TEST(test_z_order);
+
+    return 0;
+}
diff --git a/tests/test-output-pgsql.cpp b/tests/test-output-pgsql.cpp
index 42f490a..b7a4053 100644
--- a/tests/test-output-pgsql.cpp
+++ b/tests/test-output-pgsql.cpp
@@ -71,6 +71,27 @@ void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
     }
 }
 
+void check_number(pg::conn_ptr &conn, double expected, const std::string &query) {
+    pg::result_ptr res = conn->exec(query);
+
+    int ntuples = PQntuples(res->get());
+    if (ntuples != 1) {
+        throw std::runtime_error((boost::format("Expected only one tuple from a query, "
+                                                " but got %1%. Query was: %2%.")
+                                  % ntuples % query).str());
+    }
+
+    std::string numstr = PQgetvalue(res->get(), 0, 0);
+    double num = boost::lexical_cast<double>(numstr);
+
+    // floating point isn't exact, so allow a 0.01% difference
+    if ((num > 1.0001*expected) || (num < 0.9999*expected)) {
+        throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+                                                "query: %3%.")
+                                  % expected % num % query).str());
+    }
+}
+
 void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
     std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
                                        "where relname = '%1%'")
@@ -100,8 +121,6 @@ void test_regression_simple() {
     options.conninfo = db->conninfo().c_str();
     options.num_procs = 1;
     options.prefix = "osm2pgsql_test";
-    options.tblsslim_index = "tablespacetest";
-    options.tblsslim_data = "tablespacetest";
     options.slim = 1;
     options.style = "default.style";
 
@@ -133,8 +152,85 @@ void test_regression_simple() {
     check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line");
     check_count(test_conn,  375, "SELECT count(*) FROM osm2pgsql_test_roads");
     check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+
+    // Check size of lines
+    check_number(test_conn, 1696.04, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+    check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+
+    check_number(test_conn, 311.21, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+    check_number(test_conn, 311.21, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+    check_number(test_conn, 143.81, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+
+    // Check a point's location
+    check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=900913;POINT(1062645.12 5972593.4)'::geometry, 0.1)");
+}
+
+void test_latlong() {
+    boost::scoped_ptr<pg::tempdb> db;
+
+    try {
+        db.reset(new pg::tempdb);
+    } catch (const std::exception &e) {
+        std::cerr << "Unable to setup database: " << e.what() << "\n";
+        throw skip_test();
+    }
+
+    std::string proc_name("test-output-pgsql"), input_file("-");
+    char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+    boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+    options_t options = options_t::parse(2, argv);
+    options.conninfo = db->conninfo().c_str();
+    options.num_procs = 1;
+    options.prefix = "osm2pgsql_test";
+    options.slim = 1;
+    options.style = "default.style";
+
+    options.projection.reset(new reprojection(PROJ_LATLONG));
+    options.scale = (options.projection->get_proj_id() == PROJ_LATLONG) ? 10000000 : 100;
+
+    boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+    osmdata_t osmdata(mid_pgsql, out_test);
+
+    boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+    osmdata.start();
+
+    if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) {
+        throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'.");
+    }
+
+    parser.reset(NULL);
+
+    osmdata.stop();
+
+    // start a new connection to run tests on
+    pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+    assert_has_table(test_conn, "osm2pgsql_test_point");
+    assert_has_table(test_conn, "osm2pgsql_test_line");
+    assert_has_table(test_conn, "osm2pgsql_test_polygon");
+    assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+    check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point");
+    check_count(test_conn, 3298, "SELECT count(*) FROM osm2pgsql_test_line");
+    check_count(test_conn, 374, "SELECT count(*) FROM osm2pgsql_test_roads");
+    check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+
+    // Check size of lines
+    check_number(test_conn, 0.0105343, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+    check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+
+    check_number(test_conn, 1.70718e-08, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+    check_number(test_conn, 1.70718e-08, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+    check_number(test_conn, 143.845, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+
+    // Check a point's location
+    check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=4326;POINT(9.5459035 47.1866494)'::geometry, 0.00001)");
 }
 
+
 void test_area_way_simple() {
     boost::scoped_ptr<pg::tempdb> db;
 
@@ -259,8 +355,6 @@ void test_clone() {
     options.conninfo = db->conninfo().c_str();
     options.num_procs = 1;
     options.prefix = "osm2pgsql_test";
-    options.tblsslim_index = "tablespacetest";
-    options.tblsslim_data = "tablespacetest";
     options.slim = 1;
     options.style = "default.style";
 
@@ -302,6 +396,7 @@ void test_clone() {
 
 int main(int argc, char *argv[]) {
     RUN_TEST(test_regression_simple);
+    RUN_TEST(test_latlong);
     RUN_TEST(test_clone);
     RUN_TEST(test_area_way_simple);
     RUN_TEST(test_route_rel);
diff --git a/tests/test_output_pgsql_z_order.osm b/tests/test_output_pgsql_z_order.osm
new file mode 100644
index 0000000..76ddd9a
--- /dev/null
+++ b/tests/test_output_pgsql_z_order.osm
@@ -0,0 +1,75 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<osm version='0.6' generator='hand'>
+  <node id='1' version='1' visible='true' lat='49' lon='-122.5'>
+    <tag k='highway' v='bus_stop' />
+  </node>
+  <node id='2' version='1' visible='true' lat='49.005' lon='-122.51'>
+    <tag k='highway' v='bus_stop' />
+  </node>
+  <node id='3' version='1' visible='true' lat='49.01' lon='-122.5' />
+  <!-- yes, all the ways have the same linestring -->
+  <way id='1' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='motorway' />
+  </way>
+  <way id='2' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='motorway_link' />
+  </way>
+  <way id='3' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='trunk' />
+  </way>
+  <way id='4' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='trunk_link' />
+  </way>
+  <way id='5' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='primary' />
+  </way>
+  <way id='6' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='primary_link' />
+  </way>
+  <way id='7' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='secondary' />
+  </way>
+  <way id='8' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='secondary_link' />
+  </way>
+  <way id='9' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='tertiary' />
+  </way>
+  <way id='10' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='tertiary_link' />
+  </way>
+  <way id='11' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <tag k='highway' v='residential' />
+    <tag k='layer' v='5' />
+  </way>
+  <!-- put *something* in the polygon table -->
+  <way id='1000' version='1' visible='true'>
+    <nd ref='1' />
+    <nd ref='2' />
+    <nd ref='3' />
+    <nd ref='1' />
+    <tag k='building' v='yes' />
+  </way>
+</osm>

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-grass/osm2pgsql.git



More information about the Pkg-grass-devel mailing list