[osm2pgsql] 01/01: Imported Upstream version 0.88.0~rc1
Sebastiaan Couwenberg
sebastic at moszumanska.debian.org
Sat Jul 11 09:02:52 UTC 2015
This is an automated email from the git hooks/post-receive script.
sebastic pushed a commit to branch upstream
in repository osm2pgsql.
commit 0a8b63fb27b8084adec80eb729bbb98ad7628a21
Author: Bas Couwenberg <sebastic at xs4all.nl>
Date: Sat Jul 11 10:20:57 2015 +0200
Imported Upstream version 0.88.0~rc1
---
.gitignore | 5 +-
AUTHORS | 4 +-
CONTRIBUTING.md | 102 ++++++++++++++
Makefile.am | 15 ++-
README.md | 25 +---
configure.ac | 2 +-
docs/migrations.md | 15 +++
geometry-builder.cpp | 57 ++++----
node-persistent-cache.cpp | 89 ++++++------
node-ram-cache.cpp | 53 +++++---
node-ram-cache.hpp | 21 ++-
options.cpp | 14 +-
pgsql-id-tracker.cpp | 153 ---------------------
pgsql-id-tracker.hpp | 31 -----
tagtransform.cpp | 52 ++++---
tests/common-pg.cpp | 35 ++---
tests/common-pg.hpp | 2 +
tests/regression-test.py | 4 +-
tests/test-middle-flat.cpp | 112 ++++++++++++++++
tests/test-output-multi-line.cpp | 2 -
tests/test-output-multi-point-multi-table.cpp | 2 -
tests/test-output-multi-point.cpp | 2 -
tests/test-output-multi-polygon.cpp | 2 -
tests/test-output-pgsql-tablespace.cpp | 146 ++++++++++++++++++++
tests/test-output-pgsql-z_order.cpp | 186 ++++++++++++++++++++++++++
tests/test-output-pgsql.cpp | 103 +++++++++++++-
tests/test_output_pgsql_z_order.osm | 75 +++++++++++
27 files changed, 938 insertions(+), 371 deletions(-)
diff --git a/.gitignore b/.gitignore
index 6ca8a4d..a4cd9b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -45,6 +45,7 @@ tests/.dirstamp
tests/test-parse-xml2
tests/test-middle-ram
tests/test-middle-pgsql
+tests/test-middle-flat
tests/test-pgsql-escape
tests/test-parse-options
tests/test-output-multi-tags
@@ -55,10 +56,12 @@ tests/test-output-multi-point-multi-table
tests/test-output-multi-polygon
tests/test-output-multi-poly-trivial
tests/test-output-pgsql
+tests/test-output-pgsql-tablespace
+tests/test-output-pgsql-z_order
tests/test-expire-tiles
tests/*.log
tests/*.trs
-tests/test_output_pgsql_area_way.flat.nodes.bin
+tests/*.flat.nodes.bin
.libs/
*.lo
diff --git a/AUTHORS b/AUTHORS
index af8498b..0f6bbfa 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -1,3 +1,3 @@
osm2pgsql was written by Jon Burgess, Artem Pavlenko, Martijn van Oosterhout
-Sarah Hoffman, Kai Krueger, Frederik Ramm, Brian Quinion and other
-OpenStreetMap project members.
\ No newline at end of file
+Sarah Hoffmann, Kai Krueger, Frederik Ramm, Brian Quinion, Matt Amos,
+Kevin Kreiser, Paul Norman and other OpenStreetMap project members.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..27bb5ca
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,102 @@
+# Osm2pgsql contribution guidelines
+
+## Workflow
+
+We operate the "Fork & Pull" model explained at
+
+https://help.github.com/articles/using-pull-requests
+
+You should fork the project into your own repo, create a topic branch
+there and then make one or more pull requests back to the openstreetmap repository.
+Your pull requests will then be reviewed and discussed.
+
+## History
+
+To understand the osm2pgsql code, it helps to know some history on it. Osm2pgsql
+was written in C in 2007 as a port of an older Python utility. In 2014 it was
+ported to C++ by MapQuest and the last C version was released as 0.86.0. In it's
+time, it has had varying contribution activity, including times with no
+maintainer or active developers.
+
+Parts of the codebase still clearly show their C origin and could use rewriting
+in modern C++, making use of data structures in the standard library.
+
+## Versioning
+
+Osm2pgsql uses a X.Y.Z version number, where Y tells you if you are on a stable
+or development series. Like the Linux Kernel, even numbers are stable and
+development versions are odd.
+
+Old branches and versions are not generally maintained.
+
+## Code style
+
+The current codebase is a mix of styles, but new code should be written in the
+[K&R 1TBS style](https://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS) with
+4 spaces indentation. Tabs should never be used in the C++ code.
+
+e.g.
+
+```
+int main(int argc, char *argv[])
+{
+ ...
+ while (x == y) {
+ something();
+ somethingelse();
+
+ if (some_error) {
+ do_correct();
+ } else {
+ continue_as_usual();
+ }
+ }
+
+ finalthing();
+ ...
+}
+```
+
+Names should use underscores, not camel case, with class/struct names ending in `_t`.
+
+## Platforms targeted
+
+Ideally osm2pgsql should compile on Linux, OS X, FreeBSD and Windows. It is
+actively tested on Debian, Ubuntu and FreeBSD by the maintainers.
+
+## Testing
+
+The code also comes with a suite of tests which can be run by
+executing ``make check``.
+
+Most of these tests depend on being able to set up a database and run osm2pgsql
+against it. You need to ensure that PostgreSQL is running and that your user is
+a superuser of that system. To do that, run:
+
+```sh
+sudo -u postgres createuser -s $USER
+sudo mkdir -p /tmp/psql-tablespace
+sudo chown postgres.postgres /tmp/psql-tablespace
+psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres
+```
+
+Once this is all set up, all the tests should run (no SKIPs), and pass
+(no FAILs). If you encounter a failure, you can find more information
+by looking in the `test-suite.log`. If you find something which seems
+to be a bug, please check to see if it is a known issue at
+https://github.com/openstreetmap/osm2pgsql/issues and, if it's not
+already known, report it there.
+
+If running the tests in a virtual machine, allocate sufficient disk space for a
+20GB flat nodes file.
+
+### Performance Testing
+
+If performance testing with a full planet import is required, indicate what
+needs testing in a pull request.
+
+## Maintainers
+
+The current maintainers of osm2pgsql are [Sarah Hoffmann](https://github.com/lonvia/)
+and [Paul Norman](https://github.com/pnorman/). Sarah has more experience with
+the gazetteer backend and Paul with the pgsql and multi backends.
diff --git a/Makefile.am b/Makefile.am
index 8ce3f4f..c91d408 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -32,7 +32,6 @@ libosm2pgsql_la_SOURCES = \
parse-pbf.cpp \
parse-xml2.cpp \
pgsql.cpp \
- pgsql-id-tracker.cpp \
processor-line.cpp \
processor-point.cpp \
processor-polygon.cpp \
@@ -51,6 +50,7 @@ check_PROGRAMS = \
tests/test-parse-xml2 \
tests/test-middle-ram \
tests/test-middle-pgsql \
+ tests/test-middle-flat \
tests/test-output-multi-tags \
tests/test-output-multi-line \
tests/test-output-multi-line-storage \
@@ -59,6 +59,8 @@ check_PROGRAMS = \
tests/test-output-multi-polygon \
tests/test-output-multi-poly-trivial \
tests/test-output-pgsql \
+ tests/test-output-pgsql-z_order \
+ tests/test-output-pgsql-tablespace \
tests/test-pgsql-escape \
tests/test-parse-options \
tests/test-expire-tiles
@@ -69,6 +71,8 @@ tests_test_middle_ram_SOURCES = tests/test-middle-ram.cpp tests/middle-tests.cpp
tests_test_middle_ram_LDADD = libosm2pgsql.la
tests_test_middle_pgsql_SOURCES = tests/test-middle-pgsql.cpp tests/middle-tests.cpp tests/common-pg.cpp
tests_test_middle_pgsql_LDADD = libosm2pgsql.la
+tests_test_middle_flat_SOURCES = tests/test-middle-flat.cpp tests/middle-tests.cpp tests/common-pg.cpp
+tests_test_middle_flat_LDADD = libosm2pgsql.la
tests_test_output_multi_tags_SOURCES = tests/test-output-multi-tags.cpp tests/common-pg.cpp
tests_test_output_multi_tags_LDADD = libosm2pgsql.la
tests_test_output_multi_line_SOURCES = tests/test-output-multi-line.cpp tests/common-pg.cpp
@@ -85,6 +89,10 @@ tests_test_output_multi_poly_trivial_SOURCES = tests/test-output-multi-poly-triv
tests_test_output_multi_poly_trivial_LDADD = libosm2pgsql.la
tests_test_output_pgsql_SOURCES = tests/test-output-pgsql.cpp tests/common-pg.cpp
tests_test_output_pgsql_LDADD = libosm2pgsql.la
+tests_test_output_pgsql_tablespace_SOURCES = tests/test-output-pgsql-tablespace.cpp tests/common-pg.cpp
+tests_test_output_pgsql_tablespace_LDADD = libosm2pgsql.la
+tests_test_output_pgsql_z_order_SOURCES = tests/test-output-pgsql-z_order.cpp tests/common-pg.cpp
+tests_test_output_pgsql_z_order_LDADD = libosm2pgsql.la
tests_test_pgsql_escape_SOURCES = tests/test-pgsql-escape.cpp
tests_test_pgsql_escape_LDADD = libosm2pgsql.la
tests_test_parse_options_SOURCES = tests/test-parse-options.cpp
@@ -92,6 +100,8 @@ tests_test_parse_options_LDADD = libosm2pgsql.la
tests_test_expire_tiles_SOURCES = tests/test-expire-tiles.cpp
tests_test_expire_tiles_LDADD = libosm2pgsql.la
+MOSTLYCLEANFILES = tests/test_middle_flat.flat.nodes.bin tests/test_output_pgsql_area_way.flat.nodes.bin
+
TESTS = $(check_PROGRAMS) tests/regression-test.sh
TEST_EXTENSIONS = .sh
SH_LOG_COMPILER = sh
@@ -138,6 +148,7 @@ osm2pgsql_LDADD += $(GLOBAL_LDFLAGS)
tests_test_parse_xml2_LDADD += $(GLOBAL_LDFLAGS)
tests_test_middle_ram_LDADD += $(GLOBAL_LDFLAGS)
tests_test_middle_pgsql_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_middle_flat_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_multi_tags_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_multi_line_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_multi_line_storage_LDADD += $(GLOBAL_LDFLAGS)
@@ -146,6 +157,8 @@ tests_test_output_multi_point_multi_table_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_multi_polygon_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_multi_poly_trivial_LDADD += $(GLOBAL_LDFLAGS)
tests_test_output_pgsql_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_output_pgsql_tablespace_LDADD += $(GLOBAL_LDFLAGS)
+tests_test_output_pgsql_z_order_LDADD += $(GLOBAL_LDFLAGS)
tests_test_pgsql_escape_LDADD += $(GLOBAL_LDFLAGS)
tests_test_parse_options_LDADD += $(GLOBAL_LDFLAGS)
tests_test_expire_tiles_LDADD += $(GLOBAL_LDFLAGS)
diff --git a/README.md b/README.md
index f6bdd68..b601b8c 100644
--- a/README.md
+++ b/README.md
@@ -143,33 +143,12 @@ postgres tables instead of those provided in the pgsql backend.
Any questions should be directed at the osm dev list
http://wiki.openstreetmap.org/index.php/Mailing_lists
-## Testing ##
-
-The code also comes with a suite of tests which can be run by
-executing ``make check``.
-
-Some of these tests depend on being able to set up a database and run
-osm2pgsql against it. You need to ensure that PostgreSQL is running
-and that your user is a superuser of that system. To do that, run:
-
-```sh
-sudo -u postgres createuser -s $USER
-sudo mkdir -p /tmp/psql-tablespace
-sudo chown postgres.postgres /tmp/psql-tablespace
-psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres
-```
-
-Once this is all set up, all the tests should run (no SKIPs), and pass
-(no FAILs). If you encounter a failure, you can find more information
-by looking in the `test-suite.log`. If you find something which seems
-to be a bug, please check to see if it is a known issue at
-https://github.com/openstreetmap/osm2pgsql/issues and, if it's not
-already known, report it there.
-
## Contributing ##
We welcome contributions to osm2pgsql. If you would like to report an issue,
please use the [issue tracker on GitHub](https://github.com/openstreetmap/osm2pgsql/issues).
+More information can be found in [CONTRIBUTING.md](CONTRIBUTING.md).
+
General queries can be sent to the tile-serving@ or dev@
[mailing lists](http://wiki.openstreetmap.org/wiki/Mailing_lists).
diff --git a/configure.ac b/configure.ac
index 278de33..a545f9d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,5 +1,5 @@
dnl Process this file with autoconf to produce a configure script.
-AC_INIT(osm2pgsql, 0.87.4)
+AC_INIT(osm2pgsql, 0.88.0-RC1)
dnl Required autoconf version
AC_PREREQ(2.61)
diff --git a/docs/migrations.md b/docs/migrations.md
index 10457c8..af1c70a 100644
--- a/docs/migrations.md
+++ b/docs/migrations.md
@@ -4,6 +4,21 @@ Some osm2pgsql changes have slightly changed the database schema it expects. If
updating an old database, a migration may be needed. The migrations here assume
the default `planet_osm` prefix.
+## 0.88.0 z_order changes ##
+
+0.88.0 z_order logic was changed, requuiring an increase in z_order values. To
+migrate to the new range of values, run
+
+```sql
+UPDATE planet_osm_line SET z_order = z_order * 10;
+UPDATE planet_osm_roads SET z_order = z_order * 10;
+```
+
+This will not apply the new logic, but will get the existing z_orders in the right
+group of 100 for the new logic.
+
+If not using osm2pgsql z_orders, this change may be ignored.
+
## 0.87.0 pending removal ##
0.87.0 moved the in-database tracking of pending ways and relations to
diff --git a/geometry-builder.cpp b/geometry-builder.cpp
index 143a1cc..5c4a6b5 100644
--- a/geometry-builder.cpp
+++ b/geometry-builder.cpp
@@ -81,9 +81,9 @@ struct polygondata
};
struct polygondata_comparearea {
- bool operator()(const polygondata& lhs, const polygondata& rhs) {
- return lhs.area > rhs.area;
- }
+ bool operator()(const polygondata& lhs, const polygondata& rhs) {
+ return lhs.area > rhs.area;
+ }
};
} // anonymous namespace
@@ -190,25 +190,25 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t
// length of the line in `segment` over the `split_at` distance.
if (distance + delta > split_at) {
- const size_t splits = std::floor((distance + delta) / split_at);
- // use the splitting distance to split the current segment up
- // into as many parts as necessary to keep each part below
- // the `split_at` distance.
- for (size_t i = 0; i < splits; ++i) {
- double frac = (double(i + 1) * split_at - distance) / delta;
- const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x,
- frac * (this_pt.y - prev_pt.y) + prev_pt.y);
- segment->add(interpolated);
- geom_ptr geom = geom_ptr(gf.createLineString(segment.release()));
+ const size_t splits = std::floor((distance + delta) / split_at);
+ // use the splitting distance to split the current segment up
+ // into as many parts as necessary to keep each part below
+ // the `split_at` distance.
+ for (size_t i = 0; i < splits; ++i) {
+ double frac = (double(i + 1) * split_at - distance) / delta;
+ const Coordinate interpolated(frac * (this_pt.x - prev_pt.x) + prev_pt.x,
+ frac * (this_pt.y - prev_pt.y) + prev_pt.y);
+ segment->add(interpolated);
+ geom_ptr geom = geom_ptr(gf.createLineString(segment.release()));
- //copy of an empty one should be cheapest
- wkts->push_back(geometry_builder::wkt_t());
- //then we set on the one we already have
- wkts->back().geom = writer.write(geom.get());
- wkts->back().area = 0;
+ //copy of an empty one should be cheapest
+ wkts->push_back(geometry_builder::wkt_t());
+ //then we set on the one we already have
+ wkts->back().geom = writer.write(geom.get());
+ wkts->back().area = 0;
- segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
- segment->add(interpolated);
+ segment.reset(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
+ segment->add(interpolated);
}
// reset the distance based on the final splitting point for
// the next iteration.
@@ -255,12 +255,12 @@ geometry_builder::maybe_wkts_t geometry_builder::get_wkt_split(const nodelist_t
}
int geometry_builder::parse_wkt(const char * wkt, multinodelist_t &nodes, int *polygon) {
- GeometryFactory gf;
- WKTReader reader(&gf);
- std::string wkt_string(wkt);
- GeometryCollection * gc;
- CoordinateSequence * coords;
- size_t num_geometries;
+ GeometryFactory gf;
+ WKTReader reader(&gf);
+ std::string wkt_string(wkt);
+ GeometryCollection * gc;
+ CoordinateSequence * coords;
+ size_t num_geometries;
*polygon = 0;
try {
@@ -320,7 +320,6 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel
maybe_wkts_t wkts(new std::vector<geometry_builder::wkt_t>);
-
try
{
for (multinodelist_t::const_iterator it = xnodes.begin(); it != xnodes.end(); ++it) {
@@ -422,7 +421,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_polygons(const multinodel
}
}
}
- pgf.destroy(preparedtoplevelpolygon);
+ pgf.destroy(preparedtoplevelpolygon);
}
// polys now is a list of polygons tagged with which ones are inside each other
@@ -608,7 +607,7 @@ geometry_builder::maybe_wkts_t geometry_builder::build_both(const multinodelist_
}
else
{
- //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl;
+ //std::cerr << "polygon(" << osm_id << ") is no good: points(" << pline->getNumPoints() << "), closed(" << pline->isClosed() << "). " << writer.write(pline.get()) << std::endl;
double distance = 0;
std::auto_ptr<CoordinateSequence> segment;
segment = std::auto_ptr<CoordinateSequence>(gf.getCoordinateSequenceFactory()->create((size_t)0, (size_t)2));
diff --git a/node-persistent-cache.cpp b/node-persistent-cache.cpp
index 61a4804..3689ce7 100644
--- a/node-persistent-cache.cpp
+++ b/node-persistent-cache.cpp
@@ -49,10 +49,10 @@ void node_persistent_cache::writeout_dirty_nodes()
{
for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
{
- if (readNodeBlockCache[i].dirty)
+ if (readNodeBlockCache[i].dirty())
{
if (lseek64(node_cache_fd,
- (readNodeBlockCache[i].block_offset
+ ((osmid_t) readNodeBlockCache[i].block_offset
<< READ_NODE_BLOCK_SHIFT)
* sizeof(ramNode)
+ sizeof(persistentCacheHeader),
@@ -70,7 +70,7 @@ void node_persistent_cache::writeout_dirty_nodes()
util::exit_nicely();
}
}
- readNodeBlockCache[i].dirty = 0;
+ readNodeBlockCache[i].reset_used();
}
}
@@ -85,9 +85,9 @@ int node_persistent_cache::replace_block()
for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
{
- if (readNodeBlockCache[i].used < min_used)
+ if (readNodeBlockCache[i].used() < min_used)
{
- min_used = readNodeBlockCache[i].used;
+ min_used = readNodeBlockCache[i].used();
block_id = i;
}
}
@@ -95,9 +95,9 @@ int node_persistent_cache::replace_block()
{
for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
{
- if (readNodeBlockCache[i].used > 1)
+ if (readNodeBlockCache[i].used() > 1)
{
- readNodeBlockCache[i].used--;
+ readNodeBlockCache[i].dec_used();
}
}
}
@@ -214,10 +214,10 @@ int node_persistent_cache::load_block(osmid_t block_offset)
{
const int block_id = replace_block();
- if (readNodeBlockCache[block_id].dirty)
+ if (readNodeBlockCache[block_id].dirty())
{
if (lseek64(node_cache_fd,
- (readNodeBlockCache[block_id].block_offset
+ ((osmid_t) readNodeBlockCache[block_id].block_offset
<< READ_NODE_BLOCK_SHIFT) * sizeof(ramNode)
+ sizeof(struct persistentCacheHeader), SEEK_SET) < 0) {
fprintf(stderr, "Failed to seek to correct position in node cache: %s\n",
@@ -232,13 +232,21 @@ int node_persistent_cache::load_block(osmid_t block_offset)
strerror(errno));
util::exit_nicely();
}
- readNodeBlockCache[block_id].dirty = 0;
+ readNodeBlockCache[block_id].reset_used();
}
- remove_from_cache_idx(readNodeBlockCache[block_id].block_offset);
- new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE];
+ if (readNodeBlockCache[block_id].nodes) {
+ remove_from_cache_idx((osmid_t) readNodeBlockCache[block_id].block_offset);
+ new(readNodeBlockCache[block_id].nodes) ramNode[READ_NODE_BLOCK_SIZE];
+ } else {
+ readNodeBlockCache[block_id].nodes = new ramNode[READ_NODE_BLOCK_SIZE];
+ if (!readNodeBlockCache[block_id].nodes) {
+ fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
+ util::exit_nicely();
+ }
+ }
readNodeBlockCache[block_id].block_offset = block_offset;
- readNodeBlockCache[block_id].used = READ_NODE_CACHE_SIZE;
+ readNodeBlockCache[block_id].set_used(READ_NODE_CACHE_SIZE);
/* Make sure the node cache is correctly initialised for the block that will be read */
if (cacheHeader.max_initialised_id
@@ -289,7 +297,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
* node cache file in buffer cache therefore duplicates the data wasting 16GB of ram.
* Therefore tell the OS not to cache the node-persistent-cache during initial import.
* */
- if (sync_file_range(node_cache_fd, writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+ if (sync_file_range(node_cache_fd, (osmid_t) writeNodeBlock.block_offset*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode),
SYNC_FILE_RANGE_WRITE) < 0) {
fprintf(stderr, "Info: Sync_file_range writeout has an issue. This shouldn't be anything to worry about.: %s\n",
@@ -297,7 +305,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
};
if (writeNodeBlock.block_offset > 16) {
- if(sync_file_range(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+ if(sync_file_range(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode),
SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_AFTER) < 0) {
fprintf(stderr, "Info: Sync_file_range block has an issue. This shouldn't be anything to worry about.: %s\n",
@@ -305,7 +313,7 @@ void node_persistent_cache::nodes_set_create_writeout_block()
}
#ifdef HAVE_POSIX_FADVISE
- if (posix_fadvise(node_cache_fd, (writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
+ if (posix_fadvise(node_cache_fd, ((osmid_t) writeNodeBlock.block_offset - 16)*WRITE_NODE_BLOCK_SIZE * sizeof(ramNode) +
sizeof(persistentCacheHeader), WRITE_NODE_BLOCK_SIZE * sizeof(ramNode), POSIX_FADV_DONTNEED) !=0 ) {
fprintf(stderr, "Info: Posix_fadvise failed. This shouldn't be anything to worry about.: %s\n",
strerror(errno));
@@ -320,23 +328,22 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon)
assert(!append_mode);
assert(!read_mode);
- osmid_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT;
+ int32_t block_offset = id >> WRITE_NODE_BLOCK_SHIFT;
if (writeNodeBlock.block_offset != block_offset)
{
- if (writeNodeBlock.dirty)
+ if (writeNodeBlock.dirty())
{
nodes_set_create_writeout_block();
- writeNodeBlock.dirty = 0;
/* After writing out the node block, the file pointer is at the next block level */
writeNodeBlock.block_offset++;
- cacheHeader.max_initialised_id = (writeNodeBlock.block_offset
+ cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset
<< WRITE_NODE_BLOCK_SHIFT) - 1;
}
if (writeNodeBlock.block_offset > block_offset)
{
fprintf(stderr,
- "ERROR: Block_offset not in sequential order: %" PRIdOSMID "%" PRIdOSMID "\n",
+ "ERROR: Block_offset not in sequential order: %d %d\n",
writeNodeBlock.block_offset, block_offset);
util::exit_nicely();
}
@@ -353,7 +360,7 @@ int node_persistent_cache::set_create(osmid_t id, double lat, double lon)
}
writeNodeBlock.nodes[id & WRITE_NODE_BLOCK_MASK] = ramNode(lon, lat);
- writeNodeBlock.dirty = 1;
+ writeNodeBlock.set_dirty();
return 0;
}
@@ -373,8 +380,8 @@ int node_persistent_cache::set_append(osmid_t id, double lat, double lon)
readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode();
else
readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK] = ramNode(lon, lat);
- readNodeBlockCache[block_id].used++;
- readNodeBlockCache[block_id].dirty = 1;
+ readNodeBlockCache[block_id].inc_used();
+ readNodeBlockCache[block_id].set_dirty();
return 1;
}
@@ -399,7 +406,7 @@ int node_persistent_cache::get(osmNode *out, osmid_t id)
block_id = load_block(block_offset);
}
- readNodeBlockCache[block_id].used++;
+ readNodeBlockCache[block_id].inc_used();
if (!readNodeBlockCache[block_id].nodes[id & READ_NODE_BLOCK_MASK].is_valid())
return 1;
@@ -451,13 +458,12 @@ void node_persistent_cache::set_read_mode()
if (read_mode)
return;
- if (writeNodeBlock.dirty > 0) {
+ if (writeNodeBlock.dirty()) {
assert(!append_mode);
- fprintf(stderr, "Switching to read mode\n");
nodes_set_create_writeout_block();
- writeNodeBlock.dirty = 0;
+ writeNodeBlock.reset_used();
writeNodeBlock.block_offset++;
- cacheHeader.max_initialised_id = (writeNodeBlock.block_offset
+ cacheHeader.max_initialised_id = ((osmid_t) writeNodeBlock.block_offset
<< WRITE_NODE_BLOCK_SHIFT) - 1;
/* write out the header */
@@ -475,8 +481,6 @@ void node_persistent_cache::set_read_mode()
}
read_mode = true;
-
- fprintf(stderr, "Switching to read mode done\n");
}
node_persistent_cache::node_persistent_cache(const options_t *options, int append,
@@ -484,7 +488,6 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
: node_cache_fd(0), node_cache_fname(NULL), append_mode(0), cacheHeader(),
writeNodeBlock(), readNodeBlockCache(NULL), read_mode(ro), ram_cache(ptr)
{
- int i, err;
append_mode = append;
if (options->flat_node_file) {
node_cache_fname = options->flat_node_file->c_str();
@@ -533,12 +536,10 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
};
writeNodeBlock.block_offset = 0;
- writeNodeBlock.dirty = 0;
- writeNodeBlock.nodes = 0;
if (!read_mode)
{
-
+ int err;
#ifdef HAVE_POSIX_FALLOCATE
if ((err = posix_fallocate(node_cache_fd, 0,
sizeof(ramNode) * MAXIMUM_INITIAL_ID)) != 0)
@@ -606,27 +607,16 @@ node_persistent_cache::node_persistent_cache(const options_t *options, int appen
fprintf(stderr,"Maximum node in persistent node cache: %" PRIdOSMID "\n", cacheHeader.max_initialised_id);
- readNodeBlockCache = new ramNodeBlock [READ_NODE_CACHE_SIZE];
+ readNodeBlockCache = new ramNodeBlock[READ_NODE_CACHE_SIZE];
if (!readNodeBlockCache) {
fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
util::exit_nicely();
}
- for (i = 0; i < READ_NODE_CACHE_SIZE; i++)
- {
- readNodeBlockCache[i].nodes = new ramNode[READ_NODE_BLOCK_SIZE];
- if (!readNodeBlockCache[i].nodes) {
- fprintf(stderr, "Out of memory: Failed to allocate node read cache\n");
- util::exit_nicely();
- }
- readNodeBlockCache[i].block_offset = -1;
- readNodeBlockCache[i].used = 0;
- readNodeBlockCache[i].dirty = 0;
- }
}
node_persistent_cache::~node_persistent_cache()
{
- if (writeNodeBlock.dirty > 0)
+ if (writeNodeBlock.dirty())
nodes_set_create_writeout_block();
writeout_dirty_nodes();
@@ -659,7 +649,8 @@ node_persistent_cache::~node_persistent_cache()
if (readNodeBlockCache) {
for (int i = 0; i < READ_NODE_CACHE_SIZE; i++)
{
- delete[] readNodeBlockCache[i].nodes;
+ if (readNodeBlockCache[i].nodes)
+ delete[] readNodeBlockCache[i].nodes;
}
delete[] readNodeBlockCache;
}
diff --git a/node-ram-cache.cpp b/node-ram-cache.cpp
index 167f6b7..23f8d24 100644
--- a/node-ram-cache.cpp
+++ b/node-ram-cache.cpp
@@ -60,7 +60,7 @@
int ramNode::scale;
-static int id2block(osmid_t id)
+static int32_t id2block(osmid_t id)
{
/* + NUM_BLOCKS/2 allows for negative IDs */
return (id >> BLOCK_SHIFT) + NUM_BLOCKS/2;
@@ -71,7 +71,7 @@ static int id2offset(osmid_t id)
return id & (PER_BLOCK-1);
}
-static osmid_t block2id(int block, int offset)
+static osmid_t block2id(int32_t block, int offset)
{
return (((osmid_t) block - NUM_BLOCKS/2) << BLOCK_SHIFT) + (osmid_t) offset;
}
@@ -84,7 +84,7 @@ void node_ram_cache::percolate_up( int pos )
while( i > 0 )
{
int parent = (i-1)>>1;
- if( queue[i]->used < queue[parent]->used )
+ if( queue[i]->used() < queue[parent]->used() )
{
Swap( queue[i], queue[parent] )
i = parent;
@@ -108,7 +108,11 @@ ramNode *node_ram_cache::next_chunk() {
int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
- if ((sizeSparseTuples > maxSparseTuples) || ( cacheUsed > cacheSize)) {
+ // Sparse cache depends on ordered nodes, reject out-of-order ids.
+ // Also check that there is still space.
+ if ((maxSparseId && id < maxSparseId)
+ || (sizeSparseTuples > maxSparseTuples)
+ || ( cacheUsed > cacheSize)) {
if ((allocStrategy & ALLOC_LOSSY) > 0)
return 1;
else {
@@ -116,6 +120,7 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
util::exit_nicely();
}
}
+ maxSparseId = id;
sparseBlock[sizeSparseTuples].id = id;
sparseBlock[sizeSparseTuples].coord = coord;
@@ -126,9 +131,8 @@ int node_ram_cache::set_sparse(osmid_t id, const ramNode &coord) {
}
int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
- int block = id2block(id);
- int offset = id2offset(id);
- int i = 0;
+ int32_t const block = id2block(id);
+ int const offset = id2offset(id);
if (maxBlocks == 0) return 1;
@@ -143,7 +147,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
* to store it in dense representation. If not, push all elements of the block
* to the sparse node cache and reuse memory of the previous block for the current block */
if ( ((allocStrategy & ALLOC_SPARSE) == 0) ||
- ((queue[usedBlocks - 1]->used / (double)(1<< BLOCK_SHIFT)) >
+ ((queue[usedBlocks - 1]->used() / (double)(1<< BLOCK_SHIFT)) >
(sizeof(ramNode) / (double)sizeof(ramNodeID)))) {
/* Block has reached the level to keep it in dense representation */
/* We've just finished with the previous block, so we need to percolate it up the queue to its correct position */
@@ -152,7 +156,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
blocks[block].nodes = next_chunk();
} else {
/* previous block was not dense enough, so push it into the sparse node cache instead */
- for (i = 0; i < (1 << BLOCK_SHIFT); i++) {
+ for (int i = 0; i < (1 << BLOCK_SHIFT); i++) {
if (queue[usedBlocks -1]->nodes[i].is_valid()) {
set_sparse(block2id(queue[usedBlocks - 1]->block_offset, i),
queue[usedBlocks -1]->nodes[i]);
@@ -160,7 +164,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
}
}
/* reuse previous block, as its content is now in the sparse representation */
- storedNodes -= queue[usedBlocks - 1]->used;
+ storedNodes -= queue[usedBlocks - 1]->used();
blocks[block].nodes = queue[usedBlocks - 1]->nodes;
blocks[queue[usedBlocks - 1]->block_offset].nodes = NULL;
usedBlocks--;
@@ -170,7 +174,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
blocks[block].nodes = next_chunk();
}
- blocks[block].used = 0;
+ blocks[block].reset_used();
blocks[block].block_offset = block;
if (!blocks[block].nodes) {
fprintf(stderr, "Error allocating nodes\n");
@@ -194,17 +198,17 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
* current head of the tree down to the right level to restore the
* priority queue invariant. Upto log(maxBlocks) iterations */
- i=0;
+ int i = 0;
while( 2*i+1 < usedBlocks - 1 ) {
- if( queue[2*i+1]->used <= queue[2*i+2]->used ) {
- if( queue[i]->used > queue[2*i+1]->used ) {
+ if( queue[2*i+1]->used() <= queue[2*i+2]->used() ) {
+ if( queue[i]->used() > queue[2*i+1]->used() ) {
Swap( queue[i], queue[2*i+1] );
i = 2*i+1;
}
else
break;
} else {
- if( queue[i]->used > queue[2*i+2]->used ) {
+ if( queue[i]->used() > queue[2*i+2]->used() ) {
Swap( queue[i], queue[2*i+2] );
i = 2*i+2;
} else
@@ -213,13 +217,13 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
}
/* Now the head of the queue is the smallest, so it becomes our replacement candidate */
blocks[block].nodes = queue[0]->nodes;
- blocks[block].used = 0;
+ blocks[block].reset_used();
new(blocks[block].nodes) ramNode[PER_BLOCK];
/* Clear old head block and point to new block */
- storedNodes -= queue[0]->used;
+ storedNodes -= queue[0]->used();
queue[0]->nodes = NULL;
- queue[0]->used = 0;
+ queue[0]->reset_used();
queue[0] = &blocks[block];
}
} else {
@@ -243,7 +247,7 @@ int node_ram_cache::set_dense(osmid_t id, const ramNode &coord) {
}
blocks[block].nodes[offset] = coord;
- blocks[block].used++;
+ blocks[block].inc_used();
storedNodes++;
return 0;
}
@@ -275,8 +279,8 @@ int node_ram_cache::get_sparse(osmNode *out, osmid_t id) {
}
int node_ram_cache::get_dense(osmNode *out, osmid_t id) {
- int block = id2block(id);
- int offset = id2offset(id);
+ int32_t const block = id2block(id);
+ int const offset = id2offset(id);
if (!blocks[block].nodes)
return 1;
@@ -294,7 +298,7 @@ int node_ram_cache::get_dense(osmNode *out, osmid_t id) {
node_ram_cache::node_ram_cache( int strategy, int cacheSizeMB, int fixpointscale )
: allocStrategy(ALLOC_DENSE), blocks(NULL), usedBlocks(0),
maxBlocks(0), blockCache(NULL), queue(NULL), sparseBlock(NULL),
- maxSparseTuples(0), sizeSparseTuples(0), cacheUsed(0),
+ maxSparseTuples(0), sizeSparseTuples(0), maxSparseId(0), cacheUsed(0),
cacheSize(0), storedNodes(0), totalNodes(0), nodesCacheHits(0),
nodesCacheLookups(0), warn_node_order(0) {
@@ -390,6 +394,11 @@ node_ram_cache::~node_ram_cache() {
}
int node_ram_cache::set(osmid_t id, double lat, double lon, const taglist_t &) {
+ if ((id > 0 && id >> BLOCK_SHIFT >> 32) || (id < 0 && ~id >> BLOCK_SHIFT >> 32 )) {
+ fprintf(stderr, "\nAbsolute node IDs must not be larger than %lld (got %lld)\n",
+ 1ULL << 42, (long long) id);
+ util::exit_nicely();
+ }
totalNodes++;
/* if ALLOC_DENSE and ALLOC_SPARSE are set, send it through
* ram_nodes_set_dense. If a block is non dense, it will automatically
diff --git a/node-ram-cache.hpp b/node-ram-cache.hpp
index 0404904..c292c2a 100644
--- a/node-ram-cache.hpp
+++ b/node-ram-cache.hpp
@@ -83,11 +83,23 @@ struct ramNodeID {
ramNode coord;
};
-struct ramNodeBlock {
+class ramNodeBlock {
+public:
+ ramNodeBlock() : nodes(NULL), block_offset(-1), _used(0) {}
+
+ void set_dirty() { _used |= 1; }
+ bool dirty() const { return _used & 1; }
+
+ void reset_used() { _used = 0; }
+ void inc_used() { _used += 2; }
+ void dec_used() { _used -= 2; }
+ void set_used(int used) { _used = (used << 1) || (_used & 1); }
+ int used() const { return _used >> 1; }
+
ramNode *nodes;
- osmid_t block_offset;
- int used;
- int dirty;
+ int32_t block_offset;
+private:
+ int32_t _used; // 0-bit indicates dirty
};
struct node_ram_cache : public boost::noncopyable
@@ -120,6 +132,7 @@ private:
ramNodeID *sparseBlock;
int64_t maxSparseTuples;
int64_t sizeSparseTuples;
+ osmid_t maxSparseId;
int64_t cacheUsed, cacheSize;
osmid_t storedNodes, totalNodes;
diff --git a/options.cpp b/options.cpp
index 19462b4..8c2ed03 100644
--- a/options.cpp
+++ b/options.cpp
@@ -385,7 +385,7 @@ options_t options_t::parse(int argc, char *argv[])
break;
case 'k':
if (options.hstore_mode != HSTORE_NONE) {
- throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n");
+ throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n");
}
options.hstore_mode = HSTORE_NORM;
break;
@@ -394,7 +394,7 @@ options_t options_t::parse(int argc, char *argv[])
break;
case 'j':
if (options.hstore_mode != HSTORE_NONE) {
- throw std::runtime_error("ERROR: You can not specify both --hstore (-k) and --hstore-all (-j)\n");
+ throw std::runtime_error("You can not specify both --hstore (-k) and --hstore-all (-j)\n");
}
options.hstore_mode = HSTORE_ALL;
break;
@@ -425,7 +425,7 @@ options_t options_t::parse(int argc, char *argv[])
else if (strcmp(optarg, "optimized") == 0)
options.alloc_chunkwise = ALLOC_DENSE | ALLOC_SPARSE;
else {
- throw std::runtime_error((boost::format("ERROR: Unrecognized cache strategy %1%.\n") % optarg).str());
+ throw std::runtime_error((boost::format("Unrecognized cache strategy %1%.\n") % optarg).str());
}
break;
case 205:
@@ -481,11 +481,15 @@ options_t options_t::parse(int argc, char *argv[])
}
if (options.append && options.create) {
- throw std::runtime_error("Error: --append and --create options can not be used at the same time!\n");
+ throw std::runtime_error("--append and --create options can not be used at the same time!\n");
+ }
+
+ if (options.append && !options.slim) {
+ throw std::runtime_error("--append can only be used with slim mode!\n");
}
if (options.droptemp && !options.slim) {
- throw std::runtime_error("Error: --drop only makes sense with --slim.\n");
+ throw std::runtime_error("--drop only makes sense with --slim.\n");
}
if (options.unlogged && !options.create) {
diff --git a/pgsql-id-tracker.cpp b/pgsql-id-tracker.cpp
deleted file mode 100644
index bcd54af..0000000
--- a/pgsql-id-tracker.cpp
+++ /dev/null
@@ -1,153 +0,0 @@
-#include "pgsql-id-tracker.hpp"
-
-#include <cassert>
-#include <cstdio>
-#include <limits>
-#include <libpq-fe.h>
-#include <boost/format.hpp>
-
-#include "osmtypes.hpp"
-#include "pgsql.hpp"
-#include "util.hpp"
-
-struct pgsql_id_tracker::pimpl {
- pimpl(const std::string &conninfo,
- const std::string &prefix,
- const std::string &type,
- bool owns_table);
- ~pimpl();
-
- PGconn *conn;
- std::string table_name;
- bool owns_table;
- osmid_t old_id;
-};
-
-pgsql_id_tracker::pimpl::pimpl(const std::string &conninfo,
- const std::string &prefix,
- const std::string &type,
- bool owns_table_)
- : conn(PQconnectdb(conninfo.c_str())),
- table_name((boost::format("%1%_%2%") % prefix % type).str()),
- owns_table(owns_table_),
- old_id(0) {
- if (PQstatus(conn) != CONNECTION_OK) {
- fprintf(stderr, "Connection to database failed: %s\n", PQerrorMessage(conn));
- util::exit_nicely();
- }
- if (owns_table) {
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "DROP TABLE IF EXISTS \"%s\"",
- table_name.c_str());
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "CREATE TABLE \"%s\" (id " POSTGRES_OSMID_TYPE ")",
- table_name.c_str());
- }
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "PREPARE set_mark(" POSTGRES_OSMID_TYPE ") AS INSERT INTO \"%s\" (id) "
- "SELECT $1 WHERE NOT EXISTS (SELECT id FROM \"%s\" WHERE id = $1)",
- table_name.c_str(), table_name.c_str());
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "PREPARE get_mark(" POSTGRES_OSMID_TYPE ") AS SELECT id FROM \"%s\" "
- "WHERE id = $1",
- table_name.c_str());
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "PREPARE get_min AS SELECT min(id) AS id FROM \"%s\"",
- table_name.c_str());
- pgsql_exec(conn, PGRES_COMMAND_OK,
- "PREPARE drop_mark(" POSTGRES_OSMID_TYPE ") AS DELETE FROM \"%s\" "
- "WHERE id = $1",
- table_name.c_str());
- pgsql_exec(conn, PGRES_COMMAND_OK, "BEGIN");
-}
-
-pgsql_id_tracker::pimpl::~pimpl() {
- if (conn) {
- pgsql_exec(conn, PGRES_COMMAND_OK, "COMMIT");
- if (owns_table) {
- pgsql_exec(conn, PGRES_COMMAND_OK, "DROP TABLE \"%s\"", table_name.c_str());
- }
- PQfinish(conn);
- }
- conn = NULL;
-}
-
-pgsql_id_tracker::pgsql_id_tracker(const std::string &conninfo,
- const std::string &prefix,
- const std::string &type,
- bool owns_table)
- : impl() {
- impl.reset(new pimpl(conninfo, prefix, type, owns_table));
-}
-
-pgsql_id_tracker::~pgsql_id_tracker() {
-}
-
-void pgsql_id_tracker::mark(osmid_t id) {
- char tmp[16];
- char const *paramValues[1];
-
- snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
- paramValues[0] = tmp;
-
- pgsql_execPrepared(impl->conn, "set_mark", 1, paramValues, PGRES_COMMAND_OK);
-}
-
-bool pgsql_id_tracker::is_marked(osmid_t id) {
- char tmp[16];
- char const *paramValues[1] = {NULL};
- PGresult *result = NULL;
-
- snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
- paramValues[0] = tmp;
-
- result = pgsql_execPrepared(impl->conn, "get_mark", 1, paramValues, PGRES_TUPLES_OK);
- bool done = PQntuples(result) > 0;
- PQclear(result);
- return done;
-}
-
-osmid_t pgsql_id_tracker::pop_mark() {
- osmid_t id = std::numeric_limits<osmid_t>::max();
- PGresult *result = NULL;
-
- result = pgsql_execPrepared(impl->conn, "get_min", 0, NULL, PGRES_TUPLES_OK);
- if ((PQntuples(result) == 1) &&
- (PQgetisnull(result, 0, 0) == 0)) {
- id = strtoosmid(PQgetvalue(result, 0, 0), NULL, 10);
- }
-
- PQclear(result);
-
- if (id != std::numeric_limits<osmid_t>::max()) {
- unmark(id);
- }
-
- assert((id > impl->old_id) || (id == std::numeric_limits<osmid_t>::max()));
- impl->old_id = id;
-
- return id;
-}
-
-void pgsql_id_tracker::unmark(osmid_t id) {
- char tmp[16];
- char const *paramValues[1] = {NULL};
-
- snprintf(tmp, sizeof(tmp), "%" PRIdOSMID, id);
- paramValues[0] = tmp;
-
- pgsql_execPrepared(impl->conn, "drop_mark", 1, paramValues, PGRES_COMMAND_OK);
-}
-
-void pgsql_id_tracker::commit() {
- if (impl->owns_table) {
- pgsql_exec(impl->conn, PGRES_COMMAND_OK, "CREATE INDEX ON \"%s\" (id)", impl->table_name.c_str());
- }
- pgsql_exec(impl->conn, PGRES_COMMAND_OK, "COMMIT");
- pgsql_exec(impl->conn, PGRES_COMMAND_OK, "BEGIN");
-}
-
-void pgsql_id_tracker::force_release() {
- impl->owns_table = false;
- impl->conn = NULL;
-}
diff --git a/pgsql-id-tracker.hpp b/pgsql-id-tracker.hpp
deleted file mode 100644
index 4a140ee..0000000
--- a/pgsql-id-tracker.hpp
+++ /dev/null
@@ -1,31 +0,0 @@
-#ifndef PGSQL_ID_TRACKER_HPP
-#define PGSQL_ID_TRACKER_HPP
-
-#include "id-tracker.hpp"
-
-#include <string>
-#include <boost/smart_ptr/scoped_ptr.hpp>
-
-struct pgsql_id_tracker : public id_tracker {
- pgsql_id_tracker(const std::string &conninfo,
- const std::string &prefix,
- const std::string &type,
- bool owns_table);
- ~pgsql_id_tracker();
-
- void mark(osmid_t id);
- bool is_marked(osmid_t id);
-
- osmid_t pop_mark();
-
- void commit();
- void force_release(); // to avoid brain-damages with fork()
-
-private:
- void unmark(osmid_t id);
-
- struct pimpl;
- boost::scoped_ptr<pimpl> impl;
-};
-
-#endif /* PGSQL_ID_TRACKER_HPP */
diff --git a/tagtransform.cpp b/tagtransform.cpp
index 208b33c..dd7d9fc 100644
--- a/tagtransform.cpp
+++ b/tagtransform.cpp
@@ -25,20 +25,33 @@ static const struct {
const char *highway;
int roads;
} layers[] = {
- { 3, "minor", 0 },
- { 3, "road", 0 },
- { 3, "unclassified", 0 },
- { 3, "residential", 0 },
- { 4, "tertiary_link", 0 },
- { 4, "tertiary", 0 },
- { 6, "secondary_link",1 },
- { 6, "secondary", 1 },
- { 7, "primary_link", 1 },
- { 7, "primary", 1 },
- { 8, "trunk_link", 1 },
- { 8, "trunk", 1 },
- { 9, "motorway_link", 1 },
- { 9, "motorway", 1 }
+ { 1, "proposed", 0 },
+ { 2, "construction", 0 },
+ { 10, "steps", 0 },
+ { 10, "cycleway", 0 },
+ { 10, "bridleway", 0 },
+ { 10, "footway", 0 },
+ { 10, "path", 0 },
+ { 11, "track", 0 },
+ { 15, "service", 0 },
+
+ { 24, "tertiary_link", 0 },
+ { 25, "secondary_link",1 },
+ { 27, "primary_link", 1 },
+ { 28, "trunk_link", 1 },
+ { 29, "motorway_link", 1 },
+
+ { 30, "raceway", 0 },
+ { 31, "pedestrian", 0 },
+ { 32, "living_street", 0 },
+ { 33, "road", 0 },
+ { 33, "unclassified", 0 },
+ { 33, "residential", 0 },
+ { 34, "tertiary", 0 },
+ { 36, "secondary", 1 },
+ { 37, "primary", 1 },
+ { 38, "trunk", 1 },
+ { 39, "motorway", 1 }
};
static const unsigned int nLayers = (sizeof(layers)/sizeof(*layers));
@@ -56,12 +69,11 @@ void add_z_order(taglist_t &tags, int *roads)
int z_order = 0;
int l = layer ? strtol(layer->c_str(), NULL, 10) : 0;
- z_order = 10 * l;
+ z_order = 100 * l;
*roads = 0;
if (highway) {
for (unsigned i = 0; i < nLayers; i++) {
- //if (layers[i].highway == *highway) {
if (!strcmp(layers[i].highway, highway->c_str())) {
z_order += layers[i].offset;
*roads = layers[i].roads;
@@ -71,7 +83,7 @@ void add_z_order(taglist_t &tags, int *roads)
}
if (railway && !railway->empty()) {
- z_order += 5;
+ z_order += 35;
*roads = 1;
}
/* Administrative boundaries are rendered at low zooms so we prefer to use the roads table */
@@ -79,10 +91,10 @@ void add_z_order(taglist_t &tags, int *roads)
*roads = 1;
if (bridge)
- z_order += 10;
+ z_order += 100;
if (tunnel)
- z_order -= 10;
+ z_order -= 100;
char z[13];
snprintf(z, sizeof(z), "%d", z_order);
@@ -118,7 +130,7 @@ unsigned int c_filter_rel_member_tags(const taglist_t &rel_tags,
if (is_route && (it->key == "name"))
out_tags.push_dedupe(tag("route_name", it->value));
//copy all other tags except for "type"
- else if (it->key != "type")
+ if (it->key != "type")
out_tags.push_dedupe(*it);
}
diff --git a/tests/common-pg.cpp b/tests/common-pg.cpp
index 48f5633..19d7e81 100644
--- a/tests/common-pg.cpp
+++ b/tests/common-pg.cpp
@@ -75,22 +75,7 @@ result::~result() {
tempdb::tempdb()
: m_conn(conn::connect("dbname=postgres")) {
- result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE "
- "spcname = 'tablespacetest'");
-
- if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) ||
- (PQntuples(res->get()) != 1)) {
- std::ostringstream out;
- out << "The test needs a temporary tablespace to run in, but it does not "
- << "exist. Please create the temporary tablespace. On Linux, you can "
- << "do this by running:\n"
- << " sudo mkdir -p /tmp/psql-tablespace\n"
- << " sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n"
- << " psql -c \"CREATE TABLESPACE tablespacetest LOCATION "
- << "'/tmp/psql-tablespace'\" postgres\n";
- throw std::runtime_error(out.str());
- }
-
+ result_ptr res = NULL;
m_db_name = (boost::format("osm2pgsql-test-%1%-%2%") % getpid() % time(NULL)).str();
m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name);
//tests can be run concurrently which means that this query can collide with other similar ones
@@ -116,6 +101,24 @@ tempdb::tempdb()
setup_extension(db, "hstore", NULL);
}
+void tempdb::check_tblspc() {
+ result_ptr res = m_conn->exec("SELECT spcname FROM pg_tablespace WHERE "
+ "spcname = 'tablespacetest'");
+ if ((PQresultStatus(res->get()) != PGRES_TUPLES_OK) ||
+ (PQntuples(res->get()) != 1)) {
+ std::ostringstream out;
+ out << "The test needs a temporary tablespace to run in, but it does not "
+ << "exist. Please create the temporary tablespace. On Linux, you can "
+ << "do this by running:\n"
+ << " sudo mkdir -p /tmp/psql-tablespace\n"
+ << " sudo /bin/chown postgres.postgres /tmp/psql-tablespace\n"
+ << " psql -c \"CREATE TABLESPACE tablespacetest LOCATION "
+ << "'/tmp/psql-tablespace'\" postgres\n";
+ throw std::runtime_error(out.str());
+ }
+
+}
+
tempdb::~tempdb() {
if (m_conn) {
m_conn->exec(boost::format("DROP DATABASE IF EXISTS \"%1%\"") % m_db_name);
diff --git a/tests/common-pg.hpp b/tests/common-pg.hpp
index bb3cb08..faa2888 100644
--- a/tests/common-pg.hpp
+++ b/tests/common-pg.hpp
@@ -56,6 +56,8 @@ struct tempdb
const std::string &conninfo() const;
+ void check_tblspc();
+
private:
void setup_extension(conn_ptr db, const std::string &extension, ...);
diff --git a/tests/regression-test.py b/tests/regression-test.py
index 617fd50..4ebcef6 100755
--- a/tests/regression-test.py
+++ b/tests/regression-test.py
@@ -168,11 +168,11 @@ sql_test_statements=[
( 92, 'Basic line length', 'SELECT round(sum(ST_Length(way))) FROM planet_osm_roads;', 2032023),
( 93, 'Basic number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4228),
( 94, 'Basic number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2316),
- ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 10897),
+ ( 95, 'Basic number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11131),
( 96, 'Basic number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9540),
( 97, 'Diff import number of hstore points tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_point;', 4352),
( 98, 'Diff import number of hstore roads tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_roads;', 2340),
- ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11020),
+ ( 99, 'Diff import number of hstore lines tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_line;', 11254),
( 100, 'Diff import number of hstore polygons tags', 'SELECT sum(array_length(akeys(tags),1)) FROM planet_osm_polygon;', 9834),
#**** Tests to check if inner polygon appears when outer tags change after initially identicall inner and outer way tags in a multi-polygon ****
#**** These tests are currently broken and noted in trac ticket #2853 ****
diff --git a/tests/test-middle-flat.cpp b/tests/test-middle-flat.cpp
new file mode 100644
index 0000000..bb9e45d
--- /dev/null
+++ b/tests/test-middle-flat.cpp
@@ -0,0 +1,112 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "output-null.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+/* This is basically the same as test-middle-pgsql, but with flat nodes. */
+
+void run_tests(options_t options, const std::string cache_type) {
+ options.append = 0;
+ options.create = 1;
+ options.flat_node_cache_enabled = true;
+ // flat nodes truncates the file each time it's started, so we can reuse the same file
+ options.flat_node_file = boost::optional<std::string>("tests/test_middle_flat.flat.nodes.bin");
+
+ {
+ middle_pgsql_t mid_pgsql;
+ output_null_t out_test(&mid_pgsql, options);
+
+ mid_pgsql.start(&options);
+
+ if (test_node_set(&mid_pgsql) != 0) { throw std::runtime_error("test_node_set failed."); }
+
+ mid_pgsql.commit();
+ mid_pgsql.stop();
+ }
+ {
+ middle_pgsql_t mid_pgsql;
+ output_null_t out_test(&mid_pgsql, options);
+
+ mid_pgsql.start(&options);
+
+ if (test_nodes_comprehensive_set(&mid_pgsql) != 0) { throw std::runtime_error("test_nodes_comprehensive_set failed."); }
+
+ mid_pgsql.commit();
+ mid_pgsql.stop();
+ }
+ /* This should work, but doesn't. More tests are needed that look at updates
+ without the complication of ways.
+ */
+/* {
+ middle_pgsql_t mid_pgsql;
+ output_null_t out_test(&mid_pgsql, options);
+
+ mid_pgsql.start(&options);
+ mid_pgsql.commit();
+ mid_pgsql.stop();
+ // Switch to append mode because this tests updates
+ options.append = 1;
+ options.create = 0;
+ mid_pgsql.start(&options);
+ if (test_way_set(&mid_pgsql) != 0) { throw std::runtime_error("test_way_set failed."); }
+
+ mid_pgsql.commit();
+ mid_pgsql.stop();
+ }*/
+}
+int main(int argc, char *argv[]) {
+ boost::scoped_ptr<pg::tempdb> db;
+
+ try {
+ db.reset(new pg::tempdb);
+ } catch (const std::exception &e) {
+ std::cerr << "Unable to setup database: " << e.what() << "\n";
+ return 77; // <-- code to skip this test.
+ }
+
+ try {
+ options_t options;
+ options.conninfo = db->conninfo().c_str();
+ options.scale = 10000000;
+ options.cache = 1;
+ options.num_procs = 1;
+ options.prefix = "osm2pgsql_test";
+ options.slim = 1;
+
+ options.alloc_chunkwise = ALLOC_SPARSE | ALLOC_DENSE; // what you get with optimized
+ run_tests(options, "optimized");
+ options.alloc_chunkwise = ALLOC_SPARSE;
+ run_tests(options, "sparse");
+
+ options.alloc_chunkwise = ALLOC_DENSE;
+ run_tests(options, "dense");
+
+ options.alloc_chunkwise = ALLOC_DENSE | ALLOC_DENSE_CHUNK; // what you get with chunk
+ run_tests(options, "chunk");
+ } catch (const std::exception &e) {
+ std::cerr << "ERROR: " << e.what() << std::endl;
+ return 1;
+ } catch (...) {
+ std::cerr << "UNKNOWN ERROR" << std::endl;
+ return 1;
+ }
+ return 0;
+}
diff --git a/tests/test-output-multi-line.cpp b/tests/test-output-multi-line.cpp
index cf53067..fe6b60b 100644
--- a/tests/test-output-multi-line.cpp
+++ b/tests/test-output-multi-line.cpp
@@ -61,8 +61,6 @@ int main(int argc, char *argv[]) {
options_t options;
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
boost::shared_ptr<geometry_processor> processor =
diff --git a/tests/test-output-multi-point-multi-table.cpp b/tests/test-output-multi-point-multi-table.cpp
index d35f5bb..f76c147 100644
--- a/tests/test-output-multi-point-multi-table.cpp
+++ b/tests/test-output-multi-point-multi-table.cpp
@@ -67,8 +67,6 @@ int main(int argc, char *argv[]) {
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
options.prefix = "osm2pgsql_test";
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
export_list columns;
diff --git a/tests/test-output-multi-point.cpp b/tests/test-output-multi-point.cpp
index 36e102a..fe29481 100644
--- a/tests/test-output-multi-point.cpp
+++ b/tests/test-output-multi-point.cpp
@@ -62,8 +62,6 @@ int main(int argc, char *argv[]) {
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
options.prefix = "osm2pgsql_test";
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
boost::shared_ptr<geometry_processor> processor =
diff --git a/tests/test-output-multi-polygon.cpp b/tests/test-output-multi-polygon.cpp
index 38be794..98370c6 100644
--- a/tests/test-output-multi-polygon.cpp
+++ b/tests/test-output-multi-polygon.cpp
@@ -62,8 +62,6 @@ int main(int argc, char *argv[]) {
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
options.prefix = "osm2pgsql_test";
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
boost::shared_ptr<geometry_processor> processor = geometry_processor::create("polygon", &options);
diff --git a/tests/test-output-pgsql-tablespace.cpp b/tests/test-output-pgsql-tablespace.cpp
new file mode 100755
index 0000000..3f75299
--- /dev/null
+++ b/tests/test-output-pgsql-tablespace.cpp
@@ -0,0 +1,146 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "osmdata.hpp"
+#include "output-pgsql.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+#include "middle-ram.hpp"
+#include "taginfo_impl.hpp"
+#include "parse.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+#include <boost/lexical_cast.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+namespace {
+
+struct skip_test : public std::exception {
+ const char *what() { return "Test skipped."; }
+};
+
+void run_test(const char* test_name, void (*testfunc)()) {
+ try {
+ fprintf(stderr, "%s\n", test_name);
+ testfunc();
+
+ } catch (const skip_test &) {
+ exit(77); // <-- code to skip this test.
+
+ } catch (const std::exception& e) {
+ fprintf(stderr, "%s\n", e.what());
+ fprintf(stderr, "FAIL\n");
+ exit(EXIT_FAILURE);
+ }
+
+ fprintf(stderr, "PASS\n");
+}
+#define RUN_TEST(x) run_test(#x, &(x))
+
+void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
+ pg::result_ptr res = conn->exec(query);
+
+ int ntuples = PQntuples(res->get());
+ if (ntuples != 1) {
+ throw std::runtime_error((boost::format("Expected only one tuple from a query "
+ "to check COUNT(*), but got %1%. Query "
+ "was: %2%.")
+ % ntuples % query).str());
+ }
+
+ std::string numstr = PQgetvalue(res->get(), 0, 0);
+ int count = boost::lexical_cast<int>(numstr);
+
+ if (count != expected) {
+ throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+ "query: %3%.")
+ % expected % count % query).str());
+ }
+}
+
+void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
+ std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
+ "where relname = '%1%'")
+ % table_name).str();
+
+ check_count(test_conn, 1, query);
+}
+
+// "simple" test modeled on the basic regression test from
+// the python script. this is just to check everything is
+// working as expected before we start the complex stuff.
+void test_regression_simple() {
+ boost::scoped_ptr<pg::tempdb> db;
+
+ try {
+ db.reset(new pg::tempdb);
+ db->check_tblspc(); // Unlike others, these tests require a test tablespace
+ } catch (const std::exception &e) {
+ std::cerr << "Unable to setup database: " << e.what() << "\n";
+ throw skip_test();
+ }
+
+ std::string proc_name("test-output-pgsql"), input_file("-");
+ char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+ boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+ options_t options = options_t::parse(2, argv);
+ options.conninfo = db->conninfo().c_str();
+ options.num_procs = 1;
+ options.prefix = "osm2pgsql_test";
+ options.slim = 1;
+ options.style = "default.style";
+
+ options.tblsslim_index = "tablespacetest";
+ options.tblsslim_data = "tablespacetest";
+
+ boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+ osmdata_t osmdata(mid_pgsql, out_test);
+
+ boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+ osmdata.start();
+
+ if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) {
+ throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'.");
+ }
+
+ parser.reset(NULL);
+
+ osmdata.stop();
+
+ // start a new connection to run tests on
+ pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+ assert_has_table(test_conn, "osm2pgsql_test_point");
+ assert_has_table(test_conn, "osm2pgsql_test_line");
+ assert_has_table(test_conn, "osm2pgsql_test_polygon");
+ assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+ check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point");
+ check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line");
+ check_count(test_conn, 375, "SELECT count(*) FROM osm2pgsql_test_roads");
+ check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+}
+
+} // anonymous namespace
+
+int main(int argc, char *argv[]) {
+ RUN_TEST(test_regression_simple);
+
+ return 0;
+}
diff --git a/tests/test-output-pgsql-z_order.cpp b/tests/test-output-pgsql-z_order.cpp
new file mode 100644
index 0000000..c940f44
--- /dev/null
+++ b/tests/test-output-pgsql-z_order.cpp
@@ -0,0 +1,186 @@
+#include <iostream>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <cassert>
+#include <sstream>
+#include <stdexcept>
+#include <memory>
+
+#include "osmtypes.hpp"
+#include "osmdata.hpp"
+#include "output-pgsql.hpp"
+#include "options.hpp"
+#include "middle-pgsql.hpp"
+#include "middle-ram.hpp"
+#include "taginfo_impl.hpp"
+#include "parse.hpp"
+
+#include <libpq-fe.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <boost/scoped_ptr.hpp>
+#include <boost/lexical_cast.hpp>
+
+#include "tests/middle-tests.hpp"
+#include "tests/common-pg.hpp"
+
+namespace {
+
+struct skip_test : public std::exception {
+ const char *what() { return "Test skipped."; }
+};
+
+void run_test(const char* test_name, void (*testfunc)()) {
+ try {
+ fprintf(stderr, "%s\n", test_name);
+ testfunc();
+
+ } catch (const skip_test &) {
+ exit(77); // <-- code to skip this test.
+
+ } catch (const std::exception& e) {
+ fprintf(stderr, "%s\n", e.what());
+ fprintf(stderr, "FAIL\n");
+ exit(EXIT_FAILURE);
+ }
+
+ fprintf(stderr, "PASS\n");
+}
+#define RUN_TEST(x) run_test(#x, &(x))
+
+void check_string(pg::conn_ptr &conn, std::string expected, const std::string &query) {
+ pg::result_ptr res = conn->exec(query);
+
+ int ntuples = PQntuples(res->get());
+ if (ntuples != 1) {
+ throw std::runtime_error((boost::format("Expected only one tuple from a query "
+ "to check a string, but got %1%. Query "
+ "was: %2%.")
+ % ntuples % query).str());
+ }
+
+ std::string actual = PQgetvalue(res->get(), 0, 0);
+
+ if (actual != expected) {
+ throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+ "query: %3%.")
+ % expected % actual % query).str());
+ }
+}
+
+
+void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
+ pg::result_ptr res = conn->exec(query);
+
+ int ntuples = PQntuples(res->get());
+ if (ntuples != 1) {
+ throw std::runtime_error((boost::format("Expected only one tuple from a query "
+ "to check COUNT(*), but got %1%. Query "
+ "was: %2%.")
+ % ntuples % query).str());
+ }
+
+ std::string numstr = PQgetvalue(res->get(), 0, 0);
+ int count = boost::lexical_cast<int>(numstr);
+
+ if (count != expected) {
+ throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+ "query: %3%.")
+ % expected % count % query).str());
+ }
+}
+
+void check_number(pg::conn_ptr &conn, double expected, const std::string &query) {
+ pg::result_ptr res = conn->exec(query);
+
+ int ntuples = PQntuples(res->get());
+ if (ntuples != 1) {
+ throw std::runtime_error((boost::format("Expected only one tuple from a query, "
+ " but got %1%. Query was: %2%.")
+ % ntuples % query).str());
+ }
+
+ std::string numstr = PQgetvalue(res->get(), 0, 0);
+ double num = boost::lexical_cast<double>(numstr);
+
+ // floating point isn't exact, so allow a 0.01% difference
+ if ((num > 1.0001*expected) || (num < 0.9999*expected)) {
+ throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+ "query: %3%.")
+ % expected % num % query).str());
+ }
+}
+
+void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
+ std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
+ "where relname = '%1%'")
+ % table_name).str();
+
+ check_count(test_conn, 1, query);
+}
+
+// "simple" test modeled on the basic regression test from
+// the python script. this is just to check everything is
+// working as expected before we start the complex stuff.
+void test_z_order() {
+ boost::scoped_ptr<pg::tempdb> db;
+
+ try {
+ db.reset(new pg::tempdb);
+ } catch (const std::exception &e) {
+ std::cerr << "Unable to setup database: " << e.what() << "\n";
+ throw skip_test();
+ }
+
+ std::string proc_name("test-output-pgsql"), input_file("-");
+ char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+ boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+ options_t options = options_t::parse(2, argv);
+ options.conninfo = db->conninfo().c_str();
+ options.num_procs = 1;
+ options.prefix = "osm2pgsql_test";
+ options.style = "default.style";
+
+ boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+ osmdata_t osmdata(mid_pgsql, out_test);
+
+ boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+ osmdata.start();
+
+ if (parser->streamFile("libxml2", "tests/test_output_pgsql_z_order.osm", options.sanitize, &osmdata) != 0) {
+ throw std::runtime_error("Unable to read input file `tests/test_output_pgsql_z_order.osm'.");
+ }
+
+ parser.reset(NULL);
+
+ osmdata.stop();
+
+ // start a new connection to run tests on
+ pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+ assert_has_table(test_conn, "osm2pgsql_test_point");
+ assert_has_table(test_conn, "osm2pgsql_test_line");
+ assert_has_table(test_conn, "osm2pgsql_test_polygon");
+ assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+ check_string(test_conn, "motorway", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 0");
+ check_string(test_conn, "trunk", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 1");
+ check_string(test_conn, "primary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 2");
+ check_string(test_conn, "secondary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 3");
+ check_string(test_conn, "tertiary", "SELECT highway FROM osm2pgsql_test_line WHERE layer IS NULL ORDER BY z_order DESC LIMIT 1 OFFSET 4");
+
+ check_string(test_conn, "residential", "SELECT highway FROM osm2pgsql_test_line ORDER BY z_order DESC LIMIT 1 OFFSET 0");
+}
+
+} // anonymous namespace
+
+int main(int argc, char *argv[]) {
+ RUN_TEST(test_z_order);
+
+ return 0;
+}
diff --git a/tests/test-output-pgsql.cpp b/tests/test-output-pgsql.cpp
index 42f490a..b7a4053 100644
--- a/tests/test-output-pgsql.cpp
+++ b/tests/test-output-pgsql.cpp
@@ -71,6 +71,27 @@ void check_count(pg::conn_ptr &conn, int expected, const std::string &query) {
}
}
+void check_number(pg::conn_ptr &conn, double expected, const std::string &query) {
+ pg::result_ptr res = conn->exec(query);
+
+ int ntuples = PQntuples(res->get());
+ if (ntuples != 1) {
+ throw std::runtime_error((boost::format("Expected only one tuple from a query, "
+ " but got %1%. Query was: %2%.")
+ % ntuples % query).str());
+ }
+
+ std::string numstr = PQgetvalue(res->get(), 0, 0);
+ double num = boost::lexical_cast<double>(numstr);
+
+ // floating point isn't exact, so allow a 0.01% difference
+ if ((num > 1.0001*expected) || (num < 0.9999*expected)) {
+ throw std::runtime_error((boost::format("Expected %1%, but got %2%, when running "
+ "query: %3%.")
+ % expected % num % query).str());
+ }
+}
+
void assert_has_table(pg::conn_ptr &test_conn, const std::string &table_name) {
std::string query = (boost::format("select count(*) from pg_catalog.pg_class "
"where relname = '%1%'")
@@ -100,8 +121,6 @@ void test_regression_simple() {
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
options.prefix = "osm2pgsql_test";
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
options.style = "default.style";
@@ -133,8 +152,85 @@ void test_regression_simple() {
check_count(test_conn, 3300, "SELECT count(*) FROM osm2pgsql_test_line");
check_count(test_conn, 375, "SELECT count(*) FROM osm2pgsql_test_roads");
check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+
+ // Check size of lines
+ check_number(test_conn, 1696.04, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+ check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+
+ check_number(test_conn, 311.21, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+ check_number(test_conn, 311.21, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+ check_number(test_conn, 143.81, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+
+ // Check a point's location
+ check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=900913;POINT(1062645.12 5972593.4)'::geometry, 0.1)");
+}
+
+void test_latlong() {
+ boost::scoped_ptr<pg::tempdb> db;
+
+ try {
+ db.reset(new pg::tempdb);
+ } catch (const std::exception &e) {
+ std::cerr << "Unable to setup database: " << e.what() << "\n";
+ throw skip_test();
+ }
+
+ std::string proc_name("test-output-pgsql"), input_file("-");
+ char *argv[] = { &proc_name[0], &input_file[0], NULL };
+
+ boost::shared_ptr<middle_pgsql_t> mid_pgsql(new middle_pgsql_t());
+ options_t options = options_t::parse(2, argv);
+ options.conninfo = db->conninfo().c_str();
+ options.num_procs = 1;
+ options.prefix = "osm2pgsql_test";
+ options.slim = 1;
+ options.style = "default.style";
+
+ options.projection.reset(new reprojection(PROJ_LATLONG));
+ options.scale = (options.projection->get_proj_id() == PROJ_LATLONG) ? 10000000 : 100;
+
+ boost::shared_ptr<output_pgsql_t> out_test(new output_pgsql_t(mid_pgsql.get(), options));
+
+ osmdata_t osmdata(mid_pgsql, out_test);
+
+ boost::scoped_ptr<parse_delegate_t> parser(new parse_delegate_t(options.extra_attributes, options.bbox, options.projection));
+
+ osmdata.start();
+
+ if (parser->streamFile("pbf", "tests/liechtenstein-2013-08-03.osm.pbf", options.sanitize, &osmdata) != 0) {
+ throw std::runtime_error("Unable to read input file `tests/liechtenstein-2013-08-03.osm.pbf'.");
+ }
+
+ parser.reset(NULL);
+
+ osmdata.stop();
+
+ // start a new connection to run tests on
+ pg::conn_ptr test_conn = pg::conn::connect(db->conninfo());
+
+ assert_has_table(test_conn, "osm2pgsql_test_point");
+ assert_has_table(test_conn, "osm2pgsql_test_line");
+ assert_has_table(test_conn, "osm2pgsql_test_polygon");
+ assert_has_table(test_conn, "osm2pgsql_test_roads");
+
+ check_count(test_conn, 1342, "SELECT count(*) FROM osm2pgsql_test_point");
+ check_count(test_conn, 3298, "SELECT count(*) FROM osm2pgsql_test_line");
+ check_count(test_conn, 374, "SELECT count(*) FROM osm2pgsql_test_roads");
+ check_count(test_conn, 4128, "SELECT count(*) FROM osm2pgsql_test_polygon");
+
+ // Check size of lines
+ check_number(test_conn, 0.0105343, "SELECT ST_Length(way) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+ check_number(test_conn, 1151.26, "SELECT ST_Length(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_line WHERE osm_id = 44822682");
+
+ check_number(test_conn, 1.70718e-08, "SELECT way_area FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+ check_number(test_conn, 1.70718e-08, "SELECT ST_Area(way) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+ check_number(test_conn, 143.845, "SELECT ST_Area(ST_Transform(way,4326)::geography) FROM osm2pgsql_test_polygon WHERE osm_id = 157261342");
+
+ // Check a point's location
+ check_count(test_conn, 1, "SELECT count(*) FROM osm2pgsql_test_point WHERE ST_DWithin(way, 'SRID=4326;POINT(9.5459035 47.1866494)'::geometry, 0.00001)");
}
+
void test_area_way_simple() {
boost::scoped_ptr<pg::tempdb> db;
@@ -259,8 +355,6 @@ void test_clone() {
options.conninfo = db->conninfo().c_str();
options.num_procs = 1;
options.prefix = "osm2pgsql_test";
- options.tblsslim_index = "tablespacetest";
- options.tblsslim_data = "tablespacetest";
options.slim = 1;
options.style = "default.style";
@@ -302,6 +396,7 @@ void test_clone() {
int main(int argc, char *argv[]) {
RUN_TEST(test_regression_simple);
+ RUN_TEST(test_latlong);
RUN_TEST(test_clone);
RUN_TEST(test_area_way_simple);
RUN_TEST(test_route_rel);
diff --git a/tests/test_output_pgsql_z_order.osm b/tests/test_output_pgsql_z_order.osm
new file mode 100644
index 0000000..76ddd9a
--- /dev/null
+++ b/tests/test_output_pgsql_z_order.osm
@@ -0,0 +1,75 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<osm version='0.6' generator='hand'>
+ <node id='1' version='1' visible='true' lat='49' lon='-122.5'>
+ <tag k='highway' v='bus_stop' />
+ </node>
+ <node id='2' version='1' visible='true' lat='49.005' lon='-122.51'>
+ <tag k='highway' v='bus_stop' />
+ </node>
+ <node id='3' version='1' visible='true' lat='49.01' lon='-122.5' />
+ <!-- yes, all the ways have the same linestring -->
+ <way id='1' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='motorway' />
+ </way>
+ <way id='2' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='motorway_link' />
+ </way>
+ <way id='3' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='trunk' />
+ </way>
+ <way id='4' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='trunk_link' />
+ </way>
+ <way id='5' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='primary' />
+ </way>
+ <way id='6' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='primary_link' />
+ </way>
+ <way id='7' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='secondary' />
+ </way>
+ <way id='8' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='secondary_link' />
+ </way>
+ <way id='9' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='tertiary' />
+ </way>
+ <way id='10' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='tertiary_link' />
+ </way>
+ <way id='11' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <tag k='highway' v='residential' />
+ <tag k='layer' v='5' />
+ </way>
+ <!-- put *something* in the polygon table -->
+ <way id='1000' version='1' visible='true'>
+ <nd ref='1' />
+ <nd ref='2' />
+ <nd ref='3' />
+ <nd ref='1' />
+ <tag k='building' v='yes' />
+ </way>
+</osm>
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-grass/osm2pgsql.git
More information about the Pkg-grass-devel
mailing list