[mkgmap-splitter] 01/06: Imported Upstream version 0.0.0+svn468
Bas Couwenberg
sebastic at debian.org
Thu Dec 1 17:45:26 UTC 2016
This is an automated email from the git hooks/post-receive script.
sebastic pushed a commit to branch master
in repository mkgmap-splitter.
commit cb4c3e3566066427c7a46066439c841c1a6a8f6f
Author: Bas Couwenberg <sebastic at xs4all.nl>
Date: Thu Dec 1 18:31:18 2016 +0100
Imported Upstream version 0.0.0+svn468
---
doc/splitter.1 | 930 ++++++-------
doc/splitter.1.xml | 1450 ++++++++++----------
doc/splitter.txt | 2 +-
resources/splitter-version.properties | 4 +-
src/uk/me/parabola/splitter/AbstractOSMWriter.java | 12 -
src/uk/me/parabola/splitter/Area.java | 42 +-
...erDictionaryInt.java => AreaDictionaryInt.java} | 59 +-
...ctionaryShort.java => AreaDictionaryShort.java} | 140 +-
.../splitter/{WriterGrid.java => AreaGrid.java} | 124 +-
.../{WriterGridResult.java => AreaGridResult.java} | 8 +-
.../splitter/{WriterIndex.java => AreaIndex.java} | 18 +-
src/uk/me/parabola/splitter/AreaList.java | 95 +-
src/uk/me/parabola/splitter/AreasCalculator.java | 422 ++++++
src/uk/me/parabola/splitter/BinaryMapWriter.java | 14 +-
src/uk/me/parabola/splitter/DataStorer.java | 301 ++--
src/uk/me/parabola/splitter/KmlWriter.java | 1 +
src/uk/me/parabola/splitter/Main.java | 1236 +++--------------
.../me/parabola/splitter/MultiTileProcessor.java | 51 +-
src/uk/me/parabola/splitter/O5mMapWriter.java | 45 +-
src/uk/me/parabola/splitter/OSMFileHandler.java | 113 ++
src/uk/me/parabola/splitter/OSMWriter.java | 9 -
.../me/parabola/splitter/PolygonDescProcessor.java | 18 +-
.../me/parabola/splitter/ProblemListProcessor.java | 201 ++-
src/uk/me/parabola/splitter/ProblemLists.java | 240 ++++
src/uk/me/parabola/splitter/PseudoOSMWriter.java | 17 +-
.../parabola/splitter/SparseLong2ShortMapHuge.java | 2 +-
.../splitter/SparseLong2ShortMapInline.java | 2 +-
src/uk/me/parabola/splitter/SplitProcessor.java | 52 +-
.../parabola/splitter/SplittableDensityArea.java | 6 +-
29 files changed, 2844 insertions(+), 2770 deletions(-)
diff --git a/doc/splitter.1 b/doc/splitter.1
index db02a05..e64d064 100644
--- a/doc/splitter.1
+++ b/doc/splitter.1
@@ -1,465 +1,465 @@
-'\" -*- coding: us-ascii -*-
-.if \n(.g .ds T< \\FC
-.if \n(.g .ds T> \\F[\n[.fam]]
-.de URL
-\\$2 \(la\\$1\(ra\\$3
-..
-.if \n(.g .mso www.tmac
-.TH mkgmap-splitter 1 "9 January 2015" "" ""
-.SH NAME
-mkgmap-splitter \- tile splitter for mkgmap
-.SH SYNOPSIS
-'nh
-.fi
-.ad l
-\fBmkgmap-splitter\fR \kx
-.if (\nx>(\n(.l/2)) .nr x (\n(.l/5)
-'in \n(.iu+\nxu
-[\fIoptions\fR] \fIfile.osm\fR
-'in \n(.iu-\nxu
-.ad b
-'hy
-> \fI\*(T<\fIsplitter.log\fR\*(T>\fR
-.SH DESCRIPTION
-\fBmkgmap-splitter\fR splits an .osm file that contains
-large well mapped regions into a number of smaller tiles, to fit within
-the maximum size used for the Garmin maps format.
-.PP
-The two most important features are:
-.TP 0.2i
-\(bu
-Variable sized tiles to prevent a large number of tiny files.
-.TP 0.2i
-\(bu
-Tiles join exactly with no overlap or gaps.
-.PP
-You will need a lot of memory on your computer if you intend to split a
-large area.
-A few options allow configuring how much memory you need.
-With the default parameters, you need about 4-5 bytes for every node and
-way.
-This doesn't sound a lot but there are about 1700 million nodes in the
-whole planet file and so you cannot process the whole planet in one pass
-file on a 32 bit machine using this utility as the maximum java heap
-space is 2G.
-It is possible with 64 bit java and about 7GB of heap or with multiple
-passes.
-.PP
-The Europe extract from Cloudmade or Geofabrik can be processed within
-the 2G limit if you have sufficient memory.
-With the default options europe is split into about 750 tiles.
-The Europe extract is about half of the size of the complete planet file.
-.PP
-On the other hand a single country, even a well mapped one such as
-Germany or the UK, will be possible on a modest machine, even a netbook.
-.SH USAGE
-Splitter requires java 1.6 or higher.
-Basic usage is as follows.
-.PP
-.nf
-\*(T<
-\fBmkgmap\-splitter\fR \fI\fIfile.osm\fR\fR > \fI\fIsplitter.log\fR\fR
- \*(T>
-.fi
-.PP
-If you have less than 2 GB of memory on your computer you should reduce
-the \*(T<\fB\-Xmx\fR\*(T> option by setting the JAVA_OPTS environment
-variable.
-.PP
-.nf
-\*(T<
-JAVA_OPTS="\fI\-Xmx512m\fR" \fBmkgmap\-splitter\fR \fI\fIfile.osm\fR\fR > \fI\fIsplitter.log\fR\fR
- \*(T>
-.fi
-.PP
-This will produce a number of .osm.pbf files that can be read by
-\fBmkgmap\fR(1).
-There are also other files produced:
-.PP
-The \*(T<\fItemplate.args\fR\*(T> file is a file that can
-be used with the \*(T<\fB\-c\fR\*(T> option of
-\fBmkgmap\fR that will compile all the files.
-You can use it as is or you can copy it and edit it to include
-your own options.
-For example instead of each description being "OSM Map" it could
-be "NW Scotland" as appropriate.
-.PP
-The \*(T<\fIareas.list\fR\*(T> file is the list of bounding
-boxes that were calculated.
-If you want you can use this on a subsequent call the the
-splitter using the \*(T<\fB\-\-split\-file\fR\*(T> option to use
-exactly the same areas as last time.
-This might be useful if you produce a map regularly and want to
-keep the tile areas the same from month to month.
-It is also useful to avoid the time it takes to regenerate the
-file each time (currently about a third of the overall time
-taken to perform the split).
-Of course if the map grows enough that one of the tiles overflows
-you will have to re-calculate the areas again.
-.PP
-The \*(T<\fIareas.poly\fR\*(T> file contains the bounding
-polygon of the calculated areas.
-See option \*(T<\fB\-\-polygon\-file\fR\*(T> how this can be used.
-.PP
-The \*(T<\fIdensities\-out.txt\fR\*(T> file is written when
-no split-file is given and contains debugging information only.
-.PP
-You can also use a gzip'ed or bz2'ed compressed .osm file as the input
-file.
-Note that this can slow down the splitter considerably (particularly true
-for bz2) because decompressing the .osm file can take quite a lot of CPU
-power.
-If you are likely to be processing a file several times you're probably
-better off converting the file to one of the binary formats pbf or o5m.
-The o5m format is faster to read, but requires more space on the disk.
-.SH OPTIONS
-There are a number of options to fine tune things that you might want to
-try.
-.TP
-\*(T<\fB\-\-boundary\-tags=\fR\*(T>\fIstring\fR
-A comma separated list of tag values for relations.
-Used to filter multipolygon and boundary relations for
-problem-list processing. See also option \-\-wanted\-admin\-level.
-Default: use-exclude-list
-.TP
-\*(T<\fB\-\-cache=\fR\*(T>\fIstring\fR
-Deprecated, now does nothing
-.TP
-\*(T<\fB\-\-description=\fR\*(T>\fIstring\fR
-Sets the desciption to be written in to the
-\*(T<\fItemplate.args\fR\*(T> file.
-.TP
-\*(T<\fB\-\-geonames\-file=\fR\*(T>\fIstring\fR
-The name of a GeoNames file to use for determining tile names.
-Typically \*(T<\fIcities15000.zip\fR\*(T> from
-.URL http://download.geonames.org/export/dump geonames
-\&.
-.TP
-\*(T<\fB\-\-keep\-complete=\fR\*(T>\fIboolean\fR
-Use \*(T<\fB\-\-keep\-complete=false\fR\*(T> to disable two
-additional program phases between the split and the final
-distribution phase (not recommended).
-The first phase, called gen-problem-list, detects all ways and
-relations that are crossing the borders of one or more output
-files.
-The second phase, called handle-problem-list, collects the
-coordinates of these ways and relations and calculates all output
-files that are crossed or enclosed.
-The information is passed to the final dist-phase in three
-temporary files.
-This avoids broken polygons, but be aware that it requires to read
-the input files at least two additional times.
-
-Do not specify it with \*(T<\fB\-\-overlap\fR\*(T> unless you have
-a good reason to do so.
-
-Defaulte: true
-.TP
-\*(T<\fB\-\-mapid=\fR\*(T>\fIint\fR
-Set the filename for the split files.
-In the example the first file will be called
-\*(T<\fI63240001.osm.pbf\fR\*(T> and the next one will be
-\*(T<\fI63240002.osm.pbf\fR\*(T> and so on.
-
-Default: 63240001
-.TP
-\*(T<\fB\-\-max\-areas=\fR\*(T>\fIint\fR
-The maximum number of areas that can be processed in a single pass
-during the second stage of processing.
-This must be a number from 1 to 4096.
-Higher numbers mean fewer passes over the source file and hence
-quicker overall processing, but also require more memory.
-If you find you are running out of memory but don't want to
-increase your \*(T<\fB\-\-max\-nodes\fR\*(T> value, try reducing
-this instead.
-Changing this will have no effect on the result of the split, it's
-purely to let you trade off memory for performance.
-Note that the first stage of the processing has a fixed memory
-overhead regardless of what this is set to so if you are running
-out of memory before the \*(T<\fIareas.list\fR\*(T> file is
-generated, you need to either increase your \*(T<\fB\-Xmx\fR\*(T>
-value or reduce the size of the input file you're trying to split.
-
-Default: 512
-.TP
-\*(T<\fB\-\-max\-nodes=\fR\*(T>\fIint\fR
-The maximum number of nodes that can be in any of the resultant
-files.
-The default is fairly conservative, you could increase it quite a
-lot before getting any 'map too big' messages.
-Not much experimentation has been done.
-Also the bigger this value, the less memory is required during the
-splitting stage.
-
-Default: 1600000
-.TP
-\*(T<\fB\-\-max\-threads=\fR\*(T>\fIvalue\fR
-The maximum number of threads used by
-\fBmkgmap-splitter\fR.
-
-Default: 4 (auto)
-.TP
-\*(T<\fB\-\-mixed=\fR\*(T>\fIboolean\fR
-Specify this if the input osm file has nodes, ways and relations
-intermingled or the ids are not strictly sorted.
-To increase performance, use the \fBosmosis\fR sort
-function.
-
-Default: false
-.TP
-\*(T<\fB\-\-no\-trim=\fR\*(T>\fIboolean\fR
-Don't trim empty space off the edges of tiles.
-This option is ignored when \*(T<\fB\-\-polygon\-file\fR\*(T> is
-used.
-
-Default: false
-.TP
-\*(T<\fB\-\-num\-tiles=\fR\*(T>\fIvalue\fR\*(T<\fBstring\fR\*(T>
-A target value that is used when no split-file is given.
-Splitting is done so that the given number of tiles is produced.
-The \*(T<\fB\-\-max\-nodes\fR\*(T> value is ignored if this option
-is given.
-.TP
-\*(T<\fB\-\-output=\fR\*(T>\fIstring\fR
-The format in which the output files are written.
-Possible values are xml, pbf, o5m, and simulate.
-The default is pbf, which produces the smallest file sizes.
-The o5m format is faster to write, but creates around 40% larger
-files.
-The simulate option is for debugging purposes.
-.TP
-\*(T<\fB\-\-output\-dir=\fR\*(T>\fIpath\fR
-The directory to which splitter should write the output files.
-If the specified path to a directory doesn't exist,
-\fBmkgmap-splitter\fR tries to create it.
-Defaults to the current working directory.
-.TP
-\*(T<\fB\-\-overlap=\fR\*(T>\fIstring\fR
-Deprecated since r279.
-With \*(T<\fB\-\-keep\-complete=false\fR\*(T>,
-\fBmkgmap-splitter\fR should include nodes outside
-the bounding box, so that \fBmkgmap\fR can neatly
-crop exactly at the border.
-This parameter controls the size of that overlap.
-It is in map units, a default of 2000 is used which means about
-0.04 degrees of latitude or longitude.
-If \*(T<\fB\-\-keep\-complete=true\fR\*(T> is active and
-\*(T<\fB\-\-overlap\fR\*(T> is given, a warning will be printed
-because this combination rarely makes sense.
-.TP
-\*(T<\fB\-\-polygon\-desc\-file=\fR\*(T>\fIpath\fR
-An osm file (.o5m, .pbf, .osm) with named ways that describe
-bounding polygons with OSM ways having tags name and mapid.
-.TP
-\*(T<\fB\-\-polygon\-file=\fR\*(T>\fIpath\fR
-The name of a file containing a bounding polygon in the
-.URL "" "osmosis polygon file format"
-\&.
-\fBmkgmap-splitter\fR uses this file when calculating
-the areas.
-It first calculates a grid using the given
-\*(T<\fB\-\-resolution\fR\*(T>.
-The input file is read and for each node, a counter is increased
-for the related grid area.
-If the input file contains a bounding box, this is applied to the
-grid so that nodes outside of the bounding box are ignored.
-Next, if specified, the bounding polygon is used to zero those
-grid elements outside of the bounding polygon area.
-If the polygon area(s) describe(s) a rectilinear area with no more
-than 40 vertices, \fBmkgmap-splitter\fR will try to
-create output files that fit exactly into the area, otherwise it
-will approximate the polygon area with rectangles.
-.TP
-\*(T<\fB\-\-precomp\-sea=\fR\*(T>\fIpath\fR
-The name of a directory containing precompiled sea tiles.
-If given, \fBmkgmap-splitter\fR will use the
-precompiled sea tiles in the same way as \fBmkgmap\fR
-does.
-Use this if you want to use a polygon-file or
-\*(T<\fB\-\-no\-trim=true\fR\*(T> and \fBmkgmap\fR
-creates empty *.img files combined with a message starting "There
-is not enough room in a single garmin map for all the input data".
-.TP
-\*(T<\fB\-\-problem\-file=\fR\*(T>\fIpath\fR
-The name of a file containing ways and relations that are known to
-cause problems in the split process.
-Use this option if \*(T<\fB\-\-keep\-complete\fR\*(T> requires too
-much time or memory and \*(T<\fB\-\-overlap\fR\*(T> doesn't solve
-your problem.
-
-Syntax of problem file:
-
-.nf
-\*(T<
-way:<id> # comment...
-rel:<id> # comment...
- \*(T>
-.fi
-
-example:
-
-.nf
-\*(T<
-way:2784765 # Ferry Guernsey \- Jersey
- \*(T>
-.fi
-.TP
-\*(T<\fB\-\-problem\-report=\fR\*(T>\fIpath\fR
-The name of a file to write the generated problem list created with
-\*(T<\fB\-\-keep\-complete\fR\*(T>.
-The parameter is ignored if \*(T<\fB\-\-keep\-complete=false\fR\*(T>.
-You can reuse this file with the \*(T<\fB\-\-problem\-file\fR\*(T>
-parameter, but do this only if you use the same values for
-\*(T<\fB\-\-max\-nodes\fR\*(T> and \*(T<\fB\-\-resolution\fR\*(T>.
-.TP
-\*(T<\fB\-\-resolution=\fR\*(T>\fIint\fR
-The resolution of the density map produced during the first phase.
-A value between 1 and 24.
-Default is 13.
-Increasing the value to 14 requires four times more memory in the
-split phase.
-The value is ignored if a \*(T<\fB\-\-split\-file\fR\*(T> is given.
-.TP
-\*(T<\fB\-\-search\-limit=\fR\*(T>\fIint\fR
-Search limit in split algo.
-Higher values may find better splits, but will take longer.
-
-Default: 200000
-.TP
-\*(T<\fB\-\-split\-file=\fR\*(T>\fIpath\fR
-Use the previously calculated tile areas instead of calculating
-them from scratch.
-The file can be in .list or .kml format.
-.TP
-\*(T<\fB\-\-status\-freq=\fR\*(T>\fIint\fR
-Displays the amount of memory used by the JVM every
-\*(T<\fB\-\-status\-freq\fR\*(T> seconds.
-Set =0 to disable.
-
-Default: 120
-.TP
-\*(T<\fB\-\-stop\-after=\fR\*(T>\fIstring\fR
-Debugging: stop after a given program phase.
-Can be split, gen-problem-list, or handle-problem-list.
-Default is dist which means execute all phases.
-.TP
-\*(T<\fB\-\-wanted\-admin\-level=\fR\*(T>\fIint\fR
-Specifies the lowest admin_level value of boundary relations that
-should be kept complete. Used to filter boundary relations for
-problem-list processing. The default value 5 means that
-boundary relations are kept complete when the admin_level is
-5 or higher (5..11).
-The parameter is ignored if \*(T<\fB\-\-keep\-complete=false\fR\*(T>.
-Default: 5
-.TP
-\*(T<\fB\-\-write\-kml=\fR\*(T>\fIpath\fR
-The name of a kml file to write out the areas to.
-This is in addition to \*(T<\fIareas.list\fR\*(T>
-(which is always written out).
-.PP
-Special options
-.TP
-\*(T<\fB\-\-version\fR\*(T>
-If the parameter \*(T<\fB\-\-version\fR\*(T> is found somewhere in
-the options, \fBmkgmap-splitter\fR will just print
-the version info and exit.
-Version info looks like this:
-
-.nf
-\*(T<
-splitter 279 compiled 2013\-01\-12T01:45:02+0000
- \*(T>
-.fi
-.TP
-\*(T<\fB\-\-help\fR\*(T>
-If the parameter \*(T<\fB\-\-help\fR\*(T> is found somewhere in
-the options, \fBmkgmap-splitter\fR will print a list
-of all known normal options together with a short help and exit.
-.SH TUNING
-Tuning for best performance
-.PP
-A few hints for those that are using \fBmkgmap-splitter\fR
-to split large files.
-.TP 0.2i
-\(bu
-For faster processing with \*(T<\fB\-\-keep\-complete=true\fR\*(T>,
-convert the input file to o5m format using:
-
-.nf
-\*(T<
-\fBosmconvert\fR \fB\-\-drop\-version\fR \fIfile.osm\fR \fB\-o=\fR\fB\fIfile.o5m\fR\fR
- \*(T>
-.fi
-.TP 0.2i
-\(bu
-The option \*(T<\fB\-\-drop\-version\fR\*(T> is optional, it reduces
-the file to that data that is needed by
-\fBmkgmap-splitter\fR and \fBmkgmap\fR.
-.TP 0.2i
-\(bu
-If you still experience poor performance, look into
-\*(T<\fIsplitter.log\fR\*(T>.
-Search for the word Distributing.
-You may find something like this in the next line:
-
-.nf
-\*(T<
-Processing 1502 areas in 3 passes, 501 areas at a time
- \*(T>
-.fi
-
-This means splitter has to read the input file input three times
-because the \*(T<\fB\-\-max\-areas\fR\*(T> parameter was much smaller
-than the number of areas.
-If you have enough heap, set \*(T<\fB\-\-max\-areas\fR\*(T> value to a
-value that is higher than the number of areas, e.g.
-\*(T<\fB\-\-max\-areas=2048\fR\*(T>.
-Execute \fBmkgmap-splitter\fR again and you should find
-
-.nf
-\*(T<
-Processing 1502 areas in a single pass
- \*(T>
-.fi
-.TP 0.2i
-\(bu
-More areas require more memory.
-Make sure that \fBmkgmap-splitter\fR has enough heap
-(increase the \*(T<\fB\-Xmx\fR\*(T> parameter) so that it doesn't
-waste much time in the garbage collector (GC), but keep as much
-memory as possible for the systems I/O caches.
-.TP 0.2i
-\(bu
-If available, use two different disks for input file and output
-directory, esp. when you use o5m format for input and output.
-.TP 0.2i
-\(bu
-If you use \fBmkgmap\fR r2415 or later and disk space
-is no concern, consider to use \*(T<\fB\-\-output=o5m\fR\*(T> to
-speed up processing.
-.PP
-Tuning for low memory requirements
-.PP
-If your machine has less than 1 GB free memory (eg. a netbook), you can
-still use \fBmkgmap-splitter\fR, but you might have to be
-patient if you use the parameter \*(T<\fB\-\-keep\-complete\fR\*(T> and
-want to split a file like \*(T<\fIgermany.osm.pbf\fR\*(T> or a
-larger one.
-If needed, reduce the number of parrallel processed areas to 50 with the
-\*(T<\fB\-\-max\-areas\fR\*(T> parameter.
-You have to use \*(T<\fB\-\-keep\-complete=false\fR\*(T> when splitting an
-area like Europe.
-.SH NOTES
-.TP 0.2i
-\(bu
-There is no longer an upper limit on the number of areas that can be
-output (previously it was 255).
-More areas just mean potentially more passes being required over the
-\&.osm file, and hence the splitter will take longer to run.
-.TP 0.2i
-\(bu
-There is no longer a limit on how many areas a way or relation can
-belong to (previously it was 4).
-.SH "SEE ALSO"
-\fBmkgmap\fR(1),
-\fBosmconvert\fR(1)
+'\" -*- coding: us-ascii -*-
+.if \n(.g .ds T< \\FC
+.if \n(.g .ds T> \\F[\n[.fam]]
+.de URL
+\\$2 \(la\\$1\(ra\\$3
+..
+.if \n(.g .mso www.tmac
+.TH mkgmap-splitter 1 "9 January 2015" "" ""
+.SH NAME
+mkgmap-splitter \- tile splitter for mkgmap
+.SH SYNOPSIS
+'nh
+.fi
+.ad l
+\fBmkgmap-splitter\fR \kx
+.if (\nx>(\n(.l/2)) .nr x (\n(.l/5)
+'in \n(.iu+\nxu
+[\fIoptions\fR] \fIfile.osm\fR
+'in \n(.iu-\nxu
+.ad b
+'hy
+> \fI\*(T<\fIsplitter.log\fR\*(T>\fR
+.SH DESCRIPTION
+\fBmkgmap-splitter\fR splits an .osm file that contains
+large well mapped regions into a number of smaller tiles, to fit within
+the maximum size used for the Garmin maps format.
+.PP
+The two most important features are:
+.TP 0.2i
+\(bu
+Variable sized tiles to prevent a large number of tiny files.
+.TP 0.2i
+\(bu
+Tiles join exactly with no overlap or gaps.
+.PP
+You will need a lot of memory on your computer if you intend to split a
+large area.
+A few options allow configuring how much memory you need.
+With the default parameters, you need about 4-5 bytes for every node and
+way.
+This doesn't sound a lot but there are about 1700 million nodes in the
+whole planet file and so you cannot process the whole planet in one pass
+file on a 32 bit machine using this utility as the maximum java heap
+space is 2G.
+It is possible with 64 bit java and about 7GB of heap or with multiple
+passes.
+.PP
+The Europe extract from Cloudmade or Geofabrik can be processed within
+the 2G limit if you have sufficient memory.
+With the default options europe is split into about 750 tiles.
+The Europe extract is about half of the size of the complete planet file.
+.PP
+On the other hand a single country, even a well mapped one such as
+Germany or the UK, will be possible on a modest machine, even a netbook.
+.SH USAGE
+Splitter requires java 1.6 or higher.
+Basic usage is as follows.
+.PP
+.nf
+\*(T<
+\fBmkgmap\-splitter\fR \fI\fIfile.osm\fR\fR > \fI\fIsplitter.log\fR\fR
+ \*(T>
+.fi
+.PP
+If you have less than 2 GB of memory on your computer you should reduce
+the \*(T<\fB\-Xmx\fR\*(T> option by setting the JAVA_OPTS environment
+variable.
+.PP
+.nf
+\*(T<
+JAVA_OPTS="\fI\-Xmx512m\fR" \fBmkgmap\-splitter\fR \fI\fIfile.osm\fR\fR > \fI\fIsplitter.log\fR\fR
+ \*(T>
+.fi
+.PP
+This will produce a number of .osm.pbf files that can be read by
+\fBmkgmap\fR(1).
+There are also other files produced:
+.PP
+The \*(T<\fItemplate.args\fR\*(T> file is a file that can
+be used with the \*(T<\fB\-c\fR\*(T> option of
+\fBmkgmap\fR that will compile all the files.
+You can use it as is or you can copy it and edit it to include
+your own options.
+For example instead of each description being "OSM Map" it could
+be "NW Scotland" as appropriate.
+.PP
+The \*(T<\fIareas.list\fR\*(T> file is the list of bounding
+boxes that were calculated.
+If you want you can use this on a subsequent call the the
+splitter using the \*(T<\fB\-\-split\-file\fR\*(T> option to use
+exactly the same areas as last time.
+This might be useful if you produce a map regularly and want to
+keep the tile areas the same from month to month.
+It is also useful to avoid the time it takes to regenerate the
+file each time (currently about a third of the overall time
+taken to perform the split).
+Of course if the map grows enough that one of the tiles overflows
+you will have to re-calculate the areas again.
+.PP
+The \*(T<\fIareas.poly\fR\*(T> file contains the bounding
+polygon of the calculated areas.
+See option \*(T<\fB\-\-polygon\-file\fR\*(T> how this can be used.
+.PP
+The \*(T<\fIdensities\-out.txt\fR\*(T> file is written when
+no split-file is given and contains debugging information only.
+.PP
+You can also use a gzip'ed or bz2'ed compressed .osm file as the input
+file.
+Note that this can slow down the splitter considerably (particularly true
+for bz2) because decompressing the .osm file can take quite a lot of CPU
+power.
+If you are likely to be processing a file several times you're probably
+better off converting the file to one of the binary formats pbf or o5m.
+The o5m format is faster to read, but requires more space on the disk.
+.SH OPTIONS
+There are a number of options to fine tune things that you might want to
+try.
+.TP
+\*(T<\fB\-\-boundary\-tags=\fR\*(T>\fIstring\fR
+A comma separated list of tag values for relations.
+Used to filter multipolygon and boundary relations for
+problem-list processing. See also option \-\-wanted\-admin\-level.
+Default: use-exclude-list
+.TP
+\*(T<\fB\-\-cache=\fR\*(T>\fIstring\fR
+Deprecated, now does nothing
+.TP
+\*(T<\fB\-\-description=\fR\*(T>\fIstring\fR
+Sets the desciption to be written in to the
+\*(T<\fItemplate.args\fR\*(T> file.
+.TP
+\*(T<\fB\-\-geonames\-file=\fR\*(T>\fIstring\fR
+The name of a GeoNames file to use for determining tile names.
+Typically \*(T<\fIcities15000.zip\fR\*(T> from
+.URL http://download.geonames.org/export/dump geonames
+\&.
+.TP
+\*(T<\fB\-\-keep\-complete=\fR\*(T>\fIboolean\fR
+Use \*(T<\fB\-\-keep\-complete=false\fR\*(T> to disable two
+additional program phases between the split and the final
+distribution phase (not recommended).
+The first phase, called gen-problem-list, detects all ways and
+relations that are crossing the borders of one or more output
+files.
+The second phase, called handle-problem-list, collects the
+coordinates of these ways and relations and calculates all output
+files that are crossed or enclosed.
+The information is passed to the final dist-phase in three
+temporary files.
+This avoids broken polygons, but be aware that it requires to read
+the input files at least two additional times.
+
+Do not specify it with \*(T<\fB\-\-overlap\fR\*(T> unless you have
+a good reason to do so.
+
+Defaulte: true
+.TP
+\*(T<\fB\-\-mapid=\fR\*(T>\fIint\fR
+Set the filename for the split files.
+In the example the first file will be called
+\*(T<\fI63240001.osm.pbf\fR\*(T> and the next one will be
+\*(T<\fI63240002.osm.pbf\fR\*(T> and so on.
+
+Default: 63240001
+.TP
+\*(T<\fB\-\-max\-areas=\fR\*(T>\fIint\fR
+The maximum number of areas that can be processed in a single pass
+during the second stage of processing.
+This must be a number from 1 to 4096.
+Higher numbers mean fewer passes over the source file and hence
+quicker overall processing, but also require more memory.
+If you find you are running out of memory but don't want to
+increase your \*(T<\fB\-\-max\-nodes\fR\*(T> value, try reducing
+this instead.
+Changing this will have no effect on the result of the split, it's
+purely to let you trade off memory for performance.
+Note that the first stage of the processing has a fixed memory
+overhead regardless of what this is set to so if you are running
+out of memory before the \*(T<\fIareas.list\fR\*(T> file is
+generated, you need to either increase your \*(T<\fB\-Xmx\fR\*(T>
+value or reduce the size of the input file you're trying to split.
+
+Default: 512
+.TP
+\*(T<\fB\-\-max\-nodes=\fR\*(T>\fIint\fR
+The maximum number of nodes that can be in any of the resultant
+files.
+The default is fairly conservative, you could increase it quite a
+lot before getting any 'map too big' messages.
+Not much experimentation has been done.
+Also the bigger this value, the less memory is required during the
+splitting stage.
+
+Default: 1600000
+.TP
+\*(T<\fB\-\-max\-threads=\fR\*(T>\fIvalue\fR
+The maximum number of threads used by
+\fBmkgmap-splitter\fR.
+
+Default: 4 (auto)
+.TP
+\*(T<\fB\-\-mixed=\fR\*(T>\fIboolean\fR
+Specify this if the input osm file has nodes, ways and relations
+intermingled or the ids are not strictly sorted.
+To increase performance, use the \fBosmosis\fR sort
+function.
+
+Default: false
+.TP
+\*(T<\fB\-\-no\-trim=\fR\*(T>\fIboolean\fR
+Don't trim empty space off the edges of tiles.
+This option is ignored when \*(T<\fB\-\-polygon\-file\fR\*(T> is
+used.
+
+Default: false
+.TP
+\*(T<\fB\-\-num\-tiles=\fR\*(T>\fIvalue\fR\*(T<\fBstring\fR\*(T>
+A target value that is used when no split-file is given.
+Splitting is done so that the given number of tiles is produced.
+The \*(T<\fB\-\-max\-nodes\fR\*(T> value is ignored if this option
+is given.
+.TP
+\*(T<\fB\-\-output=\fR\*(T>\fIstring\fR
+The format in which the output files are written.
+Possible values are xml, pbf, o5m, and simulate.
+The default is pbf, which produces the smallest file sizes.
+The o5m format is faster to write, but creates around 40% larger
+files.
+The simulate option is for debugging purposes.
+.TP
+\*(T<\fB\-\-output\-dir=\fR\*(T>\fIpath\fR
+The directory to which splitter should write the output files.
+If the specified path to a directory doesn't exist,
+\fBmkgmap-splitter\fR tries to create it.
+Defaults to the current working directory.
+.TP
+\*(T<\fB\-\-overlap=\fR\*(T>\fIstring\fR
+Deprecated since r279.
+With \*(T<\fB\-\-keep\-complete=false\fR\*(T>,
+\fBmkgmap-splitter\fR should include nodes outside
+the bounding box, so that \fBmkgmap\fR can neatly
+crop exactly at the border.
+This parameter controls the size of that overlap.
+It is in map units, a default of 2000 is used which means about
+0.04 degrees of latitude or longitude.
+If \*(T<\fB\-\-keep\-complete=true\fR\*(T> is active and
+\*(T<\fB\-\-overlap\fR\*(T> is given, a warning will be printed
+because this combination rarely makes sense.
+.TP
+\*(T<\fB\-\-polygon\-desc\-file=\fR\*(T>\fIpath\fR
+An osm file (.o5m, .pbf, .osm) with named ways that describe
+bounding polygons with OSM ways having tags name and mapid.
+.TP
+\*(T<\fB\-\-polygon\-file=\fR\*(T>\fIpath\fR
+The name of a file containing a bounding polygon in the
+.URL "" "osmosis polygon file format"
+\&.
+\fBmkgmap-splitter\fR uses this file when calculating
+the areas.
+It first calculates a grid using the given
+\*(T<\fB\-\-resolution\fR\*(T>.
+The input file is read and for each node, a counter is increased
+for the related grid area.
+If the input file contains a bounding box, this is applied to the
+grid so that nodes outside of the bounding box are ignored.
+Next, if specified, the bounding polygon is used to zero those
+grid elements outside of the bounding polygon area.
+If the polygon area(s) describe(s) a rectilinear area with no more
+than 40 vertices, \fBmkgmap-splitter\fR will try to
+create output files that fit exactly into the area, otherwise it
+will approximate the polygon area with rectangles.
+.TP
+\*(T<\fB\-\-precomp\-sea=\fR\*(T>\fIpath\fR
+The name of a directory containing precompiled sea tiles.
+If given, \fBmkgmap-splitter\fR will use the
+precompiled sea tiles in the same way as \fBmkgmap\fR
+does.
+Use this if you want to use a polygon-file or
+\*(T<\fB\-\-no\-trim=true\fR\*(T> and \fBmkgmap\fR
+creates empty *.img files combined with a message starting "There
+is not enough room in a single garmin map for all the input data".
+.TP
+\*(T<\fB\-\-problem\-file=\fR\*(T>\fIpath\fR
+The name of a file containing ways and relations that are known to
+cause problems in the split process.
+Use this option if \*(T<\fB\-\-keep\-complete\fR\*(T> requires too
+much time or memory and \*(T<\fB\-\-overlap\fR\*(T> doesn't solve
+your problem.
+
+Syntax of problem file:
+
+.nf
+\*(T<
+way:<id> # comment...
+rel:<id> # comment...
+ \*(T>
+.fi
+
+example:
+
+.nf
+\*(T<
+way:2784765 # Ferry Guernsey \- Jersey
+ \*(T>
+.fi
+.TP
+\*(T<\fB\-\-problem\-report=\fR\*(T>\fIpath\fR
+The name of a file to write the generated problem list created with
+\*(T<\fB\-\-keep\-complete\fR\*(T>.
+The parameter is ignored if \*(T<\fB\-\-keep\-complete=false\fR\*(T>.
+You can reuse this file with the \*(T<\fB\-\-problem\-file\fR\*(T>
+parameter, but do this only if you use the same values for
+\*(T<\fB\-\-max\-nodes\fR\*(T> and \*(T<\fB\-\-resolution\fR\*(T>.
+.TP
+\*(T<\fB\-\-resolution=\fR\*(T>\fIint\fR
+The resolution of the density map produced during the first phase.
+A value between 1 and 24.
+Default is 13.
+Increasing the value to 14 requires four times more memory in the
+split phase.
+The value is ignored if a \*(T<\fB\-\-split\-file\fR\*(T> is given.
+.TP
+\*(T<\fB\-\-search\-limit=\fR\*(T>\fIint\fR
+Search limit in split algo.
+Higher values may find better splits, but will take longer.
+
+Default: 200000
+.TP
+\*(T<\fB\-\-split\-file=\fR\*(T>\fIpath\fR
+Use the previously calculated tile areas instead of calculating
+them from scratch.
+The file can be in .list or .kml format.
+.TP
+\*(T<\fB\-\-status\-freq=\fR\*(T>\fIint\fR
+Displays the amount of memory used by the JVM every
+\*(T<\fB\-\-status\-freq\fR\*(T> seconds.
+Set =0 to disable.
+
+Default: 120
+.TP
+\*(T<\fB\-\-stop\-after=\fR\*(T>\fIstring\fR
+Debugging: stop after a given program phase.
+Can be split, gen-problem-list, or handle-problem-list.
+Default is dist which means execute all phases.
+.TP
+\*(T<\fB\-\-wanted\-admin\-level=\fR\*(T>\fIint\fR
+Specifies the lowest admin_level value of boundary relations that
+should be kept complete. Used to filter boundary relations for
+problem-list processing. The default value 5 means that
+boundary relations are kept complete when the admin_level is
+5 or higher (5..11).
+The parameter is ignored if \*(T<\fB\-\-keep\-complete=false\fR\*(T>.
+Default: 5
+.TP
+\*(T<\fB\-\-write\-kml=\fR\*(T>\fIpath\fR
+The name of a kml file to write out the areas to.
+This is in addition to \*(T<\fIareas.list\fR\*(T>
+(which is always written out).
+.PP
+Special options
+.TP
+\*(T<\fB\-\-version\fR\*(T>
+If the parameter \*(T<\fB\-\-version\fR\*(T> is found somewhere in
+the options, \fBmkgmap-splitter\fR will just print
+the version info and exit.
+Version info looks like this:
+
+.nf
+\*(T<
+splitter 279 compiled 2013\-01\-12T01:45:02+0000
+ \*(T>
+.fi
+.TP
+\*(T<\fB\-\-help\fR\*(T>
+If the parameter \*(T<\fB\-\-help\fR\*(T> is found somewhere in
+the options, \fBmkgmap-splitter\fR will print a list
+of all known normal options together with a short help and exit.
+.SH TUNING
+Tuning for best performance
+.PP
+A few hints for those that are using \fBmkgmap-splitter\fR
+to split large files.
+.TP 0.2i
+\(bu
+For faster processing with \*(T<\fB\-\-keep\-complete=true\fR\*(T>,
+convert the input file to o5m format using:
+
+.nf
+\*(T<
+\fBosmconvert\fR \fB\-\-drop\-version\fR \fIfile.osm\fR \fB\-o=\fR\fB\fIfile.o5m\fR\fR
+ \*(T>
+.fi
+.TP 0.2i
+\(bu
+The option \*(T<\fB\-\-drop\-version\fR\*(T> is optional, it reduces
+the file to that data that is needed by
+\fBmkgmap-splitter\fR and \fBmkgmap\fR.
+.TP 0.2i
+\(bu
+If you still experience poor performance, look into
+\*(T<\fIsplitter.log\fR\*(T>.
+Search for the word Distributing.
+You may find something like this in the next line:
+
+.nf
+\*(T<
+Processing 1502 areas in 3 passes, 501 areas at a time
+ \*(T>
+.fi
+
+This means splitter has to read the input file input three times
+because the \*(T<\fB\-\-max\-areas\fR\*(T> parameter was much smaller
+than the number of areas.
+If you have enough heap, set \*(T<\fB\-\-max\-areas\fR\*(T> value to a
+value that is higher than the number of areas, e.g.
+\*(T<\fB\-\-max\-areas=2048\fR\*(T>.
+Execute \fBmkgmap-splitter\fR again and you should find
+
+.nf
+\*(T<
+Processing 1502 areas in a single pass
+ \*(T>
+.fi
+.TP 0.2i
+\(bu
+More areas require more memory.
+Make sure that \fBmkgmap-splitter\fR has enough heap
+(increase the \*(T<\fB\-Xmx\fR\*(T> parameter) so that it doesn't
+waste much time in the garbage collector (GC), but keep as much
+memory as possible for the systems I/O caches.
+.TP 0.2i
+\(bu
+If available, use two different disks for input file and output
+directory, esp. when you use o5m format for input and output.
+.TP 0.2i
+\(bu
+If you use \fBmkgmap\fR r2415 or later and disk space
+is no concern, consider to use \*(T<\fB\-\-output=o5m\fR\*(T> to
+speed up processing.
+.PP
+Tuning for low memory requirements
+.PP
+If your machine has less than 1 GB free memory (eg. a netbook), you can
+still use \fBmkgmap-splitter\fR, but you might have to be
+patient if you use the parameter \*(T<\fB\-\-keep\-complete\fR\*(T> and
+want to split a file like \*(T<\fIgermany.osm.pbf\fR\*(T> or a
+larger one.
+If needed, reduce the number of parallel processed areas to 50 with the
+\*(T<\fB\-\-max\-areas\fR\*(T> parameter.
+You have to use \*(T<\fB\-\-keep\-complete=false\fR\*(T> when splitting an
+area like Europe.
+.SH NOTES
+.TP 0.2i
+\(bu
+There is no longer an upper limit on the number of areas that can be
+output (previously it was 255).
+More areas just mean potentially more passes being required over the
+\&.osm file, and hence the splitter will take longer to run.
+.TP 0.2i
+\(bu
+There is no longer a limit on how many areas a way or relation can
+belong to (previously it was 4).
+.SH "SEE ALSO"
+\fBmkgmap\fR(1),
+\fBosmconvert\fR(1)
diff --git a/doc/splitter.1.xml b/doc/splitter.1.xml
index ee2f661..a093a47 100644
--- a/doc/splitter.1.xml
+++ b/doc/splitter.1.xml
@@ -1,725 +1,725 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
-<refentry id='mkgmap-splitter'>
-
- <refmeta>
- <refentrytitle>mkgmap-splitter</refentrytitle>
- <manvolnum>1</manvolnum>
- </refmeta>
-
- <refnamediv>
- <refname>mkgmap-splitter</refname>
- <refpurpose>tile splitter for mkgmap</refpurpose>
- </refnamediv>
-
- <refsynopsisdiv id='synopsis'>
- <cmdsynopsis>
- <command>mkgmap-splitter</command>
- <arg choice='opt'><replaceable>options</replaceable></arg>
- <arg choice='plain'><replaceable><filename>file.osm</filename></replaceable></arg>
- </cmdsynopsis>
- > <replaceable><filename>splitter.log</filename></replaceable>
- </refsynopsisdiv>
-
- <refsect1 id='description'>
- <title>DESCRIPTION</title>
- <para>
- <command>mkgmap-splitter</command> splits an .osm file that contains
- large well mapped regions into a number of smaller tiles, to fit within
- the maximum size used for the Garmin maps format.
- </para>
- <para>
- The two most important features are:
- <itemizedlist>
- <listitem>
- <para>
- Variable sized tiles to prevent a large number of tiny files.
- </para>
- </listitem>
- <listitem>
- <para>
- Tiles join exactly with no overlap or gaps.
- </para>
- </listitem>
- </itemizedlist>
- </para>
- <para>
- You will need a lot of memory on your computer if you intend to split a
- large area.
- A few options allow configuring how much memory you need.
- With the default parameters, you need about 4-5 bytes for every node and
- way.
- This doesn't sound a lot but there are about 1700 million nodes in the
- whole planet file and so you cannot process the whole planet in one pass
- file on a 32 bit machine using this utility as the maximum java heap
- space is 2G.
- It is possible with 64 bit java and about 7GB of heap or with multiple
- passes.
- </para>
- <para>
- The Europe extract from Cloudmade or Geofabrik can be processed within
- the 2G limit if you have sufficient memory.
- With the default options europe is split into about 750 tiles.
- The Europe extract is about half of the size of the complete planet file.
- </para>
- <para>
- On the other hand a single country, even a well mapped one such as
- Germany or the UK, will be possible on a modest machine, even a netbook.
- </para>
- </refsect1>
-
- <refsect1 id='usage'>
- <title>USAGE</title>
- <para>
- Splitter requires java 1.6 or higher.
- Basic usage is as follows.
- </para>
- <screen>
-<command>mkgmap-splitter</command> <replaceable><filename>file.osm</filename></replaceable> > <replaceable><filename>splitter.log</filename></replaceable>
- </screen>
- <para>
- If you have less than 2 GB of memory on your computer you should reduce
- the <option>-Xmx</option> option by setting the JAVA_OPTS environment
- variable.
- </para>
- <screen>
-JAVA_OPTS="<replaceable>-Xmx512m</replaceable>" <command>mkgmap-splitter</command> <replaceable><filename>file.osm</filename></replaceable> > <replaceable><filename>splitter.log</filename></replaceable>
- </screen>
- <para>
- This will produce a number of .osm.pbf files that can be read by
- <citerefentry>
- <refentrytitle>mkgmap</refentrytitle>
- <manvolnum>1</manvolnum>
- </citerefentry>.
- There are also other files produced:
- </para>
- <para>
- The <filename>template.args</filename> file is a file that can
- be used with the <option>-c</option> option of
- <command>mkgmap</command> that will compile all the files.
- You can use it as is or you can copy it and edit it to include
- your own options.
- For example instead of each description being "OSM Map" it could
- be "NW Scotland" as appropriate.
- </para>
- <para>
- The <filename>areas.list</filename> file is the list of bounding
- boxes that were calculated.
- If you want you can use this on a subsequent call the the
- splitter using the <option>--split-file</option> option to use
- exactly the same areas as last time.
- This might be useful if you produce a map regularly and want to
- keep the tile areas the same from month to month.
- It is also useful to avoid the time it takes to regenerate the
- file each time (currently about a third of the overall time
- taken to perform the split).
- Of course if the map grows enough that one of the tiles overflows
- you will have to re-calculate the areas again.
- </para>
- <para>
- The <filename>areas.poly</filename> file contains the bounding
- polygon of the calculated areas.
- See option <option>--polygon-file</option> how this can be used.
- </para>
- <para>
- The <filename>densities-out.txt</filename> file is written when
- no split-file is given and contains debugging information only.
- </para>
- <para>
- You can also use a gzip'ed or bz2'ed compressed .osm file as the input
- file.
- Note that this can slow down the splitter considerably (particularly true
- for bz2) because decompressing the .osm file can take quite a lot of CPU
- power.
- If you are likely to be processing a file several times you're probably
- better off converting the file to one of the binary formats pbf or o5m.
- The o5m format is faster to read, but requires more space on the disk.
- </para>
- </refsect1>
-
- <refsect1 id='options'>
- <title>OPTIONS</title>
- <para>
- There are a number of options to fine tune things that you might want to
- try.
- </para>
- <variablelist>
-
- <varlistentry>
- <term><option>--boundary-tags=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- A comma separated list of tag values for relations.
- Used to filter multipolygon and boundary relations for
- problem-list processing.
- See also option <option>--wanted-admin-level</option>.
- Default: use-exclude-list
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--cache=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- Deprecated, now does nothing
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--description=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- Sets the desciption to be written in to the
- <filename>template.args</filename> file.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--geonames-file=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- The name of a GeoNames file to use for determining tile names.
- Typically <filename>cities15000.zip</filename> from
- <ulink url="http://download.geonames.org/export/dump">geonames</ulink>.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--keep-complete=<replaceable>boolean</replaceable></option></term>
- <listitem>
- <para>
- Use <option>--keep-complete=false</option> to disable two
- additional program phases between the split and the final
- distribution phase (not recommended).
- The first phase, called gen-problem-list, detects all ways and
- relations that are crossing the borders of one or more output
- files.
- The second phase, called handle-problem-list, collects the
- coordinates of these ways and relations and calculates all output
- files that are crossed or enclosed.
- The information is passed to the final dist-phase in three
- temporary files.
- This avoids broken polygons, but be aware that it requires to read
- the input files at least two additional times.
- </para>
- <para>
- Do not specify it with <option>--overlap</option> unless you have
- a good reason to do so.
- </para>
- <para>
- Defaulte: true
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--mapid=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- Set the filename for the split files.
- In the example the first file will be called
- <filename>63240001.osm.pbf</filename> and the next one will be
- <filename>63240002.osm.pbf</filename> and so on.
- </para>
- <para>
- Default: 63240001
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--max-areas=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- The maximum number of areas that can be processed in a single pass
- during the second stage of processing.
- This must be a number from 1 to 4096.
- Higher numbers mean fewer passes over the source file and hence
- quicker overall processing, but also require more memory.
- If you find you are running out of memory but don't want to
- increase your <option>--max-nodes</option> value, try reducing
- this instead.
- Changing this will have no effect on the result of the split, it's
- purely to let you trade off memory for performance.
- Note that the first stage of the processing has a fixed memory
- overhead regardless of what this is set to so if you are running
- out of memory before the <filename>areas.list</filename> file is
- generated, you need to either increase your <option>-Xmx</option>
- value or reduce the size of the input file you're trying to split.
- </para>
- <para>
- Default: 512
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--max-nodes=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- The maximum number of nodes that can be in any of the resultant
- files.
- The default is fairly conservative, you could increase it quite a
- lot before getting any 'map too big' messages.
- Not much experimentation has been done.
- Also the bigger this value, the less memory is required during the
- splitting stage.
- </para>
- <para>
- Default: 1600000
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--max-threads=<replaceable>value</replaceable></option></term>
- <listitem>
- <para>
- The maximum number of threads used by
- <command>mkgmap-splitter</command>.
- </para>
- <para>
- Default: 4 (auto)
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--mixed=<replaceable>boolean</replaceable></option></term>
- <listitem>
- <para>
- Specify this if the input osm file has nodes, ways and relations
- intermingled or the ids are not strictly sorted.
- To increase performance, use the <command>osmosis</command> sort
- function.
- </para>
- <para>
- Default: false
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--no-trim=<replaceable>boolean</replaceable></option></term>
- <listitem>
- <para>
- Don't trim empty space off the edges of tiles.
- This option is ignored when <option>--polygon-file</option> is
- used.
- </para>
- <para>
- Default: false
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--num-tiles=<replaceable>value</replaceable>string</option></term>
- <listitem>
- <para>
- A target value that is used when no split-file is given.
- Splitting is done so that the given number of tiles is produced.
- The <option>--max-nodes</option> value is ignored if this option
- is given.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--output=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- The format in which the output files are written.
- Possible values are xml, pbf, o5m, and simulate.
- The default is pbf, which produces the smallest file sizes.
- The o5m format is faster to write, but creates around 40% larger
- files.
- The simulate option is for debugging purposes.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--output-dir=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The directory to which splitter should write the output files.
- If the specified path to a directory doesn't exist,
- <command>mkgmap-splitter</command> tries to create it.
- Defaults to the current working directory.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--overlap=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- Deprecated since r279.
- With <option>--keep-complete=false</option>,
- <command>mkgmap-splitter</command> should include nodes outside
- the bounding box, so that <command>mkgmap</command> can neatly
- crop exactly at the border.
- This parameter controls the size of that overlap.
- It is in map units, a default of 2000 is used which means about
- 0.04 degrees of latitude or longitude.
- If <option>--keep-complete=true</option> is active and
- <option>--overlap</option> is given, a warning will be printed
- because this combination rarely makes sense.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--polygon-desc-file=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- An osm file (.o5m, .pbf, .osm) with named ways that describe
- bounding polygons with OSM ways having tags name and mapid.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--polygon-file=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The name of a file containing a bounding polygon in the
- <ulink url="">osmosis polygon file format</ulink>.
- <command>mkgmap-splitter</command> uses this file when calculating
- the areas.
- It first calculates a grid using the given
- <option>--resolution</option>.
- The input file is read and for each node, a counter is increased
- for the related grid area.
- If the input file contains a bounding box, this is applied to the
- grid so that nodes outside of the bounding box are ignored.
- Next, if specified, the bounding polygon is used to zero those
- grid elements outside of the bounding polygon area.
- If the polygon area(s) describe(s) a rectilinear area with no more
- than 40 vertices, <command>mkgmap-splitter</command> will try to
- create output files that fit exactly into the area, otherwise it
- will approximate the polygon area with rectangles.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--precomp-sea=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The name of a directory containing precompiled sea tiles.
- If given, <command>mkgmap-splitter</command> will use the
- precompiled sea tiles in the same way as <command>mkgmap</command>
- does.
- Use this if you want to use a polygon-file or
- <option>--no-trim=true</option> and <command>mkgmap</command>
- creates empty *.img files combined with a message starting "There
- is not enough room in a single garmin map for all the input data".
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--problem-file=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The name of a file containing ways and relations that are known to
- cause problems in the split process.
- Use this option if <option>--keep-complete</option> requires too
- much time or memory and <option>--overlap</option> doesn't solve
- your problem.
- </para>
- <para>
- Syntax of problem file:
- </para>
- <programlisting>
-way:<id> # comment...
-rel:<id> # comment...
- </programlisting>
- <para>
- example:
- </para>
- <programlisting>
-way:2784765 # Ferry Guernsey - Jersey
- </programlisting>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--problem-report=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The name of a file to write the generated problem list created with
- <option>--keep-complete</option>.
- The parameter is ignored if <option>--keep-complete=false</option>.
- You can reuse this file with the <option>--problem-file</option>
- parameter, but do this only if you use the same values for
- <option>--max-nodes</option> and <option>--resolution</option>.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--resolution=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- The resolution of the density map produced during the first phase.
- A value between 1 and 24.
- Default is 13.
- Increasing the value to 14 requires four times more memory in the
- split phase.
- The value is ignored if a <option>--split-file</option> is given.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--search-limit=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- Search limit in split algo.
- Higher values may find better splits, but will take longer.
- </para>
- <para>
- Default: 200000
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--split-file=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- Use the previously calculated tile areas instead of calculating
- them from scratch.
- The file can be in .list or .kml format.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--status-freq=<replaceable>int</replaceable></option></term>
- <listitem>
- <para>
- Displays the amount of memory used by the JVM every
- <option>--status-freq</option> seconds.
- Set =0 to disable.
- </para>
- <para>
- Default: 120
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--stop-after=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- Debugging: stop after a given program phase.
- Can be split, gen-problem-list, or handle-problem-list.
- Default is dist which means execute all phases.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--wanted-admin-level=<replaceable>string</replaceable></option></term>
- <listitem>
- <para>
- Specifies the lowest admin_level value of boundary relations that
- should be kept complete. Used to filter boundary relations for
- problem-list processing. The default value 5 means that
- boundary relations are kept complete when the admin_level is
- 5 or higher (5..11).
- The parameter is ignored if <option>--keep-complete=false</option>.
- Default: 5
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--write-kml=<replaceable>path</replaceable></option></term>
- <listitem>
- <para>
- The name of a kml file to write out the areas to.
- This is in addition to <filename>areas.list</filename>
- (which is always written out).
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- <para>
- Special options
- </para>
- <variablelist>
-
- <varlistentry>
- <term><option>--version</option></term>
- <listitem>
- <para>
- If the parameter <option>--version</option> is found somewhere in
- the options, <command>mkgmap-splitter</command> will just print
- the version info and exit.
- Version info looks like this:
- <screen>
-splitter 279 compiled 2013-01-12T01:45:02+0000
- </screen>
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><option>--help</option></term>
- <listitem>
- <para>
- If the parameter <option>--help</option> is found somewhere in
- the options, <command>mkgmap-splitter</command> will print a list
- of all known normal options together with a short help and exit.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- </refsect1>
-
- <refsect1 id='tuning'>
- <title>TUNING</title>
-
- <para>
- Tuning for best performance
- </para>
- <para>
- A few hints for those that are using <command>mkgmap-splitter</command>
- to split large files.
- </para>
- <itemizedlist>
-
- <listitem>
- <para>
- For faster processing with <option>--keep-complete=true</option>,
- convert the input file to o5m format using:
- <screen>
-<command>osmconvert</command> <option>--drop-version</option> <filename>file.osm</filename> <option>-o=<filename>file.o5m</filename></option>
- </screen>
- </para>
- </listitem>
-
- <listitem>
- <para>
- The option <option>--drop-version</option> is optional, it reduces
- the file to that data that is needed by
- <command>mkgmap-splitter</command> and <command>mkgmap</command>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- If you still experience poor performance, look into
- <filename>splitter.log</filename>.
- Search for the word Distributing.
- You may find something like this in the next line:
- <screen>
-Processing 1502 areas in 3 passes, 501 areas at a time
- </screen>
- This means splitter has to read the input file input three times
- because the <option>--max-areas</option> parameter was much smaller
- than the number of areas.
- If you have enough heap, set <option>--max-areas</option> value to a
- value that is higher than the number of areas, e.g.
- <option>--max-areas=2048</option>.
- Execute <command>mkgmap-splitter</command> again and you should find
- <screen>
-Processing 1502 areas in a single pass
- </screen>
- </para>
- </listitem>
-
- <listitem>
- <para>
- More areas require more memory.
- Make sure that <command>mkgmap-splitter</command> has enough heap
- (increase the <option>-Xmx</option> parameter) so that it doesn't
- waste much time in the garbage collector (GC), but keep as much
- memory as possible for the systems I/O caches.
- </para>
- </listitem>
-
- <listitem>
- <para>
- If available, use two different disks for input file and output
- directory, esp. when you use o5m format for input and output.
- </para>
- </listitem>
-
- <listitem>
- <para>
- If you use <command>mkgmap</command> r2415 or later and disk space
- is no concern, consider to use <option>--output=o5m</option> to
- speed up processing.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- Tuning for low memory requirements
- </para>
- <para>
- If your machine has less than 1 GB free memory (eg. a netbook), you can
- still use <command>mkgmap-splitter</command>, but you might have to be
- patient if you use the parameter <option>--keep-complete</option> and
- want to split a file like <filename>germany.osm.pbf</filename> or a
- larger one.
- If needed, reduce the number of parrallel processed areas to 50 with the
- <option>--max-areas</option> parameter.
- You have to use <option>--keep-complete=false</option> when splitting an
- area like Europe.
- </para>
- </refsect1>
-
- <refsect1 id='notes'>
- <title>NOTES</title>
- <itemizedlist>
-
- <listitem>
- <para>
- There is no longer an upper limit on the number of areas that can be
- output (previously it was 255).
- More areas just mean potentially more passes being required over the
- .osm file, and hence the splitter will take longer to run.
- </para>
- </listitem>
-
- <listitem>
- <para>
- There is no longer a limit on how many areas a way or relation can
- belong to (previously it was 4).
- </para>
- </listitem>
-
- </itemizedlist>
- </refsect1>
-
- <refsect1 id='see-also'>
- <title>SEE ALSO</title>
-
- <citerefentry>
- <refentrytitle>mkgmap</refentrytitle>
- <manvolnum>1</manvolnum>
- </citerefentry>,
- <citerefentry>
- <refentrytitle>osmconvert</refentrytitle>
- <manvolnum>1</manvolnum>
- </citerefentry>
-
- </refsect1>
-
-</refentry>
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
+<refentry id='mkgmap-splitter'>
+
+ <refmeta>
+ <refentrytitle>mkgmap-splitter</refentrytitle>
+ <manvolnum>1</manvolnum>
+ </refmeta>
+
+ <refnamediv>
+ <refname>mkgmap-splitter</refname>
+ <refpurpose>tile splitter for mkgmap</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv id='synopsis'>
+ <cmdsynopsis>
+ <command>mkgmap-splitter</command>
+ <arg choice='opt'><replaceable>options</replaceable></arg>
+ <arg choice='plain'><replaceable><filename>file.osm</filename></replaceable></arg>
+ </cmdsynopsis>
+ > <replaceable><filename>splitter.log</filename></replaceable>
+ </refsynopsisdiv>
+
+ <refsect1 id='description'>
+ <title>DESCRIPTION</title>
+ <para>
+ <command>mkgmap-splitter</command> splits an .osm file that contains
+ large well mapped regions into a number of smaller tiles, to fit within
+ the maximum size used for the Garmin maps format.
+ </para>
+ <para>
+ The two most important features are:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Variable sized tiles to prevent a large number of tiny files.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Tiles join exactly with no overlap or gaps.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ You will need a lot of memory on your computer if you intend to split a
+ large area.
+ A few options allow configuring how much memory you need.
+ With the default parameters, you need about 4-5 bytes for every node and
+ way.
+ This doesn't sound a lot but there are about 1700 million nodes in the
+ whole planet file and so you cannot process the whole planet in one pass
+ file on a 32 bit machine using this utility as the maximum java heap
+ space is 2G.
+ It is possible with 64 bit java and about 7GB of heap or with multiple
+ passes.
+ </para>
+ <para>
+ The Europe extract from Cloudmade or Geofabrik can be processed within
+ the 2G limit if you have sufficient memory.
+ With the default options europe is split into about 750 tiles.
+ The Europe extract is about half of the size of the complete planet file.
+ </para>
+ <para>
+ On the other hand a single country, even a well mapped one such as
+ Germany or the UK, will be possible on a modest machine, even a netbook.
+ </para>
+ </refsect1>
+
+ <refsect1 id='usage'>
+ <title>USAGE</title>
+ <para>
+ Splitter requires java 1.6 or higher.
+ Basic usage is as follows.
+ </para>
+ <screen>
+<command>mkgmap-splitter</command> <replaceable><filename>file.osm</filename></replaceable> > <replaceable><filename>splitter.log</filename></replaceable>
+ </screen>
+ <para>
+ If you have less than 2 GB of memory on your computer you should reduce
+ the <option>-Xmx</option> option by setting the JAVA_OPTS environment
+ variable.
+ </para>
+ <screen>
+JAVA_OPTS="<replaceable>-Xmx512m</replaceable>" <command>mkgmap-splitter</command> <replaceable><filename>file.osm</filename></replaceable> > <replaceable><filename>splitter.log</filename></replaceable>
+ </screen>
+ <para>
+ This will produce a number of .osm.pbf files that can be read by
+ <citerefentry>
+ <refentrytitle>mkgmap</refentrytitle>
+ <manvolnum>1</manvolnum>
+ </citerefentry>.
+ There are also other files produced:
+ </para>
+ <para>
+ The <filename>template.args</filename> file is a file that can
+ be used with the <option>-c</option> option of
+ <command>mkgmap</command> that will compile all the files.
+ You can use it as is or you can copy it and edit it to include
+ your own options.
+ For example instead of each description being "OSM Map" it could
+ be "NW Scotland" as appropriate.
+ </para>
+ <para>
+ The <filename>areas.list</filename> file is the list of bounding
+ boxes that were calculated.
+ If you want you can use this on a subsequent call the the
+ splitter using the <option>--split-file</option> option to use
+ exactly the same areas as last time.
+ This might be useful if you produce a map regularly and want to
+ keep the tile areas the same from month to month.
+ It is also useful to avoid the time it takes to regenerate the
+ file each time (currently about a third of the overall time
+ taken to perform the split).
+ Of course if the map grows enough that one of the tiles overflows
+ you will have to re-calculate the areas again.
+ </para>
+ <para>
+ The <filename>areas.poly</filename> file contains the bounding
+ polygon of the calculated areas.
+ See option <option>--polygon-file</option> how this can be used.
+ </para>
+ <para>
+ The <filename>densities-out.txt</filename> file is written when
+ no split-file is given and contains debugging information only.
+ </para>
+ <para>
+ You can also use a gzip'ed or bz2'ed compressed .osm file as the input
+ file.
+ Note that this can slow down the splitter considerably (particularly true
+ for bz2) because decompressing the .osm file can take quite a lot of CPU
+ power.
+ If you are likely to be processing a file several times you're probably
+ better off converting the file to one of the binary formats pbf or o5m.
+ The o5m format is faster to read, but requires more space on the disk.
+ </para>
+ </refsect1>
+
+ <refsect1 id='options'>
+ <title>OPTIONS</title>
+ <para>
+ There are a number of options to fine tune things that you might want to
+ try.
+ </para>
+ <variablelist>
+
+ <varlistentry>
+ <term><option>--boundary-tags=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ A comma separated list of tag values for relations.
+ Used to filter multipolygon and boundary relations for
+ problem-list processing.
+ See also option <option>--wanted-admin-level</option>.
+ Default: use-exclude-list
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--cache=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ Deprecated, now does nothing
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--description=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ Sets the desciption to be written in to the
+ <filename>template.args</filename> file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--geonames-file=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a GeoNames file to use for determining tile names.
+ Typically <filename>cities15000.zip</filename> from
+ <ulink url="http://download.geonames.org/export/dump">geonames</ulink>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--keep-complete=<replaceable>boolean</replaceable></option></term>
+ <listitem>
+ <para>
+ Use <option>--keep-complete=false</option> to disable two
+ additional program phases between the split and the final
+ distribution phase (not recommended).
+ The first phase, called gen-problem-list, detects all ways and
+ relations that are crossing the borders of one or more output
+ files.
+ The second phase, called handle-problem-list, collects the
+ coordinates of these ways and relations and calculates all output
+ files that are crossed or enclosed.
+ The information is passed to the final dist-phase in three
+ temporary files.
+ This avoids broken polygons, but be aware that it requires to read
+ the input files at least two additional times.
+ </para>
+ <para>
+ Do not specify it with <option>--overlap</option> unless you have
+ a good reason to do so.
+ </para>
+ <para>
+ Defaulte: true
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--mapid=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ Set the filename for the split files.
+ In the example the first file will be called
+ <filename>63240001.osm.pbf</filename> and the next one will be
+ <filename>63240002.osm.pbf</filename> and so on.
+ </para>
+ <para>
+ Default: 63240001
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--max-areas=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of areas that can be processed in a single pass
+ during the second stage of processing.
+ This must be a number from 1 to 4096.
+ Higher numbers mean fewer passes over the source file and hence
+ quicker overall processing, but also require more memory.
+ If you find you are running out of memory but don't want to
+ increase your <option>--max-nodes</option> value, try reducing
+ this instead.
+ Changing this will have no effect on the result of the split, it's
+ purely to let you trade off memory for performance.
+ Note that the first stage of the processing has a fixed memory
+ overhead regardless of what this is set to so if you are running
+ out of memory before the <filename>areas.list</filename> file is
+ generated, you need to either increase your <option>-Xmx</option>
+ value or reduce the size of the input file you're trying to split.
+ </para>
+ <para>
+ Default: 512
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--max-nodes=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of nodes that can be in any of the resultant
+ files.
+ The default is fairly conservative, you could increase it quite a
+ lot before getting any 'map too big' messages.
+ Not much experimentation has been done.
+ Also the bigger this value, the less memory is required during the
+ splitting stage.
+ </para>
+ <para>
+ Default: 1600000
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--max-threads=<replaceable>value</replaceable></option></term>
+ <listitem>
+ <para>
+ The maximum number of threads used by
+ <command>mkgmap-splitter</command>.
+ </para>
+ <para>
+ Default: 4 (auto)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--mixed=<replaceable>boolean</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify this if the input osm file has nodes, ways and relations
+ intermingled or the ids are not strictly sorted.
+ To increase performance, use the <command>osmosis</command> sort
+ function.
+ </para>
+ <para>
+ Default: false
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--no-trim=<replaceable>boolean</replaceable></option></term>
+ <listitem>
+ <para>
+ Don't trim empty space off the edges of tiles.
+ This option is ignored when <option>--polygon-file</option> is
+ used.
+ </para>
+ <para>
+ Default: false
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--num-tiles=<replaceable>value</replaceable>string</option></term>
+ <listitem>
+ <para>
+ A target value that is used when no split-file is given.
+ Splitting is done so that the given number of tiles is produced.
+ The <option>--max-nodes</option> value is ignored if this option
+ is given.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--output=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ The format in which the output files are written.
+ Possible values are xml, pbf, o5m, and simulate.
+ The default is pbf, which produces the smallest file sizes.
+ The o5m format is faster to write, but creates around 40% larger
+ files.
+ The simulate option is for debugging purposes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--output-dir=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The directory to which splitter should write the output files.
+ If the specified path to a directory doesn't exist,
+ <command>mkgmap-splitter</command> tries to create it.
+ Defaults to the current working directory.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--overlap=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ Deprecated since r279.
+ With <option>--keep-complete=false</option>,
+ <command>mkgmap-splitter</command> should include nodes outside
+ the bounding box, so that <command>mkgmap</command> can neatly
+ crop exactly at the border.
+ This parameter controls the size of that overlap.
+ It is in map units, a default of 2000 is used which means about
+ 0.04 degrees of latitude or longitude.
+ If <option>--keep-complete=true</option> is active and
+ <option>--overlap</option> is given, a warning will be printed
+ because this combination rarely makes sense.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--polygon-desc-file=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ An osm file (.o5m, .pbf, .osm) with named ways that describe
+ bounding polygons with OSM ways having tags name and mapid.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--polygon-file=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a file containing a bounding polygon in the
+ <ulink url="">osmosis polygon file format</ulink>.
+ <command>mkgmap-splitter</command> uses this file when calculating
+ the areas.
+ It first calculates a grid using the given
+ <option>--resolution</option>.
+ The input file is read and for each node, a counter is increased
+ for the related grid area.
+ If the input file contains a bounding box, this is applied to the
+ grid so that nodes outside of the bounding box are ignored.
+ Next, if specified, the bounding polygon is used to zero those
+ grid elements outside of the bounding polygon area.
+ If the polygon area(s) describe(s) a rectilinear area with no more
+ than 40 vertices, <command>mkgmap-splitter</command> will try to
+ create output files that fit exactly into the area, otherwise it
+ will approximate the polygon area with rectangles.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--precomp-sea=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a directory containing precompiled sea tiles.
+ If given, <command>mkgmap-splitter</command> will use the
+ precompiled sea tiles in the same way as <command>mkgmap</command>
+ does.
+ Use this if you want to use a polygon-file or
+ <option>--no-trim=true</option> and <command>mkgmap</command>
+ creates empty *.img files combined with a message starting "There
+ is not enough room in a single garmin map for all the input data".
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--problem-file=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a file containing ways and relations that are known to
+ cause problems in the split process.
+ Use this option if <option>--keep-complete</option> requires too
+ much time or memory and <option>--overlap</option> doesn't solve
+ your problem.
+ </para>
+ <para>
+ Syntax of problem file:
+ </para>
+ <programlisting>
+way:<id> # comment...
+rel:<id> # comment...
+ </programlisting>
+ <para>
+ example:
+ </para>
+ <programlisting>
+way:2784765 # Ferry Guernsey - Jersey
+ </programlisting>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--problem-report=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a file to write the generated problem list created with
+ <option>--keep-complete</option>.
+ The parameter is ignored if <option>--keep-complete=false</option>.
+ You can reuse this file with the <option>--problem-file</option>
+ parameter, but do this only if you use the same values for
+ <option>--max-nodes</option> and <option>--resolution</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--resolution=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ The resolution of the density map produced during the first phase.
+ A value between 1 and 24.
+ Default is 13.
+ Increasing the value to 14 requires four times more memory in the
+ split phase.
+ The value is ignored if a <option>--split-file</option> is given.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--search-limit=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ Search limit in split algo.
+ Higher values may find better splits, but will take longer.
+ </para>
+ <para>
+ Default: 200000
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--split-file=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ Use the previously calculated tile areas instead of calculating
+ them from scratch.
+ The file can be in .list or .kml format.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--status-freq=<replaceable>int</replaceable></option></term>
+ <listitem>
+ <para>
+ Displays the amount of memory used by the JVM every
+ <option>--status-freq</option> seconds.
+ Set =0 to disable.
+ </para>
+ <para>
+ Default: 120
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--stop-after=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ Debugging: stop after a given program phase.
+ Can be split, gen-problem-list, or handle-problem-list.
+ Default is dist which means execute all phases.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--wanted-admin-level=<replaceable>string</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the lowest admin_level value of boundary relations that
+ should be kept complete. Used to filter boundary relations for
+ problem-list processing. The default value 5 means that
+ boundary relations are kept complete when the admin_level is
+ 5 or higher (5..11).
+ The parameter is ignored if <option>--keep-complete=false</option>.
+ Default: 5
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--write-kml=<replaceable>path</replaceable></option></term>
+ <listitem>
+ <para>
+ The name of a kml file to write out the areas to.
+ This is in addition to <filename>areas.list</filename>
+ (which is always written out).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <para>
+ Special options
+ </para>
+ <variablelist>
+
+ <varlistentry>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ If the parameter <option>--version</option> is found somewhere in
+ the options, <command>mkgmap-splitter</command> will just print
+ the version info and exit.
+ Version info looks like this:
+ <screen>
+splitter 279 compiled 2013-01-12T01:45:02+0000
+ </screen>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ If the parameter <option>--help</option> is found somewhere in
+ the options, <command>mkgmap-splitter</command> will print a list
+ of all known normal options together with a short help and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </refsect1>
+
+ <refsect1 id='tuning'>
+ <title>TUNING</title>
+
+ <para>
+ Tuning for best performance
+ </para>
+ <para>
+ A few hints for those that are using <command>mkgmap-splitter</command>
+ to split large files.
+ </para>
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ For faster processing with <option>--keep-complete=true</option>,
+ convert the input file to o5m format using:
+ <screen>
+<command>osmconvert</command> <option>--drop-version</option> <filename>file.osm</filename> <option>-o=<filename>file.o5m</filename></option>
+ </screen>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The option <option>--drop-version</option> is optional, it reduces
+ the file to that data that is needed by
+ <command>mkgmap-splitter</command> and <command>mkgmap</command>.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ If you still experience poor performance, look into
+ <filename>splitter.log</filename>.
+ Search for the word Distributing.
+ You may find something like this in the next line:
+ <screen>
+Processing 1502 areas in 3 passes, 501 areas at a time
+ </screen>
+ This means splitter has to read the input file input three times
+ because the <option>--max-areas</option> parameter was much smaller
+ than the number of areas.
+ If you have enough heap, set <option>--max-areas</option> value to a
+ value that is higher than the number of areas, e.g.
+ <option>--max-areas=2048</option>.
+ Execute <command>mkgmap-splitter</command> again and you should find
+ <screen>
+Processing 1502 areas in a single pass
+ </screen>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ More areas require more memory.
+ Make sure that <command>mkgmap-splitter</command> has enough heap
+ (increase the <option>-Xmx</option> parameter) so that it doesn't
+ waste much time in the garbage collector (GC), but keep as much
+ memory as possible for the systems I/O caches.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ If available, use two different disks for input file and output
+ directory, esp. when you use o5m format for input and output.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ If you use <command>mkgmap</command> r2415 or later and disk space
+ is no concern, consider to use <option>--output=o5m</option> to
+ speed up processing.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ <para>
+ Tuning for low memory requirements
+ </para>
+ <para>
+ If your machine has less than 1 GB free memory (eg. a netbook), you can
+ still use <command>mkgmap-splitter</command>, but you might have to be
+ patient if you use the parameter <option>--keep-complete</option> and
+ want to split a file like <filename>germany.osm.pbf</filename> or a
+ larger one.
+ If needed, reduce the number of parallel processed areas to 50 with the
+ <option>--max-areas</option> parameter.
+ You have to use <option>--keep-complete=false</option> when splitting an
+ area like Europe.
+ </para>
+ </refsect1>
+
+ <refsect1 id='notes'>
+ <title>NOTES</title>
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ There is no longer an upper limit on the number of areas that can be
+ output (previously it was 255).
+ More areas just mean potentially more passes being required over the
+ .osm file, and hence the splitter will take longer to run.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ There is no longer a limit on how many areas a way or relation can
+ belong to (previously it was 4).
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </refsect1>
+
+ <refsect1 id='see-also'>
+ <title>SEE ALSO</title>
+
+ <citerefentry>
+ <refentrytitle>mkgmap</refentrytitle>
+ <manvolnum>1</manvolnum>
+ </citerefentry>,
+ <citerefentry>
+ <refentrytitle>osmconvert</refentrytitle>
+ <manvolnum>1</manvolnum>
+ </citerefentry>
+
+ </refsect1>
+
+</refentry>
diff --git a/doc/splitter.txt b/doc/splitter.txt
index b14ee53..bc62a73 100644
--- a/doc/splitter.txt
+++ b/doc/splitter.txt
@@ -282,7 +282,7 @@ use --output=o5m to speed up processing.
If your machine has less than 1GB free memory (eg. a netbook), you can still
use splitter, but you might have to be patient if you use the
parameter --keep-complete and want to split a file like germany.osm.pbf or a
-larger one. If needed, reduce the number of parrallel processed areas to 50
+larger one. If needed, reduce the number of parallel processed areas to 50
with the max-areas parameter. You have to use --keep-complete=false when
splitting an area like Europe.
diff --git a/resources/splitter-version.properties b/resources/splitter-version.properties
index 0c859d8..b55d073 100644
--- a/resources/splitter-version.properties
+++ b/resources/splitter-version.properties
@@ -1,2 +1,2 @@
-svn.version: 440
-build.timestamp: 2016-11-16T11:19:34+0000
+svn.version: 468
+build.timestamp: 2016-11-30T15:40:42+0000
diff --git a/src/uk/me/parabola/splitter/AbstractOSMWriter.java b/src/uk/me/parabola/splitter/AbstractOSMWriter.java
index 66023b6..d9a4919 100644
--- a/src/uk/me/parabola/splitter/AbstractOSMWriter.java
+++ b/src/uk/me/parabola/splitter/AbstractOSMWriter.java
@@ -67,16 +67,4 @@ public abstract class AbstractOSMWriter implements OSMWriter{
public Rectangle getBBox(){
return bbox;
}
-
- public boolean nodeBelongsToThisArea(Node node) {
- return (extendedBounds.contains(node.getMapLat(), node.getMapLon()));
- }
-
- public boolean coordsBelongToThisArea(int mapLat, int mapLon) {
- return (extendedBounds.contains(mapLat,mapLon));
- }
-
- public boolean areaIsPseudo(){
- return false;
- }
}
diff --git a/src/uk/me/parabola/splitter/Area.java b/src/uk/me/parabola/splitter/Area.java
index b730a81..43bced5 100644
--- a/src/uk/me/parabola/splitter/Area.java
+++ b/src/uk/me/parabola/splitter/Area.java
@@ -32,9 +32,7 @@ public class Area {
private final int maxLat;
private final int maxLong;
private Rectangle javaRect;
- private java.awt.geom.Area javaArea;
private boolean isJoinable = true;
- private boolean isResultOfSplitting;
private boolean isPseudoArea;
public boolean isJoinable() {
@@ -110,10 +108,12 @@ public class Area {
return javaRect;
}
+ /**
+ *
+ * @return a new {@link java.awt.geom.Area} instance
+ */
public java.awt.geom.Area getJavaArea(){
- if (javaArea == null)
- javaArea = new java.awt.geom.Area(getRect());
- return javaArea;
+ return new java.awt.geom.Area(getRect());
}
public void setMapId(int mapId) {
this.mapId = mapId;
@@ -179,13 +179,8 @@ public class Area {
&& lon <= maxLong;
}
- public Area add(Area area) {
- return new Area(
- Math.min(minLat, area.minLat),
- Math.min(minLong, area.minLong),
- Math.max(maxLat, area.maxLat),
- Math.max(maxLong, area.maxLong)
- );
+ public boolean contains(Node node) {
+ return contains(node.getMapLat(), node.getMapLon());
}
/**
@@ -200,6 +195,28 @@ public class Area {
&& other.getMaxLong() <= maxLong;
}
+ /**
+ * Checks if this area intersects the given bounding box at least
+ * in one point.
+ *
+ * @param bbox an area
+ * @return <code>true</code> if this area intersects the bbox;
+ * <code>false</code> else
+ */
+ public final boolean intersects(Area bbox) {
+ return minLat <= bbox.getMaxLat() && maxLat >= bbox.getMinLat() &&
+ minLong <= bbox.getMaxLong() && maxLong >= bbox.getMinLong();
+ }
+
+ public Area add(Area area) {
+ return new Area(
+ Math.min(minLat, area.minLat),
+ Math.min(minLong, area.minLong),
+ Math.max(maxLat, area.maxLat),
+ Math.max(maxLong, area.maxLong)
+ );
+ }
+
public boolean isPseudoArea() {
return isPseudoArea;
}
@@ -207,4 +224,5 @@ public class Area {
public void setPseudoArea(boolean isPseudoArea) {
this.isPseudoArea = isPseudoArea;
}
+
}
diff --git a/src/uk/me/parabola/splitter/WriterDictionaryInt.java b/src/uk/me/parabola/splitter/AreaDictionaryInt.java
similarity index 57%
rename from src/uk/me/parabola/splitter/WriterDictionaryInt.java
rename to src/uk/me/parabola/splitter/AreaDictionaryInt.java
index 0bf2b47..ef0d350 100644
--- a/src/uk/me/parabola/splitter/WriterDictionaryInt.java
+++ b/src/uk/me/parabola/splitter/AreaDictionaryInt.java
@@ -17,56 +17,57 @@ import java.util.BitSet;
import java.util.HashMap;
/**
- * Maps a BitSet containing the used writers to an integer value.
- * An OSM element is written to one or more writers. Every used
- * combination of writers is translated to an integer.
- * Use this dictionary if you expect many different writer combinations,
+ * Maps a BitSet containing the used areas to an integer value.
+ * An OSM element is written to one or more areas. Every used
+ * combination of areas is translated to an integer.
+ * Use this dictionary if you expect many different area combinations,
* e.g. for relations and their members.
- * @author GerdP
+ * @author Gerd Petermann
*
*/
-public class WriterDictionaryInt{
+public class AreaDictionaryInt{
public final static int UNASSIGNED = -1;
private final ArrayList<BitSet> sets;
- private final int numOfWriters;
+ private final int numOfAreas;
private final HashMap<BitSet, Integer> index;
/**
- * Create a dictionary for a given number of writers
- * @param numOfWriters the number of writers that are used
+ * Create a dictionary for a given array of areas
+ * @param num the number of areas that are used
*/
- WriterDictionaryInt (OSMWriter [] writers){
- this.numOfWriters = writers.length;
- sets = new ArrayList<BitSet>();
- index = new HashMap<BitSet, Integer>();
+ AreaDictionaryInt (int num){
+ this.numOfAreas = num;
+ sets = new ArrayList<>();
+ index = new HashMap<>();
init();
}
/**
- * initialize the dictionary with sets containing a single writer.
+ * initialize the dictionary with sets containing a single area.
*/
private void init(){
- ArrayList<BitSet> writerSets = new ArrayList<BitSet>(numOfWriters);
- for (int i=0; i < numOfWriters; i++){
+ ArrayList<BitSet> areaSets = new ArrayList<>(numOfAreas);
+ for (int i = 0; i < numOfAreas; i++) {
BitSet b = new BitSet();
b.set(i);
translate(b);
- writerSets.add(b);
+ areaSets.add(b);
}
}
+
/**
* Calculate the integer value for a given BitSet. The BitSet must not
- * contain values higher than numOfWriters.
- * @param writerSet the BitSet
+ * contain values higher than numOfAreas.
+ * @param areaSet the BitSet
* @return an int value that identifies this BitSet
*/
- public int translate(final BitSet writerSet){
- Integer combiIndex = index.get(writerSet);
+ public Integer translate(final BitSet areaSet){
+ Integer combiIndex = index.get(areaSet);
if (combiIndex == null){
BitSet bnew = new BitSet();
- bnew.or(writerSet);
+ bnew.or(areaSet);
combiIndex = sets.size();
sets.add(bnew);
index.put(bnew, combiIndex);
@@ -92,18 +93,4 @@ public class WriterDictionaryInt{
public int size(){
return sets.size();
}
-
- public int getNumOfWriters(){
- return numOfWriters;
- }
-
- /**
- * return the id of a single writer or
- * @param writerIdx
- * @return
- */
- public boolean isSingleWriterIdx(int writerIdx) {
- return (writerIdx < numOfWriters);
- }
-
}
diff --git a/src/uk/me/parabola/splitter/WriterDictionaryShort.java b/src/uk/me/parabola/splitter/AreaDictionaryShort.java
similarity index 55%
rename from src/uk/me/parabola/splitter/WriterDictionaryShort.java
rename to src/uk/me/parabola/splitter/AreaDictionaryShort.java
index bb1d2b2..c1de7cc 100644
--- a/src/uk/me/parabola/splitter/WriterDictionaryShort.java
+++ b/src/uk/me/parabola/splitter/AreaDictionaryShort.java
@@ -16,76 +16,82 @@ import it.unimi.dsi.fastutil.shorts.ShortArrayList;
import java.awt.Rectangle;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.BitSet;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
/**
- * Maps a BitSet containing the used writers to a short value.
- * An OSM element is written to one or more writers. Every used
- * combination of writers is translated to a short.
+ * Maps a BitSet containing the used areas to a short value.
+ * An OSM element is written to one or more areas. Every used
+ * combination of areas is translated to a short.
* @author GerdP
*
*/
-public class WriterDictionaryShort{
- public final static int DICT_START = -1 * (Short.MIN_VALUE + 1);
- private OSMWriter[] writers;
+public class AreaDictionaryShort{
+ private final static int DICT_START = Short.MAX_VALUE;
+ private final Area[] areas;
private final ArrayList<BitSet> sets;
private final ArrayList<ShortArrayList> arrays;
- private final int numOfWriters;
+ private final int numOfAreas;
private final HashMap<BitSet, Short> index;
- private final HashSet<Short> simpleNeighbours = new HashSet<Short>();
+ private final HashSet<Short> simpleNeighbours = new HashSet<>();
+ private final int overlapAmount;
/**
- * Create a dictionary for a given array of writers
- * @param writers the array of writers
+ * Create a dictionary for a given array of areas
+ * @param overlapAmount
+ * @param areas the array of areas
*/
- WriterDictionaryShort (OSMWriter [] writers){
- this.writers = writers;
- this.numOfWriters = writers.length;
- sets = new ArrayList<BitSet>();
- arrays = new ArrayList<ShortArrayList>();
- index = new HashMap<BitSet, Short>();
+ AreaDictionaryShort (List<Area> areas, int overlapAmount){
+ this.areas = areas.toArray(new Area[areas.size()]);
+ this.overlapAmount = overlapAmount;
+ this.numOfAreas = areas.size();
+ sets = new ArrayList<>();
+ arrays = new ArrayList<>();
+ index = new HashMap<>();
init();
}
/**
- * initialize the dictionary with sets containing a single writer.
+ * initialize the dictionary with sets containing a single area.
*/
private void init(){
- ArrayList<Rectangle> rectangles = new ArrayList<Rectangle>(numOfWriters);
- ArrayList<BitSet> writerSets = new ArrayList<BitSet>(numOfWriters);
- for (int i=0; i < numOfWriters; i++){
+ ArrayList<Rectangle> rectangles = new ArrayList<>(numOfAreas);
+ ArrayList<BitSet> areaSets = new ArrayList<>(numOfAreas);
+ for (int i=0; i < numOfAreas; i++){
BitSet b = new BitSet();
b.set(i);
translate(b);
- rectangles.add(Utils.area2Rectangle(writers[i].getBounds(), 0));
- writerSets.add(b);
+ rectangles.add(Utils.area2Rectangle(areas[i], 0));
+ areaSets.add(b);
}
- findSimpleNeigbours(rectangles, writerSets);
+ findSimpleNeigbours(rectangles, areaSets);
System.out.println("cached " + simpleNeighbours.size() + " combinations of areas that form rectangles.");
return;
}
/**
* Calculate the short value for a given BitSet. The BitSet must not
- * contain values higher than numOfWriters.
- * @param writerSet the BitSet
+ * contain values higher than numOfAreas.
+ * @param areaSet the BitSet
* @return a short value that identifies this BitSet
*/
- public short translate(final BitSet writerSet){
- Short combiIndex = index.get(writerSet);
+ public Short translate(final BitSet areaSet){
+ Short combiIndex = index.get(areaSet);
if (combiIndex == null){
BitSet bnew = new BitSet();
- bnew.or(writerSet);
+ bnew.or(areaSet);
ShortArrayList a = new ShortArrayList();
- for (int i = writerSet.nextSetBit(0); i >= 0; i = writerSet.nextSetBit(i + 1)) {
+ for (int i = areaSet.nextSetBit(0); i >= 0; i = areaSet.nextSetBit(i + 1)) {
a.add((short) i);
}
combiIndex = (short) (sets.size() - DICT_START);
if (combiIndex == Short.MAX_VALUE){
- throw new SplitFailedException("writerDictionary is full. Decrease --max-areas value");
+ throw new SplitFailedException("areaDictionary is full. Decrease --max-areas value");
}
sets.add(bnew);
arrays.add(a);
@@ -99,9 +105,9 @@ public class WriterDictionaryShort{
* added together. A way or relation that lies exactly within
* such a combination cannot cross other areas.
*/
- private void findSimpleNeigbours(ArrayList<Rectangle> rectangles, ArrayList<BitSet> writerSets){
- ArrayList<Rectangle> newRectangles = new ArrayList<Rectangle>();
- ArrayList<BitSet> newWriterSets = new ArrayList<BitSet>();
+ private void findSimpleNeigbours(ArrayList<Rectangle> rectangles, ArrayList<BitSet> areaSets){
+ ArrayList<Rectangle> newRectangles = new ArrayList<>();
+ ArrayList<BitSet> newAreaSets = new ArrayList<>();
for (int i = 0; i < rectangles.size(); i++){
Rectangle r1 = rectangles.get(i);
@@ -116,17 +122,17 @@ public class WriterDictionaryShort{
isSimple = true;
if (isSimple){
BitSet simpleNeighbour = new BitSet();
- simpleNeighbour.or(writerSets.get(i));
- simpleNeighbour.or(writerSets.get(j));
+ simpleNeighbour.or(areaSets.get(i));
+ simpleNeighbour.or(areaSets.get(j));
if (simpleNeighbour.cardinality() <= 10){
- short idx = translate(simpleNeighbour);
+ Short idx = translate(simpleNeighbour);
if (simpleNeighbours.contains(idx) == false){
simpleNeighbours.add(idx);
//System.out.println("simple neighbor: " + getMapIds(simpleNeighbour));
Rectangle pair = new Rectangle(r1);
pair.add(r2);
newRectangles.add(pair);
- newWriterSets.add(simpleNeighbour);
+ newAreaSets.add(simpleNeighbour);
}
}
}
@@ -134,11 +140,11 @@ public class WriterDictionaryShort{
}
if (newRectangles.isEmpty() == false){
rectangles.addAll(newRectangles);
- writerSets.addAll(newWriterSets);
+ areaSets.addAll(newAreaSets);
newRectangles = null;
- newWriterSets = null;
+ newAreaSets = null;
if (simpleNeighbours.size() < 1000)
- findSimpleNeigbours(rectangles,writerSets);
+ findSimpleNeigbours(rectangles,areaSets);
}
}
/**
@@ -153,10 +159,10 @@ public class WriterDictionaryShort{
}
/**
- * Return a list containing the writer ids for the given
+ * Return a list containing the area ids for the given
* short value.
- * @param idx a short value that was returned by the translate()
- * @return a list containing the writer ids
+ * @param idx a short value that was returned by the translate() method
+ * @return a list containing the area ids
*/
public ShortArrayList getList (final short idx){
return arrays.get(DICT_START + idx);
@@ -170,39 +176,37 @@ public class WriterDictionaryShort{
return sets.size();
}
- public int getNumOfWriters(){
- return numOfWriters;
+ public int getNumOfAreas(){
+ return numOfAreas;
}
- public OSMWriter[] getWriters(){
- return writers;
- }
-
- public boolean mayCross(short writerIdx){
- if (writerIdx + DICT_START < numOfWriters)
+ public boolean mayCross(short areaIdx){
+ if (areaIdx + DICT_START < numOfAreas)
return false;
- if (simpleNeighbours.contains(writerIdx))
+ if (simpleNeighbours.contains(areaIdx))
return false;
return true;
}
- public String getMapIds(BitSet writerSet){
- StringBuilder sb = new StringBuilder("{");
- for (int k = 0;k<numOfWriters;k++){
- if (writerSet.get(k)) {
- sb.append(writers[k].getMapId());
- sb.append(", ");
- }
- }
- return sb.substring(0, sb.length()-2) + "}";
+ public Area getArea(int idx) {
+ return areas[idx];
}
- /**
- * return the id of a single writer or
- * @param writerIdx
- * @return
- */
- public boolean isSingleWriterIdx(short writerIdx) {
- return (writerIdx + DICT_START < numOfWriters);
+ public Area getExtendedArea(int idx) {
+ Area bounds = areas[idx];
+ if (overlapAmount == 0)
+ return bounds;
+ return new Area(bounds.getMinLat() - overlapAmount,
+ bounds.getMinLong() - overlapAmount,
+ bounds.getMaxLat() + overlapAmount,
+ bounds.getMaxLong() + overlapAmount);
}
+
+ public List<Area> getAreas() {
+ return Collections.unmodifiableList(Arrays.asList(areas));
+ }
+
+ public static short translate(short lastUsedWriter) {
+ return (short) (lastUsedWriter - DICT_START);
+ }
}
diff --git a/src/uk/me/parabola/splitter/WriterGrid.java b/src/uk/me/parabola/splitter/AreaGrid.java
similarity index 59%
rename from src/uk/me/parabola/splitter/WriterGrid.java
rename to src/uk/me/parabola/splitter/AreaGrid.java
index d7f9337..9a65aec 100644
--- a/src/uk/me/parabola/splitter/WriterGrid.java
+++ b/src/uk/me/parabola/splitter/AreaGrid.java
@@ -16,27 +16,27 @@ package uk.me.parabola.splitter;
import java.util.BitSet;
/**
- * A grid that covers the area covered by all writers. Each grid element contains
+ * A grid that covers the area covered by all areas. Each grid element contains
* information about the tiles that are intersecting the grid element and whether
* the grid element lies completely within such a tile area.
* This is used to minimize the needed tests when analyzing coordinates of node coordinates.
* @author GerdP
*
*/
-public class WriterGrid implements WriterIndex{
+public class AreaGrid implements AreaIndex{
private final Area bounds;
private final Grid grid;
- protected final WriterGridResult r;
- protected final WriterDictionaryShort writerDictionary;
+ protected final AreaGridResult r;
+ protected final AreaDictionaryShort areaDictionary;
/**
- * Create a grid to speed up the search of writer candidates.
- * @param writerDictionary
+ * Create a grid to speed up the search of area candidates.
+ * @param areaDictionary
* @param withOuter
*/
- WriterGrid(WriterDictionaryShort writerDictionary){
- this.writerDictionary = writerDictionary;
- r = new WriterGridResult();
+ AreaGrid(AreaDictionaryShort areaDictionary){
+ this.areaDictionary = areaDictionary;
+ r = new AreaGridResult();
long start = System.currentTimeMillis();
grid = new Grid(null, null);
@@ -49,11 +49,11 @@ public class WriterGrid implements WriterIndex{
return bounds;
}
- public WriterGridResult get (final Node n){
+ public AreaGridResult get (final Node n){
return grid.get(n.getMapLat(),n.getMapLon());
}
- public WriterGridResult get (int lat, int lon){
+ public AreaGridResult get (int lat, int lon){
return grid.get(lat, lon);
}
@@ -77,9 +77,9 @@ public class WriterGrid implements WriterIndex{
private final int gridDimLon;
private final int gridDimLat;
- public Grid(BitSet UsedWriters, Area bounds) {
- // each element contains an index to the writerDictionary or unassigned
- if (UsedWriters == null){
+ public Grid(BitSet usedAreas, Area bounds) {
+ // each element contains an index to the areaDictionary or unassigned
+ if (usedAreas == null){
gridDimLon = TOP_GRID_DIM_LON;
gridDimLat = TOP_GRID_DIM_LAT;
}
@@ -88,33 +88,32 @@ public class WriterGrid implements WriterIndex{
gridDimLat = SUB_GRID_DIM_LAT;
}
grid = new short[gridDimLon + 1][gridDimLat + 1];
- // is true for an element if the list of writers needs to be tested
+ // is true for an element if the list of areas needs to be tested
testGrid = new boolean[gridDimLon + 1][gridDimLat + 1];
this.bounds = bounds;
- maxCompares = fillGrid(UsedWriters);
+ maxCompares = fillGrid(usedAreas);
}
public Area getBounds() {
return bounds;
}
/**
* Create the grid and fill each element
- * @param usedWriters
+ * @param usedAreas
* @param testGrid
* @param grid
* @return
*/
- private int fillGrid(BitSet usedWriters) {
+ private int fillGrid(BitSet usedAreas) {
int gridStepLon, gridStepLat;
- OSMWriter[] writers = writerDictionary.getWriters();
if (bounds == null){
// calculate grid area
Area tmpBounds = null;
- for (int i = 0; i<writers.length; i++){
- OSMWriter w = writers[i];
- if (usedWriters == null || usedWriters.get(i))
- tmpBounds = (tmpBounds ==null) ? w.getExtendedBounds() : tmpBounds.add(w.getExtendedBounds());
+ for (int i = 0; i < areaDictionary.getNumOfAreas(); i++) {
+ Area extBounds = areaDictionary.getExtendedArea(i);
+ if (usedAreas == null || usedAreas.get(i))
+ tmpBounds = (tmpBounds ==null) ? extBounds : tmpBounds.add(extBounds);
}
- // create new Area to make sure that we don't update the writer area
+ // create new Area to make sure that we don't update the existing area
bounds = new Area(tmpBounds.getMinLat() , tmpBounds.getMinLong(), tmpBounds.getMaxLat(), tmpBounds.getMaxLong());
}
// save these results for later use
@@ -130,35 +129,34 @@ public class WriterGrid implements WriterIndex{
assert gridStepLon * gridDimLon >= gridWidth : "gridStepLon is too small";
assert gridStepLat * gridDimLat >= gridHeight : "gridStepLat is too small";
- int maxWriterSearch = 0;
- BitSet writerSet = new BitSet();
- BitSet[][] gridWriters = new BitSet[gridDimLon+1][gridDimLat+1];
+ int maxAreaSearch = 0;
+ BitSet areaSet = new BitSet();
+ BitSet[][] gridAreas = new BitSet[gridDimLon+1][gridDimLat+1];
- int numWriters = writerDictionary.getNumOfWriters();
- for (int j = 0; j < numWriters; j++) {
- OSMWriter w = writers[j];
- if (!(usedWriters == null || usedWriters.get(j)))
+ for (int j = 0; j < areaDictionary.getNumOfAreas(); j++) {
+ Area extBounds = areaDictionary.getExtendedArea(j);
+ if (!(usedAreas == null || usedAreas.get(j)))
continue;
- int minLonWriter = w.getExtendedBounds().getMinLong();
- int maxLonWriter = w.getExtendedBounds().getMaxLong();
- int minLatWriter = w.getExtendedBounds().getMinLat();
- int maxLatWriter = w.getExtendedBounds().getMaxLat();
- int startLon = Math.max(0,(minLonWriter- gridMinLon ) / gridDivLon);
- int endLon = Math.min(gridDimLon,(maxLonWriter - gridMinLon ) / gridDivLon);
- int startLat = Math.max(0,(minLatWriter- gridMinLat ) / gridDivLat);
- int endLat = Math.min(gridDimLat,(maxLatWriter - gridMinLat ) / gridDivLat);
- // add this writer to all grid elements that intersect with the writer bbox
+ int minLonArea = extBounds.getMinLong();
+ int maxLonArea = extBounds.getMaxLong();
+ int minLatArea = extBounds.getMinLat();
+ int maxLatArea = extBounds.getMaxLat();
+ int startLon = Math.max(0,(minLonArea- gridMinLon ) / gridDivLon);
+ int endLon = Math.min(gridDimLon,(maxLonArea - gridMinLon ) / gridDivLon);
+ int startLat = Math.max(0,(minLatArea- gridMinLat ) / gridDivLat);
+ int endLat = Math.min(gridDimLat,(maxLatArea - gridMinLat ) / gridDivLat);
+ // add this area to all grid elements that intersect with it
for (int lon = startLon; lon <= endLon; lon++) {
int testMinLon = gridMinLon + gridStepLon * lon;
for (int lat = startLat; lat <= endLat; lat++) {
int testMinLat = gridMinLat + gridStepLat * lat;
- if (gridWriters[lon][lat]== null)
- gridWriters[lon][lat] = new BitSet();
- // add this writer
- gridWriters[lon][lat].set(j);
- if (!w.getExtendedBounds().contains(testMinLat, testMinLon)
- || !w.getExtendedBounds().contains(testMinLat+ gridStepLat, testMinLon+ gridStepLon)){
- // grid area is not completely within writer area
+ if (gridAreas[lon][lat]== null)
+ gridAreas[lon][lat] = new BitSet();
+ // add this area
+ gridAreas[lon][lat].set(j);
+ if (!extBounds.contains(testMinLat, testMinLon)
+ || !extBounds.contains(testMinLat+ gridStepLat, testMinLon+ gridStepLon)){
+ // grid area is not completely within area
testGrid[lon][lat] = true;
}
}
@@ -166,12 +164,12 @@ public class WriterGrid implements WriterIndex{
}
for (int lon = 0; lon <= gridDimLon; lon++) {
for (int lat = 0; lat <= gridDimLat; lat++) {
- writerSet = (gridWriters[lon][lat]);
- if (writerSet == null)
+ areaSet = (gridAreas[lon][lat]);
+ if (areaSet == null)
grid[lon][lat] = AbstractMapProcessor.UNASSIGNED;
else {
if (testGrid[lon][lat]){
- int numTests = writerSet.cardinality();
+ int numTests = areaSet.cardinality();
if (numTests > MAX_TESTS){
if (gridStepLat > MIN_GRID_LAT && gridStepLon > MIN_GRID_LON){
Area gridPart = new Area(gridMinLat + gridStepLat * lat, gridMinLon + gridStepLon * lon,
@@ -182,21 +180,21 @@ public class WriterGrid implements WriterIndex{
subGrid = new Grid [gridDimLon + 1][gridDimLat + 1];
usedSubGridElems++;
- subGrid[lon][lat] = new Grid(writerSet, gridPart);
+ subGrid[lon][lat] = new Grid(areaSet, gridPart);
numTests = subGrid[lon][lat].getMaxCompares() + 1;
- maxWriterSearch = Math.max(maxWriterSearch, numTests);
+ maxAreaSearch = Math.max(maxAreaSearch, numTests);
continue;
}
}
- maxWriterSearch = Math.max(maxWriterSearch, numTests);
+ maxAreaSearch = Math.max(maxAreaSearch, numTests);
}
- grid[lon][lat] = writerDictionary.translate(writerSet);
+ grid[lon][lat] = areaDictionary.translate(areaSet);
}
}
}
- System.out.println("WriterGridTree [" + gridDimLon + "][" + gridDimLat + "] for grid area " + bounds +
- " requires max. " + maxWriterSearch + " checks for each node (" + usedSubGridElems + " sub grid(s))" );
- return maxWriterSearch;
+ System.out.println("AreaGridTree [" + gridDimLon + "][" + gridDimLat + "] for grid area " + bounds +
+ " requires max. " + maxAreaSearch + " checks for each node (" + usedSubGridElems + " sub grid(s))" );
+ return maxAreaSearch;
}
@@ -208,13 +206,13 @@ public class WriterGrid implements WriterIndex{
return maxCompares;
}
/**
- * For a given node, return the list of writers that may contain it
+ * For a given node, return the list of areas that may contain it
* @param node the node
- * @return a reference to an {@link WriterGridResult} instance that contains
+ * @return a reference to an {@link AreaGridResult} instance that contains
* the list of candidates and a boolean that shows whether this list
* has to be verified or not.
*/
- public WriterGridResult get(final int lat, final int lon){
+ public AreaGridResult get(final int lat, final int lon){
if (!bounds.contains(lat, lon))
return null;
int gridLonIdx = (lon - gridMinLon ) / gridDivLon;
@@ -223,16 +221,16 @@ public class WriterGrid implements WriterIndex{
if (subGrid != null){
Grid sub = subGrid[gridLonIdx][gridLatIdx];
if (sub != null){
- // get list of writer candidates from sub grid
+ // get list of area candidates from sub grid
return sub.get(lat, lon);
}
}
- // get list of writer candidates from grid
+ // get list of area candidates from grid
short idx = grid[gridLonIdx][gridLatIdx];
if (idx == AbstractMapProcessor.UNASSIGNED)
return null;
r.testNeeded = testGrid[gridLonIdx][gridLatIdx];
- r.l = writerDictionary.getList(idx);
+ r.l = areaDictionary.getList(idx);
return r;
}
}
diff --git a/src/uk/me/parabola/splitter/WriterGridResult.java b/src/uk/me/parabola/splitter/AreaGridResult.java
similarity index 77%
rename from src/uk/me/parabola/splitter/WriterGridResult.java
rename to src/uk/me/parabola/splitter/AreaGridResult.java
index be3192c..d4b396b 100644
--- a/src/uk/me/parabola/splitter/WriterGridResult.java
+++ b/src/uk/me/parabola/splitter/AreaGridResult.java
@@ -16,12 +16,12 @@ package uk.me.parabola.splitter;
import it.unimi.dsi.fastutil.shorts.ShortArrayList;
/**
- * A helper class to combine the results of the {@link WriterGrid}
+ * A helper class to combine the results of the {@link AreaGrid}
* @author GerdP
*
*/
-public class WriterGridResult{
- ShortArrayList l; // list of indexes to the writer dictionary
- boolean testNeeded; // true: the list must be checked with the nodeBelongsToThisArea() method
+public class AreaGridResult{
+ ShortArrayList l; // list of indexes to the area dictionary
+ boolean testNeeded; // true: the list must be checked with the Area.contains() method
}
diff --git a/src/uk/me/parabola/splitter/WriterIndex.java b/src/uk/me/parabola/splitter/AreaIndex.java
similarity index 62%
rename from src/uk/me/parabola/splitter/WriterIndex.java
rename to src/uk/me/parabola/splitter/AreaIndex.java
index 5896c3c..a76c526 100644
--- a/src/uk/me/parabola/splitter/WriterIndex.java
+++ b/src/uk/me/parabola/splitter/AreaIndex.java
@@ -15,27 +15,27 @@ package uk.me.parabola.splitter;
/**
*
- * @author GerdP
+ * @author Gerd Petermann
*
*/
-public interface WriterIndex{
+public interface AreaIndex{
/**
- * @return the bounding box of the writer areas.
+ * @return the bounding box of the areas.
*/
public Area getBounds();
/**
- * Return a set of writer candidates for this node.
+ * Return a set of area candidates for this node.
* @param n the node
- * @return a reference to a static WriterGridResult instance
+ * @return a reference to a static AreaGridResult instance
*/
- public WriterGridResult get (final Node n);
+ public AreaGridResult get (final Node n);
/**
- * Return a set of writer candidates for these coordinates
+ * Return a set of area candidates for these coordinates
* @param lat the latitude value in map units
* @param lon the longitude value in map units
- * @return a reference to a static WriterGridResult instance
+ * @return a reference to a static AreaGridResult instance
*/
- public WriterGridResult get (int lat, int lon);
+ public AreaGridResult get (int lat, int lon);
}
diff --git a/src/uk/me/parabola/splitter/AreaList.java b/src/uk/me/parabola/splitter/AreaList.java
index 46c6b9b..64260c5 100644
--- a/src/uk/me/parabola/splitter/AreaList.java
+++ b/src/uk/me/parabola/splitter/AreaList.java
@@ -25,26 +25,36 @@ import java.util.Collections;
import java.util.Date;
import java.util.List;
import java.util.Locale;
+import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.xmlpull.v1.XmlPullParserException;
+import uk.me.parabola.splitter.geo.City;
+import uk.me.parabola.splitter.geo.CityFinder;
+import uk.me.parabola.splitter.geo.CityLoader;
+import uk.me.parabola.splitter.geo.DefaultCityFinder;
+
/**
* A list of areas. It can be read and written to a file.
*/
public class AreaList {
- private List<Area> areas;
-
- public AreaList(List<Area> areas) {
- this.areas = areas;
- }
-
+ private final List<Area> areas;
+ private final String description;
+ private String geoNamesFile;
+
/**
* This constructor is called when you are going to be reading in the list from
* a file, rather than making it from an already constructed list.
*/
- public AreaList() {
+ public AreaList(String description) {
+ this(new ArrayList<Area>(), description);
+ }
+
+ public AreaList(List<Area> areas, String description) {
+ this.description = description;
+ this.areas = areas;
}
/**
@@ -76,16 +86,6 @@ public class AreaList {
}
}
- /**
- * Write out a KML file containing the areas that we calculated. This KML file
- * can be opened in Google Earth etc to see the areas that were split.
- *
- * @param filename The KML filename to write to.
- */
- public void writeKml(String filename) {
- KmlWriter.writeKml(filename, areas);
- }
-
public void read(String filename) throws IOException {
String lower = filename.toLowerCase();
if (lower.endsWith(".kml") || lower.endsWith(".kml.gz") || lower.endsWith(".kml.bz2")) {
@@ -100,7 +100,7 @@ public class AreaList {
* Obviously other tools could create the file too.
*/
private void readList(String filename) throws IOException {
- areas = new ArrayList<>();
+ areas.clear();
Pattern pattern = Pattern.compile("([0-9]{8}):" +
" ([\\p{XDigit}x-]+),([\\p{XDigit}x-]+)" +
@@ -138,14 +138,16 @@ public class AreaList {
KmlParser parser = new KmlParser();
parser.setReader(Utils.openFile(filename, false));
parser.parse();
- areas = parser.getAreas();
+ List<Area> newAreas = parser.getAreas();
+ areas.clear();
+ areas.addAll(newAreas);
} catch (XmlPullParserException e) {
throw new IOException("Unable to parse KML file " + filename, e);
}
}
public List<Area> getAreas() {
- return areas;
+ return Collections.unmodifiableList(areas);
}
public void dump() {
@@ -178,13 +180,12 @@ public class AreaList {
pw.println(i+1);
else
pw.println("!" + (i+1));
- Point point = null,lastPoint = null;
+ Point point = null;
for (int j = 0; j < shape.size(); j++){
- if (j > 0)
- lastPoint = point;
point = shape.get(j);
if (j > 0 && j+1 < shape.size()){
- Point nextPoint = shape.get(j+1);
+ Point lastPoint = shape.get(j - 1);
+ Point nextPoint = shape.get(j + 1);
if (point.x == nextPoint.x && point.x == lastPoint.x)
continue;
if (point.y == nextPoint.y && point.y == lastPoint.y)
@@ -248,4 +249,50 @@ public class AreaList {
}
}
+ public void setAreaNames() {
+ CityFinder cityFinder = null;
+ if (geoNamesFile != null){
+ CityLoader cityLoader = new CityLoader(true);
+ List<City> cities = cityLoader.load(geoNamesFile);
+ if (cities == null)
+ return;
+
+ cityFinder = new DefaultCityFinder(cities);
+ }
+ for (Area area : getAreas()) {
+ area.setName(description);
+ if (cityFinder == null)
+ continue;
+
+ // Decide what to call the area
+ Set<City> found = cityFinder.findCities(area);
+ City bestMatch = null;
+ for (City city : found) {
+ if (bestMatch == null || city.getPopulation() > bestMatch.getPopulation()) {
+ bestMatch = city;
+ }
+ }
+ if (bestMatch != null)
+ area.setName(bestMatch.getCountryCode() + '-' + bestMatch.getName());
+ }
+ }
+
+ /**
+ *
+ * @param mapId
+ */
+ public void setMapIds(int mapId) {
+ for (Area area : getAreas()) {
+ area.setMapId(mapId++);
+ }
+ }
+
+ public void setGeoNamesFile(String geoNamesFile) {
+ this.geoNamesFile = geoNamesFile;
+ }
+
+ public void setAreas(List<Area> calculateAreas) {
+ areas.clear();
+ areas.addAll(calculateAreas);
+ }
}
diff --git a/src/uk/me/parabola/splitter/AreasCalculator.java b/src/uk/me/parabola/splitter/AreasCalculator.java
new file mode 100644
index 0000000..1c011b6
--- /dev/null
+++ b/src/uk/me/parabola/splitter/AreasCalculator.java
@@ -0,0 +1,422 @@
+package uk.me.parabola.splitter;
+
+import java.awt.Point;
+import java.awt.Rectangle;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+
+import org.openstreetmap.osmosis.core.filter.common.PolygonFileReader;
+
+import it.unimi.dsi.fastutil.longs.LongArrayList;
+
+public class AreasCalculator {
+ private final List<PolygonDesc> polygons = new ArrayList<>();
+ private int resolution = 13;
+ private PolygonDescProcessor polygonDescProcessor;
+
+ public AreasCalculator() {
+ }
+
+
+ public void setResolution(int resolution) {
+ this.resolution = resolution;
+ }
+
+
+ /**
+ * Check if the bounding polygons are usable.
+ * @param polygon
+ * @return
+ */
+ public boolean checkPolygons() {
+ for (PolygonDesc pd : polygons){
+ if (checkPolygon(pd.area) == false)
+ return false;
+ }
+ return true;
+ }
+
+
+ /**
+ * Check if the bounding polygon is usable.
+ * @param polygon
+ * @return
+ */
+ private boolean checkPolygon(java.awt.geom.Area mapPolygonArea) {
+ List<List<Point>> shapes = Utils.areaToShapes(mapPolygonArea);
+ int shift = 24 - resolution;
+ long rectangleWidth = 1L << shift;
+ for (List<Point> shape: shapes){
+ int estimatedPoints = 0;
+ Point p1 = shape.get(0);
+ for (int i = 1; i < shape.size(); i++){
+ Point p2 = shape.get(i);
+ if (p1.x != p2.x && p1.y != p2.y){
+ // diagonal line
+ int width = Math.abs(p1.x-p2.x);
+ int height = Math.abs(p1.y-p2.y);
+ estimatedPoints += (Math.min(width, height) / rectangleWidth) * 2;
+ }
+
+ if (estimatedPoints > SplittableDensityArea.MAX_SINGLE_POLYGON_VERTICES)
+ return false; // too complex
+
+ p1 = p2;
+ }
+ }
+ return true;
+ }
+
+ public void readPolygonFile(String polygonFile, int mapId) {
+ polygons.clear();
+ File f = new File(polygonFile);
+
+ if (!f.exists()){
+ throw new IllegalArgumentException("polygon file doesn't exist: " + polygonFile);
+ }
+ PolygonFileReader polyReader = new PolygonFileReader(f);
+ java.awt.geom.Area polygonInDegrees = polyReader.loadPolygon();
+ PolygonDesc pd = new PolygonDesc(polyReader.getPolygonName(),
+ Utils.AreaDegreesToMapUnit(polygonInDegrees),
+ mapId);
+ polygons.add(pd);
+ }
+
+
+ public void readPolygonDescFile(String polygonDescFile) {
+ polygons.clear();
+ File f = new File(polygonDescFile);
+
+ if (!f.exists()){
+ System.out.println("Error: polygon desc file doesn't exist: " + polygonDescFile);
+ System.exit(-1);
+ }
+ polygonDescProcessor = new PolygonDescProcessor(resolution);
+ OSMFileHandler polyDescHandler = new OSMFileHandler();
+ polyDescHandler.setFileNames(Arrays.asList(polygonDescFile));
+ polyDescHandler.setMixed(false);
+ polyDescHandler.process(polygonDescProcessor);
+ polygons.addAll(polygonDescProcessor.getPolygons());
+ }
+
+
+ public void writeListFiles(File outputDir, List<Area> areas, String kmlOutputFile,
+ String outputType) throws IOException {
+ if (polygonDescProcessor != null)
+ polygonDescProcessor.writeListFiles(outputDir, areas, kmlOutputFile, outputType);
+ }
+
+
+ public List<PolygonDesc> getPolygons() {
+ return Collections.unmodifiableList(polygons);
+ }
+ /**
+ * Make sure that our areas cover the planet. This is done by adding
+ * pseudo-areas if needed.
+ * @param realAreas list of areas (read from split-file or calculated)
+ * @return new list of areas containing the real areas and additional areas
+ */
+ public static List<Area> addPseudoAreas(List<Area> realAreas){
+ ArrayList<Area> areas = new ArrayList<>(realAreas);
+ Rectangle planetBounds = new Rectangle(Utils.toMapUnit(-180.0), Utils.toMapUnit(-90.0), 2* Utils.toMapUnit(180.0), 2 * Utils.toMapUnit(90.0));
+
+ while (!checkIfCovered(planetBounds, areas)){
+ boolean changed = addPseudoArea(areas);
+
+ if (!changed){
+ throw new SplitFailedException("Failed to fill planet with pseudo-areas");
+ }
+ }
+ return areas;
+ }
+ /**
+ * Work around for possible rounding errors in area.subtract processing
+ * @param area an area that is considered to be empty or a rectangle
+ * @return
+ */
+ private static java.awt.geom.Area simplifyArea(java.awt.geom.Area area) {
+ if (area.isEmpty() || area.isRectangular())
+ return area;
+ // area.isRectugular() may returns false although the shape is a
+ // perfect rectangle :-( If we subtract the area from its bounding
+ // box we get better results.
+ java.awt.geom.Area bbox = new java.awt.geom.Area (area.getBounds2D());
+ bbox.subtract(area);
+ if (bbox.isEmpty()) // bbox equals area: is a rectangle
+ return new java.awt.geom.Area (area.getBounds2D());
+ return area;
+ }
+
+ private static boolean checkIfCovered(Rectangle bounds, ArrayList<Area> areas){
+ java.awt.geom.Area bbox = new java.awt.geom.Area(bounds);
+ long sumTiles = 0;
+
+ for (Area area: areas){
+ sumTiles += (long)area.getHeight() * (long)area.getWidth();
+ bbox.subtract(area.getJavaArea());
+ }
+ long areaBox = (long) bounds.height*(long)bounds.width;
+
+ if (sumTiles != areaBox)
+ return false;
+
+ return bbox.isEmpty();
+ }
+
+ /**
+ * Create a list of areas that do not overlap. If areas in the original
+ * list are overlapping, they can be replaced by up to 5 disjoint areas.
+ * This is done if parameter makeDisjoint is true
+ * @param realAreas the list of areas
+ * @return the new list
+ */
+ public static ArrayList<Area> getNonOverlappingAreas(final List<Area> realAreas){
+ java.awt.geom.Area covered = new java.awt.geom.Area();
+ ArrayList<Area> splitList = new ArrayList<>();
+ int artificialId = -99999999;
+ boolean foundOverlap = false;
+ for (Area area1 : realAreas) {
+ Rectangle r1 = area1.getRect();
+ if (covered.intersects(r1) == false){
+ splitList.add(area1);
+ }
+ else {
+ if (foundOverlap == false){
+ foundOverlap = true;
+ System.out.println("Removing overlaps from tiles...");
+ }
+ //String msg = "splitting " + area1.getMapId() + " " + (i+1) + "/" + realAreas.size() + " overlapping ";
+ // find intersecting areas in the already covered part
+ ArrayList<Area> splitAreas = new ArrayList<>();
+
+ for (int j = 0; j < splitList.size(); j++){
+ Area area2 = splitList.get(j);
+ if (area2 == null)
+ continue;
+ Rectangle r2 = area2.getRect();
+ if (r1.intersects(r2)){
+ java.awt.geom.Area overlap = new java.awt.geom.Area(area1.getRect());
+ overlap.intersect(area2.getJavaArea());
+ Rectangle ro = overlap.getBounds();
+ if (ro.height == 0 || ro.width == 0)
+ continue;
+ //msg += area2.getMapId() + " ";
+ Area aNew = new Area(ro.y, ro.x, (int)ro.getMaxY(),(int)ro.getMaxX());
+ aNew.setMapId(artificialId++);
+ aNew.setName("" + area1.getMapId());
+ aNew.setJoinable(false);
+ covered.subtract(area2.getJavaArea());
+ covered.add(overlap);
+ splitList.set(j, aNew);
+
+ java.awt.geom.Area coveredByPair = new java.awt.geom.Area(r1);
+ coveredByPair.add(new java.awt.geom.Area(r2));
+
+ java.awt.geom.Area originalPair = new java.awt.geom.Area(coveredByPair);
+
+ int minX = coveredByPair.getBounds().x;
+ int minY = coveredByPair.getBounds().y;
+ int maxX = (int) coveredByPair.getBounds().getMaxX();
+ int maxY = (int) coveredByPair.getBounds().getMaxY();
+ coveredByPair.subtract(overlap);
+ if (coveredByPair.isEmpty())
+ continue; // two equal areas a
+
+ coveredByPair.subtract(covered);
+ java.awt.geom.Area testSplit = new java.awt.geom.Area(overlap);
+
+ Rectangle[] rectPair = {r1,r2};
+ Area[] areaPair = {area1,area2};
+ int lx = minX;
+ int lw = ro.x-minX;
+ int rx = (int)ro.getMaxX();
+ int rw = maxX - rx;
+ int uy = (int)ro.getMaxY();
+ int uh = maxY - uy;
+ int by = minY;
+ int bh = ro.y - by;
+ Rectangle[] clippers = {
+ new Rectangle(lx, minY, lw, bh), // lower left
+ new Rectangle(ro.x, minY, ro.width, bh), // lower middle
+ new Rectangle(rx, minY, rw, bh), // lower right
+ new Rectangle(lx, ro.y, lw, ro.height), // left
+ new Rectangle(rx, ro.y, rw, ro.height), // right
+ new Rectangle(lx, uy, lw, uh), // upper left
+ new Rectangle(ro.x, uy, ro.width, uh), // upper middle
+ new Rectangle(rx, uy, rw, uh) // upper right
+ };
+
+ for (Rectangle clipper: clippers){
+ for (int k = 0; k <= 1; k++){
+ Rectangle test = clipper.intersection(rectPair[k]);
+ if (!test.isEmpty()){
+ testSplit.add(new java.awt.geom.Area(test));
+ if (k==1 || covered.intersects(test) == false){
+ aNew = new Area(test.y,test.x,(int)test.getMaxY(),(int)test.getMaxX());
+ aNew.setMapId(areaPair[k].getMapId());
+ splitAreas.add(aNew);
+ covered.add(aNew.getJavaArea());
+ }
+ }
+ }
+ }
+ assert testSplit.equals(originalPair);
+ }
+ }
+
+ // recombine parts that form a rectangle
+ for (Area splitArea: splitAreas){
+ if (splitArea.isJoinable()){
+ for (int j = 0; j < splitList.size(); j++){
+ Area area = splitList.get(j);
+ if (area == null || area.isJoinable() == false || area.getMapId() != splitArea.getMapId() )
+ continue;
+ boolean doJoin = false;
+ if (splitArea.getMaxLat() == area.getMaxLat()
+ && splitArea.getMinLat() == area.getMinLat()
+ && (splitArea.getMinLong() == area.getMaxLong() || splitArea.getMaxLong() == area.getMinLong()))
+ doJoin = true;
+ else if (splitArea.getMinLong() == area.getMinLong()
+ && splitArea.getMaxLong()== area.getMaxLong()
+ && (splitArea.getMinLat() == area.getMaxLat() || splitArea.getMaxLat() == area.getMinLat()))
+ doJoin = true;
+ if (doJoin){
+ splitArea = area.add(splitArea);
+ splitArea.setMapId(area.getMapId());
+ splitList.set(j, splitArea);
+ splitArea = null; // don't add later
+ break;
+ }
+ }
+ }
+ if (splitArea != null){
+ splitList.add(splitArea);
+ }
+ }
+ /*
+ if (msg.isEmpty() == false)
+ System.out.println(msg);
+ */
+ }
+ covered.add(new java.awt.geom.Area(r1));
+ }
+ covered.reset();
+ Iterator <Area> iter = splitList.iterator();
+ while (iter.hasNext()){
+ Area a = iter.next();
+ if (a == null)
+ iter.remove();
+ else {
+ Rectangle r1 = a.getRect();
+ if (covered.intersects(r1) == true){
+ throw new SplitFailedException("Failed to create list of distinct areas");
+ }
+ covered.add(a.getJavaArea());
+ }
+ }
+ return splitList;
+ }
+
+ /**
+ * Fill uncovered parts of the planet with pseudo-areas.
+ * TODO: check if better algorithm reduces run time in ProblemListProcessor
+ * We want a small number of pseudo areas because many of them will
+ * require more memory or more passes, esp. when processing whole planet.
+ * Also, the total length of all edges should be small.
+ * @param areas list of areas (either real or pseudo)
+ * @return true if pseudo-areas were added
+ */
+ private static boolean addPseudoArea(ArrayList<Area> areas) {
+ int oldSize = areas.size();
+ Rectangle planetBounds = new Rectangle(Utils.toMapUnit(-180.0), Utils.toMapUnit(-90.0), 2* Utils.toMapUnit(180.0), 2 * Utils.toMapUnit(90.0));
+ java.awt.geom.Area uncovered = new java.awt.geom.Area(planetBounds);
+ java.awt.geom.Area covered = new java.awt.geom.Area();
+ for (Area area: areas){
+ uncovered.subtract(area.getJavaArea());
+ covered.add(area.getJavaArea());
+ }
+ Rectangle rCov = covered.getBounds();
+ Rectangle[] topAndBottom = {
+ new Rectangle(planetBounds.x,(int)rCov.getMaxY(),planetBounds.width, (int)(planetBounds.getMaxY()-rCov.getMaxY())), // top
+ new Rectangle(planetBounds.x,planetBounds.y,planetBounds.width,rCov.y-planetBounds.y)}; // bottom
+ for (Rectangle border: topAndBottom){
+ if (!border.isEmpty()){
+ uncovered.subtract(new java.awt.geom.Area(border));
+ covered.add(new java.awt.geom.Area(border));
+ Area pseudo = new Area(border.y,border.x,(int)border.getMaxY(),(int)border.getMaxX());
+ pseudo.setMapId(-1 * (areas.size()+1));
+ pseudo.setPseudoArea(true);
+ areas.add(pseudo);
+ }
+ }
+ while (uncovered.isEmpty() == false){
+ boolean changed = false;
+ List<List<Point>> shapes = Utils.areaToShapes(uncovered);
+ // we divide planet into stripes for all vertices of the uncovered area
+ int minX = uncovered.getBounds().x;
+ int nextX = Integer.MAX_VALUE;
+ for (int i = 0; i < shapes.size(); i++){
+ List<Point> shape = shapes.get(i);
+ for (Point point: shape){
+ int lon = point.x;
+ if (lon < nextX && lon > minX)
+ nextX = lon;
+ }
+ }
+ java.awt.geom.Area stripeLon = new java.awt.geom.Area(new Rectangle(minX, planetBounds.y, nextX - minX, planetBounds.height));
+ // cut out already covered area
+ stripeLon.subtract(covered);
+ assert stripeLon.isEmpty() == false;
+ // the remaining area must be a set of zero or more disjoint rectangles
+ List<List<Point>> stripeShapes = Utils.areaToShapes(stripeLon);
+ for (int j = 0; j < stripeShapes .size(); j++){
+ List<Point> rectShape = stripeShapes .get(j);
+ java.awt.geom.Area test = Utils.shapeToArea(rectShape);
+ test = simplifyArea(test);
+ assert test.isRectangular();
+ Rectangle pseudoRect = test.getBounds();
+ if (uncovered.contains(pseudoRect)){
+ assert test.getBounds().width == stripeLon.getBounds().width;
+ boolean wasMerged = false;
+ // check if new area can be merged with last rectangles
+ for (int k=areas.size()-1; k >= oldSize; k--){
+ Area prev = areas.get(k);
+ if (prev.getMaxLong() < pseudoRect.x || prev.isPseudoArea() == false)
+ continue;
+ if (prev.getHeight() == pseudoRect.height && prev.getMaxLong() == pseudoRect.x && prev.getMinLat() == pseudoRect.y){
+ // merge
+ Area pseudo = prev.add(new Area(pseudoRect.y,pseudoRect.x,(int)pseudoRect.getMaxY(),(int)pseudoRect.getMaxX()));
+ pseudo.setMapId(prev.getMapId());
+ pseudo.setPseudoArea(true);
+ areas.set(k, pseudo);
+ //System.out.println("Enlarged pseudo area " + pseudo.getMapId() + " " + pseudo);
+ wasMerged = true;
+ break;
+ }
+ }
+
+ if (!wasMerged){
+ Area pseudo = new Area(pseudoRect.y, pseudoRect.x, (int)pseudoRect.getMaxY(), (int)pseudoRect.getMaxX());
+ pseudo.setMapId(-1 * (areas.size()+1));
+ pseudo.setPseudoArea(true);
+ //System.out.println("Adding pseudo area " + pseudo.getMapId() + " " + pseudo);
+ areas.add(pseudo);
+ }
+ uncovered.subtract(test);
+ covered.add(test);
+ changed = true;
+ }
+ }
+ if (!changed)
+ break;
+ }
+ return oldSize != areas.size();
+ }
+
+
+}
diff --git a/src/uk/me/parabola/splitter/BinaryMapWriter.java b/src/uk/me/parabola/splitter/BinaryMapWriter.java
index 21ae043..b87390b 100644
--- a/src/uk/me/parabola/splitter/BinaryMapWriter.java
+++ b/src/uk/me/parabola/splitter/BinaryMapWriter.java
@@ -52,7 +52,7 @@ public class BinaryMapWriter extends AbstractOSMWriter {
/** Base class containing common code needed for serializing each type of primitives. */
private abstract class Prim<T extends Element> {
/** Queue that tracks the list of all primitives. */
- ArrayList<T> contents = new ArrayList<T>();
+ ArrayList<T> contents = new ArrayList<>();
/** Add to the queue.
* @param item The entity to add */
@@ -473,12 +473,12 @@ public class BinaryMapWriter extends AbstractOSMWriter {
Osmformat.HeaderBlock.Builder headerblock = Osmformat.HeaderBlock
.newBuilder();
- Osmformat.HeaderBBox.Builder bbox = Osmformat.HeaderBBox.newBuilder();
- bbox.setLeft(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMinLong())));
- bbox.setBottom(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMinLat())));
- bbox.setRight(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMaxLong())));
- bbox.setTop(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMaxLat())));
- headerblock.setBbox(bbox);
+ Osmformat.HeaderBBox.Builder pbfBbox = Osmformat.HeaderBBox.newBuilder();
+ pbfBbox.setLeft(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMinLong())));
+ pbfBbox.setBottom(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMinLat())));
+ pbfBbox.setRight(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMaxLong())));
+ pbfBbox.setTop(serializer.mapRawDegrees(Utils.toDegrees(bounds.getMaxLat())));
+ headerblock.setBbox(pbfBbox);
// headerblock.setSource("splitter"); //TODO: entity.getOrigin());
finishHeader(headerblock);
diff --git a/src/uk/me/parabola/splitter/DataStorer.java b/src/uk/me/parabola/splitter/DataStorer.java
index 52164f1..0329769 100644
--- a/src/uk/me/parabola/splitter/DataStorer.java
+++ b/src/uk/me/parabola/splitter/DataStorer.java
@@ -10,128 +10,199 @@
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*/
- package uk.me.parabola.splitter;
+package uk.me.parabola.splitter;
import java.io.File;
import java.io.IOException;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
/**
* Stores data that is needed in different passes of the program.
+ *
* @author GerdP
*
*/
-public class DataStorer{
- public static final int NODE_TYPE = 0;
- public static final int WAY_TYPE = 1;
- public static final int REL_TYPE = 2;
-
- private final int numOfWriters;
- private final Long2IntClosedMapFunction[] maps = new Long2IntClosedMapFunction[3];
-
- private final WriterDictionaryShort writerDictionary;
- private final WriterDictionaryInt multiTileWriterDictionary;
- private final WriterIndex writerIndex;
- private SparseLong2ShortMapFunction usedWays = null;
- private final OSMId2ObjectMap<Integer> usedRels = new OSMId2ObjectMap<Integer>();
- private boolean idsAreNotSorted;
-
- /**
- * Create a dictionary for a given number of writers
- * @param numOfWriters the number of writers that are used
- */
- DataStorer (OSMWriter [] writers){
- this.numOfWriters = writers.length;
- this.writerDictionary = new WriterDictionaryShort(writers);
- this.multiTileWriterDictionary = new WriterDictionaryInt(writers);
- this.writerIndex = new WriterGrid(writerDictionary);
- return;
- }
-
- public int getNumOfWriters(){
- return numOfWriters;
- }
-
- public WriterDictionaryShort getWriterDictionary() {
- return writerDictionary;
- }
-
-
- public void setWriterMap(int type, Long2IntClosedMapFunction nodeWriterMap){
- maps[type] = nodeWriterMap;
- }
- public Long2IntClosedMapFunction getWriterMap(int type){
- return maps[type];
- }
-
- public WriterIndex getGrid() {
- return writerIndex;
- }
-
- public WriterDictionaryInt getMultiTileWriterDictionary() {
- return multiTileWriterDictionary;
- }
-
- public SparseLong2ShortMapFunction getUsedWays() {
- return usedWays;
- }
-
- public OSMId2ObjectMap<Integer> getUsedRels() {
- return usedRels;
- }
-
- public void setUsedWays(SparseLong2ShortMapFunction ways) {
- usedWays = ways;
- }
-
- public boolean isIdsAreNotSorted() {
- return idsAreNotSorted;
- }
-
- public void setIdsAreNotSorted(boolean idsAreNotSorted) {
- this.idsAreNotSorted = idsAreNotSorted;
- }
-
- public void restartWriterMaps() {
- for (Long2IntClosedMapFunction map: maps){
- if (map != null){
- try {
- map.close();
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- }
- }
-
- }
-
- public void switchToSeqAccess(File fileOutputDir) throws IOException {
- boolean msgWritten = false;
- long start = System.currentTimeMillis();
- for (Long2IntClosedMapFunction map: maps){
- if (map != null){
- if (!msgWritten){
- System.out.println("Writing results of MultiTileAnalyser to temp files ...");
- msgWritten = true;
- }
- map.switchToSeqAccess(fileOutputDir);
- }
- }
- System.out.println("Writing temp files took " + (System.currentTimeMillis()-start) + " ms");
- }
-
- public void finish() {
- for (Long2IntClosedMapFunction map: maps){
- if (map != null)
- map.finish();
- }
- }
-
- public void stats(final String prefix) {
- for (Long2IntClosedMapFunction map: maps){
- if (map != null)
- map.stats(prefix);
- }
-
- }
+public class DataStorer {
+ public static final int NODE_TYPE = 0;
+ public static final int WAY_TYPE = 1;
+ public static final int REL_TYPE = 2;
+
+ private final int numOfAreas;
+
+ private final Long2IntClosedMapFunction[] maps = new Long2IntClosedMapFunction[3];
+
+ private final AreaDictionaryShort areaDictionary;
+ private final AreaDictionaryInt multiTileDictionary;
+ private final AreaIndex areaIndex;
+ private SparseLong2ShortMapFunction usedWays = null;
+ private final OSMId2ObjectMap<Integer> usedRels = new OSMId2ObjectMap<>();
+ private boolean idsAreNotSorted;
+ private OSMWriter[] writers;
+ /** map with relations that should be complete and are written to only one tile */
+ private final Long2ObjectOpenHashMap<Integer> oneDistinctAreaOnlyRels = new Long2ObjectOpenHashMap<>();
+ private final OSMId2ObjectMap<Integer> oneTileOnlyRels = new OSMId2ObjectMap<>();
+
+ /**
+ * Create a dictionary for a given number of writers
+ *
+ * @param overlapAmount
+ * @param numOfWriters
+ * the number of writers that are used
+ */
+ DataStorer(List<Area> areas, int overlapAmount) {
+ this.numOfAreas = areas.size();
+ this.areaDictionary = new AreaDictionaryShort(areas, overlapAmount);
+ this.multiTileDictionary = new AreaDictionaryInt(numOfAreas);
+ this.areaIndex = new AreaGrid(areaDictionary);
+ return;
+ }
+
+ public int getNumOfAreas() {
+ return numOfAreas;
+ }
+
+ public AreaDictionaryShort getAreaDictionary() {
+ return areaDictionary;
+ }
+
+ public Area getArea(int idx) {
+ return areaDictionary.getArea(idx);
+ }
+
+ public Area getExtendedArea(int idx) {
+ return areaDictionary.getExtendedArea(idx);
+ }
+
+ public void setWriters(OSMWriter[] writers) {
+ this.writers = writers;
+ }
+
+ public void setWriterMap(int type, Long2IntClosedMapFunction nodeWriterMap) {
+ maps[type] = nodeWriterMap;
+ }
+
+ public Long2IntClosedMapFunction getWriterMap(int type) {
+ return maps[type];
+ }
+
+ public AreaIndex getGrid() {
+ return areaIndex;
+ }
+
+ public AreaDictionaryInt getMultiTileDictionary() {
+ return multiTileDictionary;
+ }
+
+ public SparseLong2ShortMapFunction getUsedWays() {
+ return usedWays;
+ }
+
+ public OSMId2ObjectMap<Integer> getUsedRels() {
+ return usedRels;
+ }
+
+ public void setUsedWays(SparseLong2ShortMapFunction ways) {
+ usedWays = ways;
+ }
+
+ public boolean isIdsAreNotSorted() {
+ return idsAreNotSorted;
+ }
+
+ public void setIdsAreNotSorted(boolean idsAreNotSorted) {
+ this.idsAreNotSorted = idsAreNotSorted;
+ }
+
+ public void restartWriterMaps() {
+ for (Long2IntClosedMapFunction map : maps) {
+ if (map != null) {
+ try {
+ map.close();
+ } catch (IOException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+ }
+ }
+ }
+
+ }
+
+ public void switchToSeqAccess(File fileOutputDir) throws IOException {
+ boolean msgWritten = false;
+ long start = System.currentTimeMillis();
+ for (Long2IntClosedMapFunction map : maps) {
+ if (map != null) {
+ if (!msgWritten) {
+ System.out.println("Writing results of MultiTileAnalyser to temp files ...");
+ msgWritten = true;
+ }
+ map.switchToSeqAccess(fileOutputDir);
+ }
+ }
+ System.out
+ .println("Writing temp files took " + (System.currentTimeMillis() - start) + " ms");
+ }
+
+ public void finish() {
+ for (Long2IntClosedMapFunction map : maps) {
+ if (map != null)
+ map.finish();
+ }
+ }
+
+ public void stats(final String prefix) {
+ for (Long2IntClosedMapFunction map : maps) {
+ if (map != null)
+ map.stats(prefix);
+ }
+ }
+
+ public OSMWriter[] getWriters() {
+ return writers;
+ }
+
+ public void storeRelationArea(long id, Integer areaIdx) {
+ oneDistinctAreaOnlyRels.put(id, areaIdx);
+ }
+
+ public Integer getOneTileOnlyRels(long id) {
+ return oneTileOnlyRels.get(id);
+ }
+
+ /**
+ * If the BitSet ids in oneTileOnlyRels were produced with a different set of
+ * areas we have to translate the values
+ * @param distinctAreas list of distinct (non-overlapping) areas
+ * @param distinctDataStorer
+ */
+ public void translateDistinctToRealAreas(DataStorer distinctDataStorer) {
+ List<Area> distinctAreas = distinctDataStorer.getAreaDictionary().getAreas();
+ Map<Area, Integer> map = new HashMap<>();
+ for (Area distinctArea : distinctAreas) {
+ if (distinctArea.getMapId() < 0 && !distinctArea.isPseudoArea()) {
+ BitSet w = new BitSet();
+ for (int i = 0; i < getNumOfAreas(); i++) {
+ if (this.areaDictionary.getArea(i).contains(distinctArea)) {
+ w.set(i);
+ }
+ }
+ map.put(distinctArea, this.multiTileDictionary.translate(w));
+ }
+ }
+
+ for ( Entry<Long, Integer> e: distinctDataStorer.oneDistinctAreaOnlyRels.entrySet()) {
+ if (e.getValue() >= 0 && !distinctAreas.get(e.getValue()).isPseudoArea()) {
+ Integer areaIdx = map.get(distinctAreas.get(e.getValue()));
+ oneTileOnlyRels.put(e.getKey(), areaIdx != null ? areaIdx: e.getValue());
+ } else {
+ oneTileOnlyRels.put(e.getKey(), AreaDictionaryInt.UNASSIGNED);
+ }
+
+ }
+ }
}
diff --git a/src/uk/me/parabola/splitter/KmlWriter.java b/src/uk/me/parabola/splitter/KmlWriter.java
index 0bcf292..456aa94 100644
--- a/src/uk/me/parabola/splitter/KmlWriter.java
+++ b/src/uk/me/parabola/splitter/KmlWriter.java
@@ -133,6 +133,7 @@ public class KmlWriter {
* @param filename The KML filename to write to.
*/
public static void writeKml(String filename, List<Area> areas) {
+ System.out.println("Writing KML file to " + filename);
try (PrintWriter pw = new PrintWriter(filename);) {
writeKmlHeader(pw);
for (Area area : areas) {
diff --git a/src/uk/me/parabola/splitter/Main.java b/src/uk/me/parabola/splitter/Main.java
index deca79b..45ff360 100644
--- a/src/uk/me/parabola/splitter/Main.java
+++ b/src/uk/me/parabola/splitter/Main.java
@@ -13,44 +13,18 @@
package uk.me.parabola.splitter;
-import crosby.binary.file.BlockInputStream;
-
-import org.openstreetmap.osmosis.core.filter.common.PolygonFileReader;
import org.xmlpull.v1.XmlPullParserException;
import uk.me.parabola.splitter.args.ParamParser;
import uk.me.parabola.splitter.args.SplitterParams;
-import uk.me.parabola.splitter.geo.City;
-import uk.me.parabola.splitter.geo.CityFinder;
-import uk.me.parabola.splitter.geo.CityLoader;
-import uk.me.parabola.splitter.geo.DefaultCityFinder;
-import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
-import it.unimi.dsi.fastutil.longs.LongArrayList;
-import it.unimi.dsi.fastutil.shorts.ShortArrayList;
-
-import java.awt.Point;
import java.awt.Rectangle;
import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileWriter;
import java.io.IOException;
-import java.io.InputStream;
-import java.io.InputStreamReader;
-import java.io.LineNumberReader;
-import java.io.PrintWriter;
-import java.io.Reader;
-import java.util.ArrayList;
import java.util.Arrays;
-import java.util.BitSet;
+import java.util.Collections;
import java.util.Date;
-import java.util.HashMap;
-import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Set;
-import java.util.TreeSet;
import java.util.regex.Pattern;
/**
@@ -103,8 +77,6 @@ public class Main {
private boolean mixed;
// A polygon file in osmosis polygon format
private String polygonFile;
- private List<PolygonDesc> polygons = new ArrayList<>();
- Rectangle polgonsBoundingBox = null;
// The path where the results are written out to.
private File fileOutputDir;
@@ -130,31 +102,21 @@ public class Main {
private String[] boundaryTags;
- private LongArrayList problemWays = new LongArrayList();
- private LongArrayList problemRels = new LongArrayList();
- private TreeSet<Long> calculatedProblemWays = new TreeSet<>();
- private TreeSet<Long> calculatedProblemRels = new TreeSet<>();
-
- // map with relations that should be complete and are written to only one tile
- private final Long2ObjectOpenHashMap<Integer> oneTileOnlyRels = new Long2ObjectOpenHashMap<>();
-
- // for faster access on blocks in pbf files
- private final HashMap<String, ShortArrayList> blockTypeMap = new HashMap<>();
- // for faster access on blocks in o5m files
- private final HashMap<String, long[]> skipArrayMap = new HashMap<>();
-
private String stopAfter;
private String precompSeaDir;
private String polygonDescFile;
- private PolygonDescProcessor polygonDescProcessor;
private int searchLimit;
private String handleElementVersion;
private boolean ignoreBoundsTags;
+
+ private final OSMFileHandler osmFileHandler = new OSMFileHandler();
+ private final AreasCalculator areasCalculator = new AreasCalculator();
+ private final ProblemLists problemList = new ProblemLists();
public static void main(String[] args) {
Main m = new Main();
@@ -189,7 +151,16 @@ public class Main {
long start = System.currentTimeMillis();
System.out.println("Time started: " + new Date());
try {
- split();
+ List<Area> areas = split();
+ DataStorer dataStorer;
+ if (keepComplete) {
+ dataStorer = calcProblemLists(areas);
+ useProblemLists(dataStorer);
+ } else {
+ dataStorer = new DataStorer(areas, overlapAmount);
+ }
+ writeTiles(dataStorer);
+ dataStorer.finish();
} catch (IOException e) {
System.err.println("Error opening or reading file " + e);
e.printStackTrace();
@@ -238,101 +209,96 @@ public class Main {
*/
}
- private void split() throws IOException, XmlPullParserException {
-
- File outputDir = fileOutputDir;
- if (!outputDir.exists()) {
- System.out.println("Output directory not found. Creating directory '" + fileOutputDir + "'");
- if (!outputDir.mkdirs()) {
- System.err.println("Unable to create output directory! Using default directory instead");
- fileOutputDir = new File(DEFAULT_DIR);
- }
- } else if (!outputDir.isDirectory()) {
- System.err.println("The --output-dir parameter must specify a directory. The --output-dir parameter is being ignored, writing to default directory instead.");
- fileOutputDir = new File(DEFAULT_DIR);
- }
-
- if (fileNameList.isEmpty()) {
- throw new IllegalArgumentException("No input files were supplied");
- }
-
- if (areaList == null) {
- int alignment = 1 << (24 - resolution);
- System.out.println("Map is being split for resolution " + resolution + ':');
- System.out.println(" - area boundaries are aligned to 0x" + Integer.toHexString(alignment) + " map units (" + Utils.toDegrees(alignment) + " degrees)");
- System.out.println(" - areas are multiples of 0x" + Integer.toHexString(alignment) + " map units wide and high");
- areaList = calculateAreas();
- if (areaList == null || areaList.getAreas().isEmpty()){
- System.err.println("Failed to calculate areas. See stdout messages for details.");
- System.out.println("Failed to calculate areas.");
- System.out.println("Sorry. Cannot split the file without creating huge, almost empty, tiles.");
- System.out.println("Please specify a bounding polygon with the --polygon-file parameter.");
- throw new SplitFailedException("");
- }
- if (mapId + areaList.getAreas().size() > 99999999){
- System.err.println("Too many areas for initial mapid " + mapId + ", resetting to 63240001");
- mapId = 63240001;
- }
- for (Area area : areaList.getAreas()) {
- area.setMapId(mapId++);
- if (description != null)
- area.setName(description);
- }
- nameAreas();
- areaList.write(new File(fileOutputDir, "areas.list").getPath());
- areaList.writePoly(new File(fileOutputDir, "areas.poly").getPath());
- } else {
- nameAreas();
- }
-
- List<Area> areas = areaList.getAreas();
-
- if (kmlOutputFile != null) {
- File out = new File(kmlOutputFile);
- if (!out.isAbsolute())
- out = new File(fileOutputDir, kmlOutputFile);
- System.out.println("Writing KML file to " + out.getPath());
- areaList.writeKml(out.getPath());
- }
- if (polygonDescProcessor != null)
- polygonDescProcessor.writeListFiles(outputDir, areas, kmlOutputFile, outputType);
- areaList.writeArgsFile(new File(fileOutputDir, "template.args").getPath(), outputType, -1);
-
- if ("split".equals(stopAfter)){
- try {Thread.sleep(1000);}catch (InterruptedException e) {}
- System.err.println("stopped after " + stopAfter);
- throw new StopNoErrorException("stopped after " + stopAfter);
- }
-
- System.out.println(areas.size() + " areas:");
- for (Area area : areas) {
- System.out.format("Area %08d: %d,%d to %d,%d covers %s",
- area.getMapId(),
- area.getMinLat(), area.getMinLong(),
- area.getMaxLat(), area.getMaxLong(),
- area.toHexString());
-
- if (area.getName() != null)
- System.out.print(' ' + area.getName());
- System.out.println();
- }
-
- List<Area> distinctAreas = null;
- if (keepComplete){
- distinctAreas = genProblemLists(areas);
- if ("gen-problem-list".equals(stopAfter)){
- try {Thread.sleep(1000);}catch (InterruptedException e) {}
- System.err.println("stopped after " + stopAfter);
- throw new StopNoErrorException("stopped after " + stopAfter);
- }
-
- }
- writeAreas(areas, distinctAreas);
- }
-
- private int getAreasPerPass(int areaCount) {
- return (int) Math.ceil((double) areaCount / (double) maxAreasPerPass);
- }
+ private List<Area> split() throws IOException, XmlPullParserException {
+
+ File outputDir = fileOutputDir;
+ if (!outputDir.exists()) {
+ System.out.println("Output directory not found. Creating directory '" + fileOutputDir + "'");
+ if (!outputDir.mkdirs()) {
+ System.err.println("Unable to create output directory! Using default directory instead");
+ fileOutputDir = new File(DEFAULT_DIR);
+ }
+ } else if (!outputDir.isDirectory()) {
+ System.err.println("The --output-dir parameter must specify a directory. The --output-dir parameter is being ignored, writing to default directory instead.");
+ fileOutputDir = new File(DEFAULT_DIR);
+ }
+
+ if (fileNameList.isEmpty()) {
+ throw new IllegalArgumentException("No input files were supplied");
+ }
+
+ boolean writeAreas = false;
+ if (areaList.getAreas().isEmpty()) {
+ writeAreas = true;
+ int alignment = 1 << (24 - resolution);
+ System.out.println("Map is being split for resolution " + resolution + ':');
+ System.out.println(" - area boundaries are aligned to 0x" + Integer.toHexString(alignment) + " map units (" + Utils.toDegrees(alignment) + " degrees)");
+ System.out.println(" - areas are multiples of 0x" + Integer.toHexString(alignment) + " map units wide and high");
+ areaList.setAreas(calculateAreas());
+ if (areaList == null || areaList.getAreas().isEmpty()){
+ System.err.println("Failed to calculate areas. See stdout messages for details.");
+ System.out.println("Failed to calculate areas.");
+ System.out.println("Sorry. Cannot split the file without creating huge, almost empty, tiles.");
+ System.out.println("Please specify a bounding polygon with the --polygon-file parameter.");
+ throw new SplitFailedException("");
+ }
+ if (mapId + areaList.getAreas().size() > 99999999){
+ System.err.println("Too many areas for initial mapid " + mapId + ", resetting to 63240001");
+ mapId = 63240001;
+ }
+ areaList.setMapIds(mapId);
+ }
+ areaList.setAreaNames();
+ if (writeAreas) {
+ areaList.write(new File(fileOutputDir, "areas.list").getPath());
+ areaList.writePoly(new File(fileOutputDir, "areas.poly").getPath());
+ }
+
+ List<Area> areas = areaList.getAreas();
+
+ if (kmlOutputFile != null) {
+ File out = new File(kmlOutputFile);
+ if (!out.isAbsolute())
+ out = new File(fileOutputDir, kmlOutputFile);
+ KmlWriter.writeKml(out.getPath(), areas);
+ }
+ areasCalculator.writeListFiles(outputDir, areas, kmlOutputFile, outputType);
+ areaList.writeArgsFile(new File(fileOutputDir, "template.args").getPath(), outputType, -1);
+
+ if ("split".equals(stopAfter)){
+ try {Thread.sleep(1000);}catch (InterruptedException e) {}
+ System.err.println("stopped after " + stopAfter);
+ throw new StopNoErrorException("stopped after " + stopAfter);
+ }
+
+ System.out.println(areas.size() + " areas:");
+ for (Area area : areas) {
+ System.out.format("Area %08d: %d,%d to %d,%d covers %s",
+ area.getMapId(),
+ area.getMinLat(), area.getMinLong(),
+ area.getMaxLat(), area.getMaxLong(),
+ area.toHexString());
+
+ if (area.getName() != null)
+ System.out.print(' ' + area.getName());
+ System.out.println();
+ }
+ return areas;
+ }
+
+ private DataStorer calcProblemLists(List<Area> areas) {
+ DataStorer dataStorer = problemList.calcProblemLists(osmFileHandler, areas, wantedAdminLevel, boundaryTags, maxAreasPerPass, overlapAmount);
+ if (problemReport != null){
+ problemList.writeProblemList(fileOutputDir, problemReport);
+ }
+
+ if ("gen-problem-list".equals(stopAfter)){
+ try {Thread.sleep(1000);}catch (InterruptedException e) {}
+ System.err.println("stopped after " + stopAfter);
+ throw new StopNoErrorException("stopped after " + stopAfter);
+ }
+ return dataStorer;
+ }
/**
* Deal with the command line arguments.
@@ -373,6 +339,7 @@ public class Main {
System.out.println("Make sure that option parameters start with -- " );
throw new IllegalArgumentException();
}
+ osmFileHandler.setFileNames(fileNameList);
mapId = params.getMapid();
if (mapId > 99999999) {
mapId = 63240001;
@@ -399,11 +366,13 @@ public class Main {
}
}
description = params.getDescription();
+ areaList = new AreaList(description);
geoNamesFile = params.getGeonamesFile();
if (geoNamesFile != null){
if (testAndReportFname(geoNamesFile, "geonames-file") == false){
throw new IllegalArgumentException();
}
+ areaList.setGeoNamesFile (geoNamesFile);
}
resolution = params.getResolution();
trim = !params.isNoTrim();
@@ -418,6 +387,7 @@ public class Main {
resolution = 13;
}
mixed = params.isMixed();
+ osmFileHandler.setMixed(mixed);
statusFreq = params.getStatusFreq();
String outputDir = params.getOutputDir();
@@ -434,7 +404,7 @@ public class Main {
problemFile = params.getProblemFile();
if (problemFile != null){
- if (!readProblemIds(problemFile))
+ if (!problemList.readProblemIds(problemFile))
throw new IllegalArgumentException();
}
String splitFile = params.getSplitFile();
@@ -463,8 +433,7 @@ public class Main {
problemReport = params.getProblemReport();
String boundaryTagsParm = params.getBoundaryTags();
if ("use-exclude-list".equals(boundaryTagsParm) == false){
- Pattern csvSplitter = Pattern.compile(Pattern.quote(","));
- boundaryTags = csvSplitter.split(boundaryTagsParm);
+ boundaryTags = boundaryTagsParm.split(Pattern.quote(","));
}
if (keepComplete){
@@ -504,7 +473,7 @@ public class Main {
}
if (splitFile != null) {
try {
- areaList = new AreaList();
+ areaList = new AreaList(description);
areaList.read(splitFile);
areaList.dump();
} catch (IOException e) {
@@ -519,17 +488,7 @@ public class Main {
if (splitFile != null){
System.out.println("Warning: parameter polygon-file is ignored because split-file is used.");
} else {
- File f = new File(polygonFile);
-
- if (!f.exists()){
- throw new IllegalArgumentException("polygon file doesn't exist: " + polygonFile);
- }
- PolygonFileReader polyReader = new PolygonFileReader(f);
- java.awt.geom.Area polygonInDegrees = polyReader.loadPolygon();
- PolygonDesc pd = new PolygonDesc(polyReader.getPolygonName(),
- Utils.AreaDegreesToMapUnit(polygonInDegrees),
- mapId);
- polygons.add(pd);
+ areasCalculator.readPolygonFile(polygonFile, mapId);
}
}
polygonDescFile = params.getPolygonDescFile();
@@ -537,32 +496,14 @@ public class Main {
if (splitFile != null){
System.out.println("Warning: parameter polygon-desc-file is ignored because split-file is used.");
} else {
- File f = new File(polygonDescFile);
-
- if (!f.exists()){
- System.out.println("Error: polygon desc file doesn't exist: " + polygonDescFile);
- System.exit(-1);
- }
- polygonDescProcessor = new PolygonDescProcessor(resolution);
- try {
- processOSMFiles(polygonDescProcessor, Arrays.asList(polygonDescFile));
- polygons = polygonDescProcessor.getPolygons();
- } catch (XmlPullParserException e) {
- polygons = null;
- polygonDescProcessor = null;
- System.err.println("Could not read polygon desc file");
- e.printStackTrace();
- }
- }
- }
- if (polygons.isEmpty() == false){
- if (polygons.size() == 1){
- polgonsBoundingBox = polygons.get(0).area.getBounds();
- }
- if (checkPolygons() == false){
- System.out.println("Warning: Bounding polygon is complex. Splitter might not be able to fit all tiles into the polygon!");
+ areasCalculator.readPolygonDescFile(polygonDescFile);
}
}
+ areasCalculator.setResolution(resolution);
+ if (!areasCalculator.checkPolygons()) {
+ System.out.println(
+ "Warning: Bounding polygon is complex. Splitter might not be able to fit all tiles into the polygon!");
+ }
stopAfter = params.getStopAfter();
if (Arrays.asList("split", "gen-problem-list" , "handle-problem-list", "dist").contains(stopAfter) == false){
throw new IllegalArgumentException("the --stop-after parameter must be either split, gen-problem-list, handle-problem-list, or dist.");
@@ -575,8 +516,9 @@ public class Main {
throw new IllegalArgumentException("precomp-sea directory doesn't exist or is not readable: " + precompSeaDir);
}
}
- if (polygons.isEmpty() == false && numTiles > 0){
- if (polygons.size() == 1){
+ int numPolygons = areasCalculator.getPolygons().size();
+ if (numPolygons > 0 && numTiles > 0){
+ if (numPolygons == 1){
System.out.println("Warning: Parameter polygon-file is only used to calculate the bounds because --num-tiles is used");
} else {
System.out.println("Warning: parameter polygon-file is ignored because --num-tiles is used");
@@ -598,257 +540,87 @@ public class Main {
* Calculate the areas that we are going to split into by getting the total area and
* then subdividing down until each area has at most max-nodes nodes in it.
*/
- private AreaList calculateAreas() throws XmlPullParserException {
-
- DensityMapCollector pass1Collector = new DensityMapCollector(resolution, ignoreBoundsTags);
- MapProcessor processor = pass1Collector;
-
- File densityData = new File("densities.txt");
- File densityOutData = null;
- if (densityData.exists() && densityData.isFile()){
- System.err.println("reading density data from " + densityData.getAbsolutePath());
- pass1Collector.readMap(densityData.getAbsolutePath());
- }
- else {
- densityOutData = new File(fileOutputDir,"densities-out.txt");
- processMap(processor);
- }
- System.out.println("in " + fileNameList.size() + (fileNameList.size() == 1 ? " file" : " files"));
- System.out.println("Time: " + new Date());
- if (densityOutData != null )
- pass1Collector.saveMap(densityOutData.getAbsolutePath());
-
- Area exactArea = pass1Collector.getExactArea();
- System.out.println("Exact map coverage read from input file(s) is " + exactArea);
- if (polgonsBoundingBox != null){
- exactArea = Area.calcArea(exactArea, polgonsBoundingBox);
- if (exactArea != null)
- System.out.println("Exact map coverage after applying bounding box of polygon-file is " + exactArea);
- else {
- System.out.println("Exact map coverage after applying bounding box of polygon-file is an empty area" );
- return new AreaList(new ArrayList<Area>());
- }
- }
-
- if (precompSeaDir != null){
- System.out.println("Counting nodes of precompiled sea data ...");
- long startSea = System.currentTimeMillis();
- DensityMapCollector seaCollector = new DensityMapCollector(resolution, true);
- PrecompSeaReader precompSeaReader = new PrecompSeaReader(exactArea, new File(precompSeaDir));
- precompSeaReader.processMap(seaCollector);
- pass1Collector.mergeSeaData(seaCollector, trim, resolution);
- System.out.println("Precompiled sea data pass took " + (System.currentTimeMillis()-startSea) + " ms");
- }
- Area roundedBounds = RoundingUtils.round(exactArea, resolution);
- SplittableDensityArea splittableArea = pass1Collector.getSplitArea(searchLimit, roundedBounds);
- if (splittableArea.hasData() == false){
- System.out.println("input file(s) have no data inside calculated bounding box");
- return new AreaList(new ArrayList<Area>());
- }
- System.out.println("Rounded map coverage is " + splittableArea.getBounds());
-
- splittableArea.setTrim(trim);
- splittableArea.setMapId(mapId);
- long startSplit = System.currentTimeMillis();
- List<Area> areas ;
- if (numTiles >= 2){
- System.out.println("Splitting nodes into " + numTiles + " areas");
- areas = splittableArea.split(numTiles);
- }
- else {
- System.out.println("Splitting nodes into areas containing a maximum of " + Utils.format(maxNodes) + " nodes each...");
- splittableArea.setMaxNodes(maxNodes);
- areas = splittableArea.split(polygons);
- }
- if (areas != null && areas.isEmpty() == false)
- System.out.println("Creating the initial areas took " + (System.currentTimeMillis()- startSplit) + " ms");
- return new AreaList(areas);
- }
-
- private void nameAreas() {
- CityFinder cityFinder = null;
- if (geoNamesFile != null){
- CityLoader cityLoader = new CityLoader(true);
- List<City> cities = cityLoader.load(geoNamesFile);
- if (cities == null)
- return;
-
- cityFinder = new DefaultCityFinder(cities);
- }
- for (Area area : areaList.getAreas()) {
- area.setName(description);
- if (cityFinder == null)
- continue;
-
- // Decide what to call the area
- Set<City> found = cityFinder.findCities(area);
- City bestMatch = null;
- for (City city : found) {
- if (bestMatch == null || city.getPopulation() > bestMatch.getPopulation()) {
- bestMatch = city;
- }
- }
- if (bestMatch != null)
- area.setName(bestMatch.getCountryCode() + '-' + bestMatch.getName());
- }
- }
-
- /**
- * Calculate lists of ways and relations that will be split for a given list
- * of areas.
- * @param distinctAreas the list of areas
- * @return
- * @throws IOException
- * @throws XmlPullParserException
- */
- private ArrayList<Area> genProblemLists(List<Area> realAreas) throws IOException, XmlPullParserException {
- long startProblemListGenerator = System.currentTimeMillis();
-
- ArrayList<Area> distinctAreas = getNonOverlappingAreas(realAreas);
- if (distinctAreas.size() > realAreas.size()) {
- System.err.println("Waring: The areas given in --split-file are overlapping. Support for this might be removed in future versions.");
- Set<Integer> overlappingTiles = new TreeSet<>();
- for (int i = 0; i < realAreas.size(); i++) {
- Area a1 = realAreas.get(i);
- for (int j = i+1; j < realAreas.size(); j++) {
- Area a2 = realAreas.get(j);
- if (a1.getRect().intersects(a2.getRect())) {
- overlappingTiles.add(a1.getMapId());
- overlappingTiles.add(a2.getMapId());
- }
- }
- }
- if (!overlappingTiles.isEmpty()) {
- System.out.println("Overlaping tiles: " + overlappingTiles.toString());
- }
- }
- System.out.println("Generating problem list for " + distinctAreas.size() + " distinct areas");
- List<Area> workAreas = addPseudoWriters(distinctAreas);
-
-
- // debugging
- /*
- AreaList planet = new AreaList(workAreas);
- String planetName = "planet-partition-" + partition + ".kml";
- File out = new File(planetName);
- if (!out.isAbsolute())
- kmlOutputFile = new File(fileOutputDir, planetName).getPath();
- System.out.println("Writing planet KML file " + kmlOutputFile);
- planet.writeKml(kmlOutputFile);
- */
- int numPasses = getAreasPerPass(workAreas.size());
- int areasPerPass = (int) Math.ceil((double) workAreas.size() / (double) numPasses);
- if (numPasses > 1) {
- System.out.println("Processing " + distinctAreas.size() + " areas in " + numPasses + " passes, " + areasPerPass + " areas at a time");
- } else {
- System.out.println("Processing " + distinctAreas.size() + " areas in a single pass");
- }
-
- OSMWriter [] writers = new OSMWriter[workAreas.size()];
- ArrayList<Area> allAreas = new ArrayList<>();
-
- System.out.println("Pseudo-Writers:");
- for (int j = 0;j < writers.length; j++){
- Area area = workAreas.get(j);
- allAreas.add(area);
- writers[j] = new PseudoOSMWriter(area, area.getMapId(), area.isPseudoArea(), 0);
- if (area.isPseudoArea())
- System.out.println("Pseudo area " + area.getMapId() + " covers " + area);
- }
- DataStorer dataStorer = new DataStorer(writers);
- System.out.println("Starting problem-list-generator pass(es)");
- LongArrayList problemWaysThisPart = new LongArrayList();
- LongArrayList problemRelsThisPart = new LongArrayList();
- for (int pass = 0; pass < numPasses; pass++) {
- System.out.println("-----------------------------------");
- System.out.println("Starting problem-list-generator pass " + (pass+1) + " of " + numPasses);
- long startThisPass = System.currentTimeMillis();
- int writerOffset = pass * areasPerPass;
- int numWritersThisPass = Math.min(areasPerPass, workAreas.size() - pass * areasPerPass);
- ProblemListProcessor processor = new ProblemListProcessor(
- dataStorer, writerOffset, numWritersThisPass,
- problemWaysThisPart, problemRelsThisPart, oneTileOnlyRels, boundaryTags);
- processor.setWantedAdminLevel(wantedAdminLevel);
-
- boolean done = false;
- while (!done){
- done = processMap(processor);
- }
- System.out.println("Problem-list-generator pass " + (pass+1) + " took " + (System.currentTimeMillis() - startThisPass) + " ms");
- }
- //writeProblemList("problem-candidates-partition-" + partition + ".txt", problemWaysThisPart, problemRelsThisPart);
- calculatedProblemWays.addAll(problemWaysThisPart);
- calculatedProblemRels.addAll(problemRelsThisPart);
- System.out.println("Problem-list-generator pass(es) took " + (System.currentTimeMillis() - startProblemListGenerator) + " ms");
- if (distinctAreas.size() > realAreas.size()) {
- // correct wrong entries caused by partitioning
- for (Long id: calculatedProblemRels){
- oneTileOnlyRels.remove(id);
- }
- }
- if (problemReport != null){
- writeProblemList(problemReport,
- calculatedProblemWays,
- calculatedProblemRels);
- }
- return distinctAreas;
+ private List<Area> calculateAreas() throws XmlPullParserException {
+
+ DensityMapCollector pass1Collector = new DensityMapCollector(resolution, ignoreBoundsTags);
+ MapProcessor processor = pass1Collector;
+
+ File densityData = new File("densities.txt");
+ File densityOutData = null;
+ if (densityData.exists() && densityData.isFile()){
+ System.err.println("reading density data from " + densityData.getAbsolutePath());
+ pass1Collector.readMap(densityData.getAbsolutePath());
+ }
+ else {
+ densityOutData = new File(fileOutputDir,"densities-out.txt");
+ osmFileHandler.process(processor);
+ }
+ System.out.println("in " + fileNameList.size() + (fileNameList.size() == 1 ? " file" : " files"));
+ System.out.println("Time: " + new Date());
+ if (densityOutData != null )
+ pass1Collector.saveMap(densityOutData.getAbsolutePath());
+
+ Area exactArea = pass1Collector.getExactArea();
+ System.out.println("Exact map coverage read from input file(s) is " + exactArea);
+ if (areasCalculator.getPolygons().size() == 1){
+ Rectangle polgonsBoundingBox = areasCalculator.getPolygons().get(0).area.getBounds();
+ exactArea = Area.calcArea(exactArea, polgonsBoundingBox);
+ if (exactArea != null)
+ System.out.println("Exact map coverage after applying bounding box of polygon-file is " + exactArea);
+ else {
+ System.out.println("Exact map coverage after applying bounding box of polygon-file is an empty area" );
+ return Collections.emptyList();
+ }
+ }
+
+ if (precompSeaDir != null){
+ System.out.println("Counting nodes of precompiled sea data ...");
+ long startSea = System.currentTimeMillis();
+ DensityMapCollector seaCollector = new DensityMapCollector(resolution, true);
+ PrecompSeaReader precompSeaReader = new PrecompSeaReader(exactArea, new File(precompSeaDir));
+ precompSeaReader.processMap(seaCollector);
+ pass1Collector.mergeSeaData(seaCollector, trim, resolution);
+ System.out.println("Precompiled sea data pass took " + (System.currentTimeMillis()-startSea) + " ms");
+ }
+ Area roundedBounds = RoundingUtils.round(exactArea, resolution);
+ SplittableDensityArea splittableArea = pass1Collector.getSplitArea(searchLimit, roundedBounds);
+ if (splittableArea.hasData() == false){
+ System.out.println("input file(s) have no data inside calculated bounding box");
+ return Collections.emptyList();
+ }
+ System.out.println("Rounded map coverage is " + splittableArea.getBounds());
+
+ splittableArea.setTrim(trim);
+ splittableArea.setMapId(mapId);
+ long startSplit = System.currentTimeMillis();
+ List<Area> areas ;
+ if (numTiles >= 2){
+ System.out.println("Splitting nodes into " + numTiles + " areas");
+ areas = splittableArea.split(numTiles);
+ }
+ else {
+ System.out.println("Splitting nodes into areas containing a maximum of " + Utils.format(maxNodes) + " nodes each...");
+ splittableArea.setMaxNodes(maxNodes);
+ areas = splittableArea.split(areasCalculator.getPolygons());
+ }
+ if (areas != null && areas.isEmpty() == false)
+ System.out.println("Creating the initial areas took " + (System.currentTimeMillis()- startSplit) + " ms");
+ return areas;
}
- /**
- * Final pass(es), we have the areas so parse the file(s) again.
- *
- * @param areas Area list determined on the first pass.
- * @param distinctAreas
- */
- private void writeAreas(List<Area> areas, List<Area> distinctAreas) throws IOException, XmlPullParserException {
+ private OSMWriter[] createWriters(List<Area> areas) {
OSMWriter[] allWriters = new OSMWriter[areas.size()];
- Map<String, byte[]> wellKnownTagKeys = null;
- Map<String, byte[]> wellKnownTagVals = null;
- if ("o5m".equals(outputType)){
- wellKnownTagKeys = new HashMap<>();
- wellKnownTagVals = new HashMap<>();
-
- String[] tagKeys = { "1", "1outer", "1inner", "type", // relation specific
- // 50 most often used keys (taken from taginfo 2014-05-19)
- "source", "building",
- "highway", "name", "addr:housenumber", "addr:street",
- "addr:city", "addr:postcode", /*"created_by", */"addr:country",
- "natural", "source:date", "tiger:cfcc", "tiger:county",
- "tiger:reviewed", "landuse", "waterway", "wall", "surface",
- "attribution", "power", "tiger:source", "tiger:tlid",
- "tiger:name_base", "oneway", "amenity", "start_date",
- "tiger:name_type", "ref:bag", "tiger:upload_uuid",
- "tiger:separated", "ref", "yh:WIDTH", "tiger:zip_left",
- "note", "source_ref", "tiger:zip_right", "access",
- "yh:STRUCTURE", "yh:TYPE", "yh:TOTYUMONO", "yh:WIDTH_RANK",
- "maxspeed", "lanes", "service", "barrier", "source:addr",
- "tracktype", "is_in", "layer" , "place"};
-
- for (String s:tagKeys){
- wellKnownTagKeys.put(s, s.getBytes("UTF-8"));
- }
-
- String[] tagVals = { "yes", "no", "residential", "water", "tower",
- "footway", "Bing", "PGS", "private", "stream", "service",
- "house", "unclassified", "track", "traffic_signals","restaurant","entrance"};
-
- for (String s:tagVals){
- wellKnownTagVals.put(s, s.getBytes("UTF-8"));
- }
-
- }
-
for (int j = 0; j < allWriters.length; j++) {
Area area = areas.get(j);
AbstractOSMWriter w;
if ("pbf".equals(outputType))
- w = new BinaryMapWriter(area, fileOutputDir, area.getMapId(), overlapAmount );
+ w = new BinaryMapWriter(area, fileOutputDir, area.getMapId(), overlapAmount);
else if ("o5m".equals(outputType))
- w = new O5mMapWriter(area, fileOutputDir, area.getMapId(), overlapAmount, wellKnownTagKeys,wellKnownTagVals);
+ w = new O5mMapWriter(area, fileOutputDir, area.getMapId(), overlapAmount);
else if ("simulate".equals(outputType))
- w = new PseudoOSMWriter(area, area.getMapId(), false, overlapAmount);
+ w = new PseudoOSMWriter(area);
else
- w = new OSMXMLWriter(area, fileOutputDir, area.getMapId(), overlapAmount );
+ w = new OSMXMLWriter(area, fileOutputDir, area.getMapId(), overlapAmount);
switch (handleElementVersion) {
case "keep":
w.setVersionMethod(AbstractOSMWriter.KEEP_VERSION);
@@ -861,52 +633,33 @@ public class Main {
}
allWriters[j] = w;
}
-
- int numPasses = getAreasPerPass(areas.size());
- int areasPerPass = (int) Math.ceil((double) areas.size() / (double) numPasses);
- DataStorer dataStorer = new DataStorer(allWriters);
- translateDistinctToRealAreas (dataStorer, distinctAreas, areas);
- // add the user given problem polygons
- problemWays.addAll(calculatedProblemWays);
- calculatedProblemWays = null;
- problemRels.addAll(calculatedProblemRels);
- calculatedProblemRels = null;
- if (problemWays.size() > 0 || problemRels.size() > 0){
- // calculate which ways and relations are written to multiple areas.
- MultiTileProcessor multiProcessor = new MultiTileProcessor(dataStorer, problemWays, problemRels);
- // return memory to GC
- problemRels = null;
- problemWays = null;
-
- boolean done = false;
- long startThisPhase = System.currentTimeMillis();
- int prevPhase = -1;
- while(!done){
- int phase = multiProcessor.getPhase();
- if (prevPhase != phase){
- startThisPhase = System.currentTimeMillis();
- System.out.println("-----------------------------------");
- System.out.println("Executing multi-tile analyses phase " + phase);
- }
- done = processMap(multiProcessor);
- prevPhase = phase;
- if (done || (phase != multiProcessor.getPhase())){
- System.out.println("Multi-tile analyses phase " + phase + " took " + (System.currentTimeMillis() - startThisPhase) + " ms");
- }
- }
-
- System.out.println("-----------------------------------");
- }
- if ("handle-problem-list".equals(stopAfter)){
- try {Thread.sleep(1000);}catch (InterruptedException e) {}
- System.err.println("stopped after " + stopAfter);
- throw new StopNoErrorException("stopped after " + stopAfter);
- }
-
- // the final split passes
+ return allWriters;
+ }
+
+ private void useProblemLists(DataStorer dataStorer) {
+ problemList.calcMultiTileElements(dataStorer, osmFileHandler);
+ if ("handle-problem-list".equals(stopAfter)){
+ try {Thread.sleep(1000);}catch (InterruptedException e) {}
+ System.err.println("stopped after " + stopAfter);
+ throw new StopNoErrorException("stopped after " + stopAfter);
+ }
+ }
+
+ /**
+ * Final pass(es), we have the areas so parse the file(s) again.
+ * @param dataStorer collects data used in different program passes
+ */
+ private void writeTiles(DataStorer dataStorer) throws IOException {
+ List<Area> areas = dataStorer.getAreaDictionary().getAreas();
+ // the final split passes,
dataStorer.switchToSeqAccess(fileOutputDir);
+ dataStorer.setWriters(createWriters(areas));
+
System.out.println("Distributing data " + new Date());
+ int numPasses = (int) Math.ceil((double) areas.size() / maxAreasPerPass);
+ int areasPerPass = (int) Math.ceil((double) areas.size() / numPasses);
+
long startDistPass = System.currentTimeMillis();
if (numPasses > 1) {
System.out.println("Processing " + areas.size() + " areas in " + numPasses + " passes, " + areasPerPass + " areas at a time");
@@ -914,499 +667,20 @@ public class Main {
System.out.println("Processing " + areas.size() + " areas in a single pass");
}
for (int i = 0; i < numPasses; i++) {
- int writerOffset = i * areasPerPass;
- int numWritersThisPass = Math.min(areasPerPass, areas.size() - i * areasPerPass);
+ int areaOffset = i * areasPerPass;
+ int numAreasThisPass = Math.min(areasPerPass, areas.size() - i * areasPerPass);
dataStorer.restartWriterMaps();
- SplitProcessor processor = new SplitProcessor(dataStorer, oneTileOnlyRels, writerOffset, numWritersThisPass, maxThreads);
+ SplitProcessor processor = new SplitProcessor(dataStorer, areaOffset, numAreasThisPass, maxThreads);
- System.out.println("Starting distribution pass " + (i + 1) + " of " + numPasses + ", processing " + numWritersThisPass +
+ System.out.println("Starting distribution pass " + (i + 1) + " of " + numPasses + ", processing " + numAreasThisPass +
" areas (" + areas.get(i * areasPerPass).getMapId() + " to " +
- areas.get(i * areasPerPass + numWritersThisPass - 1).getMapId() + ')');
+ areas.get(i * areasPerPass + numAreasThisPass - 1).getMapId() + ')');
- processMap(processor);
+ osmFileHandler.process(processor);
}
System.out.println("Distribution pass(es) took " + (System.currentTimeMillis() - startDistPass) + " ms");
- dataStorer.finish();
-
- }
-
- /**
- * If the Bitset ids in oneTileOnlyRels were produced with a different set of
- * writers we have to translate the values
- * @param dataStorer DataStorer instance used by split processor
- * @param distinctAreas list of distinct (non-overlapping) areas
- * @param areas list of areas from split-file (or calculation)
- */
- private void translateDistinctToRealAreas(DataStorer dataStorer, List<Area> distinctAreas, List<Area> areas) {
- if (oneTileOnlyRels.isEmpty() || distinctAreas.size() == areas.size())
- return;
- Map<Area, Integer> map = new HashMap<>();
- for (Area distinctArea : distinctAreas) {
- if (distinctArea.getMapId() < 0 && !distinctArea.isPseudoArea()) {
- BitSet w = new BitSet();
- for (int i = 0; i < areas.size(); i++) {
- if (areas.get(i).contains(distinctArea)) {
- w.set(i);
- }
- }
- int id = dataStorer.getMultiTileWriterDictionary().translate(w);
- map.put(distinctArea, id);
- }
- }
- if (!map.isEmpty()) {
-
- for ( Entry<Long, Integer> e: oneTileOnlyRels.entrySet()) {
- if (e.getValue() >= 0) {
- e.setValue(map.get(distinctAreas.get(e.getValue())));
- }
- }
- }
- }
-
- private boolean processMap(MapProcessor processor) throws XmlPullParserException {
- boolean done = processOSMFiles(processor, fileNameList);
- return done;
}
- /** Read user defined problematic relations and ways */
- private boolean readProblemIds(String problemFileName) {
- File fProblem = new File(problemFileName);
- boolean ok = true;
-
- if (!fProblem.exists()) {
- System.out.println("Error: problem file doesn't exist: " + fProblem);
- return false;
- }
- try (InputStream fileStream = new FileInputStream(fProblem);
- LineNumberReader problemReader = new LineNumberReader(
- new InputStreamReader(fileStream));) {
- Pattern csvSplitter = Pattern.compile(Pattern.quote(":"));
- Pattern commentSplitter = Pattern.compile(Pattern.quote("#"));
- String problemLine;
- String[] items;
- while ((problemLine = problemReader.readLine()) != null) {
- items = commentSplitter.split(problemLine);
- if (items.length == 0 || items[0].trim().isEmpty()){
- // comment or empty line
- continue;
- }
- items = csvSplitter.split(items[0].trim());
- if (items.length != 2) {
- System.out.println("Error: Invalid format in problem file, line number " + problemReader.getLineNumber() + ": "
- + problemLine);
- ok = false;
- continue;
- }
- long id = 0;
- try{
- id = Long.parseLong(items[1]);
- }
- catch(NumberFormatException exp){
- System.out.println("Error: Invalid number format in problem file, line number " + + problemReader.getLineNumber() + ": "
- + problemLine + exp);
- ok = false;
- }
- if ("way".equals(items[0]))
- problemWays.add(id);
- else if ("rel".equals(items[0]))
- problemRels.add(id);
- else {
- System.out.println("Error in problem file: Type not way or relation, line number " + + problemReader.getLineNumber() + ": "
- + problemLine);
- ok = false;
- }
- }
- } catch (IOException exp) {
- System.out.println("Error: Cannot read problem file " + fProblem +
- exp);
- return false;
- }
- return ok;
- }
-
- /**
- * Write a file that can be given to mkgmap that contains the correct arguments
- * for the split file pieces. You are encouraged to edit the file and so it
- * contains a template of all the arguments that you might want to use.
- * @param problemRelsThisPass
- * @param problemWaysThisPass
- */
- protected void writeProblemList(String fname, Set<Long> pWays, Set<Long> pRels) {
- try (PrintWriter w = new PrintWriter(new FileWriter(new File(
- fileOutputDir, fname)));) {
-
- w.println("#");
- w.println("# This file can be given to splitter using the --problem-file option");
- w.println("#");
- w.println("# List of relations and ways that are known to cause problems");
- w.println("# in splitter or mkgmap");
- w.println("# Objects listed here are specially treated by splitter to assure");
- w.println("# that complete data is written to all related tiles");
- w.println("# Format:");
- w.println("# way:<id>");
- w.println("# rel:<id>");
- w.println("# ways");
- for (long id: pWays){
- w.println("way: " + id + " #");
- }
- w.println("# rels");
- for (long id: pRels){
- w.println("rel: " + id + " #");
- }
-
- w.println();
- } catch (IOException e) {
- System.err.println("Warning: Could not write problem-list file " + fname + ", processing continues");
- }
- }
-
- /**
- * Make sure that our writer areas cover the planet. This is done by adding
- * pseudo-writers.
- * @param realAreas
- * @return
- */
- private static List<Area> addPseudoWriters(List<Area> realAreas){
- ArrayList<Area> areas = new ArrayList<>(realAreas);
- Rectangle planetBounds = new Rectangle(Utils.toMapUnit(-180.0), Utils.toMapUnit(-90.0), 2* Utils.toMapUnit(180.0), 2 * Utils.toMapUnit(90.0));
-
- while (!checkIfCovered(planetBounds, areas)){
- boolean changed = addPseudoArea(areas);
-
- if (!changed){
- throw new SplitFailedException("Failed to fill planet with pseudo-areas");
- }
- }
- return areas;
- }
- /**
- * Work around for possible rounding errors in area.subtract processing
- * @param area an area that is considered to be empty or a rectangle
- * @return
- */
- private static java.awt.geom.Area simplifyArea(java.awt.geom.Area area) {
- if (area.isEmpty() || area.isRectangular())
- return area;
- // area.isRectugular() may returns false although the shape is a
- // perfect rectangle :-( If we subtract the area from its bounding
- // box we get better results.
- java.awt.geom.Area bbox = new java.awt.geom.Area (area.getBounds2D());
- bbox.subtract(area);
- if (bbox.isEmpty()) // bbox equals area: is a rectangle
- return new java.awt.geom.Area (area.getBounds2D());
- return area;
- }
-
- private static boolean checkIfCovered(Rectangle bounds, ArrayList<Area> areas){
- java.awt.geom.Area bbox = new java.awt.geom.Area(bounds);
- long sumTiles = 0;
-
- for (Area area: areas){
- sumTiles += (long)area.getHeight() * (long)area.getWidth();
- bbox.subtract(area.getJavaArea());
- }
- long areaBox = (long) bounds.height*(long)bounds.width;
-
- if (sumTiles != areaBox)
- return false;
-
- return bbox.isEmpty();
- }
-
- /**
- * Create a list of areas that do not overlap. If areas in the original
- * list are overlapping, they can be replaced by up to 5 disjoint areas.
- * This is done if parameter makeDisjoint is true
- * @param realAreas the list of areas
- * @return the new list
- */
- private static ArrayList<Area> getNonOverlappingAreas(final List<Area> realAreas){
- java.awt.geom.Area covered = new java.awt.geom.Area();
- ArrayList<Area> splitList = new ArrayList<>();
- int artificialId = -99999999;
- boolean foundOverlap = false;
- for (Area area1 : realAreas) {
- Rectangle r1 = area1.getRect();
- if (covered.intersects(r1) == false){
- splitList.add(area1);
- }
- else {
- if (foundOverlap == false){
- foundOverlap = true;
- System.out.println("Removing overlaps from tiles...");
- }
- //String msg = "splitting " + area1.getMapId() + " " + (i+1) + "/" + realAreas.size() + " overlapping ";
- // find intersecting areas in the already covered part
- ArrayList<Area> splitAreas = new ArrayList<>();
-
- for (int j = 0; j < splitList.size(); j++){
- Area area2 = splitList.get(j);
- if (area2 == null)
- continue;
- Rectangle r2 = area2.getRect();
- if (r1.intersects(r2)){
- java.awt.geom.Area overlap = new java.awt.geom.Area(area1.getRect());
- overlap.intersect(area2.getJavaArea());
- Rectangle ro = overlap.getBounds();
- if (ro.height == 0 || ro.width == 0)
- continue;
- //msg += area2.getMapId() + " ";
- Area aNew = new Area(ro.y, ro.x, (int)ro.getMaxY(),(int)ro.getMaxX());
- aNew.setMapId(artificialId++);
- aNew.setName("" + area1.getMapId());
- aNew.setJoinable(false);
- covered.subtract(area2.getJavaArea());
- covered.add(overlap);
- splitList.set(j, aNew);
-
- java.awt.geom.Area coveredByPair = new java.awt.geom.Area(r1);
- coveredByPair.add(new java.awt.geom.Area(r2));
-
- java.awt.geom.Area originalPair = new java.awt.geom.Area(coveredByPair);
-
- int minX = coveredByPair.getBounds().x;
- int minY = coveredByPair.getBounds().y;
- int maxX = (int) coveredByPair.getBounds().getMaxX();
- int maxY = (int) coveredByPair.getBounds().getMaxY();
- coveredByPair.subtract(overlap);
- if (coveredByPair.isEmpty())
- continue; // two equal areas a
-
- coveredByPair.subtract(covered);
- java.awt.geom.Area testSplit = new java.awt.geom.Area(overlap);
-
- Rectangle[] rectPair = {r1,r2};
- Area[] areaPair = {area1,area2};
- int lx = minX;
- int lw = ro.x-minX;
- int rx = (int)ro.getMaxX();
- int rw = maxX - rx;
- int uy = (int)ro.getMaxY();
- int uh = maxY - uy;
- int by = minY;
- int bh = ro.y - by;
- Rectangle[] clippers = {
- new Rectangle(lx, minY, lw, bh), // lower left
- new Rectangle(ro.x, minY, ro.width, bh), // lower middle
- new Rectangle(rx, minY, rw, bh), // lower right
- new Rectangle(lx, ro.y, lw, ro.height), // left
- new Rectangle(rx, ro.y, rw, ro.height), // right
- new Rectangle(lx, uy, lw, uh), // upper left
- new Rectangle(ro.x, uy, ro.width, uh), // upper middle
- new Rectangle(rx, uy, rw, uh) // upper right
- };
-
- for (Rectangle clipper: clippers){
- for (int k = 0; k <= 1; k++){
- Rectangle test = clipper.intersection(rectPair[k]);
- if (!test.isEmpty()){
- testSplit.add(new java.awt.geom.Area(test));
- if (k==1 || covered.intersects(test) == false){
- aNew = new Area(test.y,test.x,(int)test.getMaxY(),(int)test.getMaxX());
- aNew.setMapId(areaPair[k].getMapId());
- splitAreas.add(aNew);
- covered.add(aNew.getJavaArea());
- }
- }
- }
- }
- assert testSplit.equals(originalPair);
- }
- }
-
- // recombine parts that form a rectangle
- for (Area splitArea: splitAreas){
- if (splitArea.isJoinable()){
- for (int j = 0; j < splitList.size(); j++){
- Area area = splitList.get(j);
- if (area == null || area.isJoinable() == false || area.getMapId() != splitArea.getMapId() )
- continue;
- boolean doJoin = false;
- if (splitArea.getMaxLat() == area.getMaxLat()
- && splitArea.getMinLat() == area.getMinLat()
- && (splitArea.getMinLong() == area.getMaxLong() || splitArea.getMaxLong() == area.getMinLong()))
- doJoin = true;
- else if (splitArea.getMinLong() == area.getMinLong()
- && splitArea.getMaxLong()== area.getMaxLong()
- && (splitArea.getMinLat() == area.getMaxLat() || splitArea.getMaxLat() == area.getMinLat()))
- doJoin = true;
- if (doJoin){
- splitArea = area.add(splitArea);
- splitArea.setMapId(area.getMapId());
- splitList.set(j, splitArea);
- splitArea = null; // don't add later
- break;
- }
- }
- }
- if (splitArea != null){
- splitList.add(splitArea);
- }
- }
- /*
- if (msg.isEmpty() == false)
- System.out.println(msg);
- */
- }
- covered.add(new java.awt.geom.Area(r1));
- }
- covered.reset();
- Iterator <Area> iter = splitList.iterator();
- while (iter.hasNext()){
- Area a = iter.next();
- if (a == null)
- iter.remove();
- else {
- Rectangle r1 = a.getRect();
- if (covered.intersects(r1) == true){
- throw new SplitFailedException("Failed to create list of distinct areas");
- }
- covered.add(a.getJavaArea());
- }
- }
- return splitList;
- }
-
- /**
- * Fill uncovered parts of the planet with pseudo-areas.
- * TODO: check if better algorithm reduces run time in ProblemListProcessor
- * We want a small number of pseudo areas because many of them will
- * require more memory or more passes, esp. when processing whole planet.
- * Also, the total length of all edges should be small.
- * @param areas list of areas (either real or pseudo)
- * @return true if pseudo-areas were added
- */
- private static boolean addPseudoArea(ArrayList<Area> areas) {
- int oldSize = areas.size();
- Rectangle planetBounds = new Rectangle(Utils.toMapUnit(-180.0), Utils.toMapUnit(-90.0), 2* Utils.toMapUnit(180.0), 2 * Utils.toMapUnit(90.0));
- java.awt.geom.Area uncovered = new java.awt.geom.Area(planetBounds);
- java.awt.geom.Area covered = new java.awt.geom.Area();
- for (Area area: areas){
- uncovered.subtract(area.getJavaArea());
- covered.add(area.getJavaArea());
- }
- Rectangle rCov = covered.getBounds();
- Rectangle[] topAndBottom = {
- new Rectangle(planetBounds.x,(int)rCov.getMaxY(),planetBounds.width, (int)(planetBounds.getMaxY()-rCov.getMaxY())), // top
- new Rectangle(planetBounds.x,planetBounds.y,planetBounds.width,rCov.y-planetBounds.y)}; // bottom
- for (Rectangle border: topAndBottom){
- if (!border.isEmpty()){
- uncovered.subtract(new java.awt.geom.Area(border));
- covered.add(new java.awt.geom.Area(border));
- Area pseudo = new Area(border.y,border.x,(int)border.getMaxY(),(int)border.getMaxX());
- pseudo.setMapId(-1 * (areas.size()+1));
- pseudo.setPseudoArea(true);
- areas.add(pseudo);
- }
- }
- while (uncovered.isEmpty() == false){
- boolean changed = false;
- List<List<Point>> shapes = Utils.areaToShapes(uncovered);
- // we divide planet into stripes for all vertices of the uncovered area
- int minX = uncovered.getBounds().x;
- int nextX = Integer.MAX_VALUE;
- for (int i = 0; i < shapes.size(); i++){
- List<Point> shape = shapes.get(i);
- for (Point point: shape){
- int lon = point.x;
- if (lon < nextX && lon > minX)
- nextX = lon;
- }
- }
- java.awt.geom.Area stripeLon = new java.awt.geom.Area(new Rectangle(minX, planetBounds.y, nextX - minX, planetBounds.height));
- // cut out already covered area
- stripeLon.subtract(covered);
- assert stripeLon.isEmpty() == false;
- // the remaining area must be a set of zero or more disjoint rectangles
- List<List<Point>> stripeShapes = Utils.areaToShapes(stripeLon);
- for (int j = 0; j < stripeShapes .size(); j++){
- List<Point> rectShape = stripeShapes .get(j);
- java.awt.geom.Area test = Utils.shapeToArea(rectShape);
- test = simplifyArea(test);
- assert test.isRectangular();
- Rectangle pseudoRect = test.getBounds();
- if (uncovered.contains(pseudoRect)){
- assert test.getBounds().width == stripeLon.getBounds().width;
- boolean wasMerged = false;
- // check if new area can be merged with last rectangles
- for (int k=areas.size()-1; k >= oldSize; k--){
- Area prev = areas.get(k);
- if (prev.getMaxLong() < pseudoRect.x || prev.isPseudoArea() == false)
- continue;
- if (prev.getHeight() == pseudoRect.height && prev.getMaxLong() == pseudoRect.x && prev.getMinLat() == pseudoRect.y){
- // merge
- Area pseudo = prev.add(new Area(pseudoRect.y,pseudoRect.x,(int)pseudoRect.getMaxY(),(int)pseudoRect.getMaxX()));
- pseudo.setMapId(prev.getMapId());
- pseudo.setPseudoArea(true);
- areas.set(k, pseudo);
- //System.out.println("Enlarged pseudo area " + pseudo.getMapId() + " " + pseudo);
- wasMerged = true;
- break;
- }
- }
-
- if (!wasMerged){
- Area pseudo = new Area(pseudoRect.y, pseudoRect.x, (int)pseudoRect.getMaxY(), (int)pseudoRect.getMaxX());
- pseudo.setMapId(-1 * (areas.size()+1));
- pseudo.setPseudoArea(true);
- //System.out.println("Adding pseudo area " + pseudo.getMapId() + " " + pseudo);
- areas.add(pseudo);
- }
- uncovered.subtract(test);
- covered.add(test);
- changed = true;
- }
- }
- if (!changed)
- break;
- }
- return oldSize != areas.size();
- }
-
- /**
- * Check if the bounding polygons are usable.
- * @param polygon
- * @return
- */
- private boolean checkPolygons() {
- for (PolygonDesc pd : polygons){
- if (checkPolygon(pd.area) == false)
- return false;
- }
- return true;
- }
-
-
- /**
- * Check if the bounding polygon is usable.
- * @param polygon
- * @return
- */
- private boolean checkPolygon(java.awt.geom.Area mapPolygonArea) {
- List<List<Point>> shapes = Utils.areaToShapes(mapPolygonArea);
- int shift = 24 - resolution;
- long rectangleWidth = 1L << shift;
- for (List<Point> shape: shapes){
- int estimatedPoints = 0;
- Point p1 = shape.get(0);
- for (int i = 1; i < shape.size(); i++){
- Point p2 = shape.get(i);
- if (p1.x != p2.x && p1.y != p2.y){
- // diagonal line
- int width = Math.abs(p1.x-p2.x);
- int height = Math.abs(p1.y-p2.y);
- estimatedPoints += (Math.min(width, height) / rectangleWidth) * 2;
- }
-
- if (estimatedPoints > SplittableDensityArea.MAX_SINGLE_POLYGON_VERTICES)
- return false; // too complex
-
- p1 = p2;
- }
- }
- return true;
- }
-
static boolean testAndReportFname(String fileName, String type){
File f = new File(fileName);
if (f.exists() == false || f.isFile() == false || f.canRead() == false){
@@ -1417,70 +691,6 @@ public class Main {
}
return true;
}
-
- private boolean processOSMFiles(MapProcessor processor, List<String> filenames) throws XmlPullParserException {
- // Create both an XML reader and a binary reader, Dispatch each input to the
- // Appropriate parser.
- OSMParser parser = new OSMParser(processor, mixed);
-
- for (int i = 0; i < filenames.size(); i++){
- String filename = filenames.get(i);
- System.out.println("Processing " + filename);
- if (i == 1 && processor instanceof DensityMapCollector){
- ((DensityMapCollector) processor).checkBounds();
- }
-
- try {
- if (filename.endsWith(".o5m")) {
- File file = new File(filename);
- try(InputStream stream = new FileInputStream(file)){
- long[] skipArray = skipArrayMap.get(filename);
- O5mMapParser o5mParser = new O5mMapParser(processor, stream, skipArray);
- o5mParser.parse();
- if (skipArray == null){
- skipArray = o5mParser.getSkipArray();
- skipArrayMap.put(filename, skipArray);
- }
- }
- }
- else if (filename.endsWith(".pbf")) {
- // Is it a binary file?
- File file = new File(filename);
- ShortArrayList blockTypes = blockTypeMap.get(filename);
- BinaryMapParser binParser = new BinaryMapParser(processor, blockTypes, 1);
- try(InputStream stream = new FileInputStream(file)){
- BlockInputStream blockinput = (new BlockInputStream(stream, binParser));
- blockinput.process();
- if (blockTypes == null){
- // remember this file
- blockTypes = binParser.getBlockList();
- blockTypeMap.put(filename, blockTypes);
- }
- }
- } else {
- // No, try XML.
- try (Reader reader = Utils.openFile(filename, maxThreads > 1)){
- parser.setReader(reader);
- parser.parse();
- }
- }
- } catch (FileNotFoundException e) {
- System.out.println(e);
- throw new SplitFailedException("ERROR: file " + filename + " was not found");
- } catch (XmlPullParserException e) {
- e.printStackTrace();
- throw new SplitFailedException("ERROR: file " + filename + " is not a valid OSM XML file");
- } catch (IllegalArgumentException e) {
- e.printStackTrace();
- throw new SplitFailedException("ERROR: file " + filename + " contains unexpected data");
- } catch (IOException e) {
- e.printStackTrace();
- throw new SplitFailedException("ERROR: file " + filename + " caused I/O exception");
- }
- }
- boolean done = processor.endMap();
- return done;
- }
}
diff --git a/src/uk/me/parabola/splitter/MultiTileProcessor.java b/src/uk/me/parabola/splitter/MultiTileProcessor.java
index a526b31..02dee4c 100644
--- a/src/uk/me/parabola/splitter/MultiTileProcessor.java
+++ b/src/uk/me/parabola/splitter/MultiTileProcessor.java
@@ -48,7 +48,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
private int phase = PHASE1_RELS_ONLY;
private final DataStorer dataStorer;
- private final WriterDictionaryInt multiTileDictionary;
+ private final AreaDictionaryInt multiTileDictionary;
private Long2ObjectLinkedOpenHashMap<MTRelation> relMap = new Long2ObjectLinkedOpenHashMap<>();
private Long2IntClosedMapFunction nodeWriterMap;
private Long2IntClosedMapFunction wayWriterMap;
@@ -61,6 +61,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
private OSMId2ObjectMap<Rectangle> wayBboxMap;
private SparseBitSet mpWays = new SparseBitSet();
private OSMId2ObjectMap<JoinedWay> mpWayEndNodesMap;
+ /** each bit represents one area/tile */
private final BitSet workWriterSet;
private long lastCoordId = Long.MIN_VALUE;
private int foundWays;
@@ -72,7 +73,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
MultiTileProcessor(DataStorer dataStorer, LongArrayList problemWayList, LongArrayList problemRelList) {
this.dataStorer = dataStorer;
- multiTileDictionary = dataStorer.getMultiTileWriterDictionary();
+ multiTileDictionary = dataStorer.getMultiTileDictionary();
for (long id: problemWayList){
neededWays.set(id);
}
@@ -165,7 +166,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
}
int wayWriterIdx;
if (workWriterSet.isEmpty())
- wayWriterIdx = WriterDictionaryInt.UNASSIGNED;
+ wayWriterIdx = AreaDictionaryInt.UNASSIGNED;
else
wayWriterIdx = multiTileDictionary.translate(workWriterSet);
@@ -182,7 +183,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
if (!neededWays.get(way.getId()))
return;
int wayWriterIdx = wayWriterMap.getRandom(way.getId());
- if (wayWriterIdx != WriterDictionaryInt.UNASSIGNED){
+ if (wayWriterIdx != AreaDictionaryInt.UNASSIGNED){
BitSet wayWriterSet = multiTileDictionary.getBitSet(wayWriterIdx);
for (long id : way.getRefs()) {
addOrMergeWriters(nodeWriterMap, wayWriterSet, wayWriterIdx, id);
@@ -239,8 +240,8 @@ class MultiTileProcessor extends AbstractMapProcessor {
stats("Finished collecting problem ways.");
neededNodesCount = neededNodes.cardinality();
// critical part: we have to allocate possibly large arrays here
- nodeWriterMap = new Long2IntClosedMap("node", neededNodesCount, WriterDictionaryInt.UNASSIGNED);
- wayWriterMap = new Long2IntClosedMap("way", foundWays, WriterDictionaryInt.UNASSIGNED);
+ nodeWriterMap = new Long2IntClosedMap("node", neededNodesCount, AreaDictionaryInt.UNASSIGNED);
+ wayWriterMap = new Long2IntClosedMap("way", foundWays, AreaDictionaryInt.UNASSIGNED);
wayBboxMap = new OSMId2ObjectMap<>();
dataStorer.setWriterMap(DataStorer.NODE_TYPE, nodeWriterMap);
dataStorer.setWriterMap(DataStorer.WAY_TYPE, wayWriterMap);
@@ -269,11 +270,11 @@ class MultiTileProcessor extends AbstractMapProcessor {
propagateWritersOfRelsToMembers();
wayBboxMap = null;
- relWriterMap = new Long2IntClosedMap("rel", relMap.size(), WriterDictionaryInt.UNASSIGNED);
+ relWriterMap = new Long2IntClosedMap("rel", relMap.size(), AreaDictionaryInt.UNASSIGNED);
for (Entry<MTRelation> entry : relMap.long2ObjectEntrySet()){
int val = entry.getValue().getMultiTileWriterIndex();
- if (val != WriterDictionaryInt.UNASSIGNED){
+ if (val != AreaDictionaryInt.UNASSIGNED){
try{
relWriterMap.add(entry.getLongKey(), val);
}catch (IllegalArgumentException e){
@@ -400,7 +401,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
}
else if (rel.memTypes[i] == MEM_WAY_TYPE){
int idx = wayWriterMap.getRandom(memId);
- if (idx != WriterDictionaryInt.UNASSIGNED){
+ if (idx != AreaDictionaryInt.UNASSIGNED){
writerSet.or(multiTileDictionary.getBitSet(idx));
memFound = true;
}
@@ -442,7 +443,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
int writerIdx = multiTileDictionary.translate(relWriters);
rel.setMultiTileWriterIndex(writerIdx);
int touchedTiles = relWriters.cardinality();
- if (touchedTiles > dataStorer.getNumOfWriters() / 2 && dataStorer.getNumOfWriters() > 10){
+ if (touchedTiles > dataStorer.getNumOfAreas() / 2 && dataStorer.getNumOfAreas() > 10){
System.out.println("Warning: rel " + rel.getId() + " touches " + touchedTiles + " tiles.");
}
}
@@ -472,7 +473,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
if (rel.wasAddedAsParent())
continue;
int relWriterIdx = rel.getMultiTileWriterIndex();
- if (relWriterIdx == WriterDictionaryInt.UNASSIGNED)
+ if (relWriterIdx == AreaDictionaryInt.UNASSIGNED)
continue;
BitSet relWriters = multiTileDictionary.getBitSet(relWriterIdx);
for (int i = 0; i < rel.numMembers; i++){
@@ -504,7 +505,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
}
int nodePos = -1;
try{
- nodePos = nodeWriterMap.add(id, WriterDictionaryInt.UNASSIGNED);
+ nodePos = nodeWriterMap.add(id, AreaDictionaryInt.UNASSIGNED);
}catch (IllegalArgumentException e){
System.err.println(e.getMessage());
throw new SplitFailedException(NOT_SORTED_MSG);
@@ -533,7 +534,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
}
BitSet relWriters = new BitSet();
int relWriterIdx = rel.getMultiTileWriterIndex();
- if (relWriterIdx != WriterDictionaryInt.UNASSIGNED)
+ if (relWriterIdx != AreaDictionaryInt.UNASSIGNED)
relWriters.or(multiTileDictionary.getBitSet(relWriterIdx));
boolean changed = false;
@@ -550,7 +551,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
orSubRelWriters(subRel, depth+1, visited);
visited.remove(visited.size()-1);
int memWriterIdx = subRel.getMultiTileWriterIndex();
- if (memWriterIdx == WriterDictionaryInt.UNASSIGNED || memWriterIdx == relWriterIdx){
+ if (memWriterIdx == AreaDictionaryInt.UNASSIGNED || memWriterIdx == relWriterIdx){
continue;
}
BitSet memWriters = multiTileDictionary.getBitSet(memWriterIdx);
@@ -603,9 +604,8 @@ class MultiTileProcessor extends AbstractMapProcessor {
private boolean checkBoundingBox(BitSet writerSet, Rectangle polygonBbox){
boolean foundIntersection = false;
if (polygonBbox != null){
- OSMWriter[] writers = dataStorer.getWriterDictionary().getWriters();
- for (int i = 0; i < writers.length; i++) {
- Rectangle writerBbox = writers[i].getBBox();
+ for (int i = 0; i < dataStorer.getNumOfAreas(); i++) {
+ Rectangle writerBbox = Utils.area2Rectangle(dataStorer.getArea(i), 1);
if (writerBbox.intersects(polygonBbox)){
writerSet.set(i);
foundIntersection = true;
@@ -628,7 +628,7 @@ class MultiTileProcessor extends AbstractMapProcessor {
if (pos < 0)
return;
int childWriterIdx = map.getRandom(childId);
- if (childWriterIdx != WriterDictionaryInt.UNASSIGNED){
+ if (childWriterIdx != AreaDictionaryInt.UNASSIGNED){
// we have already calculated writers for this child
if (parentWriterIdx == childWriterIdx)
return;
@@ -653,16 +653,15 @@ class MultiTileProcessor extends AbstractMapProcessor {
* @return true if a writer was found
*/
private boolean addWritersOfPoint(BitSet writerSet, int mapLat, int mapLon){
- WriterGridResult writerCandidates = dataStorer.getGrid().get(mapLat,mapLon);
+ AreaGridResult writerCandidates = dataStorer.getGrid().get(mapLat,mapLon);
if (writerCandidates == null)
return false;
- OSMWriter[] writers = dataStorer.getWriterDictionary().getWriters();
boolean foundWriter = false;
for (int i = 0; i < writerCandidates.l.size(); i++) {
int n = writerCandidates.l.getShort(i);
- OSMWriter w = writers[n];
- boolean found = (writerCandidates.testNeeded) ? w.coordsBelongToThisArea(mapLat, mapLon) : true;
+ Area extbbox = dataStorer.getExtendedArea(n);
+ boolean found = (writerCandidates.testNeeded) ? extbbox.contains(mapLat, mapLon) : true;
foundWriter |= found;
if (found)
writerSet.set(n);
@@ -678,10 +677,8 @@ class MultiTileProcessor extends AbstractMapProcessor {
* @param p2 second point of line
*/
private void addWritersOfCrossedTiles(BitSet writerSet, final BitSet possibleWriters, final Point p1,final Point p2){
- OSMWriter[] writers = dataStorer.getWriterDictionary().getWriters();
-
for (int i = possibleWriters.nextSetBit(0); i >= 0; i = possibleWriters.nextSetBit(i+1)){
- Rectangle writerBbox = writers[i].getBBox();
+ Rectangle writerBbox = Utils.area2Rectangle(dataStorer.getArea(i), 1);
if (writerBbox.intersectsLine(p1.x,p1.y,p2.x,p2.y))
writerSet.set(i);
}
@@ -718,8 +715,8 @@ class MultiTileProcessor extends AbstractMapProcessor {
if (numWriters == 0)
needsCrossTileCheck = true;
else if (numWriters > 1){
- short idx = dataStorer.getWriterDictionary().translate(writerSet);
- if (dataStorer.getWriterDictionary().mayCross(idx))
+ short idx = dataStorer.getAreaDictionary().translate(writerSet);
+ if (dataStorer.getAreaDictionary().mayCross(idx))
needsCrossTileCheck = true;
}
}
diff --git a/src/uk/me/parabola/splitter/O5mMapWriter.java b/src/uk/me/parabola/splitter/O5mMapWriter.java
index e684fde..faac821 100644
--- a/src/uk/me/parabola/splitter/O5mMapWriter.java
+++ b/src/uk/me/parabola/splitter/O5mMapWriter.java
@@ -22,6 +22,7 @@ import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Arrays;
+import java.util.HashMap;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;
@@ -56,8 +57,6 @@ public class O5mMapWriter extends AbstractOSMWriter{
private static final double FACTOR = 10000000;
private DataOutputStream dos;
- private Map<String, byte[]> wellKnownTagKeys;
- private Map<String, byte[]> wellKnownTagVals;
private byte[][][] stw__tab; // string table
private byte[] s1Bytes;
@@ -87,13 +86,49 @@ public class O5mMapWriter extends AbstractOSMWriter{
private short[] stw__tabhash;
private byte[] numberConversionBuf;
+
+ final static Map<String, byte[]> wellKnownTagKeys = new HashMap<>();
+ final static Map<String, byte[]> wellKnownTagVals = new HashMap<>();
+ final static String[] tagKeys = { "1", "1outer", "1inner", "type", // relation specific
+ // 50 most often used keys (taken from taginfo 2016-11-20)
+ "building", "source",
+ "highway", "addr:housenumber", "addr:street", "name",
+ "addr:city", "addr:postcode", "natural", "source:date", "addr:country",
+ "landuse", "surface", "created_by", "power",
+ "tiger:cfcc", "waterway", "tiger:county",
+ "start_date", "tiger:reviewed", "wall",
+ "amenity", "oneway", "ref:bag", "ref",
+ "attribution", "tiger:name_base", "building:levels",
+ "maxspeed", "barrier", "tiger:name_type", "height",
+ "service", "source:addr", "tiger:tlid", "tiger:source",
+ "lanes", "access", "addr:place", "tiger:zip_left",
+ "tiger:upload_uuid", "layer", "tracktype",
+ "ele", "tiger:separated", "tiger:zip_right",
+ "yh:WIDTH", "place", "foot"
+ };
+ final static String[] tagVals = { "yes", "no", "residential", "garage", "water", "tower",
+ "footway", "Bing", "PGS", "private", "stream", "service",
+ "house", "unclassified", "track", "traffic_signals","restaurant","entrance"
+ };
+
+ static {
+ try {
+ for (String s : tagKeys) {
+ wellKnownTagKeys.put(s, s.getBytes("UTF-8"));
+ }
+
+ for (String s : tagVals) {
+ wellKnownTagVals.put(s, s.getBytes("UTF-8"));
+ }
+ } catch (Exception e) {
+ // should not happen
+ }
+ }
//private long countCollisions;
- public O5mMapWriter(Area bounds, File outputDir, int mapId, int extra, Map<String, byte[]> wellKnownTagKeys, Map<String, byte[]> wellKnownTagVals) {
+ public O5mMapWriter(Area bounds, File outputDir, int mapId, int extra) {
super(bounds, outputDir, mapId, extra);
- this.wellKnownTagKeys = wellKnownTagKeys;
- this.wellKnownTagVals= wellKnownTagVals;
}
private void reset() throws IOException{
diff --git a/src/uk/me/parabola/splitter/OSMFileHandler.java b/src/uk/me/parabola/splitter/OSMFileHandler.java
new file mode 100644
index 0000000..9acd86f
--- /dev/null
+++ b/src/uk/me/parabola/splitter/OSMFileHandler.java
@@ -0,0 +1,113 @@
+package uk.me.parabola.splitter;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Reader;
+import java.util.HashMap;
+import java.util.List;
+
+import org.xmlpull.v1.XmlPullParserException;
+
+import crosby.binary.file.BlockInputStream;
+import it.unimi.dsi.fastutil.shorts.ShortArrayList;
+
+/**
+ * A class which stores parameters needed to process input (OSM) files
+ *
+ * @author Gerd Petermann
+ *
+ */
+public class OSMFileHandler {
+ /** list of OSM input files to process */
+ private List<String> filenames;
+ // for faster access on blocks in pbf files
+ private final HashMap<String, ShortArrayList> blockTypeMap = new HashMap<>();
+ // for faster access on blocks in o5m files
+ private final HashMap<String, long[]> skipArrayMap = new HashMap<>();
+
+ // Whether or not the source OSM file(s) contain strictly nodes first, then ways, then rels,
+ // or they're all mixed up. Running with mixed enabled takes longer.
+ private boolean mixed;
+
+ private int maxThreads = 1;
+
+ public void setFileNames (List<String> filenames) {
+ this.filenames = filenames;
+ }
+
+ public void setMixed(boolean f) {
+ mixed = f;
+ }
+
+ public void setMaxThreads (int maxThreads) {
+ this.maxThreads = maxThreads;
+ }
+
+ public boolean process(MapProcessor processor) {
+ // Create both an XML reader and a binary reader, Dispatch each input to the
+ // Appropriate parser.
+
+ for (int i = 0; i < filenames.size(); i++){
+ String filename = filenames.get(i);
+ System.out.println("Processing " + filename);
+ if (i == 1 && processor instanceof DensityMapCollector){
+ ((DensityMapCollector) processor).checkBounds();
+ }
+
+ try {
+ if (filename.endsWith(".o5m")) {
+ File file = new File(filename);
+ try(InputStream stream = new FileInputStream(file)){
+ long[] skipArray = skipArrayMap.get(filename);
+ O5mMapParser o5mParser = new O5mMapParser(processor, stream, skipArray);
+ o5mParser.parse();
+ if (skipArray == null){
+ skipArray = o5mParser.getSkipArray();
+ skipArrayMap.put(filename, skipArray);
+ }
+ }
+ }
+ else if (filename.endsWith(".pbf")) {
+ // Is it a binary file?
+ File file = new File(filename);
+ ShortArrayList blockTypes = blockTypeMap.get(filename);
+ BinaryMapParser binParser = new BinaryMapParser(processor, blockTypes, 1);
+ try(InputStream stream = new FileInputStream(file)){
+ BlockInputStream blockinput = (new BlockInputStream(stream, binParser));
+ blockinput.process();
+ if (blockTypes == null){
+ // remember this file
+ blockTypes = binParser.getBlockList();
+ blockTypeMap.put(filename, blockTypes);
+ }
+ }
+ } else {
+ // No, try XML.
+ try (Reader reader = Utils.openFile(filename, maxThreads > 1)){
+ OSMParser parser = new OSMParser(processor, mixed);
+ parser.setReader(reader);
+ parser.parse();
+ }
+ }
+ } catch (FileNotFoundException e) {
+ System.out.println(e);
+ throw new SplitFailedException("ERROR: file " + filename + " was not found");
+ } catch (XmlPullParserException e) {
+ e.printStackTrace();
+ throw new SplitFailedException("ERROR: file " + filename + " is not a valid OSM XML file");
+ } catch (IllegalArgumentException e) {
+ e.printStackTrace();
+ throw new SplitFailedException("ERROR: file " + filename + " contains unexpected data");
+ } catch (IOException e) {
+ e.printStackTrace();
+ throw new SplitFailedException("ERROR: file " + filename + " caused I/O exception");
+ }
+ }
+ boolean done = processor.endMap();
+ return done;
+ }
+
+}
diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java
index 4d050e4..8bf312f 100644
--- a/src/uk/me/parabola/splitter/OSMWriter.java
+++ b/src/uk/me/parabola/splitter/OSMWriter.java
@@ -41,18 +41,9 @@ public interface OSMWriter {
*/
public abstract void finishWrite();
- public boolean nodeBelongsToThisArea(Node node);
-
- public boolean coordsBelongToThisArea(int mapLat, int mapLon);
-
public abstract void write(Node node) throws IOException;
public abstract void write(Way way) throws IOException;
public abstract void write(Relation rel) throws IOException;
-
- /**
- * @return true if the area was added for the problem list generator
- */
- public boolean areaIsPseudo();
}
diff --git a/src/uk/me/parabola/splitter/PolygonDescProcessor.java b/src/uk/me/parabola/splitter/PolygonDescProcessor.java
index 0458fa0..d7f70d2 100644
--- a/src/uk/me/parabola/splitter/PolygonDescProcessor.java
+++ b/src/uk/me/parabola/splitter/PolygonDescProcessor.java
@@ -23,7 +23,7 @@ import java.util.List;
/**
*
- * Class to read a polygon description file.
+ * Class to read a polygon description file (OSM)
* Expected input are nodes and ways. Ways with
* tag name=* and mapid=nnnnnnnn should describe polygons
* which are used to calculate area lists.
@@ -95,17 +95,6 @@ class PolygonDescProcessor extends AbstractMapProcessor {
}
/**
- * @return the combined polygon
- */
- Area getCombinedPolygon(){
- Area combinedArea = new Area();
- for (PolygonDesc pd : polygonDescriptions){
- combinedArea.add(pd.area);
- }
- return combinedArea;
- }
-
- /**
* Calculate and write the area lists for each named polygon.
* @param fileOutputDir
* @param areas the list of all areas
@@ -121,7 +110,6 @@ class PolygonDescProcessor extends AbstractMapProcessor {
if (pd.area.intersects(a.getRect()))
areasPart.add(a);
}
- AreaList al = new AreaList(areasPart);
if (kmlOutputFile != null){
File out = new File(kmlOutputFile);
String kmlOutputFilePart = pd.name + "-" + out.getName();
@@ -131,9 +119,9 @@ class PolygonDescProcessor extends AbstractMapProcessor {
out = new File(kmlOutputFilePart);
if (out.getParent() == null)
out = new File(fileOutputDir, kmlOutputFilePart);
- System.out.println("Writing KML file to " + out.getPath());
- al.writeKml(out.getPath());
+ KmlWriter.writeKml(out.getPath(), areasPart);
}
+ AreaList al = new AreaList(areasPart, null);
al.writePoly(new File(fileOutputDir, pd.name + "-" + "areas.poly").getPath());
al.writeArgsFile(new File(fileOutputDir, pd.name + "-" + "template.args").getPath(), outputType, pd.mapId);
}
diff --git a/src/uk/me/parabola/splitter/ProblemListProcessor.java b/src/uk/me/parabola/splitter/ProblemListProcessor.java
index 11511a1..6ea1604 100644
--- a/src/uk/me/parabola/splitter/ProblemListProcessor.java
+++ b/src/uk/me/parabola/splitter/ProblemListProcessor.java
@@ -13,7 +13,6 @@
package uk.me.parabola.splitter;
import uk.me.parabola.splitter.Relation.Member;
-import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
import it.unimi.dsi.fastutil.longs.LongArrayList;
import java.util.Arrays;
@@ -24,49 +23,45 @@ import java.util.Iterator;
/**
* Find ways and relations that will be incomplete.
* Strategy:
- * - calculate the writers of each node, calculate and store a short that represents the combination of writers
- * (this is done by the WriterDictionary)
- * - a way is incomplete (in at least one tile) if its nodes are written to different combinations of writers
- * - a relation is incomplete (in at least one tile) if its members are written to different combinations of writers
+ * - calculate the areas of each node, calculate and store a short that represents the combination of areas
+ * (this is done by the AreaDictionary)
+ * - a way is a problem way if its nodes are found in different combinations of areas
+ * - a relation is a problem relation if its members are found in different combinations of areas
*
*/
class ProblemListProcessor extends AbstractMapProcessor {
private final static int PHASE1_NODES_AND_WAYS = 1;
private final static int PHASE2_RELS_ONLY = 2;
- private final OSMWriter[] writers;
-
- private SparseLong2ShortMapFunction coords;
- private SparseLong2ShortMapFunction ways;
+ private final SparseLong2ShortMapFunction coords;
+ private final SparseLong2ShortMapFunction ways;
- private final WriterDictionaryShort writerDictionary;
+ private final AreaDictionaryShort areaDictionary;
private final DataStorer dataStorer;
- private LongArrayList problemWays;
- private LongArrayList problemRels;
- private final Long2ObjectOpenHashMap<Integer> oneTileOnlyRels;
+ private final LongArrayList problemWays = new LongArrayList();
+ private final LongArrayList problemRels = new LongArrayList();
- private BitSet writerSet;
+ /** each bit represents one distinct area */
+ private final BitSet areaSet = new BitSet();
private int phase = PHASE1_NODES_AND_WAYS;
// for statistics
//private long countQuickTest = 0;
//private long countFullTest = 0;
private long countCoords = 0;
- private final int writerOffset;
- private final int lastWriter;
+ private final int areaOffset;
+ private final int lastAreaOffset;
private boolean isFirstPass;
private boolean isLastPass;
- private WriterIndex writerIndex;
+ private AreaIndex areaIndex;
private final HashSet<String> wantedBoundaryAdminLevels = new HashSet<>();
private final HashSet<String> wantedBoundaryTagValues;
- ProblemListProcessor(DataStorer dataStorer, int writerOffset,
- int numWritersThisPass, LongArrayList problemWays,
- LongArrayList problemRels, Long2ObjectOpenHashMap<Integer> oneTileOnlyRels,
- String[] boundaryTagList) {
+ ProblemListProcessor(DataStorer dataStorer, int areaOffset,
+ int numAreasThisPass, String[] boundaryTagList) {
this.dataStorer = dataStorer;
- this.writerDictionary = dataStorer.getWriterDictionary();
+ this.areaDictionary = dataStorer.getAreaDictionary();
if (dataStorer.getUsedWays() == null){
ways = SparseLong2ShortMap.createMap("way");
ways.defaultReturnValue(UNASSIGNED);
@@ -74,20 +69,14 @@ class ProblemListProcessor extends AbstractMapProcessor {
}
else
ways = dataStorer.getUsedWays();
- this.writers = writerDictionary.getWriters();
- //this.ways = dataStorer.getWays();
- writerSet = new BitSet(writerDictionary.getNumOfWriters());
- this.writerIndex = dataStorer.getGrid();
+ this.areaIndex = dataStorer.getGrid();
this.coords = SparseLong2ShortMap.createMap("coord");
this.coords.defaultReturnValue(UNASSIGNED);
- this.isFirstPass = (writerOffset == 0);
- this.writerOffset = writerOffset;
- this.lastWriter = writerOffset + numWritersThisPass-1;
- this.isLastPass = (writerOffset + numWritersThisPass == writers.length);
- this.problemWays = problemWays;
- this.problemRels = problemRels;
- this.oneTileOnlyRels = oneTileOnlyRels;
+ this.isFirstPass = (areaOffset == 0);
+ this.areaOffset = areaOffset;
+ this.lastAreaOffset = areaOffset + numAreasThisPass - 1;
+ this.isLastPass = (areaOffset + numAreasThisPass == dataStorer.getNumOfAreas());
if (boundaryTagList != null && boundaryTagList.length > 0)
wantedBoundaryTagValues = new HashSet<>(Arrays.asList(boundaryTagList));
else
@@ -139,24 +128,23 @@ class ProblemListProcessor extends AbstractMapProcessor {
public void processNode(Node node) {
if (phase == PHASE2_RELS_ONLY)
return;
- int countWriters = 0;
- short lastUsedWriter = UNASSIGNED;
- short writerIdx = UNASSIGNED;
- WriterGridResult writerCandidates = writerIndex.get(node);
- if (writerCandidates == null)
+ int countAreas = 0;
+ short lastUsedArea = UNASSIGNED;
+ short areaIdx = UNASSIGNED;
+ AreaGridResult areaCandidates = areaIndex.get(node);
+ if (areaCandidates == null)
return;
- if (writerCandidates.l.size() > 1)
- writerSet.clear();
- for (int i = 0; i < writerCandidates.l.size(); i++) {
- int n = writerCandidates.l.getShort(i);
- if (n < writerOffset || n > lastWriter)
+ if (areaCandidates.l.size() > 1)
+ areaSet.clear();
+ for (int i = 0; i < areaCandidates.l.size(); i++) {
+ int n = areaCandidates.l.getShort(i);
+ if (n < areaOffset || n > lastAreaOffset)
continue;
boolean found;
- if (writerCandidates.testNeeded){
- OSMWriter w = writers[n];
- found = w.nodeBelongsToThisArea(node);
+ if (areaCandidates.testNeeded){
+ found = dataStorer.getArea(n).contains(node);
//++countFullTest;
}
else{
@@ -164,20 +152,20 @@ class ProblemListProcessor extends AbstractMapProcessor {
//++countQuickTest;
}
if (found) {
- writerSet.set(n);
- ++countWriters;
- lastUsedWriter = (short) n;
+ areaSet.set(n);
+ ++countAreas;
+ lastUsedArea = (short) n;
}
}
- if (countWriters > 0){
- if (countWriters > 1)
- writerIdx = writerDictionary.translate(writerSet);
+ if (countAreas > 0){
+ if (countAreas > 1)
+ areaIdx = areaDictionary.translate(areaSet);
else
- writerIdx = (short) (lastUsedWriter - WriterDictionaryShort.DICT_START); // no need to do lookup in the dictionary
- coords.put(node.getId(), writerIdx);
+ areaIdx = AreaDictionaryShort.translate(lastUsedArea); // no need to do lookup in the dictionary
+ coords.put(node.getId(), areaIdx);
++countCoords;
if (countCoords % 10000000 == 0){
- System.out.println("coord MAP occupancy: " + Utils.format(countCoords) + ", number of area dictionary entries: " + writerDictionary.size() + " of " + ((1<<16) - 1));
+ System.out.println("coord MAP occupancy: " + Utils.format(countCoords) + ", number of area dictionary entries: " + areaDictionary.size() + " of " + ((1<<16) - 1));
coords.stats(0);
}
}
@@ -189,9 +177,8 @@ class ProblemListProcessor extends AbstractMapProcessor {
return;
boolean maybeChanged = false;
int oldclIndex = UNASSIGNED;
- short wayWriterIdx;
- //BitSet wayNodeWriterCombis = new BitSet();
- writerSet.clear();
+ short wayAreaIdx;
+ areaSet.clear();
//for (long id: way.getRefs()){
int refs = way.getRefs().size();
for (int i = 0; i < refs; i++){
@@ -202,29 +189,27 @@ class ProblemListProcessor extends AbstractMapProcessor {
continue;
}
if (oldclIndex != clIdx){
- //wayNodeWriterCombis.set(clIdx + WriterDictionaryShort.DICT_START);
- BitSet cl = writerDictionary.getBitSet(clIdx);
- writerSet.or(cl);
+ BitSet cl = areaDictionary.getBitSet(clIdx);
+ areaSet.or(cl);
oldclIndex = clIdx;
maybeChanged = true;
}
}
if (!isFirstPass && maybeChanged || isLastPass){
- wayWriterIdx = ways.get(way.getId());
- if (wayWriterIdx != UNASSIGNED)
- writerSet.or(writerDictionary.getBitSet(wayWriterIdx));
+ wayAreaIdx = ways.get(way.getId());
+ if (wayAreaIdx != UNASSIGNED)
+ areaSet.or(areaDictionary.getBitSet(wayAreaIdx));
}
if (isLastPass){
- if (checkIfMultipleWriters(writerSet)){
+ if (checkIfMultipleAreas(areaSet)){
problemWays.add(way.getId());
- //System.out.println("gen: w" + way.getId() + " touches " + writerDictionary.getMapIds(writerSet));
}
}
- if (maybeChanged && writerSet.isEmpty() == false){
- wayWriterIdx = writerDictionary.translate(writerSet);
- ways.put(way.getId(), wayWriterIdx);
+ if (maybeChanged && areaSet.isEmpty() == false){
+ wayAreaIdx = areaDictionary.translate(areaSet);
+ ways.put(way.getId(), wayAreaIdx);
}
}
// default exclude list for boundary tag
@@ -275,12 +260,12 @@ class ProblemListProcessor extends AbstractMapProcessor {
if (!useThis){
return;
}
- writerSet.clear();
- Integer relWriterIdx;
+ areaSet.clear();
+ Integer relAreaIdx;
if (!isFirstPass){
- relWriterIdx = dataStorer.getUsedRels().get(rel.getId());
- if (relWriterIdx != null)
- writerSet.or(dataStorer.getMultiTileWriterDictionary().getBitSet(relWriterIdx));
+ relAreaIdx = dataStorer.getUsedRels().get(rel.getId());
+ if (relAreaIdx != null)
+ areaSet.or(dataStorer.getMultiTileDictionary().getBitSet(relAreaIdx));
}
short oldclIndex = UNASSIGNED;
short oldwlIndex = UNASSIGNED;
@@ -292,8 +277,8 @@ class ProblemListProcessor extends AbstractMapProcessor {
if (clIdx != UNASSIGNED){
if (oldclIndex != clIdx){
- BitSet wl = writerDictionary.getBitSet(clIdx);
- writerSet.or(wl);
+ BitSet wl = areaDictionary.getBitSet(clIdx);
+ areaSet.or(wl);
}
oldclIndex = clIdx;
@@ -304,53 +289,31 @@ class ProblemListProcessor extends AbstractMapProcessor {
if (wlIdx != UNASSIGNED){
if (oldwlIndex != wlIdx){
- BitSet wl = writerDictionary.getBitSet(wlIdx);
- writerSet.or(wl);
+ BitSet wl = areaDictionary.getBitSet(wlIdx);
+ areaSet.or(wl);
}
oldwlIndex = wlIdx;
}
}
// ignore relation here
}
- if (writerSet.isEmpty())
+ if (areaSet.isEmpty())
return;
if (isLastPass){
- if (checkIfMultipleWriters(writerSet)){
+ if (checkIfMultipleAreas(areaSet)){
problemRels.add(rel.getId());
- //System.out.println("gen: r" + rel.getId() + " touches " + writerDictionary.getMapIds(writerSet));
} else {
- // the relation is only in one tile
- int newWriterIdx = -1;
- for (int i = writerSet.nextSetBit(0); i >= 0; i = writerSet.nextSetBit(i+1)){
- if (writers[i].areaIsPseudo() == false) {
- // this should be the only writer
- newWriterIdx = i;
- break;
- }
- }
- // find out if it was already processed in a previous partition of tiles
- Integer writerInOtherPartition = oneTileOnlyRels.get(rel.getId());
-
- if (newWriterIdx >= 0){
- // the relation is written to a real tile in this partition
- if (writerInOtherPartition != null && writerInOtherPartition >= 0){
- // the relation also appeared in another partition of tiles,
- // so it is a problem rel
- problemRels.add(rel.getId());
- return;
- }
- }
- // store the info that the rel is only in one tile, but
- // don't overwrite the info when it was a real tile
- if (writerInOtherPartition == null || writerInOtherPartition < 0){
- oneTileOnlyRels.put(rel.getId(), new Integer(newWriterIdx));
- }
+
+ // the relation is only in one distinct area
+ relAreaIdx = dataStorer.getMultiTileDictionary().translate(areaSet);
+ // store the info that the rel is only in one distinct area (-1 means pseudo-area)
+ dataStorer.storeRelationArea(rel.getId(), relAreaIdx);
}
return;
}
- relWriterIdx = dataStorer.getMultiTileWriterDictionary().translate(writerSet);
- dataStorer.getUsedRels().put(rel.getId(), relWriterIdx);
+ relAreaIdx = dataStorer.getMultiTileDictionary().translate(areaSet);
+ dataStorer.getUsedRels().put(rel.getId(), relAreaIdx);
}
@@ -366,7 +329,7 @@ class ProblemListProcessor extends AbstractMapProcessor {
System.out.println("");
System.out.println(" Number of stored shorts for ways: " + Utils.format(dataStorer.getUsedWays().size()));
System.out.println(" Number of stored integers for rels: " + Utils.format(dataStorer.getUsedRels().size()));
- System.out.println(" Number of stored combis in big dictionary: " + Utils.format(dataStorer.getMultiTileWriterDictionary().size()));
+ System.out.println(" Number of stored combis in big dictionary: " + Utils.format(dataStorer.getMultiTileDictionary().size()));
System.out.println(" Number of detected problem ways: " + Utils.format(problemWays.size()));
System.out.println(" Number of detected problem rels: " + Utils.format(problemRels.size()));
Utils.printMem();
@@ -378,16 +341,22 @@ class ProblemListProcessor extends AbstractMapProcessor {
}
/**
- * @param writerCombis
- * @return true if the combination of writers can contain a problem polygon
+ * @param areaCombis
+ * @return true if the combination of distinct areas can contain a problem polygon
*/
- static boolean checkIfMultipleWriters(BitSet writerCombis){
+ static boolean checkIfMultipleAreas(BitSet areaCombis){
// this returns a few false positives for those cases
// where a way or rel crosses two pseudo-areas at a
- // place that is far away from the real writers
+ // place that is far away from the real areas
// but it is difficult to detect these cases.
- return writerCombis.cardinality() > 1;
+ return areaCombis.cardinality() > 1;
}
-
+ public LongArrayList getProblemWays() {
+ return problemWays;
+ }
+
+ public LongArrayList getProblemRels() {
+ return problemRels;
+ }
}
diff --git a/src/uk/me/parabola/splitter/ProblemLists.java b/src/uk/me/parabola/splitter/ProblemLists.java
new file mode 100644
index 0000000..4b28850
--- /dev/null
+++ b/src/uk/me/parabola/splitter/ProblemLists.java
@@ -0,0 +1,240 @@
+package uk.me.parabola.splitter;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.LineNumberReader;
+import java.io.PrintWriter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.regex.Pattern;
+
+import it.unimi.dsi.fastutil.longs.LongArrayList;
+
+public class ProblemLists {
+ private final LongArrayList problemWays = new LongArrayList();
+ private final LongArrayList problemRels = new LongArrayList();
+ private final TreeSet<Long> calculatedProblemWays = new TreeSet<>();
+ private final TreeSet<Long> calculatedProblemRels = new TreeSet<>();
+
+ /**
+ * Calculate lists of ways and relations that appear in multiple areas for a given list
+ * of areas.
+ * @param osmFileHandler
+ * @param realAreas
+ * @param wantedAdminLevel
+ * @param boundaryTags
+ * @param maxAreasPerPass
+ * @param overlapAmount
+ * @return
+ */
+ public DataStorer calcProblemLists(OSMFileHandler osmFileHandler, List<Area> realAreas, int wantedAdminLevel,
+ String[] boundaryTags, int maxAreasPerPass, int overlapAmount) {
+ long startProblemListGenerator = System.currentTimeMillis();
+ ArrayList<Area> distinctAreas = AreasCalculator.getNonOverlappingAreas(realAreas);
+ if (distinctAreas.size() > realAreas.size()) {
+ System.err.println("Waring: The areas given in --split-file are overlapping.");
+ Set<Integer> overlappingTiles = new TreeSet<>();
+ for (int i = 0; i < realAreas.size(); i++) {
+ Area a1 = realAreas.get(i);
+ for (int j = i+1; j < realAreas.size(); j++) {
+ Area a2 = realAreas.get(j);
+ if (a1.intersects(a2)) {
+ overlappingTiles.add(a1.getMapId());
+ overlappingTiles.add(a2.getMapId());
+ }
+ }
+ }
+ if (!overlappingTiles.isEmpty()) {
+ System.out.println("Overlaping tiles: " + overlappingTiles.toString());
+ }
+ }
+ System.out.println("Generating problem list for " + distinctAreas.size() + " distinct areas");
+ List<Area> workAreas = AreasCalculator.addPseudoAreas(distinctAreas);
+
+ int numPasses = (int) Math.ceil((double) workAreas.size() / maxAreasPerPass);
+ int areasPerPass = (int) Math.ceil((double) workAreas.size() / numPasses);
+ if (numPasses > 1) {
+ System.out.println("Processing " + distinctAreas.size() + " areas in " + numPasses + " passes, " + areasPerPass + " areas at a time");
+ } else {
+ System.out.println("Processing " + distinctAreas.size() + " areas in a single pass");
+ }
+
+ ArrayList<Area> allAreas = new ArrayList<>();
+
+ System.out.println("Pseudo areas:");
+ for (int j = 0;j < workAreas.size(); j++){
+ Area area = workAreas.get(j);
+ allAreas.add(area);
+ if (area.isPseudoArea())
+ System.out.println("Pseudo area " + area.getMapId() + " covers " + area);
+ }
+
+ DataStorer distinctDataStorer = new DataStorer(workAreas, overlapAmount);
+ System.out.println("Starting problem-list-generator pass(es)");
+
+ for (int pass = 0; pass < numPasses; pass++) {
+ System.out.println("-----------------------------------");
+ System.out.println("Starting problem-list-generator pass " + (pass+1) + " of " + numPasses);
+ long startThisPass = System.currentTimeMillis();
+ int areaOffset = pass * areasPerPass;
+ int numAreasThisPass = Math.min(areasPerPass, workAreas.size() - pass * areasPerPass);
+ ProblemListProcessor processor = new ProblemListProcessor(distinctDataStorer, areaOffset,
+ numAreasThisPass, boundaryTags);
+ processor.setWantedAdminLevel(wantedAdminLevel);
+
+ boolean done = false;
+ while (!done){
+ done = osmFileHandler.process(processor);
+ calculatedProblemWays.addAll(processor.getProblemWays());
+ calculatedProblemRels.addAll(processor.getProblemRels());
+ }
+ System.out.println("Problem-list-generator pass " + (pass+1) + " took " + (System.currentTimeMillis() - startThisPass) + " ms");
+ }
+ System.out.println("Problem-list-generator pass(es) took " + (System.currentTimeMillis() - startProblemListGenerator) + " ms");
+ DataStorer dataStorer = new DataStorer(realAreas, overlapAmount);
+ dataStorer.translateDistinctToRealAreas(distinctDataStorer);
+ return dataStorer;
+ }
+
+ /** Read user defined problematic relations and ways */
+ public boolean readProblemIds(String problemFileName) {
+ File fProblem = new File(problemFileName);
+ boolean ok = true;
+
+ if (!fProblem.exists()) {
+ System.out.println("Error: problem file doesn't exist: " + fProblem);
+ return false;
+ }
+ try (InputStream fileStream = new FileInputStream(fProblem);
+ LineNumberReader problemReader = new LineNumberReader(
+ new InputStreamReader(fileStream));) {
+ Pattern csvSplitter = Pattern.compile(Pattern.quote(":"));
+ Pattern commentSplitter = Pattern.compile(Pattern.quote("#"));
+ String problemLine;
+ String[] items;
+ while ((problemLine = problemReader.readLine()) != null) {
+ items = commentSplitter.split(problemLine);
+ if (items.length == 0 || items[0].trim().isEmpty()){
+ // comment or empty line
+ continue;
+ }
+ items = csvSplitter.split(items[0].trim());
+ if (items.length != 2) {
+ System.out.println("Error: Invalid format in problem file, line number " + problemReader.getLineNumber() + ": "
+ + problemLine);
+ ok = false;
+ continue;
+ }
+ long id = 0;
+ try{
+ id = Long.parseLong(items[1]);
+ }
+ catch(NumberFormatException exp){
+ System.out.println("Error: Invalid number format in problem file, line number " + + problemReader.getLineNumber() + ": "
+ + problemLine + exp);
+ ok = false;
+ }
+ if ("way".equals(items[0]))
+ problemWays.add(id);
+ else if ("rel".equals(items[0]))
+ problemRels.add(id);
+ else {
+ System.out.println("Error in problem file: Type not way or relation, line number " + + problemReader.getLineNumber() + ": "
+ + problemLine);
+ ok = false;
+ }
+ }
+ } catch (IOException exp) {
+ System.out.println("Error: Cannot read problem file " + fProblem +
+ exp);
+ return false;
+ }
+ return ok;
+ }
+
+ /**
+ * Write a file that can be given to mkgmap that contains the correct arguments
+ * for the split file pieces. You are encouraged to edit the file and so it
+ * contains a template of all the arguments that you might want to use.
+ * @param problemRelsThisPass
+ * @param problemWaysThisPass
+ */
+ public void writeProblemList(File fileOutputDir, String fname) {
+ try (PrintWriter w = new PrintWriter(new FileWriter(new File(fileOutputDir, fname)));) {
+
+ w.println("#");
+ w.println("# This file can be given to splitter using the --problem-file option");
+ w.println("#");
+ w.println("# List of relations and ways that are known to cause problems");
+ w.println("# in splitter or mkgmap");
+ w.println("# Objects listed here are specially treated by splitter to assure");
+ w.println("# that complete data is written to all related tiles");
+ w.println("# Format:");
+ w.println("# way:<id>");
+ w.println("# rel:<id>");
+ w.println("# ways");
+ for (long id: calculatedProblemWays){
+ w.println("way: " + id + " #");
+ }
+ w.println("# rels");
+ for (long id: calculatedProblemRels){
+ w.println("rel: " + id + " #");
+ }
+
+ w.println();
+ } catch (IOException e) {
+ System.err.println("Warning: Could not write problem-list file " + fname + ", processing continues");
+ }
+ }
+
+ /**
+ * Calculate writers for elements which cross areas.
+ * @param dataStorer stores data that is needed in different passes of the program.
+ * @param osmFileHandler used to access OSM input files
+ */
+ public void calcMultiTileElements(DataStorer dataStorer, OSMFileHandler osmFileHandler) {
+ // merge the calculated problem ids and the user given problem ids
+ problemWays.addAll(calculatedProblemWays);
+ problemRels.addAll(calculatedProblemRels);
+ calculatedProblemRels.clear();
+ calculatedProblemWays.clear();
+
+ if (problemWays.isEmpty() && problemRels.isEmpty())
+ return;
+
+ // calculate which ways and relations are written to multiple areas.
+ MultiTileProcessor multiProcessor = new MultiTileProcessor(dataStorer, problemWays, problemRels);
+ // multiTileProcessor stores the problem relations in its own structures return memory to GC
+ problemRels.clear();
+ problemWays.clear();
+ problemRels.trim();
+ problemWays.trim();
+
+ boolean done = false;
+ long startThisPhase = System.currentTimeMillis();
+ int prevPhase = -1;
+ while(!done){
+ int phase = multiProcessor.getPhase();
+ if (prevPhase != phase){
+ startThisPhase = System.currentTimeMillis();
+ System.out.println("-----------------------------------");
+ System.out.println("Executing multi-tile analyses phase " + phase);
+ }
+ done = osmFileHandler.process(multiProcessor);
+ prevPhase = phase;
+ if (done || (phase != multiProcessor.getPhase())){
+ System.out.println("Multi-tile analyses phase " + phase + " took " + (System.currentTimeMillis() - startThisPhase) + " ms");
+ }
+ }
+
+ System.out.println("-----------------------------------");
+ }
+
+
+}
diff --git a/src/uk/me/parabola/splitter/PseudoOSMWriter.java b/src/uk/me/parabola/splitter/PseudoOSMWriter.java
index 7f9ca23..938d2b7 100644
--- a/src/uk/me/parabola/splitter/PseudoOSMWriter.java
+++ b/src/uk/me/parabola/splitter/PseudoOSMWriter.java
@@ -13,14 +13,16 @@
package uk.me.parabola.splitter;
+/**
+ * A do-nothing writer (used with --output=simulate)
+ * @author Gerd Petermann
+ *
+ */
public class PseudoOSMWriter extends AbstractOSMWriter{
- private final boolean areaIsPseudo;
- public PseudoOSMWriter(Area bounds, int mapId, boolean areaIsPseudo, int overlap) {
+ public PseudoOSMWriter(Area bounds) {
// no overlap for pseudo writers !
- super(bounds, null, mapId, overlap);
- assert areaIsPseudo && overlap==0 || !areaIsPseudo;
- this.areaIsPseudo = areaIsPseudo;
+ super(bounds, null, bounds.getMapId(), 0);
}
@Override
@@ -37,9 +39,4 @@ public class PseudoOSMWriter extends AbstractOSMWriter{
@Override
public void finishWrite() {}
-
- @Override
- public boolean areaIsPseudo() {
- return areaIsPseudo;
- }
}
diff --git a/src/uk/me/parabola/splitter/SparseLong2ShortMapHuge.java b/src/uk/me/parabola/splitter/SparseLong2ShortMapHuge.java
index 068d09d..84578c0 100644
--- a/src/uk/me/parabola/splitter/SparseLong2ShortMapHuge.java
+++ b/src/uk/me/parabola/splitter/SparseLong2ShortMapHuge.java
@@ -522,7 +522,7 @@ public class SparseLong2ShortMapHuge implements SparseLong2ShortMapFunction{
System.out.println(dataDesc + " Map details: bytes/overhead " + Utils.format(totalBytes) + " / " + Utils.format(totalOverhead) + ", overhead includes " +
topMap.size() + " arrays with " + LARGE_VECTOR_SIZE * 8/1024/1024 + " MB");
if (msgLevel > 0 & uncompressedLen > 0){
- System.out.print(dataDesc + " RLE compresion info: compressed / uncompressed size / ratio: " +
+ System.out.print(dataDesc + " RLE compression info: compressed / uncompressed size / ratio: " +
Utils.format(compressedLen) + " / "+
Utils.format(uncompressedLen) + " / "+
Utils.format(Math.round(100-(float) (compressedLen*100/uncompressedLen))) + "%");
diff --git a/src/uk/me/parabola/splitter/SparseLong2ShortMapInline.java b/src/uk/me/parabola/splitter/SparseLong2ShortMapInline.java
index 4068cf4..9e4571e 100644
--- a/src/uk/me/parabola/splitter/SparseLong2ShortMapInline.java
+++ b/src/uk/me/parabola/splitter/SparseLong2ShortMapInline.java
@@ -524,7 +524,7 @@ public class SparseLong2ShortMapInline implements SparseLong2ShortMapFunction{
System.out.println(dataDesc + " Map details: bytes/overhead " + Utils.format(totalBytes) + " / " + Utils.format(totalOverhead) + ", overhead includes " +
topMap.size() + " arrays with " + LARGE_VECTOR_SIZE * 4/1024/1024 + " MB");
if (msgLevel > 0 & uncompressedLen > 0){
- System.out.print("RLE compresion info: compressed / uncompressed size / ratio: " +
+ System.out.print("RLE compression info: compressed / uncompressed size / ratio: " +
Utils.format(compressedLen) + " / "+
Utils.format(uncompressedLen) + " / "+
Utils.format(Math.round(100-(float) (compressedLen*100/uncompressedLen))) + "%");
diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java
index 33ac5b5..b6dafa5 100644
--- a/src/uk/me/parabola/splitter/SplitProcessor.java
+++ b/src/uk/me/parabola/splitter/SplitProcessor.java
@@ -21,8 +21,6 @@ import java.util.Date;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
-import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
-
/**
* Splits a map into multiple areas.
*/
@@ -31,12 +29,11 @@ class SplitProcessor extends AbstractMapProcessor {
private SparseLong2ShortMapFunction coords;
private SparseLong2ShortMapFunction ways;
- private final WriterDictionaryShort writerDictionary;
+ private final AreaDictionaryShort writerDictionary;
private final DataStorer dataStorer;
private final Long2IntClosedMapFunction nodeWriterMap;
private final Long2IntClosedMapFunction wayWriterMap;
private final Long2IntClosedMapFunction relWriterMap;
- private final Long2ObjectOpenHashMap<Integer> oneTileOnlyRels;
// for statistics
private long countQuickTest = 0;
@@ -45,7 +42,7 @@ class SplitProcessor extends AbstractMapProcessor {
private long countWays = 0;
private final int writerOffset;
private final int lastWriter;
- private WriterIndex writerIndex;
+ private final AreaIndex writerIndex;
private final int maxThreads;
private final short unassigned = Short.MIN_VALUE;
@@ -61,12 +58,10 @@ class SplitProcessor extends AbstractMapProcessor {
private BitSet usedWriters;
- SplitProcessor(DataStorer dataStorer, Long2ObjectOpenHashMap<Integer> oneTileOnlyRels,
- int writerOffset, int numWritersThisPass, int maxThreads){
+ SplitProcessor(DataStorer dataStorer, int writerOffset, int numWritersThisPass, int maxThreads){
this.dataStorer = dataStorer;
- this.oneTileOnlyRels = oneTileOnlyRels;
- this.writerDictionary = dataStorer.getWriterDictionary();
- this.writers = writerDictionary.getWriters();
+ this.writerDictionary = dataStorer.getAreaDictionary();
+ this.writers = dataStorer.getWriters();
this.coords = SparseLong2ShortMap.createMap("coord");
this.ways = SparseLong2ShortMap.createMap("way");
this.coords.defaultReturnValue(unassigned);
@@ -112,9 +107,9 @@ class SplitProcessor extends AbstractMapProcessor {
@Override
public void processWay(Way w) {
currentWayAreaSet.clear();
- int multiTileWriterIdx = (wayWriterMap != null) ? wayWriterMap.getSeq(w.getId()): WriterDictionaryInt.UNASSIGNED;
- if (multiTileWriterIdx != WriterDictionaryInt.UNASSIGNED){
- BitSet cl = dataStorer.getMultiTileWriterDictionary().getBitSet(multiTileWriterIdx);
+ int multiTileWriterIdx = (wayWriterMap != null) ? wayWriterMap.getSeq(w.getId()): AreaDictionaryInt.UNASSIGNED;
+ if (multiTileWriterIdx != AreaDictionaryInt.UNASSIGNED){
+ BitSet cl = dataStorer.getMultiTileDictionary().getBitSet(multiTileWriterIdx);
// set only active writer bits
for(int i=cl.nextSetBit(writerOffset); i>=0 && i <= lastWriter; i=cl.nextSetBit(i+1)){
currentWayAreaSet.set(i);
@@ -163,22 +158,23 @@ class SplitProcessor extends AbstractMapProcessor {
@Override
public void processRelation(Relation rel) {
currentRelAreaSet.clear();
- Integer singleTileWriterIdx = oneTileOnlyRels.get(rel.getId());
+ Integer singleTileWriterIdx = dataStorer.getOneTileOnlyRels(rel.getId());
if (singleTileWriterIdx != null){
- if (singleTileWriterIdx < 0) {
+ if (singleTileWriterIdx == AreaDictionaryInt.UNASSIGNED) {
+ // we know that the relation is outside of all real areas
return;
}
-
- BitSet wl = dataStorer.getMultiTileWriterDictionary().getBitSet(singleTileWriterIdx);
+ // relation is within an area that is overlapped by the writer areas
+ BitSet wl = dataStorer.getMultiTileDictionary().getBitSet(singleTileWriterIdx);
// set only active writer bits
for (int i = wl.nextSetBit(writerOffset); i >= 0 && i <= lastWriter; i = wl.nextSetBit(i + 1)) {
currentRelAreaSet.set(i);
}
} else {
- int multiTileWriterIdx = (relWriterMap != null) ? relWriterMap.getSeq(rel.getId()): WriterDictionaryInt.UNASSIGNED;
- if (multiTileWriterIdx != WriterDictionaryInt.UNASSIGNED){
+ int multiTileWriterIdx = (relWriterMap != null) ? relWriterMap.getSeq(rel.getId()): AreaDictionaryInt.UNASSIGNED;
+ if (multiTileWriterIdx != AreaDictionaryInt.UNASSIGNED){
- BitSet cl = dataStorer.getMultiTileWriterDictionary().getBitSet(multiTileWriterIdx);
+ BitSet cl = dataStorer.getMultiTileDictionary().getBitSet(multiTileWriterIdx);
// set only active writer bits
for (int i = cl.nextSetBit(writerOffset); i >= 0 && i <= lastWriter; i = cl.nextSetBit(i + 1)) {
currentRelAreaSet.set(i);
@@ -267,10 +263,10 @@ class SplitProcessor extends AbstractMapProcessor {
private void writeNode(Node currentNode) throws IOException {
int countWriters = 0;
short lastUsedWriter = unassigned;
- WriterGridResult writerCandidates = writerIndex.get(currentNode);
- int multiTileWriterIdx = (nodeWriterMap != null) ? nodeWriterMap.getSeq(currentNode.getId()): WriterDictionaryInt.UNASSIGNED;
+ AreaGridResult writerCandidates = writerIndex.get(currentNode);
+ int multiTileWriterIdx = (nodeWriterMap != null) ? nodeWriterMap.getSeq(currentNode.getId()): AreaDictionaryInt.UNASSIGNED;
- boolean isSpecialNode = (multiTileWriterIdx != WriterDictionaryInt.UNASSIGNED);
+ boolean isSpecialNode = (multiTileWriterIdx != AreaDictionaryInt.UNASSIGNED);
if (writerCandidates == null && !isSpecialNode) {
return;
}
@@ -281,10 +277,10 @@ class SplitProcessor extends AbstractMapProcessor {
int n = writerCandidates.l.getShort(i);
if (n < writerOffset || n > lastWriter)
continue;
- OSMWriter w = writers[n];
+ OSMWriter writer = writers[n];
boolean found;
if (writerCandidates.testNeeded){
- found = w.nodeBelongsToThisArea(currentNode);
+ found = writer.getExtendedBounds().contains(currentNode);
++countFullTest;
}
else{
@@ -298,14 +294,14 @@ class SplitProcessor extends AbstractMapProcessor {
if (maxThreads > 1) {
addToWorkingQueue(n, currentNode);
} else {
- w.write(currentNode);
+ writer.write(currentNode);
}
}
}
}
if (isSpecialNode){
// this node is part of a multi-tile-polygon, add it to all tiles covered by the parent
- BitSet nodeWriters = dataStorer.getMultiTileWriterDictionary().getBitSet(multiTileWriterIdx);
+ BitSet nodeWriters = dataStorer.getMultiTileDictionary().getBitSet(multiTileWriterIdx);
for(int i=nodeWriters.nextSetBit(writerOffset); i>=0 && i <= lastWriter; i=nodeWriters.nextSetBit(i+1)){
if (usedWriters.get(i) )
continue;
@@ -322,7 +318,7 @@ class SplitProcessor extends AbstractMapProcessor {
if (countWriters > 1)
writersID = writerDictionary.translate(usedWriters);
else
- writersID = (short) (lastUsedWriter - WriterDictionaryShort.DICT_START); // no need to do lookup in the dictionary
+ writersID = AreaDictionaryShort.translate(lastUsedWriter); // no need to do lookup in the dictionary
coords.put(currentNode.getId(), writersID);
++countCoords;
if (countCoords % 10000000 == 0){
diff --git a/src/uk/me/parabola/splitter/SplittableDensityArea.java b/src/uk/me/parabola/splitter/SplittableDensityArea.java
index 6c0fca1..c78bbcf 100644
--- a/src/uk/me/parabola/splitter/SplittableDensityArea.java
+++ b/src/uk/me/parabola/splitter/SplittableDensityArea.java
@@ -512,7 +512,7 @@ public class SplittableDensityArea {
* @param depth recursion depth
* @param tile the tile to split
* @param rasteredPolygonArea an area describing a rectilinear shape
- * @return a solution or null if splitting failed
+ * @return a solution (maybe empty)
*/
private Solution findSolutionWithSinglePolygon(int depth, final Tile tile, java.awt.geom.Area rasteredPolygonArea) {
assert rasteredPolygonArea.isSingular();
@@ -574,7 +574,7 @@ public class SplittableDensityArea {
return part0Sol;
}
}
- return null;
+ return new Solution(maxNodes);
}
/**
@@ -757,6 +757,8 @@ public class SplittableDensityArea {
private Solution solveRectangularArea(Tile startTile){
// start values for optimization process: we make little steps towards a good solution
// spread = 7;
+ if (startTile.count == 0)
+ return new Solution(maxNodes);
searchLimit = startSearchLimit;
minNodes = Math.max(Math.min((long)(0.05 * maxNodes), extraDensityInfo.getNodeCount()), 1);
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-grass/mkgmap-splitter.git
More information about the Pkg-grass-devel
mailing list