[python-stetl] 01/07: Imported Upstream version 1.0.9+ds
Bas Couwenberg
sebastic at debian.org
Sat Jun 18 15:45:52 UTC 2016
This is an automated email from the git hooks/post-receive script.
sebastic pushed a commit to branch master
in repository python-stetl.
commit fe120972dc26324b6ca8fa072285d6183d11d06a
Author: Bas Couwenberg <sebastic at xs4all.nl>
Date: Sat Jun 18 16:14:35 2016 +0200
Imported Upstream version 1.0.9+ds
---
CHANGES.txt | 12 ++
PKG-INFO | 54 ++++++---
README.md | 38 ++++--
VERSION.txt | 2 +-
docker/Dockerfile | 44 +++++++
docker/README.md | 38 ++++++
docker/build.sh | 9 ++
docker/test/1_copystd/etl.cfg | 11 ++
docker/test/1_copystd/etl.sh | 9 ++
docker/test/1_copystd/input/cities.xml | 18 +++
docs/Makefile | 153 ++++++++++++++++++++++++
docs/cases.rst | 88 ++++++++++++++
docs/conf.py | 4 +-
docs/index.rst | 2 +
docs/install.rst | 14 +++
docs/using.rst | 51 +++++++-
examples/basics/6_cmdargs/etl.args | 4 +
examples/basics/6_cmdargs/etl.sh | 9 +-
examples/basics/readme.txt | 6 +-
examples/top10nl/etl-top10nl.cmd | 17 +--
examples/top10nl/etl-top10nl.props | 28 +++++
examples/top10nl/etl-top10nl.sh | 17 +--
examples/top10nl/readme.txt | 2 +
setup.py | 5 +-
stetl/etl.py | 2 +-
stetl/factory.py | 1 +
stetl/filters/formatconverter.py | 3 +
stetl/filters/packetwriter.py | 61 ++++++++++
stetl/filters/templatingfilter.py | 10 +-
stetl/filters/xmlelementreader.py | 133 +++++++++++++++++++++
stetl/filters/zipfileextractor.py | 70 +++++++++++
stetl/inputs/fileinput.py | 109 ++++++++++++++---
stetl/main.py | 12 +-
stetl/outputs/dboutput.py | 91 ++++++++++++--
stetl/outputs/execoutput.py | 209 +++++++++++++++++++++++++++++++++
stetl/packet.py | 1 +
stetl/util.py | 30 +++++
stetl/version.py | 2 +-
tests/__init__.pyc | Bin 0 -> 134 bytes
tests/configs/copy_in_out.cfg | 11 ++
tests/data/cities.xml | 18 +++
tests/test_chain.pyc | Bin 0 -> 994 bytes
42 files changed, 1295 insertions(+), 103 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index cbc11e6..db73902 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,18 @@
Changes
=======
+
+v1.0.9 - 17 june 2016
+---------------------
+
+See https://github.com/justb4/stetl/issues?q=milestone%3A%22Version+1.0.9%22+is%3Aclosed
+
+Highlights:
+
+- Substitutable config options in properties file (-a arg)
+- Docker support via Stetl Docker image
+- Generic ogr2ogr Input Component enhancement
+
v1.0.8 - 2 july 2015
--------------------
diff --git a/PKG-INFO b/PKG-INFO
index 0af9919..c9d6ccd 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,34 +1,35 @@
Metadata-Version: 1.1
Name: Stetl
-Version: 1.0.8
+Version: 1.0.9
Summary: Stetl provides transformation for spatial data
Home-page: http://github.com/justb4/stetl
Author: Just van den Broecke
Author-email: justb4 at gmail.com
License: GNU GPL v3
-Description: Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for the conversion of rich (as GML)
+Description: # Stetl - Streaming ETL
+
+ Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for the conversion of rich (as GML)
geospatial data conversion. Stetl is Open Source (GNU GPL v3).
+ ## Documentation
+
The main website and documentation can be found on http://stetl.org (or http://stetl.readthedocs.org).
Read a 5-minute introduction here: http://www.slideshare.net/justb4/5-minute-intro-to-setl and a longer presentation
here: http://www.slideshare.net/justb4/stetl-foss4g20131024v1.
- Stetl was presented at FOSS4G 2013 in Nottingham http://2013.foss4g.org (sat sept 21).
+ Stetl was presented at several occasions like the
+ FOSS4G 2013 in Nottingham http://2013.foss4g.org.
+ ## Origins
Stetl originated in the INSPIRE-FOSS project (www.inspire-foss.org)
and was created by Just van den Broecke. Since Stetl evolved into a wider use like
transforming Dutch GML-based datasets such as IMGEO/BGT (Large Scale Topography)
and IMKAD/BRK (Cadastral Data) it has now a repo of its own.
+ ## Design
+
Stetl basically glues together existing parsing and transformation tools like GDAL/OGR (ogr2ogr) and XSLT.
By using native tools like libxml and libxslt (via Python lxml) Stetl is speed-optimized.
- So why en when to use Stetl.
-
- * when ogr2ogr or XSLT alone cannot do the job
- * when having to deal with complex GML as source or destination
-
- So Stetl is in particularly useful for INSPIRE-related transformations and other complex GML-related ETL.
-
Stetl has a similar design as Spring (Java) and other modern frameworks based on IoC (Inversion of Control, http://en.wikipedia.org/wiki/Inversion_of_Control).
A configuration file (in Python config format) specifies your chain of ETL steps.
This chain is formed by a series of Python modules/objects and their parameters. These are
@@ -36,22 +37,33 @@ Description: Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-fra
The config file specifies the input modules (e.g. PostGIS), transformers (e.g. XSLT) and outputs (e.g. a GML file or even
WFS-T a geospatial protocol to publish GML to a server).
- There is special support now for output to the deegree WFS server (http://deegree.org), either directly in
- a so called GML Blobstore, via deegree "FSLoader" tool or via WFS-T publishing. The latter is in theory
- supported for other WFS server products like GeoServer and MapServer.
-
Stetl has been proven to handle 10's of millions of objects without any memory issues.
This is achieved through a technique called "streaming and splitting".
For example: using the OgrPostgisInput module an GML stream can be generated from the database.
A component called the GmlSplitter can split this stream into managable chunks (like 20000 features)
and feed this upstream into the ETL chain.
+ ## Examples
+
+ Stetl has been found in particularly useful for INSPIRE-related transformations and other complex GML-related ETL.
+
See examples under the examples dir.
Another example in http://code.google.com/p/inspire-foss/source/browse/trunk/etl/NL.Kadaster/Addresses
(Dutch Addresses (BAG) to INSPIRE Addresses)
- Finally, the word "stetl" is also an alternative writing for "shtetl":
+ ## Contributing
+
+ Anyone and everyone is welcome to contribute. Please take a moment to
+ review the [guidelines for contributing](CONTRIBUTING.md).
+
+ * [Bug reports](CONTRIBUTING.md#bugs)
+ * [Feature requests](CONTRIBUTING.md#features)
+ * [Pull requests](CONTRIBUTING.md#pull-requests)
+
+ ## Finally
+
+ The word "stetl" is also an alternative writing for "shtetl":
http://en.wikipedia.org/wiki/Stetl : "...Material things were neither disdained nor
extremely praised in the shtetl. Learning and education were the ultimate measures of worth in the eyes of the community,
while money was secondary to status..."
@@ -62,6 +74,18 @@ Description: Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-fra
Changes
=======
+
+ v1.0.9 - 17 june 2016
+ ---------------------
+
+ See https://github.com/justb4/stetl/issues?q=milestone%3A%22Version+1.0.9%22+is%3Aclosed
+
+ Highlights:
+
+ - Substitutable config options in properties file (-a arg)
+ - Docker support via Stetl Docker image
+ - Generic ogr2ogr Input Component enhancement
+
v1.0.8 - 2 july 2015
--------------------
diff --git a/README.md b/README.md
index 2028ecb..e83e536 100644
--- a/README.md
+++ b/README.md
@@ -1,26 +1,27 @@
+# Stetl - Streaming ETL
+
Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for the conversion of rich (as GML)
geospatial data conversion. Stetl is Open Source (GNU GPL v3).
+## Documentation
+
The main website and documentation can be found on http://stetl.org (or http://stetl.readthedocs.org).
Read a 5-minute introduction here: http://www.slideshare.net/justb4/5-minute-intro-to-setl and a longer presentation
here: http://www.slideshare.net/justb4/stetl-foss4g20131024v1.
-Stetl was presented at FOSS4G 2013 in Nottingham http://2013.foss4g.org (sat sept 21).
+Stetl was presented at several occasions like the
+FOSS4G 2013 in Nottingham http://2013.foss4g.org.
+## Origins
Stetl originated in the INSPIRE-FOSS project (www.inspire-foss.org)
and was created by Just van den Broecke. Since Stetl evolved into a wider use like
transforming Dutch GML-based datasets such as IMGEO/BGT (Large Scale Topography)
and IMKAD/BRK (Cadastral Data) it has now a repo of its own.
+## Design
+
Stetl basically glues together existing parsing and transformation tools like GDAL/OGR (ogr2ogr) and XSLT.
By using native tools like libxml and libxslt (via Python lxml) Stetl is speed-optimized.
-So why en when to use Stetl.
-
-* when ogr2ogr or XSLT alone cannot do the job
-* when having to deal with complex GML as source or destination
-
-So Stetl is in particularly useful for INSPIRE-related transformations and other complex GML-related ETL.
-
Stetl has a similar design as Spring (Java) and other modern frameworks based on IoC (Inversion of Control, http://en.wikipedia.org/wiki/Inversion_of_Control).
A configuration file (in Python config format) specifies your chain of ETL steps.
This chain is formed by a series of Python modules/objects and their parameters. These are
@@ -28,22 +29,33 @@ symbolically specified in the config file. You just invoke etl.py the main progr
The config file specifies the input modules (e.g. PostGIS), transformers (e.g. XSLT) and outputs (e.g. a GML file or even
WFS-T a geospatial protocol to publish GML to a server).
-There is special support now for output to the deegree WFS server (http://deegree.org), either directly in
-a so called GML Blobstore, via deegree "FSLoader" tool or via WFS-T publishing. The latter is in theory
-supported for other WFS server products like GeoServer and MapServer.
-
Stetl has been proven to handle 10's of millions of objects without any memory issues.
This is achieved through a technique called "streaming and splitting".
For example: using the OgrPostgisInput module an GML stream can be generated from the database.
A component called the GmlSplitter can split this stream into managable chunks (like 20000 features)
and feed this upstream into the ETL chain.
+## Examples
+
+Stetl has been found in particularly useful for INSPIRE-related transformations and other complex GML-related ETL.
+
See examples under the examples dir.
Another example in http://code.google.com/p/inspire-foss/source/browse/trunk/etl/NL.Kadaster/Addresses
(Dutch Addresses (BAG) to INSPIRE Addresses)
-Finally, the word "stetl" is also an alternative writing for "shtetl":
+## Contributing
+
+Anyone and everyone is welcome to contribute. Please take a moment to
+review the [guidelines for contributing](CONTRIBUTING.md).
+
+* [Bug reports](CONTRIBUTING.md#bugs)
+* [Feature requests](CONTRIBUTING.md#features)
+* [Pull requests](CONTRIBUTING.md#pull-requests)
+
+## Finally
+
+The word "stetl" is also an alternative writing for "shtetl":
http://en.wikipedia.org/wiki/Stetl : "...Material things were neither disdained nor
extremely praised in the shtetl. Learning and education were the ultimate measures of worth in the eyes of the community,
while money was secondary to status..."
diff --git a/VERSION.txt b/VERSION.txt
index 337a6a8..e5a4a5e 100644
--- a/VERSION.txt
+++ b/VERSION.txt
@@ -1 +1 @@
-1.0.8
\ No newline at end of file
+1.0.9
\ No newline at end of file
diff --git a/docker/Dockerfile b/docker/Dockerfile
new file mode 100644
index 0000000..fbb2b6b
--- /dev/null
+++ b/docker/Dockerfile
@@ -0,0 +1,44 @@
+FROM ubuntu:14.04
+# FROM python:2.7 would be preferred but trickier
+# TODO: use a more lightweight Linux distro than Ubuntu
+
+MAINTAINER just at justobjects.nl
+
+# Configure timezone and locale
+ARG IMAGE_TIMEZONE="Europe/Amsterdam"
+# set time right adn configure timezone and locale
+RUN echo "$IMAGE_TIMEZONE" > /etc/timezone && \
+ dpkg-reconfigure -f noninteractive tzdata
+
+# Optional: Use local cached debs from host (saves your bandwidth!)
+# Change ip below to that of your apt-cacher-ng host
+# Or comment this line out if you do not with to use caching
+# ADD 71-apt-cacher-ng /etc/apt/apt.conf.d/71-apt-cacher-ng
+
+RUN export DEBIAN_FRONTEND=noninteractive TERM=linux && \
+ echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
+
+RUN apt-get update && \
+ apt-get install -y software-properties-common
+
+# Use UbuntuGIS for latest GDAL, Proj.4, GEOS etc
+RUN apt-add-repository ppa:ubuntugis/ubuntugis-unstable
+RUN apt-get update && \
+ apt-get install -y git \
+ libproj0 \
+ libgeos-dev \
+ python-lxml \
+ python-gdal \
+ libgdal-dev \
+ build-essential \
+ python-dev \
+ python-setuptools
+
+# Get latest Stetl from git
+RUN git clone https://github.com/justb4/stetl /opt/stetl
+
+# Use Stetl installer
+RUN cd /opt/stetl; python setup.py install
+
+# Allow docker run
+ENTRYPOINT ["/usr/local/bin/stetl"]
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 0000000..3ac8aa9
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,38 @@
+# Stetl Docker
+
+Docker image to run ETL tool Stetl. See http://stetl.org.
+Public Docker images are available at: https://hub.docker.com/r/justb4/stetl/.
+
+## Status
+
+First version on June 17, 2016. Improvements welcome.
+
+## Building
+
+Use ./build.sh or
+
+ docker build -t justb4/stetl:latest .
+
+ARGs: optional --build-arg IMAGE_TIMEZONE="Europe/Amsterdam"
+
+## Running
+
+Mapping volumes (``docker -v``) can be handly to maintain all config and data on the Host, outside the Docker container.
+Note that full paths are required.
+
+Example, running a basic Stetl example:
+
+ cd test/1_copystd
+ docker run -v `pwd`:`pwd` -w `pwd` -t -i justb4/stetl:latest -c etl.cfg
+
+ or
+ ./etl.sh
+
+Many Stetl configs require a connection to PostGIS. This can be effected with a linked container: ``--link postgis``, or
+better using Docker networking.
+
+## More
+
+See https://github.com/Geonovum/smartemission/tree/master/etl for a real-world example using a single Stetl Docker image
+for multiple Stetl configurations and linking with a PostGIS Docker container. For example:
+https://github.com/Geonovum/smartemission/blob/master/etl/last.sh
diff --git a/docker/build.sh b/docker/build.sh
new file mode 100755
index 0000000..1a802e6
--- /dev/null
+++ b/docker/build.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+#
+# Build Stetl Docker image
+
+# Optional: build with custom/your timezone and forced via --no-cache
+# sudo docker build -t --no-cache --build-arg IMAGE_TIMEZONE="Europe/Amsterdam" justb4/stetl:latest .
+
+sudo docker build -t justb4/stetl:latest .
+
diff --git a/docker/test/1_copystd/etl.cfg b/docker/test/1_copystd/etl.cfg
new file mode 100644
index 0000000..fae6419
--- /dev/null
+++ b/docker/test/1_copystd/etl.cfg
@@ -0,0 +1,11 @@
+# Trivial example of an ETL Chain that just copies a file to standard output.
+
+[etl]
+chains = input_xml_file|output_std
+
+[input_xml_file]
+class = inputs.fileinput.XmlFileInput
+file_path = input/cities.xml
+
+[output_std]
+class = outputs.standardoutput.StandardXmlOutput
diff --git a/docker/test/1_copystd/etl.sh b/docker/test/1_copystd/etl.sh
new file mode 100755
index 0000000..e3f1402
--- /dev/null
+++ b/docker/test/1_copystd/etl.sh
@@ -0,0 +1,9 @@
+#!/bin/sh
+#
+# Test Stetl Docker image.
+# Simple ETL for copying a file to standard output.
+#
+#
+sudo docker run -v `pwd`:`pwd` -w `pwd` -t -i justb4/stetl:latest -c etl.cfg
+
+
diff --git a/docker/test/1_copystd/input/cities.xml b/docker/test/1_copystd/input/cities.xml
new file mode 100644
index 0000000..58721f0
--- /dev/null
+++ b/docker/test/1_copystd/input/cities.xml
@@ -0,0 +1,18 @@
+<?xml version='1.0' encoding='utf-8'?>
+<cities>
+ <city>
+ <name>Amsterdam</name>
+ <lat>52.4</lat>
+ <lon>4.9</lon>
+ </city>
+ <city>
+ <name>Bonn</name>
+ <lat>50.7</lat>
+ <lon>7.1</lon>
+ </city>
+ <city>
+ <name>Rome</name>
+ <lat>41.9</lat>
+ <lon>12.5</lon>
+ </city>
+</cities>
diff --git a/docs/Makefile b/docs/Makefile
new file mode 100644
index 0000000..729bdda
--- /dev/null
+++ b/docs/Makefile
@@ -0,0 +1,153 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS =
+SPHINXBUILD = sphinx-build
+PAPER =
+BUILDDIR = _build
+
+# Internal variables.
+PAPEROPT_a4 = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
+
+help:
+ @echo "Please use \`make <target>' where <target> is one of"
+ @echo " html to make standalone HTML files"
+ @echo " dirhtml to make HTML files named index.html in directories"
+ @echo " singlehtml to make a single large HTML file"
+ @echo " pickle to make pickle files"
+ @echo " json to make JSON files"
+ @echo " htmlhelp to make HTML files and a HTML help project"
+ @echo " qthelp to make HTML files and a qthelp project"
+ @echo " devhelp to make HTML files and a Devhelp project"
+ @echo " epub to make an epub"
+ @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+ @echo " latexpdf to make LaTeX files and run them through pdflatex"
+ @echo " text to make text files"
+ @echo " man to make manual pages"
+ @echo " texinfo to make Texinfo files"
+ @echo " info to make Texinfo files and run them through makeinfo"
+ @echo " gettext to make PO message catalogs"
+ @echo " changes to make an overview of all changed/added/deprecated items"
+ @echo " linkcheck to check all external links for integrity"
+ @echo " doctest to run all doctests embedded in the documentation (if enabled)"
+
+clean:
+ -rm -rf $(BUILDDIR)
+
+html:
+ $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+ @echo
+ @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+ $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+ @echo
+ @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+ $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+ @echo
+ @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+ $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+ @echo
+ @echo "Build finished; now you can process the pickle files."
+
+json:
+ $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+ @echo
+ @echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+ $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+ @echo
+ @echo "Build finished; now you can run HTML Help Workshop with the" \
+ ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+ $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+ @echo
+ @echo "Build finished; now you can run "qcollectiongenerator" with the" \
+ ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+ @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Stetl.qhcp"
+ @echo "To view the help file:"
+ @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Stetl.qhc"
+
+devhelp:
+ $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+ @echo
+ @echo "Build finished."
+ @echo "To view the help file:"
+ @echo "# mkdir -p $$HOME/.local/share/devhelp/Stetl"
+ @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Stetl"
+ @echo "# devhelp"
+
+epub:
+ $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+ @echo
+ @echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+ $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+ @echo
+ @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+ @echo "Run \`make' in that directory to run these through (pdf)latex" \
+ "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+ $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+ @echo "Running LaTeX files through pdflatex..."
+ $(MAKE) -C $(BUILDDIR)/latex all-pdf
+ @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+ $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+ @echo
+ @echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+ $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+ @echo
+ @echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+ $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+ @echo
+ @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+ @echo "Run \`make' in that directory to run these through makeinfo" \
+ "(use \`make info' here to do that automatically)."
+
+info:
+ $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+ @echo "Running Texinfo files through makeinfo..."
+ make -C $(BUILDDIR)/texinfo info
+ @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+ $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+ @echo
+ @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+ $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+ @echo
+ @echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+ $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+ @echo
+ @echo "Link check complete; look for any errors in the above output " \
+ "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+ $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+ @echo "Testing of doctests in the sources finished, look at the " \
+ "results in $(BUILDDIR)/doctest/output.txt."
diff --git a/docs/cases.rst b/docs/cases.rst
new file mode 100644
index 0000000..1f6b77e
--- /dev/null
+++ b/docs/cases.rst
@@ -0,0 +1,88 @@
+.. _cases:
+
+Cases
+=====
+
+This chapter lists various cases/projects where Stetl is used.
+
+NLExtract
+---------
+
+NLExtract http://nlextract.nl is a development project that aims to provide ETL-tooling for all
+Dutch Open Geo-Datasets, in particular the country wide
+"Key Registries" (Dutch: Basisregistraties) like Cadastral Parcels (BRK), Topography (BRT+BGT) and
+Buildings and Addresses (BAG). These datasets are provided as XML/GML. The ETL mostly provides
+a transformation to PostGIS. For all Key Registries, except for the BAG, Stetl is used, basically
+as-is, without extra (Python) programming. See also the NLExtract GitHub:
+https://github.com/opengeogroep/NLExtract
+
+Topography (BRT/Top10NL)
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+See https://github.com/opengeogroep/NLExtract/tree/master/top10nl/etl and the Stetl conf at
+https://github.com/opengeogroep/NLExtract/tree/master/top10nl/etl/conf/
+
+Detailed Topography (BGT)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See https://github.com/opengeogroep/NLExtract/tree/master/bgt and the Stetl conf at
+https://github.com/opengeogroep/NLExtract/blob/master/bgt/etl/conf/
+
+Cadastral Parcels (BRK)
+~~~~~~~~~~~~~~~~~~~~~~~
+
+See https://github.com/opengeogroep/NLExtract/tree/master/brk/etl
+and the Stetl conf at https://github.com/opengeogroep/NLExtract/tree/master/brk/etl/conf
+
+INSPIRE
+-------
+
+These were the origins of Stetl. This project was sponsored by Kadaster.
+See http://inspire-foss.org. The ETL involved the transformation of Dutch Key Registries (see above)
+to harmonized INSPIRE GML according to the Annexes.
+
+Addresses
+~~~~~~~~~
+
+BAG to INSPIRE Addresses Annex II Theme.
+
+See https://github.com/justb4/inspire-foss/blob/master/etl/NL.Kadaster/Addresses/
+
+Ordnance Survey
+---------------
+
+A successful Proof-of-Concept to convert Ordnance Survey Mastermap GML to PostGIS:
+
+https://github.com/justb4/stetl/tree/master/examples/ordnancesurvey
+
+SOSPilot
+--------
+
+A SensorWeb project by Geonovum, see http://sensors.geonovum.nl.
+
+Dutch AQ to WFS/WMS(-Time) and SOS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Stetl was used
+for ETL from Dutch Air Quality Data from RIVM (XML) to WMS(-Time), WFS and SOS.
+The latter was effected by SOS-Transactional publication. Documentation at
+http://sospilot.readthedocs.org and ETL on GitHub at
+https://github.com/Geonovum/sospilot/tree/master/src/rivm-lml
+
+Dutch AQ to EAI Reporting
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Stetl was used to generate XML-based reports for the EU EAI:
+
+https://github.com/Geonovum/sospilot/tree/master/src/aq-report
+
+This involved the first use of Jinja2 templating for complex XML/GML generation.
+
+Smart Emission
+--------------
+
+Sensors for air quality, meteo and audio at civilians. Project by University of Nijmegen/Gemeente Nijmegen with participation
+by Geonovum. Stetl is used to transform a low-level sensor API to PostGIS and later on WMS/WFS/SOS.
+This is also an example how to use a Stetl Docker image:
+
+See https://github.com/Geonovum/smartemission/tree/master/etl
diff --git a/docs/conf.py b/docs/conf.py
index 988e9cb..f580701 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -52,9 +52,9 @@ copyright = u'2013+, Just van den Broecke'
# built documents.
#
# The short X.Y version.
-version = '1.0.7rc7'
+version = '1.0.9'
# The full version, including alpha/beta/rc tags.
-release = '1.0.7rc7'
+release = '1.0.9'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
diff --git a/docs/index.rst b/docs/index.rst
index df60514..32036c0 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -16,6 +16,7 @@ The code is on GitHub: https://github.com/justb4/stetl.
See an `introductory Stetl presentation at Slideshare <http://www.slideshare.net/justb4/stetl-foss4g20131024v1>`_.
+This is document version |release| generated on |today|.
Contents:
@@ -26,6 +27,7 @@ Contents:
install.rst
background.rst
using.rst
+ cases.rst
code.rst
contact.rst
links.rst
diff --git a/docs/install.rst b/docs/install.rst
index 4b22fbd..1116253 100644
--- a/docs/install.rst
+++ b/docs/install.rst
@@ -23,6 +23,9 @@ installations below.
You may also want to download the complete .tar.gz distro from PyPi:
https://pypi.python.org/pypi/Stetl . This includes the examples and tests.
+**Docker:** Since version 1.0.9 Stetl also can be installed and run via `Docker <http://docker.com>`_. See
+:ref:`install_docker` below.
+
Dependencies
------------
@@ -126,8 +129,19 @@ Try running the examples when running with a downloaded distro. ::
Look for any error messages in your output.
+.. _install_docker:
+Install with Docker
+-------------------
+One of the cleanest ways to use Stetl is via `Docker <http://docker.com>`_. Your environment needs to be
+setup to use Docker and probably you want to use some tooling like `Vagrant <https://www.vagrantup.com/>`_. The author uses
+a combination of VirtualBox with Ubuntu and Vagrant on Mac OSX to run Docker, but this
+is a bit out of scope here.
+Assuming you have a working Docker environment, there are two ways to install Stetl with Docker:
+* build a Docker image yourself using the Dockerfile in https://github.com/justb4/stetl/tree/master/docker
+* use a prebuilt public Stetl Docker image: https://hub.docker.com/r/justb4/stetl
+For running Stetl using Docker see :ref:`run_docker`.
\ No newline at end of file
diff --git a/docs/using.rst b/docs/using.rst
index c5ca474..67162cc 100644
--- a/docs/using.rst
+++ b/docs/using.rst
@@ -171,6 +171,41 @@ whatever it gets as input from the previous Filter in the Chain. ::
output/gmlcities.shp
temp/gmlcities.gml
+.. _run_docker:
+
+Using Docker
+~~~~~~~~~~~~
+
+A convenient way to run Stetl is via a Docker image. See the installation instructions at
+:ref:`install_docker`. A full example can be viewed in the Smart Emission project:
+https://github.com/Geonovum/smartemission/tree/master/etl.
+
+In the simplest case you run a Stetl Docker container from your own built image or the Dockerhub-provided
+one, `justb4/stetl:latest <https://hub.docker.com/r/justb4/stetl>`_ basically as follows: ::
+
+ sudo docker run -v <host dir>:<container dir> -w <work dir> justb4/stetl:latest <any Stetl arguments>
+
+For example within the current directory you may have an ``etl.cfg`` Stetl file: ::
+
+ WORK_DIR=`pwd`
+ sudo docker run -v ${WORK_DIR}:${WORK_DIR} -w ${WORK_DIR} justb4/stetl:latest -c etl.cfg
+
+A more advanced setup would be (network-)linking to a PostGIS Docker image
+like `kartoza/postgis <https://hub.docker.com/r/kartoza/postgis/>`_: ::
+
+ # First run Postgis, remains running,
+ sudo docker run --name postgis -d -t kartoza/postgis:9.4-2.1
+
+ # Then later run Stetl
+ STETL_ARGS="-c etl.cfg -a local.args"
+ WORK_DIR="`pwd`"
+
+ sudo docker run --name stetl --link postgis:postgis -v ${WORK_DIR}:${WORK_DIR} -w ${WORK_DIR} justb4/stetl:latest ${STETL_ARGS}
+
+The last example is used within the SmartEmission project. Also with more detail and keeping
+all dynamic data (like PostGIS DB), your Stetl config and results, and logs within the host. For PostGIS see: https://github.com/Geonovum/smartemission/tree/master/services/postgis
+and Stetl see: https://github.com/Geonovum/smartemission/tree/master/etl.
+
Stetl Integration
-----------------
@@ -200,9 +235,21 @@ example `6_cmdargs <https://github.com/justb4/stetl/tree/master/examples/basics/
class = outputs.fileoutput.FileOutput
file_path = {out_xml}
-Note the symbolic input, xsl and output files. We can now perform this ETL using the `stetl -a option`. ::
+Note the symbolic input, xsl and output files. We can now perform this ETL using the `stetl -a option` in two ways.
+One, passing the arguments on the commandline, like ::
+
+ stetl -c etl.cfg -a "in_xml=input/cities.xml in_xsl=cities2gml.xsl out_xml=output/gmlcities.gml"
+
+Two, passing the arguments in a properties file, here called `etl.args` (the name of the suffix .args is not significant). ::
+
+ stetl -c etl.cfg -a etl.args
+
+Where the content of the `etl.args` properties file is: ::
- stetl -c etl.cfg -a "in_xml=input/cities.xml in_xsl=cities2gml.xsl out_xml=output/gmlcities.gml".
+ # Arguments in properties file
+ in_xml=input/cities.xml
+ in_xsl=cities2gml.xsl
+ out_xml=output/gmlcities.gml
This makes an ETL chain highly reusable. A very elaborate Stetl config with parameter substitution can be seen in the
`Top10NL ETL <https://github.com/justb4/stetl/blob/master/examples/top10nl/etl-top10nl.cfg>`_.
diff --git a/examples/basics/6_cmdargs/etl.args b/examples/basics/6_cmdargs/etl.args
new file mode 100644
index 0000000..b76b1c4
--- /dev/null
+++ b/examples/basics/6_cmdargs/etl.args
@@ -0,0 +1,4 @@
+# Arguments in properties file
+in_xml=input/cities.xml
+in_xsl=cities2gml.xsl
+out_xml=output/gmlcities.gml
diff --git a/examples/basics/6_cmdargs/etl.sh b/examples/basics/6_cmdargs/etl.sh
index 88642ea..33e5baa 100755
--- a/examples/basics/6_cmdargs/etl.sh
+++ b/examples/basics/6_cmdargs/etl.sh
@@ -4,6 +4,13 @@
#
# Author: Just van den Broecke
#
-stetl -c etl.cfg -a "in_xml=input/cities.xml in_xsl=cities2gml.xsl out_xml=output/gmlcities.gml"
+#
+#
+# Two ways to pass arguments to be substituted in the etl.cfg {arg} paramters
+
+# Option 1: using command line args
+stetl -c etl.cfg -a "in_xml=input/cities.xml in_xsl=cities2gml.xsl out_xml=output/gmlcities.gml"
+# Option 2: using a properties file
+stetl -c etl.cfg -a etl.args
diff --git a/examples/basics/readme.txt b/examples/basics/readme.txt
index 8057d4d..15b7bd8 100644
--- a/examples/basics/readme.txt
+++ b/examples/basics/readme.txt
@@ -1,4 +1,4 @@
-The directories below each show basic examples for Setl.
+The directories below each show basic examples for Stetl.
The examples build up from simple to more complex by directory number prefix.
As a general health test you may run all examples using ./runall.sh
@@ -8,7 +8,7 @@ As a general health test you may run all examples using ./runall.sh
3_shape - transform an input XML file to a GML in memory etree and output to a Shape file
4_validate - use of the validator filter after generating the GML doc
5_split - split the input XML file before transforming to GML and output to multiple GML files
-6_cmdargs - reuse ETL config file by substituting symbolic variables via command line (-a) arguments
+6_cmdargs - reuse ETL config file by substituting symbolic variables via command line (-a) arguments or properties file
7_mycomponent - adding custom/user-defined Input, Filter and/or Output Components
8_wfs - fetch input data from WFS and process it
9_string_templating - transform using standard Python string template with CSV input vars
@@ -16,6 +16,8 @@ As a general health test you may run all examples using ./runall.sh
11_formatconvert - adapt incompatible inputs to outputs via the FormatConvertFilter
12_gdal_ogr - direct OgrInput (and later output)
13_dbinput - input from SQL sources, here SLQLite Input
+14_logfileinput - input from Apache Logfile
+
diff --git a/examples/top10nl/etl-top10nl.cmd b/examples/top10nl/etl-top10nl.cmd
index 9550c12..e6baa46 100644
--- a/examples/top10nl/etl-top10nl.cmd
+++ b/examples/top10nl/etl-top10nl.cmd
@@ -1,17 +1,4 @@
rem windows version
+rem TO BE TESTED!!
-set gml_files=input
-
-rem multi_opts=-splitlistfields~-maxsubfields 1
-
-rem multi_opts=-splitlistfields
-
-rem multi_opts=-fieldTypeToString~StringList
-
-rem multi_opts=~
-
-
-set multi="multi_opts=-fieldTypeToString~StringList"
-
-python ..\..\stetl\main.py -c etl-top10nl.cfg -a "database=top10nl host=127.0.0.1 port=5432 user=top10nl password=top10nl schema=stetltest temp_dir=temp max_features=20 gml_files=%gml_files% %multi%"
-
+python ..\..\stetl\main.py -c etl-top10nl.cfg -a etl-top10.props
diff --git a/examples/top10nl/etl-top10nl.props b/examples/top10nl/etl-top10nl.props
new file mode 100644
index 0000000..392e64a
--- /dev/null
+++ b/examples/top10nl/etl-top10nl.props
@@ -0,0 +1,28 @@
+# Properties to be substituted in etl.cfg
+
+# Input dir or files
+# gml_files=/Users/just/geodata/top10nl/TOP10NL_GML_50D_Blokken_september_2012/GML_50D_Blokken/Top10NL_05Oost.gml
+gml_files=input
+
+# Misc
+temp_dir=temp
+max_features=20
+
+# How to treat multiple occurences of elems
+# May use: these options
+# Note: ' ' is used a s space separator
+# multi_opts=-splitlistfields -maxsubfields 1
+# multi_opts=-splitlistfields
+# multi_opts=-fieldTypeToString StringList
+# multi_opts=
+multi_opts=-fieldTypeToString StringList
+
+spatial_extent=120000 450000 160000 500000
+
+# Output
+database=top10nl
+host=localhost
+port=5432
+user=postgres
+password=postgres
+schema=test
diff --git a/examples/top10nl/etl-top10nl.sh b/examples/top10nl/etl-top10nl.sh
index 451d889..2963e55 100755
--- a/examples/top10nl/etl-top10nl.sh
+++ b/examples/top10nl/etl-top10nl.sh
@@ -5,21 +5,8 @@
# Author: Just van den Broecke
#
-# For substitutable args see all strings in {arg} in etl-top10nl.cfg
-# <db> <host> <port> <user> <password> <schema> <temp_dir> <max_features> <gml_files> <multi_opts>
+# For substitutable args see all strings in {arg} in etl-top10nl.cfg and example values in etl-top10.props
-# gml_files=/Users/just/geodata/top10nl/TOP10NL_GML_50D_Blokken_september_2012/GML_50D_Blokken/Top10NL_05Oost.gml
-gml_files=input
+python ../../stetl/main.py -c etl-top10nl.cfg -a etl-top10nl.props
-# May use: these options
-# Note: '~' is used a s space separator
-# multi_opts=-splitlistfields~-maxsubfields 1
-# multi_opts=-splitlistfields
-# multi_opts=-fieldTypeToString~StringList
-# multi_opts=~
-
-spatial_extent="spatial_extent=120000~450000~160000~500000"
-# spatial_extent="spatial_extent="
-multi="multi_opts=-fieldTypeToString~StringList"
-python ../../stetl/main.py -c etl-top10nl.cfg -a "database=top10nl host=localhost port=5432 user=postgres password=postgres schema=test temp_dir=temp max_features=20 gml_files=$gml_files $multi $spatial_extent"
diff --git a/examples/top10nl/readme.txt b/examples/top10nl/readme.txt
index a717094..5828ea2 100644
--- a/examples/top10nl/readme.txt
+++ b/examples/top10nl/readme.txt
@@ -1,6 +1,8 @@
Note: this example was the base for the full implementation in the NLExtract project where
it will be maintained more actively.
+
See: https://github.com/opengeogroep/NLExtract/tree/master/top10nl/etl
+USE NLExtract FOR REAL TOP10NL EXTRACT!!!
=============================
diff --git a/setup.py b/setup.py
index 7d7b9cf..1be4e58 100644
--- a/setup.py
+++ b/setup.py
@@ -2,6 +2,8 @@ import logging
import sys
from setuptools import setup, find_packages
+# To publish: python setup.py sdist upload -r pypi
+
# Have to do this after importing setuptools, which monkey patches distutils.
from distutils.extension import Extension
@@ -33,7 +35,8 @@ with open('CHANGES.txt', 'r') as f:
requirements = [
'psycopg2',
'lxml',
- 'GDAL'
+ 'GDAL<2.0',
+ 'Jinja2'
]
if sys.version_info < (2, 7):
diff --git a/stetl/etl.py b/stetl/etl.py
index e39b05a..bf1c28a 100755
--- a/stetl/etl.py
+++ b/stetl/etl.py
@@ -43,7 +43,7 @@ class ETL:
sys.exit(1)
ETL.CONFIG_DIR = os.path.dirname(os.path.abspath(config_file))
- log.info("Config/working dir =%s" % ETL.CONFIG_DIR)
+ log.info("Config/working dir = %s" % ETL.CONFIG_DIR)
self.configdict = ConfigParser()
diff --git a/stetl/factory.py b/stetl/factory.py
index 8f7cfaf..dd48701 100755
--- a/stetl/factory.py
+++ b/stetl/factory.py
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
from util import Util
+
log = Util.get_log('factory')
class Factory:
diff --git a/stetl/filters/formatconverter.py b/stetl/filters/formatconverter.py
index b9e1593..8a1e1fd 100644
--- a/stetl/filters/formatconverter.py
+++ b/stetl/filters/formatconverter.py
@@ -405,6 +405,9 @@ FORMAT_CONVERTERS = {
FORMAT.etree_doc: FormatConverter.string2etree_doc,
FORMAT.string: FormatConverter.no_op
},
+ FORMAT.line_stream: {
+ FORMAT.string: FormatConverter.no_op
+ },
FORMAT.xml_line_stream: {
FORMAT.string: FormatConverter.no_op
}
diff --git a/stetl/filters/packetwriter.py b/stetl/filters/packetwriter.py
new file mode 100644
index 0000000..a390ddc
--- /dev/null
+++ b/stetl/filters/packetwriter.py
@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+#
+# Writes the payload of a packet as a string to a file.
+# Based on outputs.fileoutput.FileOutput.
+#
+# Author: Frank Steggink
+#
+from stetl.component import Config
+from stetl.filter import Filter
+from stetl.util import Util
+from stetl.packet import FORMAT
+
+import os
+
+log = Util.get_log('packetwriter')
+
+
+class PacketWriter(Filter):
+ """
+ Writes the payload of a packet as a string to a file.
+
+ consumes=FORMAT.any, produces=FORMAT.string
+ """
+
+ # Start attribute config meta
+ @Config(ptype=str, default=None, required=True)
+ def file_path(self):
+ """
+ File path to write content to.
+
+ Required: True
+
+ Default: None
+ """
+ pass
+
+ # End attribute config meta
+
+ # Constructor
+ def __init__(self, configdict, section):
+ Filter.__init__(self, configdict, section, consumes=FORMAT.any, produces=FORMAT.string)
+ log.info("working dir %s" % os.getcwd())
+
+ def invoke(self, packet):
+ if packet.data is None:
+ return packet
+
+ file_path = self.cfg.get('file_path')
+ return self.write_file(packet, file_path)
+
+ def write_file(self, packet, file_path):
+ log.info('writing to file %s' % file_path)
+ out_file = open(file_path, 'w')
+
+ out_file.write(packet.to_string())
+
+ out_file.close()
+ log.info("written to %s" % file_path)
+
+ packet.data = file_path
+ return packet
diff --git a/stetl/filters/templatingfilter.py b/stetl/filters/templatingfilter.py
index 34a7ed6..e9cfe05 100644
--- a/stetl/filters/templatingfilter.py
+++ b/stetl/filters/templatingfilter.py
@@ -95,11 +95,11 @@ class StringTemplatingFilter(TemplatingFilter):
contains the actual values to be substituted in the template string as a record (key/value pairs).
Output is a regular string.
- consumes=FORMAT.record, produces=FORMAT.string
+ consumes=FORMAT.record or FORMAT.record_array, produces=FORMAT.string
"""
def __init__(self, configdict, section):
- TemplatingFilter.__init__(self, configdict, section, consumes=FORMAT.record)
+ TemplatingFilter.__init__(self, configdict, section, consumes=[FORMAT.record, FORMAT.record_array])
def create_template(self):
# Init once
@@ -115,7 +115,11 @@ class StringTemplatingFilter(TemplatingFilter):
self.template = Template(self.template_string)
def render_template(self, packet):
- packet.data = self.template.substitute(packet.data)
+ if type(packet.data) is list:
+ packet.data = [self.template.substitute(item) for item in packet.data]
+ else:
+ packet.data = self.template.substitute(packet.data)
+
return packet
diff --git a/stetl/filters/xmlelementreader.py b/stetl/filters/xmlelementreader.py
new file mode 100644
index 0000000..4a16497
--- /dev/null
+++ b/stetl/filters/xmlelementreader.py
@@ -0,0 +1,133 @@
+# -*- coding: utf-8 -*-
+#
+# Reads an XML file and returns XML elements.
+# Based on inputs.fileinput.XmlElementStreamFileInput.
+#
+# Author: Frank Steggink
+#
+from stetl.component import Config
+from stetl.filter import Filter
+from stetl.util import Util, etree
+from stetl.packet import FORMAT
+
+log = Util.get_log('xmlelementreader')
+
+
+class XmlElementReader(Filter):
+ """
+ Extracts XML elements from a file, outputs each feature element in Packet.
+ Parsing is streaming (no internal DOM buildup) so any file size can be handled.
+ Use this class for your big GML files!
+
+ consumes=FORMAT.string, produces=FORMAT.etree_element
+ """
+
+ # Start attribute config meta
+ @Config(ptype=list, default=None, required=True)
+ def element_tags(self):
+ """
+ Comma-separated string of XML (feature) element tag names of the elements that should be extracted
+ and added to the output element stream.
+
+ Required: True
+
+ Default: None
+ """
+ pass
+
+ @Config(ptype=bool, default=False, required=False)
+ def strip_namespaces(self):
+ """
+ should namespaces be removed from the input document and thus not be present in the output element stream?
+
+ Required: False
+
+ Default: False
+ """
+ pass
+
+ # End attribute config meta
+
+ # Constructor
+ def __init__(self, configdict, section):
+ Filter.__init__(self, configdict, section, consumes=FORMAT.string, produces=FORMAT.etree_element)
+ self.context = None
+ self.root = None
+ self.cur_file_path = None
+ self.elem_count = 0
+ log.info("Element tags to be matched: %s" % self.element_tags)
+
+ def invoke(self, packet):
+ if packet.data is None:
+ log.info("No XML file given")
+ return packet
+
+ if self.cur_file_path is None:
+ self.cur_file_path = packet.data
+
+ event = None
+ packet.data = None
+
+ if self.context is None:
+ # Open file
+ fd = open(self.cur_file_path)
+ self.elem_count = 0
+ log.info("file opened : %s" % self.cur_file_path)
+ self.context = etree.iterparse(fd, events=("start", "end"))
+ self.context = iter(self.context)
+ event, self.root = self.context.next()
+
+ packet = self.process_xml(packet)
+
+ return packet
+
+ def process_xml(self, packet):
+ while not self.context is None:
+ #while not packet.is_end_of_doc():
+ try:
+ event, elem = self.context.next()
+ except (etree.XMLSyntaxError, StopIteration):
+ # workaround for etree.XMLSyntaxError https://bugs.launchpad.net/lxml/+bug/1185701
+ self.context = None
+
+ if self.context is None:
+ # Always end of doc
+ # TODO: is this still useful for a non-input component?
+ packet.set_end_of_doc()
+ log.info("End of doc: %s elem_count=%d" % (self.cur_file_path, self.elem_count))
+
+ return packet
+
+ # Filter out Namespace from the tag
+ # this is the easiest way to go for now
+ tag = elem.tag.split('}')
+ if len(tag) == 2:
+ # Namespaced tag: 2nd is tag
+ tag = tag[1]
+ else:
+ # Non-namespaced tag: first
+ tag = tag[0]
+
+ if tag in self.element_tags:
+ if event == "start":
+ # TODO check if deepcopy is the right thing to do here.
+ # packet.data = elem
+ pass
+ # self.root.remove(elem)
+ elif event == "end":
+ # Delete the element from the tree
+ # self.root.clear()
+ packet.data = elem
+ self.elem_count += 1
+ self.root.remove(elem)
+
+ if self.strip_namespaces:
+ packet.data = Util.stripNamespaces(elem).getroot()
+
+ # If there is a next component, let it process
+ if self.next:
+ # Hand-over data (line, doc whatever) to the next component
+ packet.format = self._output_format
+ packet = self.next.process(packet)
+
+ return packet
diff --git a/stetl/filters/zipfileextractor.py b/stetl/filters/zipfileextractor.py
new file mode 100644
index 0000000..dd01cd9
--- /dev/null
+++ b/stetl/filters/zipfileextractor.py
@@ -0,0 +1,70 @@
+# -*- coding: utf-8 -*-
+#
+# Extracts a file from a ZIP file, and saves it as the given file name.
+#
+# Author: Frank Steggink
+#
+from stetl.component import Config
+from stetl.filter import Filter
+from stetl.util import Util
+from stetl.packet import FORMAT
+
+log = Util.get_log('zipfileextractor')
+
+BUFFER_SIZE = 1024 * 1024 * 1024
+
+
+class ZipFileExtractor(Filter):
+ """
+ Extracts a file from a ZIP file, and saves it as the given file name.
+
+ consumes=FORMAT.record, produces=FORMAT.string
+ """
+
+ # Start attribute config meta
+ @Config(ptype=str, default=None, required=True)
+ def file_path(self):
+ """
+ File name to write the extracted file to.
+
+ Required: True
+
+ Default: None
+ """
+ pass
+
+ # End attribute config meta
+
+ # Constructor
+ def __init__(self, configdict, section):
+ Filter.__init__(self, configdict, section, consumes=FORMAT.record, produces=FORMAT.string)
+ self.cur_file_path = self.cfg.get('file_path')
+
+ def invoke(self, packet):
+ event = None
+
+ if packet.data is None:
+ log.info("No file name given")
+ return packet
+
+ import os
+ import zipfile
+
+ with zipfile.ZipFile(packet.data['file_path']) as z:
+ with open(self.cur_file_path, 'wb') as f:
+ with z.open(packet.data['name']) as zf:
+ while True:
+ buffer = zf.read(BUFFER_SIZE)
+ if not buffer:
+ break
+ f.write(buffer)
+
+ packet.data = self.cur_file_path
+ return packet
+
+ def after_chain_invoke(self, packet):
+ import os.path
+ if os.path.isfile(self.cur_file_path):
+ os.remove(self.cur_file_path)
+
+ return True
diff --git a/stetl/inputs/fileinput.py b/stetl/inputs/fileinput.py
index fa43c28..26b5e0a 100644
--- a/stetl/inputs/fileinput.py
+++ b/stetl/inputs/fileinput.py
@@ -10,6 +10,8 @@ from stetl.util import Util, etree
from stetl.utils.apachelog import formats, parser
from stetl.packet import FORMAT
import csv
+import re
+import fnmatch
log = Util.get_log('fileinput')
@@ -22,7 +24,7 @@ class FileInput(Input):
# Start attribute config meta
# Applying Decorator pattern with the Config class to provide
# read-only config values from the configured properties.
-
+
@Config(ptype=str, default=None, required=False)
def file_path(self):
"""
@@ -35,7 +37,6 @@ class FileInput(Input):
"""
pass
-
@Config(ptype=str, default='*.[gxGX][mM][lL]', required=False)
def filename_pattern(self):
"""
@@ -70,6 +71,7 @@ class FileInput(Input):
if not len(self.file_list):
raise Exception('File list is empty!!')
+ self.cur_file_path = None
self.file_list_done = []
def read(self, packet):
@@ -186,7 +188,6 @@ class XmlFileInput(FileInput):
return data
-
class XmlElementStreamerFileInput(FileInput):
"""
Extracts XML elements from a file, outputs each feature element in Packet.
@@ -228,7 +229,6 @@ class XmlElementStreamerFileInput(FileInput):
self.file_list_done = []
self.context = None
self.root = None
- self.cur_file_path = None
self.elem_count = 0
log.info("Element tags to be matched: %s" % self.element_tags)
@@ -298,18 +298,15 @@ class XmlElementStreamerFileInput(FileInput):
return packet
-class XmlLineStreamerFileInput(FileInput):
+
+class LineStreamerFileInput(FileInput):
"""
- DEPRECATED Streams lines from an XML file(s)
+ Reads text-files, producing a stream of lines, one line per Packet.
NB assumed is that lines in the file have newlines !!
- DEPRECATED better is to use XmlElementStreamerFileInput for GML features.
-
- produces=FORMAT.xml_line_stream
"""
- # Constructor
- def __init__(self, configdict, section):
- FileInput.__init__(self, configdict, section, produces=FORMAT.xml_line_stream)
+ def __init__(self, configdict, section, produces=FORMAT.line_stream):
+ FileInput.__init__(self, configdict, section, produces)
self.file_list_done = []
self.file = None
@@ -350,9 +347,31 @@ class XmlLineStreamerFileInput(FileInput):
return packet
- packet.data = line.decode('utf-8')
+ line = line.decode('utf-8')
+ packet.data = self.process_line(line)
+
return packet
+ def process_line(self, line):
+ """
+ Override in subclass.
+ """
+ return line
+
+
+class XmlLineStreamerFileInput(LineStreamerFileInput):
+ """
+ DEPRECATED Streams lines from an XML file(s)
+ NB assumed is that lines in the file have newlines !!
+ DEPRECATED better is to use XmlElementStreamerFileInput for GML features.
+
+ produces=FORMAT.xml_line_stream
+ """
+
+ # Constructor
+ def __init__(self, configdict, section):
+ LineStreamerFileInput.__init__(self, configdict, section, produces=FORMAT.xml_line_stream)
+
class CsvFileInput(FileInput):
"""
@@ -373,7 +392,6 @@ class CsvFileInput(FileInput):
"""
pass
-
@Config(ptype=str, default='"', required=False)
def quote_char(self):
"""
@@ -453,6 +471,7 @@ class JsonFileInput(FileInput):
return file_data
+
class ApacheLogFileInput(FileInput):
"""
Parses Apache log files. Lines are converted into records based on the log format.
@@ -463,8 +482,9 @@ class ApacheLogFileInput(FileInput):
"""
@Config(ptype=dict, default=
- {'%l': 'logname', '%>s': 'status', '%D': 'deltat', '%{User-agent}i': 'agent', '%b': 'bytes', '%{Referer}i': 'referer', '%u': 'user', '%t': 'time', "'%h": 'host', '%r': 'request'}
- , required=False)
+ {'%l': 'logname', '%>s': 'status', '%D': 'deltat', '%{User-agent}i': 'agent', '%b': 'bytes',
+ '%{Referer}i': 'referer', '%u': 'user', '%t': 'time', "'%h": 'host', '%r': 'request'}
+ , required=False)
def key_map(self):
"""
Map of cryptic %-field names to readable keys in record.
@@ -494,7 +514,7 @@ class ApacheLogFileInput(FileInput):
FileInput.__init__(self, configdict, section, produces=FORMAT.record)
self.file_list_done = []
self.file = None
- self.parser = parser(self.log_format, self.key_map, options={'methods': ['GET','POST'],
+ self.parser = parser(self.log_format, self.key_map, options={'methods': ['GET', 'POST'],
'use_native_types': True,
'request_path_only': True,
'gen_key': True})
@@ -541,3 +561,58 @@ class ApacheLogFileInput(FileInput):
return packet
+class ZipFileInput(FileInput):
+ """
+ Parse ZIP file from file system or URL into a stream of records containing zipfile-path and file names.
+
+ produces=FORMAT.record
+ """
+
+ @Config(ptype=str, default='*', required=False)
+ def name_filter(self):
+ """
+ Regular "glob.glob" expression for filtering out filenames from the ZIP archive.
+
+ Required: False
+
+ Default: * (all files in zip-archive)
+ """
+ pass
+
+ def __init__(self, configdict, section):
+ FileInput.__init__(self, configdict, section, produces=FORMAT.record)
+ self.file_content = None
+
+ # Pre-compile name filter into regex object to match filenames in zip-archive(s) later
+ # See default (*) above, so we alo have a name_filter.
+ self.fname_matcher = re.compile(fnmatch.translate(self.name_filter))
+
+ def read(self, packet):
+ # No more files left and done with current file ?
+ if not self.file_content and not len(self.file_list):
+ packet.set_end_of_stream()
+ log.info("EOF file list, all files done")
+ return packet
+
+ # Done with current file or first file ?
+ if self.file_content is None:
+ self.cur_file_path = self.file_list.pop(0)
+
+ # Assemble list of file names in archive
+ import zipfile
+ zf = zipfile.ZipFile(self.cur_file_path, 'r')
+ namelist = [{'file_path': self.cur_file_path, 'name': name} for name in zf.namelist()]
+
+ # Apply filename filter to namelist (TODO could be done in single step with previous step)
+ self.file_content = [item for item in namelist if self.fname_matcher.match(item["name"])]
+
+ log.info("zip file read : %s filecount=%d" % (self.cur_file_path, len(self.file_content)))
+
+ if len(self.file_content):
+ packet.data = self.file_content.pop(0)
+ log.info("Pop file record: %s" % str(packet.data))
+
+ if not len(self.file_content):
+ self.file_content = None
+
+ return packet
diff --git a/stetl/main.py b/stetl/main.py
index d680b49..84a15d2 100755
--- a/stetl/main.py
+++ b/stetl/main.py
@@ -11,7 +11,7 @@ from util import Util
from version import __version__
import argparse #apt-get install python-argparse
import inspect
-
+import os
log = Util.get_log('main')
def parse_args():
@@ -25,7 +25,7 @@ def parse_args():
dest='config_section', required=False)
argparser.add_argument('-a ', '--args', type=str,
- help='Arguments to be substituted for {argN}s in config file, as "arg1=foo arg2=bar" etc',
+ help='Arguments or .properties file to be substituted for {argN}s in config file, as -a "arg1=foo arg2=bar" or -a args.properties',
dest='config_args', required=False)
argparser.add_argument('-d ', '--doc', type=str,
@@ -35,8 +35,12 @@ def parse_args():
args = argparser.parse_args()
if args.config_args:
- # Convert string to dict: http://stackoverflow.com/a/1248990
- args.config_args = Util.string_to_dict(args.config_args)
+ if os.path.isfile(args.config_args):
+ log.info('Found args file at: %s' % args.config_args)
+ args.config_args = Util.propsfile_to_dict(args.config_args)
+ else:
+ # Convert string to dict: http://stackoverflow.com/a/1248990
+ args.config_args = Util.string_to_dict(args.config_args)
return args
diff --git a/stetl/outputs/dboutput.py b/stetl/outputs/dboutput.py
index cc768a5..d7bc894 100644
--- a/stetl/outputs/dboutput.py
+++ b/stetl/outputs/dboutput.py
@@ -7,6 +7,7 @@
from stetl.output import Output
from stetl.util import Util
from stetl.packet import FORMAT
+from stetl.component import Config
from stetl.postgis import PostGIS
log = Util.get_log('dboutput')
@@ -31,7 +32,43 @@ class PostgresDbOutput(DbOutput):
consumes=FORMAT.string
"""
-
+ # Start attribute config meta
+ @Config(ptype=str, required=True, default=None)
+ def database(self):
+ """
+ Database name.
+ """
+ pass
+
+ @Config(ptype=str, required=False, default=None)
+ def user(self):
+ """
+ DB User name.
+ """
+ pass
+
+ @Config(ptype=str, required=False, default=None)
+ def password(self):
+ """
+ DB Password for user.
+ """
+ pass
+
+ @Config(ptype=str, required=False, default=None)
+ def host(self):
+ """
+ Hostname for DB.
+ """
+ pass
+
+ @Config(ptype=str, required=False, default='public')
+ def schema(self):
+ """
+ Postgres schema name for DB.
+ """
+ pass
+
+ # End attribute config meta
def __init__(self, configdict, section):
DbOutput.__init__(self, configdict, section, consumes=FORMAT.string)
@@ -50,15 +87,41 @@ class PostgresInsertOutput(PostgresDbOutput):
Output by inserting single record into Postgres database.
Input is a record (Python dic structure) or a Python list of dicts (records).
Creates an INSERT for Postgres to insert each single record.
+ When the "replace" parameter is True, any existing record keyed by "key" is
+ attempted to be deleted first.
+
+ NB a constraint is that each record needs to contain all values as
+ an INSERT query is built once for the columns in the first record.
consumes=FORMAT.record
"""
+ # Start attribute config meta
+ @Config(ptype=str, required=False, default='public')
+ def table(self):
+ """
+ Table for inserts.
+ """
+ pass
+
+ @Config(ptype=bool, required=False, default=False)
+ def replace(self):
+ """
+ Replace record if exists?
+ """
+ pass
+
+ @Config(ptype=str, required=False, default=None)
+ def key(self):
+ """
+ The key column name of the table, required when replacing records.
+ """
+ pass
+ # End attribute config meta
def __init__(self, configdict, section, consumes=FORMAT.record):
DbOutput.__init__(self, configdict, section, consumes=[FORMAT.record_array, FORMAT.record])
self.query = None
self.db = None
- self.key = self.cfg.get('key')
def init(self):
# Connect only once to DB
@@ -79,6 +142,16 @@ class PostgresInsertOutput(PostgresDbOutput):
log.info('query is %s', query)
return query
+ def insert(self, record):
+ if self.replace and self.key and self.key in record:
+ # Try to delete (replace option)
+ del_query = "DELETE FROM %s WHERE %s = '%s'" % (self.cfg.get('table'), self.key, record[self.key])
+ self.db.execute(del_query)
+
+ # Do insert with values from the record dict
+ self.db.execute(self.query, record.values())
+ self.db.commit(close=False)
+
def write(self, packet):
# Deal with empty or zero-length data structures (list or dict)
if packet.data is None or len(packet.data) == 0:
@@ -101,18 +174,16 @@ class PostgresInsertOutput(PostgresDbOutput):
# Check if record is single (dict) or array (list of dict)
if type(record) is dict:
# Do insert with values from the single record
- self.db.execute(self.query, record.values())
- self.db.commit(close=False)
+ self.insert(record)
# log.info('committed record key=%s' % record[self.key])
elif type(record) is list:
- # Multiple records in list
- for rec in record:
- # Do insert with values from the record
- self.db.execute(self.query, rec.values())
- self.db.commit(close=False)
+ # Multiple records in list
+ for rec in record:
+ # Do insert with values from the record
+ self.insert(rec)
- log.info('committed %d records' % len(record))
+ log.info('committed %d records' % len(record))
return packet
diff --git a/stetl/outputs/execoutput.py b/stetl/outputs/execoutput.py
new file mode 100644
index 0000000..1fe7b55
--- /dev/null
+++ b/stetl/outputs/execoutput.py
@@ -0,0 +1,209 @@
+# -*- coding: utf-8 -*-
+#
+# Output classes for ETL, executing commands.
+#
+# Author: Frank Steggink
+#
+import subprocess
+import os
+import shutil
+from stetl.component import Config
+from stetl.output import Output
+from stetl.util import Util
+from stetl.packet import FORMAT
+
+log = Util.get_log('execoutput')
+
+
+class ExecOutput(Output):
+ """
+ Executes any command (abstract base class).
+ """
+
+ def __init__(self, configdict, section, consumes):
+ Output.__init__(self, configdict, section, consumes)
+
+ def write(self, packet):
+ return packet
+
+ def execute_cmd(self, cmd):
+ use_shell = True
+ if os.name == 'nt':
+ use_shell = False
+
+ log.info("executing cmd=%s" % cmd)
+ subprocess.call(cmd, shell=use_shell)
+ log.info("execute done")
+
+
+class Ogr2OgrExecOutput(ExecOutput):
+ """
+ Executes an Ogr2Ogr command.
+ Input is a file name to be processed.
+ Output by calling Ogr2Ogr command.
+
+ consumes=FORMAT.string
+ """
+
+ # Start attribute config meta
+ # Applying Decorator pattern with the Config class to provide
+ # read-only config values from the configured properties.
+
+ @Config(ptype=str, default=None, required=True)
+ def dest_data_source(self):
+ """
+ String denoting the OGR data destination. Usually a path to a file like "path/rivers.shp" or connection string
+ to PostgreSQL like "PG: host=localhost dbname='rivers' user='postgres'".
+
+ Required: True
+
+ Default: None
+ """
+ pass
+
+ @Config(ptype=str, default=None, required=False)
+ def dest_format(self):
+ """
+ Instructs GDAL to use driver by that name to open data destination. Not required for
+ many standard formats that are self-describing like ESRI Shapefile.
+
+ Examples: 'PostgreSQL', 'GeoJSON' etc
+
+ Required: False
+
+ Default: None
+ """
+ pass
+
+ @Config(ptype=list, default=[], required=False)
+ def lco(self):
+ """
+ Options for newly created layer (-lco).
+
+ Type: list
+
+ Required: False
+
+ Default: []
+ """
+
+ pass
+
+ @Config(ptype=list, default=None, required=False)
+ def spatial_extent(self):
+ """
+ Spatial extent (-spat), to pass as xmin ymin xmax ymax
+
+ Type: list
+
+ Required: False
+
+ Default: []
+ """
+ pass
+
+ @Config(ptype=str, default=None, required=False)
+ def gfs_template(self):
+ """
+ Name of GFS template file to use during loading. Passed to ogr2ogr as
+ --config GML_GFS_TEMPLATE <name>
+
+ Required: False
+
+ Default: None
+ """
+ pass
+
+ @Config(ptype=list, default=None, required=False)
+ def options(self):
+ """
+ Miscellaneous options to pass to ogr2ogr.
+
+ Required: False
+
+ Default: None
+ """
+ pass
+
+ @Config(ptype=bool, default=False, required=False)
+ def cleanup_input(self):
+ """
+ Flag to indicate whether the input file to ogr2ogr should be cleaned up.
+
+ Type: boolean
+
+ Required: False
+
+ Default: False
+ """
+
+ pass
+
+ # End attribute config meta
+
+ def __init__(self, configdict, section):
+ ExecOutput.__init__(self, configdict, section, consumes=FORMAT.string)
+
+ # For creating tables the GFS file needs to be newer than
+ # the .gml file. -lco GML_GFS_TEMPLATE somehow does not work
+ # so we copy the .gfs file each time with the .gml file with
+ # the same base name
+ self.lco = self.cfg.get('lco')
+ self.gfs_template = self.cfg.get('gfs_template')
+ spatial_extent = self.cfg.get('spatial_extent')
+ options = self.cfg.get('options')
+
+ dest_format = self.cfg.get('dest_format')
+ dest_data_source = self.cfg.get('dest_data_source')
+
+ if not dest_format:
+ raise ValueError('Verplichte parameter dest_format ontbreekt')
+ if not dest_data_source:
+ raise ValueError('Verplichte parameter dest_data_source ontbreekt')
+
+ self.ogr2ogr_cmd = 'ogr2ogr -f ' + dest_format + ' ' + dest_data_source
+
+ if spatial_extent:
+ self.ogr2ogr_cmd += ' -spat ' + spatial_extent
+ if options:
+ self.ogr2ogr_cmd += ' ' + options
+ self.cleanup_input = self.cfg.get('cleanup_input')
+
+ self.first_run = True
+
+ def write(self, packet):
+ if packet.data is None:
+ return packet
+
+ # Execute ogr2ogr
+ ogr2ogr_cmd = self.ogr2ogr_cmd
+ if self.lco and self.first_run is True:
+ ogr2ogr_cmd += ' ' + self.lco
+ self.first_run = False
+
+ import os.path
+
+ if type(packet.data) is list:
+ for item in packet.data:
+ self.execute(ogr2ogr_cmd, item)
+ else:
+ self.execute(ogr2ogr_cmd, packet.data)
+
+ return packet
+
+ def execute(self, ogr2ogr_cmd, file_path):
+ # Copy the .gfs file if required, use the same base name
+ # so ogr2ogr will pick it up.
+ if self.gfs_template:
+ file_ext = os.path.splitext(file_path)
+ gfs_path = file_ext[0] + '.gfs'
+ shutil.copy(self.gfs_template, gfs_path)
+
+ # Append file name to command as last argument
+ self.execute_cmd(ogr2ogr_cmd + ' ' + file_path)
+
+ if self.cleanup_input:
+ os.remove(file_path)
+ if self.gfs_template:
+ os.remove(gfs_path)
+
diff --git a/stetl/packet.py b/stetl/packet.py
index 02ef49f..280771c 100644
--- a/stetl/packet.py
+++ b/stetl/packet.py
@@ -70,6 +70,7 @@ class Packet:
class FORMAT:
none = 'none'
xml_line_stream = 'xml_line_stream'
+ line_stream = 'line_stream'
etree_doc = 'etree_doc'
etree_element = 'etree_element'
etree_feature_array = 'etree_feature_array'
diff --git a/stetl/util.py b/stetl/util.py
index b326923..c1f031e 100755
--- a/stetl/util.py
+++ b/stetl/util.py
@@ -6,6 +6,7 @@
import logging, os, glob, sys, types
from time import *
+from ConfigParser import ConfigParser
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(name)s %(levelname)s %(message)s')
@@ -100,6 +101,35 @@ class Util:
return dict(dict_arr)
+ # Convert a dict to a string
+ @staticmethod
+ def dict_to_string(dict_v):
+ # Convert/flatten dict to string http://codereview.stackexchange.com/questions/7953/how-do-i-flatten-a-dictionary-into-a-string
+ return ' '.join("%s=%r" % (key, val) for (key, val) in dict_v.iteritems())
+
+ # Convert a properties file to a dict
+ @staticmethod
+ def propsfile_to_dict(file_path):
+ # See http://stackoverflow.com/questions/2819696/parsing-properties-file-in-python
+ # Need a [section] header to parse .ini files, so fake it!
+ class FakeSecHead(object):
+ def __init__(self, fp):
+ self.fp = fp
+ self.sechead = '[asection]\n'
+
+ def readline(self):
+ if self.sechead:
+ try:
+ return self.sechead
+ finally:
+ self.sechead = None
+ else:
+ return self.fp.readline()
+
+ cp = ConfigParser()
+ cp.readfp(FakeSecHead(open(file_path)))
+ return cp._sections['asection']
+
@staticmethod
def elem_to_dict(elem, strip_space=True, strip_ns=True, sub=False, attr_prefix='', gml2ogr=True, ogr2json=True):
"""Convert an Element into an internal dictionary (not JSON!)."""
diff --git a/stetl/version.py b/stetl/version.py
index e13bd59..39e0411 100644
--- a/stetl/version.py
+++ b/stetl/version.py
@@ -1 +1 @@
-__version__ = "1.0.8"
+__version__ = "1.0.9"
diff --git a/tests/__init__.pyc b/tests/__init__.pyc
new file mode 100644
index 0000000..7b7264c
Binary files /dev/null and b/tests/__init__.pyc differ
diff --git a/tests/configs/copy_in_out.cfg b/tests/configs/copy_in_out.cfg
new file mode 100644
index 0000000..8d9fc59
--- /dev/null
+++ b/tests/configs/copy_in_out.cfg
@@ -0,0 +1,11 @@
+# Trivial example of an ETL Chain that just copies a file to standard output.
+
+[etl]
+chains = input_xml_file|output_std
+
+[input_xml_file]
+class = inputs.fileinput.XmlFileInput
+file_path = tests/data/cities.xml
+
+[output_std]
+class = outputs.standardoutput.StandardOutput
diff --git a/tests/data/cities.xml b/tests/data/cities.xml
new file mode 100644
index 0000000..58721f0
--- /dev/null
+++ b/tests/data/cities.xml
@@ -0,0 +1,18 @@
+<?xml version='1.0' encoding='utf-8'?>
+<cities>
+ <city>
+ <name>Amsterdam</name>
+ <lat>52.4</lat>
+ <lon>4.9</lon>
+ </city>
+ <city>
+ <name>Bonn</name>
+ <lat>50.7</lat>
+ <lon>7.1</lon>
+ </city>
+ <city>
+ <name>Rome</name>
+ <lat>41.9</lat>
+ <lon>12.5</lon>
+ </city>
+</cities>
diff --git a/tests/test_chain.pyc b/tests/test_chain.pyc
new file mode 100644
index 0000000..8a3c06f
Binary files /dev/null and b/tests/test_chain.pyc differ
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-grass/python-stetl.git
More information about the Pkg-grass-devel
mailing list