[Pkg-javascript-commits] [node-leveldown] 52/492: KeyStream and ValueStream, version bump to 0.1.0
Andrew Kelley
andrewrk-guest at moszumanska.debian.org
Sun Jul 6 17:13:43 UTC 2014
This is an automated email from the git hooks/post-receive script.
andrewrk-guest pushed a commit to annotated tag rocksdb-0.10.1
in repository node-leveldown.
commit 522cc028bdb062cea3ef5a8a7740f4608047de5d
Author: Rod Vagg <rod at vagg.org>
Date: Fri Sep 28 16:48:43 2012 +1000
KeyStream and ValueStream, version bump to 0.1.0
---
README.md | 60 +++++++++++++++++++--
lib/errors.js | 2 +-
lib/levelup.js | 8 +++
lib/read-stream.js | 30 ++++++++---
lib/write-stream.js | 2 +-
package.json | 2 +-
src/iterator.cc | 22 ++++++--
src/iterator.h | 6 +++
test/key-value-streams-test.js | 117 +++++++++++++++++++++++++++++++++++++++++
9 files changed, 231 insertions(+), 18 deletions(-)
diff --git a/README.md b/README.md
index 61701ee..fac805e 100644
--- a/README.md
+++ b/README.md
@@ -40,6 +40,7 @@ levelup('./mydb', options, function (err, db) {
})
```
+
### Options
`levelup()` takes an optional options object as its second argument; the following properties are accepted:
@@ -57,6 +58,7 @@ levelup('./mydb', options, function (err, db) {
Additionally, each of the main interface methods accept an optional options object that can be used to override `encoding` (or `keyEncoding` & `valueEncoding`).
+
### Batch operations
For faster write operations, the `batch()` method can be used to submit an array of operations to be executed sequentially. Each operation is contained in an object having the following properties: `type`, `key`, `value`, where the *type* is either `'put'` or `'del'`. In the case of `'del'` the `'value'` property is ignored.
@@ -79,6 +81,7 @@ db.batch(ops, function (err) {
Streams
-------
+
### ReadStream
You can obtain a **ReadStream** of the full database by calling the `readStream()` method. The resulting stream is a complete Node.js-style [Readable Stream](http://nodejs.org/docs/latest/api/stream.html#stream_readable_stream) where `'data'` events emit objects with `'key'` and `'value'` pairs.
@@ -109,6 +112,51 @@ Additionally, you can supply an options object as the first parameter to `readSt
* `'reverse'`: a boolean, set to true if you want the stream to go in reverse order. Beware that due to the way LevelDB works, a reverse seek will be slower than a forward seek.
+* `'keys'`: a boolean (defaults to `true`) to indicate whether the `'data'` event should contain keys. If set to `true` and `'values'` set to `false` then `'data'` events will simply be keys, rather than objects with a `'key'` property. Used internally by the `keyStream()` method.
+
+* `'values'`: a boolean (defaults to `true`) to indicate whether the `'data'` event should contain values. If set to `true` and `'keys'` set to `false` then `'data'` events will simply be values, rather than objects with a `'value'` property. Used internally by the `valueStream()` method.
+
+
+### KeyStream
+
+A **KeyStream** is a **ReadStream** where the `'data'` events are simply the keys from the database so it can be used like a traditional stream rather than an object stream.
+
+You can obtain a KeyStream either by calling the `keyStream()` method on a LevelUP object or by passing passing an options object to `readStream()` with `keys` set to `true` and `values` set to `false`.
+
+```js
+db.keyStream()
+ .on('data', function (data) {
+ console.log('key=', data)
+ })
+
+// same as:
+db.readStream({ keys: true, values: false })
+ .on('data', function (data) {
+ console.log('key=', data)
+ })
+```
+
+
+### ValueStream
+
+A **ValueStream** is a **ReadStream** where the `'data'` events are simply the values from the database so it can be used like a traditional stream rather than an object stream.
+
+You can obtain a ValueStream either by calling the `valueStream()` method on a LevelUP object or by passing passing an options object to `readStream()` with `valuess` set to `true` and `keys` set to `false`.
+
+```js
+db.valueStream()
+ .on('data', function (data) {
+ console.log('value=', data)
+ })
+
+// same as:
+db.readStream({ keys: false, values: true })
+ .on('data', function (data) {
+ console.log('value=', data)
+ })
+```
+
+
### WriteStream
A **WriteStream** can be obtained by calling the `writeStream()` method. The resulting stream is a complete Node.js-style [Writable Stream](http://nodejs.org/docs/latest/api/stream.html#stream_writable_stream) which accepts objects with `'key'` and `'value'` pairs on its `write()` method. Tce WriteStream will buffer writes and submit them as a `batch()` operation where the writes occur on the same event loop tick, otherwise they are treated as simple `put()` operations.
@@ -130,7 +178,8 @@ db.writeStream()
The standard `write()`, `end()`, `destroy()` and `destroySoon()` methods are implemented on the WriteStream. `'drain'`, `'error'`, `'close'` and `'pipe'` events are emitted.
-### Pipes and compatibility
+
+### Pipes and Node Stream compatibility
A ReadStream can be piped directly to a WriteStream, allowing for easy copying of an entire database. A simple `copy()` operation is included in LevelUP that performs exactly this on two open databases:
@@ -142,6 +191,8 @@ function copy (srcdb, dstdb, callback) {
The ReadStream is also [fstream](https://github.com/isaacs/fstream)-compatible which means you should be able to pipe to and from fstreams. So you can serialize and deserialize an entire database to a directory where keys are filenames and values are their contents, or even into a *tar* file using [node-tar](https://github.com/isaacs/node-tar). See the [fstream functional test](https://github.com/rvagg/node-levelup/blob/master/test/functional/fstream-test.js) for an example. *(Note: I'm [...]
+KeyStreams and ValueStreams can be treated like standard streams of raw data. If `'encoding'` is set to `'binary'` the `'data'` events will simply be standard Node `Buffer` objects straight out of the data store.
+
JSON
----
@@ -155,13 +206,12 @@ Important considerations
TODO
----
-* Filter streams, e.g.: KeyReadStream, ValueReadStream
-* *Windows support, maybe*
+* Windows support (see [issue #5](https://github.com/rvagg/node-levelup/issues/5) if you would like to help)
* Benchmarks
Licence & copyright
-------------------
-LevelUP is Copyright (c) 2012 Rod Vagg <[@rvagg](https://twitter.com/rvagg)> and licenced under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.
+LevelUP is Copyright (c) 2012 Rod Vagg [@rvagg](https://twitter.com/rvagg) and licenced under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.
-LevelUP builds on the excellent work of the LevelDB and Snappy teams from Google and additional contributors. LevelDB and Snappy are both issued under the [New BSD Licence](http://opensource.org/licenses/BSD-3-Clause).
+LevelUP builds on the excellent work of the LevelDB and Snappy teams from Google and additional contributors. LevelDB and Snappy are both issued under the [New BSD Licence](http://opensource.org/licenses/BSD-3-Clause).
\ No newline at end of file
diff --git a/lib/errors.js b/lib/errors.js
index b910908..3d2569b 100644
--- a/lib/errors.js
+++ b/lib/errors.js
@@ -2,7 +2,7 @@
var errno = require('errno')
- , LevelUPError = errno.custom.createError("LevelUPError")
+ , LevelUPError = errno.custom.createError('LevelUPError')
module.exports = {
LevelUPError : LevelUPError
diff --git a/lib/levelup.js b/lib/levelup.js
index d053a97..4f63a38 100644
--- a/lib/levelup.js
+++ b/lib/levelup.js
@@ -216,6 +216,14 @@ LevelUP.prototype = {
)
}
+ , keyStream: function (options) {
+ return this.readStream(extend(options || {}, { keys: true, values: false }))
+ }
+
+ , valueStream: function (options) {
+ return this.readStream(extend(options || {}, { keys: false, values: true }))
+ }
+
, writeStream: function (options) {
return writeStream.create(
options || {}
diff --git a/lib/read-stream.js b/lib/read-stream.js
index 344049f..c45a84d 100644
--- a/lib/read-stream.js
+++ b/lib/read-stream.js
@@ -1,13 +1,28 @@
/* Copyright (c) 2012 Rod Vagg <@rvagg> */
-var Stream = require("stream").Stream
+var Stream = require('stream').Stream
, BufferStream = require('bufferstream')
, toEncoding = require('./util').toEncoding
, toBuffer = require('./util').toBuffer
, extend = require('./util').extend
- , defaultOptions = {}
+ , defaultOptions = { keys: true, values: true }
+
+ , makeKeyValueData = function (key, value) {
+ return {
+ key: toEncoding(key, this._options.keyEncoding || this._options.encoding)
+ , value: toEncoding(value, this._options.valueEncoding || this._options.encoding)
+ }
+ }
+ , makeKeyData = function (key) {
+ return toEncoding(key, this._options.keyEncoding || this._options.encoding)
+ }
+ , makeValueData = function (key, value) {
+ return toEncoding(value, this._options.valueEncoding || this._options.encoding)
+ }
+ , makeNoData = function () { return null }
+
function ReadStream (options, db, iteratorFactory) {
this.__proto__.__proto__ = Stream.prototype
@@ -24,6 +39,12 @@ function ReadStream (options, db, iteratorFactory) {
if (this._options.end)
this._options.end = toBuffer(this._options.end)
+ this._makeData = this._options.keys && this._options.values
+ ? makeKeyValueData.bind(this) : this._options.keys
+ ? makeKeyData.bind(this) : this._options.values
+ ? makeValueData.bind(this) : makeNoData
+
+
var ready = function () {
if (this._status == 'ended')
return
@@ -96,10 +117,7 @@ ReadStream.prototype = {
if (/^reading/.test(this._status))
this._status = this._status.replace(/^reading/, 'ready')
this._read()
- this.emit(this._dataEvent, {
- key : toEncoding(key , this._options.keyEncoding || this._options.encoding)
- , value : toEncoding(value , this._options.valueEncoding || this._options.encoding)
- })
+ this.emit(this._dataEvent, this._makeData(key, value))
}
, _cleanup: function (err) {
diff --git a/lib/write-stream.js b/lib/write-stream.js
index 7eb2685..0eeece9 100644
--- a/lib/write-stream.js
+++ b/lib/write-stream.js
@@ -1,6 +1,6 @@
/* Copyright (c) 2012 Rod Vagg <@rvagg> */
-var Stream = require("stream").Stream
+var Stream = require('stream').Stream
, concatStream = require('concat-stream')
, extend = require('./util').extend
diff --git a/package.json b/package.json
index 1c10798..0a7e7e2 100644
--- a/package.json
+++ b/package.json
@@ -15,7 +15,7 @@
, "storage"
, "json"
]
- , "version" : "0.0.5-1"
+ , "version" : "0.1.0"
, "main" : "lib/levelup.js"
, "dependencies" : {
"errno" : "~0.0.3"
diff --git a/src/iterator.cc b/src/iterator.cc
index 1c46563..0dcafd6 100644
--- a/src/iterator.cc
+++ b/src/iterator.cc
@@ -18,6 +18,8 @@ using namespace levelup;
LU_OPTION ( start );
LU_OPTION ( end );
LU_OPTION ( reverse );
+LU_OPTION ( keys );
+LU_OPTION ( values );
bool levelup::Iterator::GetIterator () {
if (dbIterator == NULL) {
@@ -46,9 +48,13 @@ bool levelup::Iterator::IteratorNext (string& key, string& value) {
&& (end == NULL
|| (reverse && end->compare(dbIterator->key().ToString()) <= 0)
|| (!reverse && end->compare(dbIterator->key().ToString()) >= 0))) {
- key.assign(dbIterator->key().data(), dbIterator->key().size());
- value.assign(dbIterator->value().data(), dbIterator->value().size());
+
+ if (keys)
+ key.assign(dbIterator->key().data(), dbIterator->key().size());
+ if (values)
+ value.assign(dbIterator->value().data(), dbIterator->value().size());
return true;
+
} else {
return false;
}
@@ -132,7 +138,15 @@ Handle<Value> levelup::Iterator::New (const Arguments& args) {
if (args[1]->ToObject()->Has(option_reverse)) {
reverse = args[1]->ToObject()->Get(option_reverse)->BooleanValue();
}
- Iterator* iterator = new Iterator(database, start, end, reverse);
+ bool keys = true;
+ if (args[1]->ToObject()->Has(option_keys)) {
+ keys = args[1]->ToObject()->Get(option_keys)->BooleanValue();
+ }
+ bool values = true;
+ if (args[1]->ToObject()->Has(option_values)) {
+ values = args[1]->ToObject()->Get(option_values)->BooleanValue();
+ }
+ Iterator* iterator = new Iterator(database, start, end, reverse, keys, values);
iterator->Wrap(args.This());
return args.This();
@@ -140,5 +154,5 @@ Handle<Value> levelup::Iterator::New (const Arguments& args) {
Handle<Value> levelup::CreateIterator (const Arguments& args) {
HandleScope scope;
- return scope.Close(levelup::Iterator::NewInstance(args));
+ return scope.Close(levelup::Iterator::NewInstance(args));
}
diff --git a/src/iterator.h b/src/iterator.h
index 6dcc4a8..968a2e2 100644
--- a/src/iterator.h
+++ b/src/iterator.h
@@ -32,10 +32,14 @@ private:
, Slice* start
, string* end
, bool reverse
+ , bool keys
+ , bool values
) : database(database)
, start(start)
, end(end)
, reverse(reverse)
+ , keys(keys)
+ , values(values)
{
options = new ReadOptions();
dbIterator = NULL;
@@ -55,6 +59,8 @@ private:
Slice* start;
string* end;
bool reverse;
+ bool keys;
+ bool values;
bool GetIterator ();
diff --git a/test/key-value-streams-test.js b/test/key-value-streams-test.js
new file mode 100644
index 0000000..c1ba436
--- /dev/null
+++ b/test/key-value-streams-test.js
@@ -0,0 +1,117 @@
+/* Copyright (c) 2012 Rod Vagg <@rvagg> */
+
+var buster = require('buster')
+ , assert = buster.assert
+ , common = require('./common')
+
+buster.testCase('Key and Value Streams', {
+ 'setUp': function () {
+ common.commonSetUp.call(this)
+
+ this.readySpy = this.spy()
+ this.dataSpy = this.spy()
+ this.endSpy = this.spy()
+ this.sourceData = []
+
+ for (var i = 0; i < 100; i++) {
+ var k = (i < 10 ? '0' : '') + i
+ this.sourceData.push({
+ type : 'put'
+ , key : k
+ , value : Math.random()
+ })
+ }
+
+ this.sourceKeys = Object.keys(this.sourceData)
+ .map(function (k) { return this.sourceData[k].key }.bind(this))
+ this.sourceValues = Object.keys(this.sourceData)
+ .map(function (k) { return this.sourceData[k].value }.bind(this))
+
+ this.verify = function (rs, data, done) {
+ assert.isFalse(rs.writable)
+ assert.isFalse(rs.readable)
+ assert.equals(this.readySpy.callCount, 1, 'Stream emitted single "ready" event')
+ assert.equals(this.endSpy.callCount, 1, 'Stream emitted single "end" event')
+ assert.equals(this.dataSpy.callCount, data.length, 'Stream emitted correct number of "data" events')
+ data.forEach(function (d, i) {
+ var call = this.dataSpy.getCall(i)
+ if (call) {
+ //console.log('call', i, ':', call.args[0].key, '=', call.args[0].value, '(expected', d.key, '=', d.value, ')')
+ assert.equals(call.args.length, 1, 'Stream "data" event #' + i + ' fired with 1 argument')
+ assert.equals(call.args[0], d, 'Stream correct "data" event #' + i + ': ' + d)
+ }
+ }.bind(this))
+ done()
+ }.bind(this)
+ }
+
+ , 'tearDown': common.commonTearDown
+
+ , 'test .keyStream()': function (done) {
+ this.openTestDatabase(function (db) {
+ // execute
+ db.batch(this.sourceData.slice(), function (err) {
+ refute(err)
+
+ var rs = db.keyStream()
+ assert.isFalse(rs.writable)
+ assert.isTrue(rs.readable)
+ rs.on('ready', this.readySpy)
+ rs.on('data', this.dataSpy)
+ rs.on('end', this.endSpy)
+ rs.on('close', this.verify.bind(this, rs, this.sourceKeys, done))
+ }.bind(this))
+ }.bind(this))
+ }
+
+ , 'test .readStream({keys:true,values:false})': function (done) {
+ this.openTestDatabase(function (db) {
+ // execute
+ db.batch(this.sourceData.slice(), function (err) {
+ refute(err)
+
+ var rs = db.readStream({ keys: true, values: false })
+ assert.isFalse(rs.writable)
+ assert.isTrue(rs.readable)
+ rs.on('ready', this.readySpy)
+ rs.on('data', this.dataSpy)
+ rs.on('end', this.endSpy)
+ rs.on('close', this.verify.bind(this, rs, this.sourceKeys, done))
+ }.bind(this))
+ }.bind(this))
+ }
+
+ , 'test .valueStream()': function (done) {
+ this.openTestDatabase(function (db) {
+ // execute
+ db.batch(this.sourceData.slice(), function (err) {
+ refute(err)
+
+ var rs = db.valueStream()
+ assert.isFalse(rs.writable)
+ assert.isTrue(rs.readable)
+ rs.on('ready', this.readySpy)
+ rs.on('data', this.dataSpy)
+ rs.on('end', this.endSpy)
+ rs.on('close', this.verify.bind(this, rs, this.sourceValues, done))
+ }.bind(this))
+ }.bind(this))
+ }
+
+ , 'test .readStream({keys:false,values:true})': function (done) {
+ this.openTestDatabase(function (db) {
+ // execute
+ db.batch(this.sourceData.slice(), function (err) {
+ refute(err)
+
+ var rs = db.readStream({ keys: false, values: true })
+ assert.isFalse(rs.writable)
+ assert.isTrue(rs.readable)
+ rs.on('ready', this.readySpy)
+ rs.on('data', this.dataSpy)
+ rs.on('end', this.endSpy)
+ rs.on('close', this.verify.bind(this, rs, this.sourceValues, done))
+ }.bind(this))
+ }.bind(this))
+ }
+})
\ No newline at end of file
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-javascript/node-leveldown.git
More information about the Pkg-javascript-commits
mailing list