[med-svn] [r-cran-future] 01/06: New upstream version 1.2.0
Andreas Tille
tille at debian.org
Fri Dec 9 21:49:07 UTC 2016
This is an automated email from the git hooks/post-receive script.
tille pushed a commit to branch master
in repository r-cran-future.
commit d9dd184941194186efb081c5b196280c7ce6388b
Author: Andreas Tille <tille at debian.org>
Date: Fri Dec 9 21:53:38 2016 +0100
New upstream version 1.2.0
---
DESCRIPTION | 4 +-
NAMESPACE | 20 +-
NEWS | 24 ++
R/ClusterFuture-class.R | 39 ++-
R/ClusterRegistry.R | 51 +---
R/EagerFuture-class.R | 35 ---
R/Future-class.R | 12 +-
R/FutureError-class.R | 4 +-
R/LazyFuture-class.R | 68 -----
R/UniprocessFuture-class.R | 68 ++++-
R/as.cluster.R | 57 ++++
R/eager.R | 52 ----
R/future.R | 2 +-
R/futureAssign.R | 3 -
R/futureAssign_OP.R | 22 +-
R/futureCall.R | 2 +-
R/lazy.R | 41 ---
R/makeClusterPSOCK.R | 286 +++++++++++++++++++++
R/multicore.R | 8 +-
R/multisession.R | 4 +-
R/nbrOfWorkers.R | 2 +-
R/resolve.R | 6 +-
R/sessionDetails.R | 94 +++++++
R/tweak.R | 6 +-
R/tweak_parallel_PSOCK.R | 149 -----------
R/uniprocess.R | 81 ++++++
R/utils.R | 72 +++++-
R/values.R | 20 +-
R/zzz.R | 2 +-
R/{plan.R => zzz.plan.R} | 64 ++++-
README.md | 98 +++----
cran-comments.md | 48 ++--
incl/eager.R | 22 --
incl/makeClusterPSOCK.R | 119 +++++++++
incl/plan.R | 4 +-
incl/uniprocess.R | 23 ++
inst/vignettes-static/future-1-overview.md.rsp.rsp | 10 +-
man/EagerFuture-class.Rd | 25 --
man/LazyFuture-class.Rd | 25 --
man/UniprocessFuture-class.Rd | 15 +-
man/as.cluster.Rd | 40 +++
man/eager.Rd | 40 ++-
man/future.Rd | 4 +-
man/lazy.Rd | 114 --------
man/makeClusterPSOCK.Rd | 225 ++++++++++++++++
man/multicore.Rd | 2 +-
man/plan.Rd | 6 +-
man/sessionDetails.Rd | 22 ++
man/tweak_parallel_PSOCK.Rd | 68 -----
revdep/README.md | 65 ++---
revdep/checks.rds | Bin 8524 -> 8651 bytes
revdep/problems.md | 59 ++++-
revdep/timing.md | 38 +--
tests/Future-class.R | 5 +-
tests/as.cluster.R | 64 +++++
tests/cluster.R | 32 +++
tests/eager.R | 4 +-
tests/early-signaling.R | 8 +-
tests/futures.R | 2 +-
tests/globals,manual.R | 2 +-
tests/globals,tricky.R | 2 +-
tests/incl/start,load-only.R | 1 +
tests/invalid-owner.R | 22 +-
tests/lazy.R | 2 +-
tests/multicore.R | 4 +-
tests/multisession.R | 2 +-
tests/nbrOfWorkers.R | 2 +-
tests/nested_futures,mc.cores.R | 12 +-
tests/nested_futures.R | 28 +-
tests/plan.R | 70 ++---
tests/rng.R | 2 +-
tests/sessionDetails.R | 20 ++
tests/startup.R | 4 +-
tests/transparent.R | 4 +-
tests/tweak.R | 10 +-
tests/uuid.R | 14 +-
vignettes/future-1-overview.md.rsp | 94 +++----
vignettes/future-2-issues.md.rsp | 4 +-
vignettes/future-3-topologies.md.rsp | 4 +-
79 files changed, 1777 insertions(+), 1011 deletions(-)
diff --git a/DESCRIPTION b/DESCRIPTION
index 83f705e..aafa39b 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -1,9 +1,9 @@
Package: future
-Version: 1.1.1
+Version: 1.2.0
Title: A Future API for R
Imports:
digest,
- globals (>= 0.7.0),
+ globals (>= 0.7.1),
listenv (>= 0.6.0),
parallel,
utils
diff --git a/NAMESPACE b/NAMESPACE
index 157f2aa..1546bac 100644
--- a/NAMESPACE
+++ b/NAMESPACE
@@ -1,6 +1,12 @@
# Generated by roxygen2: do not edit by hand
+S3method("[",sessionDetails)
+S3method(as.cluster,SOCK0node)
+S3method(as.cluster,SOCKnode)
+S3method(as.cluster,cluster)
+S3method(as.cluster,list)
S3method(as.raster,Mandelbrot)
+S3method(c,cluster)
S3method(futures,environment)
S3method(futures,list)
S3method(futures,listenv)
@@ -17,6 +23,10 @@ S3method(nbrOfWorkers,uniprocess)
S3method(plot,Mandelbrot)
S3method(print,Future)
S3method(print,FutureError)
+S3method(print,FutureStrategy)
+S3method(print,FutureStrategyList)
+S3method(print,future)
+S3method(print,sessionDetails)
S3method(resolve,Future)
S3method(resolve,default)
S3method(resolve,environment)
@@ -24,8 +34,8 @@ S3method(resolve,list)
S3method(resolve,listenv)
S3method(resolved,ClusterFuture)
S3method(resolved,Future)
-S3method(resolved,LazyFuture)
S3method(resolved,MulticoreFuture)
+S3method(resolved,UniprocessFuture)
S3method(resolved,default)
S3method(resolved,environment)
S3method(resolved,list)
@@ -37,7 +47,6 @@ S3method(tweak,character)
S3method(tweak,future)
S3method(value,ClusterFuture)
S3method(value,Future)
-S3method(value,LazyFuture)
S3method(value,MulticoreFuture)
S3method(values,Future)
S3method(values,environment)
@@ -61,6 +70,7 @@ export(MulticoreFuture)
export(MultiprocessFuture)
export(MultisessionFuture)
export(UniprocessFuture)
+export(as.cluster)
export(availableCores)
export(backtrace)
export(cluster)
@@ -73,6 +83,8 @@ export(futures)
export(getExpression)
export(getOutput)
export(lazy)
+export(makeClusterPSOCK)
+export(makeNodePSOCK)
export(mandelbrot)
export(mandelbrotTiles)
export(multicore)
@@ -84,6 +96,7 @@ export(remote)
export(resolve)
export(resolved)
export(run)
+export(sessionDetails)
export(supportsMulticore)
export(transparent)
export(tweak)
@@ -106,10 +119,9 @@ importFrom(listenv,map)
importFrom(listenv,parse_env_subset)
importFrom(parallel,clusterCall)
importFrom(parallel,clusterExport)
-importFrom(parallel,makeCluster)
importFrom(parallel,stopCluster)
-importFrom(utils,assignInNamespace)
importFrom(utils,capture.output)
importFrom(utils,file_test)
importFrom(utils,head)
importFrom(utils,object.size)
+importFrom(utils,sessionInfo)
diff --git a/NEWS b/NEWS
index 06ce6d5..20fc4a2 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,30 @@
Package: future
===============
+Version: 1.2.0 [2016-11-12]
+o Added makeClusterPSOCK() - a version of parallel::makePSOCKcluster()
+ that allows for more flexible control of how PSOCK cluster workers
+ are set up and how they are launched and communicated with if running
+ on external machines.
+o Added generic as.cluster() for coercing objects to cluster objects
+ to be used as in plan(cluster, workers = as.cluster(x)). Also added
+ a c() implementation for cluster objects such that multiple cluster
+ objects can be combined into a single one.
+o plan() and plan("list") now prints more user-friendly output.
+o Added sessionDetails() for gathering details of the current R session.
+o ROBUSTNESS: On Unix, internal myInternalIP() tries more alternatives
+ for finding the local IP number.
+o DEPRECATED: %<=% is deprecated. Use %<-% instead. Same for %=>%.
+o BUG FIX: values() for lists and list environments of futures where
+ one or more of the futures resolved to NULL would give an error.
+o BUG FIX: value() for ClusterFuture would give cryptic error message
+ "Error in stop(ex) : bad error message" if the cluster worker had
+ crashed / terminated. Now it will instead give an error message like
+ "Failed to retrieve the value of ClusterFuture from cluster node #1 on
+ 'localhost'. The reason reported was ‘error reading from connection".
+o BUG FIX: Argument 'user' to remote() was ignored (since 1.1.0).
+
+
Version: 1.1.1 [2016-10-10]
o FIX: For the special case where 'remote' futures use
workers = "localhost" they (again) use the exact same R executable
diff --git a/R/ClusterFuture-class.R b/R/ClusterFuture-class.R
index f329fae..420641a 100644
--- a/R/ClusterFuture-class.R
+++ b/R/ClusterFuture-class.R
@@ -50,11 +50,17 @@ ClusterFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, loc
workers <- defaultCluster()
} else if (is.character(workers) || is.numeric(workers)) {
workers <- ClusterRegistry("start", workers=workers, user=user, master=master, revtunnel=revtunnel, homogeneous=homogeneous)
- } else if (!inherits(workers, "cluster")) {
- stop("Argument 'workers' is not of class 'cluster': ", class(workers)[1])
+ } else {
+ workers <- as.cluster(workers)
+ }
+ if (!inherits(workers, "cluster")) {
+ stop("Argument 'workers' is not of class 'cluster': ", paste(sQuote(class(workers)), collapse = ", "))
}
stopifnot(length(workers) > 0)
+ ## Attaching UUID for each cluster connection, unless already done.
+ workers <- addClusterUUIDs(workers)
+
## Attach name to cluster?
name <- attr(workers, "name")
if (is.null(name)) {
@@ -235,7 +241,21 @@ value.ClusterFuture <- function(future, ...) {
}, simpleError = function(ex) ex)
if (inherits(ack, "simpleError")) {
- ex <- FutureError(sprintf("Failed to retrieve the value of %s from cluster node #%d on %s. The reason reported was %s", class(future)[1], node, cl$host, sQuote(ack$message)), call=ack$call, future=future)
+
+ ## If the connection has changed, report on that!
+ msg <- check_connection_uuid(cl[[1]], future = future)
+ if (!is.null(msg)) {
+ on_failure <- getOption("future.cluster.invalidConnection", "error")
+ if (on_failure == "error") {
+ stop(FutureError(msg, future=future))
+ }
+ warning(msg)
+ return(sprintf("EXCEPTIONAL ERROR: %s", msg))
+ }
+
+ msg <- sprintf("Failed to retrieve the value of %s from cluster node #%d on %s. ", class(future)[1], node, sQuote(cl[[1]]$host))
+ msg <- sprintf("%s The reason reported was %s", msg, sQuote(ack$message))
+ ex <- FutureError(msg, call=ack$call, future=future)
stop(ex)
}
stopifnot(isTRUE(ack))
@@ -271,6 +291,7 @@ value.ClusterFuture <- function(future, ...) {
## WORKAROUND: Need to clear cluster worker before garbage collection,
## cf. https://github.com/HenrikBengtsson/Wishlist-for-R/issues/27
+ ## UPDATE: This has been fixed in R (>= 3.3.2) /HB 2016-10-13
clusterCall(cl[1], function() NULL)
clusterCall(cl[1], gc, verbose=FALSE, reset=FALSE)
@@ -337,3 +358,15 @@ requestNode <- function(await, workers, times=getOption("future.wait.times", 600
node
}
+
+## This is needed in order to be able to assert that we later
+## actually work with the same connection. See R-devel thread
+## 'closeAllConnections() can really mess things up' on 2016-10-30
+## (https://stat.ethz.ch/pipermail/r-devel/2016-October/073331.html)
+check_connection_uuid <- function(worker, future, on_failure = "error") {
+ con <- worker$con
+ uuid <- attr(con, "uuid")
+ uuid_now <- uuid_of_connection(con, keep_source = TRUE, must_work = FALSE)
+ if (uuid_now == uuid) return(NULL)
+ sprintf("Failed to retrieve the value of %s from cluster node #%d on %s. The reason is that the socket connection to the worker has been lost. Its original UUID was %s (%s with description %s), but now it is %s (%s with description %s). This suggests that base::closeAllConnections() have been called, for instance via base::sys.save.image() which in turn is called if the R session (pid %s) is forced to terminate.", class(future)[1], future$node, sQuote(worker$host), uuid, sQuote(attr( [...]
+} ## check_connection_uuid()
diff --git a/R/ClusterRegistry.R b/R/ClusterRegistry.R
index 109b2fd..9da090d 100644
--- a/R/ClusterRegistry.R
+++ b/R/ClusterRegistry.R
@@ -1,47 +1,8 @@
-#' @importFrom parallel makeCluster stopCluster
-#' @importFrom utils capture.output
+#' @importFrom parallel stopCluster
ClusterRegistry <- local({
last <- NULL
cluster <- NULL
- .makeCluster <- function(workers, user=NULL, master=NULL, revtunnel=FALSE, ...) {
- if (is.null(workers)) return(NULL)
-
- debug <- getOption("future.debug", FALSE)
- if (debug) mdebug("ClusterRegister:::.makeCluster() ...")
-
- ## HACKS:
- ## 1. Don't pass ssh option `-l <user>` unless `user` is specified
- ## 2. Connect via reverse SSH tunneling.
- if (debug) {
- mdebug("tweak_parallel_PSOCK(user=%s, revtunnel=%s, rshopts=TRUE)",
- is.null(user), revtunnel)
- }
- on.exit(tweak_parallel_PSOCK(reset=TRUE), add=TRUE)
- tweak_parallel_PSOCK(user=is.null(user), revtunnel=revtunnel, rshopts=TRUE)
-
- if (debug) {
- on.exit(suppressMessages(untrace(system)), add=TRUE)
- suppressMessages(
- trace(system, print=FALSE, tracer=quote(message(command)))
- )
- }
-
- if (debug) {
- on.exit(mdebug("ClusterRegister:::.makeCluster() ... DONE"), add=TRUE)
- }
-
- ## This will _not_ pass `master` iff master=NULL
- args <- list(workers, revtunnel=revtunnel, ...)
- args$master <- master
-
- capture.output({
- cluster <- do.call(makeCluster, args=args)
- })
-
- cluster
- }
-
function(action=c("get", "start", "stop"), workers=NULL, ...) {
action <- match.arg(action)
@@ -56,7 +17,7 @@ ClusterRegistry <- local({
stop("Unknown mode of argument 'workers': ", mode(workers))
}
- if (is.null(cluster) && action != "stop") {
+ if (length(cluster) == 0L && action != "stop") {
cluster <<- .makeCluster(workers, ...)
last <<- workers
}
@@ -71,7 +32,7 @@ ClusterRegistry <- local({
last <<- workers
}
} else if (action == "stop") {
- if (!is.null(cluster)) try(stopCluster(cluster), silent=TRUE)
+ if (length(cluster) > 0L) try(stopCluster(cluster), silent=TRUE)
cluster <<- NULL
last <<- NULL
}
@@ -79,3 +40,9 @@ ClusterRegistry <- local({
invisible(cluster)
}
}) ## ClusterRegistry()
+
+
+.makeCluster <- function(workers, ...) {
+ if (length(workers) == 0L) return(NULL)
+ makeClusterPSOCK(workers, ...)
+} ## .makeCluster()
diff --git a/R/EagerFuture-class.R b/R/EagerFuture-class.R
deleted file mode 100644
index e5f90f2..0000000
--- a/R/EagerFuture-class.R
+++ /dev/null
@@ -1,35 +0,0 @@
-#' An eager future is a future whose value will be resolved immediately
-#'
-#' @inheritParams UniprocessFuture-class
-#'
-#' @return An object of class \code{EagerFuture}.
-#'
-#' @seealso
-#' To evaluate an expression using "eager future", see function
-#' \code{\link{eager}()}.
-#'
-#' @export
-#' @name EagerFuture-class
-#' @keywords internal
-EagerFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, globals=TRUE, local=TRUE, ...) {
- if (substitute) expr <- substitute(expr)
-
- ## Global objects
- assignToTarget <- (is.list(globals) || inherits(globals, "Globals"))
- gp <- getGlobalsAndPackages(expr, envir=envir, tweak=tweakExpression, globals=globals, resolve=TRUE)
-
- ## Assign?
- if (assignToTarget && length(gp) > 0) {
- target <- new.env(parent=envir)
- globalsT <- gp$globals
- for (name in names(globalsT)) {
- target[[name]] <- globalsT[[name]]
- }
- globalsT <- NULL
- envir <- target
- }
- gp <- NULL
-
- f <- UniprocessFuture(expr=expr, envir=envir, substitute=FALSE, local=local, ...)
- structure(f, class=c("EagerFuture", class(f)))
-}
diff --git a/R/Future-class.R b/R/Future-class.R
index a48fd5c..ba4250c 100644
--- a/R/Future-class.R
+++ b/R/Future-class.R
@@ -46,7 +46,7 @@ Future <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, local=TRUE
core <- new.env(parent=emptyenv())
core$expr <- expr
core$envir <- envir
- core$owner <- uuid()
+ core$owner <- session_uuid(attributes = TRUE)
core$local <- local
core$gc <- gc
core$earlySignal <- earlySignal
@@ -63,7 +63,7 @@ Future <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, local=TRUE
}
-#' @importFrom utils head
+#' @importFrom utils head capture.output
#' @export
print.Future <- function(x, ...) {
class <- class(x)
@@ -93,7 +93,7 @@ print.Future <- function(x, ...) {
if (exists("value", envir=x, inherits=FALSE)) {
cat("Resolved: TRUE\n")
- } else if (inherits(x, "LazyFuture")) {
+ } else if (inherits(x, "UniprocessFuture") && x$lazy) {
## FIXME: Special case; will there every be other cases
## for which we need to support this? /HB 2016-05-03
cat("Resolved: FALSE\n")
@@ -122,12 +122,12 @@ print.Future <- function(x, ...) {
## Checks whether Future is owned by the current process or not
assertOwner <- function(future, ...) {
hpid <- function(uuid) {
- info <- attr(uuid, "info")
+ info <- attr(uuid, "source")
sprintf("%s; pid %d on %s", uuid, info$pid, info$host)
}
- if (!identical(future$owner, uuid())) {
- stop(FutureError(sprintf("Invalid usage of futures: A future whose value has not yet been collected can only be queried by the R process (%s) that created it, not by any other R processes (%s): %s", hpid(future$owner), hpid(uuid()), hexpr(future$expr)), future=future))
+ if (!identical(future$owner, session_uuid(attributes = TRUE))) {
+ stop(FutureError(sprintf("Invalid usage of futures: A future whose value has not yet been collected can only be queried by the R process (%s) that created it, not by any other R processes (%s): %s", hpid(future$owner), hpid(session_uuid()), hexpr(future$expr)), future=future))
}
invisible(future)
diff --git a/R/FutureError-class.R b/R/FutureError-class.R
index 36c9f37..900b3eb 100644
--- a/R/FutureError-class.R
+++ b/R/FutureError-class.R
@@ -44,7 +44,7 @@ print.FutureError <- function(x, ...) {
future <- attr(x, "future")
output <- attr(x, "output")
if (!is.null(future) || !is.null(output)) {
- cat("\n\nDEBUG: BEGIN TROUBLESHOOING HELP\n")
+ cat("\n\nDEBUG: BEGIN TROUBLESHOOTING HELP\n")
if (!is.null(future)) {
cat("Future involved:\n")
@@ -68,7 +68,7 @@ print.FutureError <- function(x, ...) {
cat("\n\n")
}
- cat("DEBUG: END TROUBLESHOOING HELP\n")
+ cat("DEBUG: END TROUBLESHOOTING HELP\n")
}
invisible(x)
diff --git a/R/LazyFuture-class.R b/R/LazyFuture-class.R
deleted file mode 100644
index a449428..0000000
--- a/R/LazyFuture-class.R
+++ /dev/null
@@ -1,68 +0,0 @@
-#' A lazy future is a future whose value will be resolved at the time when it is requested
-#'
-#' @inheritParams UniprocessFuture-class
-#'
-#' @return An object of class \code{LazyFuture}.
-#'
-#' @seealso
-#' To evaluate an expression using "lazy future", see function
-#' \code{\link{lazy}()}.
-#'
-#' @export
-#' @name LazyFuture-class
-#' @keywords internal
-LazyFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, globals=TRUE, local=TRUE, ...) {
- if (substitute) expr <- substitute(expr)
-
- ## Evaluate in a local environment?
- if (local) {
- envir <- new.env(parent=envir)
- } else {
- if (!is.logical(globals) || globals) {
- stop("Non-supported use of lazy futures: Whenever argument 'local' is FALSE, then argument 'globals' must also be FALSE. Lazy future evaluation in the calling environment (local=FALSE) can only be done if global objects are resolved at the same time.")
- }
- }
-
- ## Global objects
- gp <- getGlobalsAndPackages(expr, envir=envir, tweak=tweakExpression, globals=globals, resolve=TRUE)
-
- ## Assign?
- if (length(gp) > 0) {
- target <- new.env(parent=envir)
- globalsT <- gp$globals
- for (name in names(globalsT)) {
- target[[name]] <- globalsT[[name]]
- }
- globalsT <- NULL
- envir <- target
- }
- gp <- NULL
-
- f <- UniprocessFuture(expr=expr, envir=envir, substitute=FALSE, local=local, ...)
- structure(f, class=c("LazyFuture", class(f)))
-}
-
-#' @export
-resolved.LazyFuture <- function(x, ...) {
- ## resolved() for LazyFuture must force value() such that
- ## the future gets resolved. The reason for this is so
- ## that polling is always possible, e.g.
- ## while(!resolved(f)) Sys.sleep(5);
- value(x, signal=FALSE)
- NextMethod("resolved")
-}
-
-#' @export
-value.LazyFuture <- function(future, signal=TRUE, ...) {
- if (future$state == 'created') {
- future <- run(future)
- }
-
- value <- future$value
- if (signal && future$state == 'failed') {
- mdebug("Future state: %s", sQuote(value))
- stop(FutureError(value, future=future))
- }
-
- value
-}
diff --git a/R/UniprocessFuture-class.R b/R/UniprocessFuture-class.R
index dbe72e6..181b180 100644
--- a/R/UniprocessFuture-class.R
+++ b/R/UniprocessFuture-class.R
@@ -1,20 +1,50 @@
#' An uniprocess future is a future whose value will be resolved synchroneously in the current process
#'
#' @inheritParams Future-class
+#' @param lazy If \code{FALSE} (default), then the setup and validation of
+#' global variables are done for eager evaluation, otherwise not.
#' @param \dots Additional named elements passed to \code{\link{Future}()}.
#'
#' @return An object of class \code{UniprocessFuture}.
#'
#' @seealso
#' To evaluate an expression using "uniprocess future", see functions
-#' \code{\link{eager}()} and \code{\link{lazy}()}.
+#' \code{\link{uniprocess}()}.
#'
#' @export
#' @name UniprocessFuture-class
#' @keywords internal
-UniprocessFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, local=TRUE, ...) {
+UniprocessFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, globals=TRUE, local=TRUE, lazy=FALSE, ...) {
if (substitute) expr <- substitute(expr)
- f <- Future(expr=expr, envir=envir, substitute=FALSE, local=local, ...)
+
+ if (lazy) {
+ ## Evaluate in a local environment?
+ if (local) {
+ envir <- new.env(parent=envir)
+ } else {
+ if (!is.logical(globals) || globals) {
+ stop("Non-supported use of lazy uniprocess futures: Whenever argument 'local' is FALSE, then argument 'globals' must also be FALSE. Lazy uniprocess future evaluation in the calling environment (local=FALSE) can only be done if global objects are resolved at the same time.")
+ }
+ }
+ }
+
+ ## Global objects
+ assignToTarget <- (is.list(globals) || inherits(globals, "Globals"))
+ gp <- getGlobalsAndPackages(expr, envir=envir, tweak=tweakExpression, globals=globals, resolve=TRUE)
+
+ ## Assign?
+ if (length(gp) > 0 && (lazy || assignToTarget)) {
+ target <- new.env(parent=envir)
+ globalsT <- gp$globals
+ for (name in names(globalsT)) {
+ target[[name]] <- globalsT[[name]]
+ }
+ globalsT <- NULL
+ envir <- target
+ }
+ gp <- NULL
+
+ f <- Future(expr=expr, envir=envir, substitute=FALSE, local=local, lazy=lazy, ...)
structure(f, class=c("UniprocessFuture", class(f)))
}
@@ -64,3 +94,35 @@ run.UniprocessFuture <- function(future, ...) {
invisible(future)
}
+
+
+
+#' @export
+resolved.UniprocessFuture <- function(x, ...) {
+ if (x$lazy) {
+ ## resolved() for lazy uniprocess futures must force value()
+ ## such that the future gets resolved. The reason for this
+ ## is so that polling is always possible, e.g.
+ ## while(!resolved(f)) Sys.sleep(5);
+ value(x, signal=FALSE)
+ }
+ NextMethod("resolved")
+}
+
+
+#' @rdname UniprocessFuture-class
+#' @export
+EagerFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, globals=TRUE, local=TRUE, lazy=FALSE, ...) {
+ if (substitute) expr <- substitute(expr)
+ f <- UniprocessFuture(expr=expr, envir=envir, substitute=FALSE, globals=globals, local=local, lazy=FALSE, ...)
+ structure(f, class=c("EagerFuture", class(f)))
+}
+
+
+#' @rdname UniprocessFuture-class
+#' @export
+LazyFuture <- function(expr=NULL, envir=parent.frame(), substitute=FALSE, globals=TRUE, local=TRUE, lazy=FALSE, ...) {
+ if (substitute) expr <- substitute(expr)
+ f <- UniprocessFuture(expr=expr, envir=envir, substitute=FALSE, globals=globals, local=local, lazy=TRUE, ...)
+ structure(f, class=c("LazyFuture", class(f)))
+}
diff --git a/R/as.cluster.R b/R/as.cluster.R
new file mode 100644
index 0000000..79fad01
--- /dev/null
+++ b/R/as.cluster.R
@@ -0,0 +1,57 @@
+#' Coerce an object to a cluster object
+#'
+#' @param x An object to be coerced.
+#' @param ... Additional arguments passed to the underlying coercion method.
+#' For \code{c(...)}, the clusters and cluster nodes to be combined.
+#'
+#' @return An object of class \code{cluster}.
+#'
+#' @export
+as.cluster <- function(x, ...) {
+ UseMethod("as.cluster")
+}
+
+#' @rdname as.cluster
+#' @export
+as.cluster.cluster <- function(x, ...) x
+
+#' @rdname as.cluster
+#' @export
+as.cluster.list <- function(x, ...) {
+ x <- lapply(x, FUN = as.cluster, ...)
+ x <- Reduce(c, x)
+}
+
+#' @rdname as.cluster
+#' @export
+as.cluster.SOCKnode <- function(x, ...) {
+ cl <- structure(list(x), class = c("SOCKcluster", "cluster"))
+}
+
+#' @rdname as.cluster
+#' @export
+as.cluster.SOCK0node <- as.cluster.SOCKnode
+
+
+#' @param recursive Not used.
+#'
+#' @return \code{c(...)} combine multiple clusters and / or cluster nodes into one cluster returned as an of class \code{cluster}.
+#'
+#' @rdname as.cluster
+#' @export
+c.cluster <- function(..., recursive = FALSE) {
+ x <- list(...)
+ x <- lapply(x, FUN = as.cluster)
+
+ ## AD HOC: Use common demonator class as new class
+ class <- lapply(x, FUN = class)
+ class <- Reduce(intersect, class)
+ stopifnot(is.element("cluster", class))
+
+ ## Combine list of clusters
+ x <- lapply(x, FUN = unclass)
+ x <- Reduce(c, x)
+ class(x) <- class
+
+ x
+}
diff --git a/R/eager.R b/R/eager.R
deleted file mode 100644
index 967cb3d..0000000
--- a/R/eager.R
+++ /dev/null
@@ -1,52 +0,0 @@
-#' Create an eager future whose value will be resolved immediately
-#'
-#' An eager future is a future that uses eager evaluation, which means
-#' that its \emph{value is computed and resolved immediately}, which is
-#' how regular expressions are evaluated in R.
-#' The only difference to R itself is that globals are validated
-#' by default just as for all other types of futures in this package.
-#'
-#' @inheritParams future
-#' @inheritParams multiprocess
-#' @param local If TRUE, the expression is evaluated such that
-#' all assignments are done to local temporary environment, otherwise
-#' the assignments are done in the calling environment.
-#'
-#' @return An \link{EagerFuture}.
-#'
-#' @example incl/eager.R
-#'
-#' @details
-#' The preferred way to create an eager future is not to call this function
-#' directly, but to register it via \code{\link{plan}(eager)} such that it
-#' becomes the default mechanism for all futures. After this
-#' \code{\link{future}()} and \code{\link{\%<-\%}} will create
-#' \emph{eager futures}.
-#'
-#' @section transparent futures:
-#' Transparent futures are eager futures configured to emulate how R
-#' evaluates expressions as far as possible. For instance, errors and
-#' warnings are signaled immediately and assignments are done to the
-#' calling environment (without \code{local()} as default for all other
-#' types of futures). This makes transparent futures ideal for
-#' troubleshooting, especially when there are errors.
-#'
-#' @aliases transparent
-#' @export transparent
-#' @export
-eager <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE, local=TRUE, earlySignal=FALSE, label=NULL, ...) {
- if (substitute) expr <- substitute(expr)
- local <- as.logical(local)
-
- future <- EagerFuture(expr=expr, envir=envir, substitute=FALSE, globals=globals, local=local, earlySignal=earlySignal, label=label)
- run(future)
-}
-class(eager) <- c("eager", "uniprocess", "future", "function")
-
-
-transparent <- function(expr, envir=parent.frame(), substitute=TRUE, globals=FALSE, local=FALSE, earlySignal=TRUE, label=NULL, ...) {
- if (substitute) expr <- substitute(expr)
- future <- eager(expr, envir=envir, substitute=FALSE, globals=globals, local=local, earlySignal=earlySignal, label=label)
- invisible(future)
-}
-class(transparent) <- c("transparent", "eager", "uniprocess", "future", "function")
diff --git a/R/future.R b/R/future.R
index 598319a..0b70538 100644
--- a/R/future.R
+++ b/R/future.R
@@ -180,7 +180,7 @@
#' @aliases futureCall
#' @export
#' @name future
-future <- function(expr, envir=parent.frame(), substitute=TRUE, evaluator=plan(), ...) {
+future <- function(expr, envir=parent.frame(), substitute=TRUE, evaluator=plan("next"), ...) {
if (substitute) expr <- substitute(expr)
if (!is.function(evaluator)) {
diff --git a/R/futureAssign.R b/R/futureAssign.R
index 11e937a..ddc3abc 100644
--- a/R/futureAssign.R
+++ b/R/futureAssign.R
@@ -5,9 +5,6 @@
#' should be assigned.
#'
#' @rdname future
-#'
-#' @aliases %<-% %->% %<=% %=>%
-#' @export %<-% %->% %<=% %=>%
#' @export
futureAssign <- function(x, value, envir=parent.frame(), assign.env=envir, substitute=TRUE) {
stopifnot(is.character(x), !is.na(x), nzchar(x))
diff --git a/R/futureAssign_OP.R b/R/futureAssign_OP.R
index c1747dd..49355f3 100644
--- a/R/futureAssign_OP.R
+++ b/R/futureAssign_OP.R
@@ -1,20 +1,36 @@
#' @usage x \%<-\% value
#'
#' @rdname future
-#'
#' @aliases %<-% %->% %<=% %=>%
#' @export %<-% %->% %<=% %=>%
-`%<-%` <- `%<=%` <- function(x, value) {
+`%<-%` <- function(x, value) {
+ target <- substitute(x)
+ expr <- substitute(value)
+ envir <- parent.frame(1)
+ futureAssignInternal(target, expr, envir=envir, substitute=FALSE)
+}
+
+`%->%` <- function(value, x) {
+ target <- substitute(x)
+ expr <- substitute(value)
+ envir <- parent.frame(1)
+ futureAssignInternal(target, expr, envir=envir, substitute=FALSE)
+}
+
+## Deprecated
+`%<=%` <- function(x, value) {
target <- substitute(x)
expr <- substitute(value)
envir <- parent.frame(1)
+ .Deprecated("%<-%")
futureAssignInternal(target, expr, envir=envir, substitute=FALSE)
}
-`%->%` <- `%=>%` <- function(value, x) {
+`%=>%` <- function(value, x) {
target <- substitute(x)
expr <- substitute(value)
envir <- parent.frame(1)
+ .Deprecated("%->%")
futureAssignInternal(target, expr, envir=envir, substitute=FALSE)
}
diff --git a/R/futureCall.R b/R/futureCall.R
index 852ea7b..8e05dc9 100644
--- a/R/futureCall.R
+++ b/R/futureCall.R
@@ -4,7 +4,7 @@
#' @rdname future
#'
#' @export
-futureCall <- function(FUN, args=NULL, envir=parent.frame(), globals=TRUE, evaluator=plan(), ...) {
+futureCall <- function(FUN, args=NULL, envir=parent.frame(), globals=TRUE, evaluator=plan("next"), ...) {
stopifnot(is.function(FUN))
stopifnot(is.list(args))
diff --git a/R/lazy.R b/R/lazy.R
deleted file mode 100644
index ce0dfe7..0000000
--- a/R/lazy.R
+++ /dev/null
@@ -1,41 +0,0 @@
-#' Create a lazy future whose value will be resolved at the time when requested
-#'
-#' A lazy future is a future that uses lazy evaluation, which means
-#' that its \emph{value is only computed and resolved at the time when the
-#' value is requested}. This means that the future will not be resolved
-#' if the value is never requested.
-#'
-#' @inheritParams future
-#' @inheritParams multiprocess
-#' @param local If TRUE, the expression is evaluated such that
-#' all assignments are done to local temporary environment, otherwise
-#' the assignments are done in the calling environment.
-#' @param globals If TRUE, global objects are resolved ("frozen") at
-#' the point of time when the future is created, otherwise they are
-#' resolved when the future is resolved.
-#'
-#' @return A \link{LazyFuture}.
-#'
-#' @example incl/lazy.R
-#'
-#' @details
-#' The preferred way to create a lazy future is not to call this function
-#' directly, but to register it via \code{\link{plan}(lazy)} such that it
-#' becomes the default mechanism for all futures. After this
-#' \code{\link{future}()} and \code{\link{\%<-\%}} will create
-#' \emph{lazy futures}.
-#'
-#' @export
-lazy <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE, local=TRUE, earlySignal=FALSE, label=NULL, ...) {
- if (substitute) expr <- substitute(expr)
- local <- as.logical(local)
-
- LazyFuture(expr=expr, envir=envir, local=local, globals=globals, earlySignal=earlySignal, label=label)
-}
-class(lazy) <- c("lazy", "uniprocess", "future", "function")
-
-## WORKAROUND:
-## Avoid lazyeval::print.lazy() being called with print(lazy())
-## https://github.com/HenrikBengtsson/future/issues/52
-class(lazy) <- c("function", class(lazy))
-
diff --git a/R/makeClusterPSOCK.R b/R/makeClusterPSOCK.R
new file mode 100644
index 0000000..4429584
--- /dev/null
+++ b/R/makeClusterPSOCK.R
@@ -0,0 +1,286 @@
+#' Create a Parallel Socket Cluster
+#'
+#' @param workers The host names of workers (as a character vector) or
+#' the number of localhost workers (as a positive integer).
+#' @param makeNode A function that creates a \code{"SOCKnode"}
+#' or \code{"SOCK0node"} object, which represents a connection
+#' to a worker.
+#' @param ... Optional arguments passed to \code{makeNode(workers[i], ..., rank=i)} where \code{i = seq_along{workers}}.
+#' @param verbose If TRUE, informative messages are outputted.
+#'
+#' @return An object of class \code{c("SOCKcluster", "cluster")} consisting
+#' of a list of \code{"SOCKnode"} or \code{"SOCK0node"} workers.
+#'
+#' @details
+#' The \code{makeClusterPSOCK()} function is similar to \code{\link[parallel:makePSOCKcluster]{makePSOCKcluster}} of the \pkg{parallel} package, but provides more flexibility in controlling the setup of the system calls that launch the background R workers and how to connect to external machines.
+#'
+#' @example incl/makeClusterPSOCK.R
+#'
+#' @export
+makeClusterPSOCK <- function(workers, makeNode = makeNodePSOCK, port = c("auto", "random"), ..., verbose = getOption("future.debug", FALSE)) {
+ if (is.numeric(workers)) {
+ if (length(workers) != 1L) {
+ stop("When numeric, argument 'workers' must be a single value: ", length(workers))
+ }
+ workers <- as.integer(workers)
+ if (is.na(workers) || workers < 1L) {
+ stop("Number of 'workers' must be one or greater: ", workers)
+ }
+ workers <- rep("localhost", times = workers)
+ }
+
+ if (is.character(port)) {
+ port <- match.arg(port)
+ if (identical(port, "auto")) {
+ port <- Sys.getenv("R_PARALLEL_PORT", NA_character_)
+ port <- as.integer(port)
+ if (is.na(port)) port <- 11000:11999
+ } else if (identical(port, "random")) {
+ port <- 11000:11999
+ }
+ } else {
+ port <- as.integer(port)
+ }
+ if (length(port) == 0L) {
+ stop("Argument 'post' must be of length one or more: 0")
+ }
+ if (length(port) > 1L) port <- sample(port, size = 1L)
+ if (is.na(port) || port < 0L || port > 65535L) {
+ stop("Invalid port: ", port)
+ }
+
+ cl <- vector("list", length = length(workers))
+ for (ii in seq_along(cl)) {
+ cl[[ii]] <- makeNode(workers[[ii]], port = port, ..., rank = ii, verbose = verbose)
+ }
+ class(cl) <- c("SOCKcluster", "cluster")
+
+ ## Attaching UUID for each cluster connection. This is done because
+ ## https://stat.ethz.ch/pipermail/r-devel/2016-October/073331.html
+ cl <- addClusterUUIDs(cl)
+
+ cl
+} ## makeClusterPSOCK()
+
+
+#' @param worker The host name or IP number of the machine where the worker should run.
+#' @param master The host name or IP number of the master / calling machine, as known to the workers. If NULL (default), then the default is \code{Sys.info()[["nodename"]]} unless \code{worker} is the localhost (\code{"localhost"} or \code{"127.0.0.1"}) or \code{revtunnel = TRUE} in case it is \code{"localhost"}.
+#' @param port The port number of the master used to for communicating with all the workers (via socket connections). If an integer vector of ports, then a random one among those is chosen. If \code{"random"}, then a random port in \code{11000:11999} is chosen. If \code{"auto"} (default), then the default is taken from environment variable \env{R_PARALLEL_PORT}, otherwise \code{"random"} is used.
+#' @param connectTimeout The maximum time (in seconds) allowed for each sockect connection between the master and a worker to be established (defaults to 2 minutes). \emph{See note below on current lack of support on Linux and macOS systems.}
+#' @param timeout The maximum time (in seconds) allowed to pass without the master and a worker communicate with each other (defaults to 30 days).
+#' @param rscript,homogeneous The system command for launching Rscript on the worker. If \code{NULL} (default), the default is \code{"Rscript"} unless \code{homogenenous} is TRUE, which in case it is \code{file.path(R.home("bin"), "Rscript")}. Argument \code{homogenenous} defaults to FALSE, unless \code{master} is the localhost (\code{"localhost"} or \code{"127.0.0.1"}).
+#' @param rscript_args Additional arguments to \code{Rscript} (as a character vector).
+#' @param methods If TRUE, then the \pkg{methods} package is also loaded.
+#' @param useXDR If TRUE, the communication between master and workers, which is binary, will be use big-endian (XDR).
+#' @param outfile Where to direct the \link[base:stdout]{stdout} and \link[base:stderr]{stderr} connection output from the workers.
+#' @param renice A numerical 'niceness' (priority) to set for the worker processes.
+#' @param rank A unique one-based index for each worker (automatically set).
+#' @param rshcmd The command to be run on the master to launch a process on another host. Only applicable if \code{machine} is not localhost.
+#' @param user (optional) The user name to be used when communicating with another host.
+#' @param revtunnel If TRUE, a reverse SSH tunneling is set up for each worker such that the worker R process sets up a socket connect to its local port \code{(port - rank + 1)} which then reaches the master on port \code{port}. If FALSE, then the worker will try to connect directly to port \code{port} on \code{master}.
+#' @param rshopts Additional arguments to \code{rshcmd} (as a character vector).
+#' @param manual If TRUE the workers will need to be run manually.
+#' @param dryrun If TRUE, nothing is set up, but a message suggesting how to launch the worker from the terminal is outputted. This is useful for troubleshooting.
+#'
+#' @return \code{makeNodePSOCK()} returns a
+#' \code{"SOCKnode"} or \code{"SOCK0node"} object
+#' representing an established connection to a worker.
+#'
+#' @details
+#' The default is to use reverse SSH tunnelling for workers
+#' running on other machines. This avoids the complication of
+#' otherwise having to configure port forwarding in firewalls,
+#' which often requires static IP address but which also most
+#' users don't have priviligies to do themselves.
+#' It also has the advantage of not having to know the internal
+#' and / or the public IP address / host name of the master.
+#'
+#' If there is no communication between the master and a
+#' worker within the \code{timeout} limit, then the corresponding
+#' socket connection will be closed automatically. This will
+#' eventually result in an error in code trying to access the
+#' connection.
+#'
+#' @section Connection time out:
+#' Argument \code{connectTimeout} does \emph{not} work properly
+#' on Unix and macOS due to limitation in \R itself. For more details
+#' on this, please R devel thread 'BUG?: On Linux setTimeLimit() fails
+#' to propagate timeout error when it occurs (works on Windows)' on
+#' 2016-10-26 (\url{https://stat.ethz.ch/pipermail/r-devel/2016-October/073309.html}). When used, the timeout will eventually trigger an error, but
+#' it won't happen until the socket connection timeout \code{timeout}
+#' itself happens.
+#'
+#' @rdname makeClusterPSOCK
+#' @export
+makeNodePSOCK <- function(worker = "localhost", master = NULL, port, connectTimeout = 2*60, timeout = 30*24*60*60, rscript = NULL, homogeneous = NULL, rscript_args = NULL, methods = TRUE, useXDR = TRUE, outfile = "/dev/null", renice = NA_integer_, rshcmd = "ssh", user = NULL, revtunnel = TRUE, rshopts = NULL, rank = 1L, manual = FALSE, dryrun = FALSE, verbose = FALSE) {
+ localMachine <- is.element(worker, c("localhost", "127.0.0.1"))
+
+ rshcmd <- as.character(rshcmd)
+ stopifnot(length(rshcmd) >= 1L)
+
+ rshopts <- as.character(rshopts)
+
+ user <- as.character(user)
+ stopifnot(length(user) <= 1L)
+
+ port <- as.integer(port)
+ if (is.na(port) || port < 0L || port > 65535L) {
+ stop("Invalid port: ", port)
+ }
+
+ revtunnel <- as.logical(revtunnel)
+ stopifnot(length(revtunnel) == 1L, !is.na(revtunnel))
+
+ if (is.null(master)) {
+ if (localMachine || revtunnel) {
+ master <- "localhost"
+ } else {
+ master <- Sys.info()[["nodename"]]
+ }
+ }
+ stopifnot(!is.null(master))
+
+ timeout <- as.numeric(timeout)
+ stopifnot(length(timeout) == 1L, !is.na(timeout), is.finite(timeout), timeout >= 0)
+
+ methods <- as.logical(methods)
+ stopifnot(length(methods) == 1L, !is.na(methods))
+
+ if (is.null(homogeneous)) {
+ homogeneous <- is.element(master, c("localhost", "127.0.0.1"))
+ }
+ homogeneous <- as.logical(homogeneous)
+ stopifnot(length(homogeneous) == 1L, !is.na(homogeneous))
+
+ if (is.null(rscript)) {
+ rscript <- "Rscript"
+ if (homogeneous) rscript <- file.path(R.home("bin"), rscript)
+ }
+ rscript <- as.character(rscript)
+ stopifnot(length(rscript) >= 1L)
+
+ rscript_args <- as.character(rscript_args)
+
+ useXDR <- as.logical(useXDR)
+ stopifnot(length(useXDR) == 1L, !is.na(useXDR))
+
+ renice <- as.integer(renice)
+ stopifnot(length(renice) == 1L)
+
+ rank <- as.integer(rank)
+ stopifnot(length(rank) == 1L, !is.na(rank))
+
+ manual <- as.logical(manual)
+ stopifnot(length(manual) == 1L, !is.na(manual))
+
+ dryrun <- as.logical(dryrun)
+ stopifnot(length(dryrun) == 1L, !is.na(dryrun))
+
+ verbose <- as.logical(verbose)
+ stopifnot(length(verbose) == 1L, !is.na(verbose))
+
+ ## .slaveRSOCK() command already specified?
+ if (!any(grepl("parallel:::.slaveRSOCK()", rscript_args, fixed = TRUE))) {
+ rscript_args <- c(rscript_args, "-e", shQuote("parallel:::.slaveRSOCK()"))
+ }
+
+ if (methods) {
+ rscript_args <- c("--default-packages=datasets,utils,grDevices,graphics,stats,methods", rscript_args)
+ }
+
+ ## Port that the Rscript should use to connect back to the master
+ if (!localMachine && revtunnel) {
+ rscript_port <- port + (rank - 1L)
+ } else {
+ rscript_port <- port
+ }
+
+ rscript <- paste(shQuote(rscript), collapse = " ")
+ rscript_args <- paste(rscript_args, collapse = " ")
+ envvars <- paste0("MASTER=", master, " PORT=", rscript_port, " OUT=", outfile, " TIMEOUT=", timeout, " XDR=", useXDR)
+
+ cmd <- paste(rscript, rscript_args, envvars)
+
+ ## Renice?
+ if (!is.na(renice) && renice > 0L) {
+ cmd <- sprintf("nice +%d %s", renice, cmd)
+ }
+
+ if (!localMachine) {
+ ## Local commands
+ rshcmd <- paste(shQuote(rshcmd), collapse = " ")
+ if (length(user) == 1L) rshopts <- c("-l", user, rshopts)
+ if (revtunnel) {
+ rshopts <- c(sprintf("-R %d:%s:%d", rscript_port, master, port), rshopts)
+ }
+ rshopts <- paste(rshopts, collapse = " ")
+ local_cmd <- paste(rshcmd, rshopts, worker, shQuote(cmd))
+ } else {
+ local_cmd <- cmd
+ }
+ stopifnot(length(local_cmd) == 1L)
+
+ if (manual || dryrun) {
+ msg <- c("----------------------------------------------------------------------", sprintf("Manually start worker #%s on %s with:", rank, sQuote(worker)), sprintf(" %s", cmd))
+ if (!localMachine) {
+ msg <- c(msg, "Alternatively, start it from the local machine with:", sprintf(" %s", local_cmd))
+ }
+ msg <- paste(c(msg, ""), collapse = "\n")
+ cat(msg)
+ utils::flush.console()
+ if (dryrun) return(NULL)
+ } else {
+ if (verbose) {
+ message(sprintf("Starting worker #%s on %s: %s", rank, sQuote(worker), local_cmd))
+ }
+ input <- if (.Platform$OS.type == "windows") "" else NULL
+ system(local_cmd, wait = FALSE, input = input)
+ }
+
+ if (verbose) {
+ message(sprintf("Waiting for worker #%s on %s to connect back", rank, sQuote(worker)))
+ }
+
+ con <- local({
+ ## Apply connection time limit "only to the rest of the current computation".
+ ## NOTE: Regardless of transient = TRUE / FALSE, it still seems we need to
+ ## undo it manually :/ /HB 2016-11-05
+ setTimeLimit(elapsed = connectTimeout)
+ on.exit(setTimeLimit(elapsed = Inf))
+
+ socketConnection("localhost", port = port, server = TRUE,
+ blocking = TRUE, open = "a+b", timeout = timeout)
+ })
+
+ if (verbose) {
+ message(sprintf("Connection with worker #%s on %s established", rank, sQuote(worker)))
+ }
+
+ structure(list(con = con, host = worker, rank = rank),
+ class = if (useXDR) "SOCKnode" else "SOCK0node")
+} ## makeNodePSOCK()
+
+
+
+## Attaching UUID for each cluster connection.
+## This is needed in order to be able to assert that we later
+## actually work with the same connection. See R-devel thread
+## 'closeAllConnections() can really mess things up' on 2016-10-30
+## (https://stat.ethz.ch/pipermail/r-devel/2016-October/073331.html)
+addClusterUUIDs <- function(cl) {
+ stopifnot(inherits(cl, "cluster"))
+
+ for (ii in seq_along(cl)) {
+ node <- cl[[ii]]
+ if (is.null(node)) next ## Happens with dryrun = TRUE
+ con <- node$con
+ uuid <- attr(con, "uuid")
+ if (is.null(uuid)) {
+ attr(con, "uuid") <- uuid_of_connection(con, keep_source = TRUE)
+ node$con <- con
+ cl[[ii]] <- node
+ }
+ }
+
+ cl
+} ## addClusterUUIDs()
diff --git a/R/multicore.R b/R/multicore.R
index 1f885da..7bcd040 100644
--- a/R/multicore.R
+++ b/R/multicore.R
@@ -12,7 +12,7 @@
#' @return A \link{MulticoreFuture}
#' If \code{workers == 1}, then all processing using done in the
#' current/main R session and we therefore fall back to using
-#' a eager future. This is also the case whenever multicore
+#' an eager future. This is also the case whenever multicore
#' processing is not supported, e.g. on Windows.
#'
#' @example incl/multicore.R
@@ -62,12 +62,12 @@ multicore <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE,
workers <- as.integer(workers)
stopifnot(is.finite(workers), workers >= 1L)
- ## Fall back to eager futures if only a single additional R process
+ ## Fall back to eager uniprocess futures if only a single additional R process
## can be spawned off, i.e. then use the current main R process.
- ## Eager futures best reflect how multicore futures handle globals.
+ ## Uniprocess futures best reflect how multicore futures handle globals.
if (workers == 1L || !supportsMulticore()) {
## covr: skip=1
- return(eager(expr, envir=envir, substitute=FALSE, globals=globals, local=TRUE, label=label))
+ return(uniprocess(expr, envir=envir, substitute=FALSE, globals=globals, local=TRUE, label=label, lazy=FALSE))
}
oopts <- options(mc.cores=workers)
diff --git a/R/multisession.R b/R/multisession.R
index cb9809e..30a6a51 100644
--- a/R/multisession.R
+++ b/R/multisession.R
@@ -66,11 +66,11 @@ multisession <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TR
workers <- as.integer(workers)
stopifnot(length(workers) == 1, is.finite(workers), workers >= 1)
- ## Fall back to eager futures if only a single R session can be used,
+ ## Fall back to lazy uniprocess futures if only a single R session can be used,
## i.e. the use the current main R process.
if (workers == 1L) {
## FIXME: How to handle argument 'persistent'? /HB 2016-03-19
- return(lazy(expr, envir=envir, substitute=FALSE, globals=globals, local=TRUE, label=label))
+ return(uniprocess(expr, envir=envir, substitute=FALSE, globals=globals, local=TRUE, label=label, lazy=TRUE))
}
## IMPORTANT: When we setup a multisession cluster, we need to
diff --git a/R/nbrOfWorkers.R b/R/nbrOfWorkers.R
index 3ae0862..7946101 100644
--- a/R/nbrOfWorkers.R
+++ b/R/nbrOfWorkers.R
@@ -35,5 +35,5 @@ nbrOfWorkers.future <- function(evaluator) {
#' @export
nbrOfWorkers.NULL <- function(evaluator) {
- nbrOfWorkers(plan())
+ nbrOfWorkers(plan("next"))
}
diff --git a/R/resolve.R b/R/resolve.R
index 05709a3..c9ea99b 100644
--- a/R/resolve.R
+++ b/R/resolve.R
@@ -57,10 +57,10 @@ resolve.Future <- function(x, idxs=NULL, value=FALSE, recursive=0, sleep=0.1, pr
msg <- sprintf("%s and its value was collected", msg)
## Recursively resolve the value?
- if (!is.atomic(v)) {
+ if (!is.atomic(v)) {
v <- resolve(v, value=TRUE, recursive=recursive-1, sleep=sleep, progress=FALSE, ...)
- msg <- sprintf("%s (and resolved itself)", msg)
- }
+ msg <- sprintf("%s (and resolved itself)", msg)
+ }
msg
} else {
diff --git a/R/sessionDetails.R b/R/sessionDetails.R
new file mode 100644
index 0000000..3244614
--- /dev/null
+++ b/R/sessionDetails.R
@@ -0,0 +1,94 @@
+#' Outputs details on the current R session
+#'
+#' @param env If TRUE, \code{Sys.getenv()} information is returned.
+#'
+#' @return Invisibly a list of all details.
+#'
+#' @details
+#' Note that remote futures use \code{persistent=TRUE} by default.
+#'
+#' @importFrom utils sessionInfo
+#' @export
+#' @keywords internal
+sessionDetails <- function(env = FALSE) {
+ ## Gather all details
+ details <- list(
+ "Sys.time()" = Sys.time(),
+ "sessionInfo()" = sessionInfo(),
+ "commandArgs()" = commandArgs(),
+ "Sys.info()" = Sys.info(),
+ "capabilities()" = capabilities(),
+ ".libPaths" = .libPaths(),
+ "Sys.getenv()" = Sys.getenv(),
+ "Sys.getlocale()" = Sys.getlocale(),
+ ".Platform" = .Platform,
+ ".Machine" = .Machine,
+ "getwd()" = getwd(),
+ "tempdir()" = tempdir(),
+ "options()" = options(),
+ "warnings()" = warnings()
+ )
+
+ ## May contain secret keys and tokens. Exclude by default.
+ if (!env) details[["Sys.getenv()"]] <- NULL
+
+ class(details) <- c("sessionDetails", class(details))
+ details
+}
+
+#' @export
+`[.sessionDetails` <- function(x, ...) {
+ class <- class(x)
+ x <- NextMethod("[")
+ class(x) <- class
+ x
+}
+
+#' @export
+print.sessionDetails <- function(x, output = c("output", "message"), headers = TRUE, width = getOption("width"), ...) {
+ banner <- function(s, width = getOption("width"), newline = TRUE) {
+ if (is.null(s)) s <- ""
+ width <- width - 1L * newline
+ if (nzchar(s)) {
+ padding <- width - nchar(s) - 3L
+ x <- c("= ", s, " ", rep("=", times = padding))
+ } else {
+ x <- rep("=", times = width)
+ }
+ if (newline) x <- c(x, "\n")
+ paste(x, collapse = "")
+ }
+
+ output <- match.arg(output)
+
+ con <- NULL
+ if (output == "output") {
+ con <- stderr()
+ } else if (output == "message") {
+ con <- stderr()
+ }
+
+ output <- capture.output({
+ if (headers) {
+ cat(banner(""))
+ cat("= SESSION DETAILS (BEGIN)\n")
+ cat(banner(""))
+ cat("\n")
+ }
+ for (kk in seq_along(x)) {
+ if (headers) cat(banner(names(x)[kk]))
+ info <- x[[kk]]
+ print(info, ...)
+ cat("\n")
+ }
+ if (headers) {
+ cat(banner(""))
+ cat("= SESSION DETAILS (STOP)\n")
+ cat(banner(""))
+ }
+ })
+ output <- paste(c(output, ""), collapse = "\n")
+ cat(output, file = con)
+
+ invisible(x)
+} ## sessionDetails()
diff --git a/R/tweak.R b/R/tweak.R
index 75703de..91af0e9 100644
--- a/R/tweak.R
+++ b/R/tweak.R
@@ -50,7 +50,11 @@ tweak.future <- function(strategy, ..., penvir=parent.frame()) {
stop("Additional arguments to tweak() must be named.")
}
-
+ ## Arguments that must not be tweaked
+ if (is.element("lazy", names)) {
+ stop("Future argument 'lazy' must not be tweaked / set via plan()")
+ }
+
## formals()<- drops any attributes including class
attrs <- attributes(strategy)
class <- class(strategy)
diff --git a/R/tweak_parallel_PSOCK.R b/R/tweak_parallel_PSOCK.R
deleted file mode 100644
index 5617de5..0000000
--- a/R/tweak_parallel_PSOCK.R
+++ /dev/null
@@ -1,149 +0,0 @@
-#' Tweak PSOCK backend of the parallel package
-#'
-#' @param user If TRUE, parallel is tweaked to only pass username to SSH if it is specified via argument \code{user}.
-#' @param revtunnel If TRUE, parallel is tweaked to make use of reverse SSH tunneling.
-#' @param rshopts If TRUE, parallel is tweaked so it is possible to specify additional command-line options to the \code{rshcmd} executable.
-#' @param use127.0.0.1 If TRUE, \code{127.0.0.1} is used instead of \code{localhost}.
-#' @param reset If TRUE, all tweaks are undone.
-#'
-#' @return Nothing.
-#'
-#' @examples
-#' \donttest{\dontrun{
-#' trace(system, tracer = quote(stop("Command: ", command)), print = FALSE)
-#'
-#' ## Without tweaks
-#' try(cl <- parallel::makeCluster("remote.myserver.org", revtunnel=TRUE,
-#' master="localhost", homogeneous=FALSE))
-#' ### Error in eval(expr, envir, enclos) :
-#' ### Command:
-#' ### ssh
-#' ### -l hb
-#' ### remote.myserver.org
-#' ### "Rscript
-#' ### --default-packages=datasets,utils,grDevices,graphics,stats,methods
-#' ### -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11099
-#' ### OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
-#'
-#' ## With tweaks
-#' future:::tweak_parallel_PSOCK()
-#' try(cl <- parallel::makeCluster("remote.myserver.org", revtunnel=TRUE,
-#' master="localhost", homogeneous=FALSE))
-#' ### Error in eval(expr, envir, enclos) :
-#' ### Command:
-#' ### ssh
-#' ### -R 11624:localhost:11624
-#' ### remote.myserver.org
-#' ### "Rscript
-#' ### --default-packages=datasets,utils,grDevices,graphics,stats,methods
-#' ### -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11099
-#' ### OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
-#'
-#' ## Undo tweaks
-#' untrace(system)
-#' future:::tweak_parallel_PSOCK(reset=TRUE)
-#' }}
-#'
-#' @references
-#' \url{https://github.com/HenrikBengtsson/Wishlist-for-R/issues/32}\cr
-#'
-#' @importFrom utils assignInNamespace
-#' @keywords internal
-tweak_parallel_PSOCK <- local({
- parallel <- NULL
-
- ## The original newPSOCKnode() and defaultClusterOptions of parallel
- newPSOCKnode_org <- defaultClusterOptions_org <- NULL
-
- .assignInNamespace <- NULL
-
- init <- function() {
- ## Already initiated?
- if (!is.null(parallel)) return()
-
- parallel <<- getNamespace("parallel")
-
- ## The original newPSOCKnode() and defaultClusterOptions of parallel
- newPSOCKnode_org <<- get("newPSOCKnode", mode="function", envir=parallel)
- defaultClusterOptions_org <<- get("defaultClusterOptions", envir=parallel)
-
- ## HACK: Trick assignInNamespace() to not complain
- .assignInNamespace <<- gsub_body("nf <- sys.nframe()", "nf <- 1L", assignInNamespace, fixed=TRUE)
- } ## init()
-
- gsub_body <- function(pattern, replacement, fun, ...) {
- stopifnot(is.function(fun))
- expr <- body(fun)
- code <- deparse(expr)
- code <- gsub(pattern=pattern, replacement=replacement, x=code, ...)
- expr <- parse(text=code)
- body(fun) <- expr
- fun
- } ## gsub_body()
-
-
-
- ## tweak_parallel_PSOCK()
- function(user=TRUE, revtunnel=TRUE, rshopts=TRUE, use127.0.0.1=FALSE, reset=FALSE) {
- ## To please R CMD check
- .assignInNamespace <- NULL; rm(list=".assignInNamespace")
-
- ## Make sure to initiate
- init()
-
- ## Original parallel setup
- newPSOCKnode <- newPSOCKnode_org
- defaultClusterOptions <- defaultClusterOptions_org
-
- if (!reset) {
- defaultClusterOptions <- as.environment(as.list(defaultClusterOptions))
-
- pattern <- 'cmd <- paste(rshcmd, "-l", user, machine, cmd)'
- replacement <- 'opts <- NULL'
-
- ## Only pass '-l <user>' if explicitly specified
- ## https://github.com/HenrikBengtsson/Wishlist-for-R/issues/31
- if (user) {
- ## Drop default 'user' (clone to avoid overwriting the original environment)
- defaultClusterOptions$user <- NULL
-
- ## newPSOCKnode() tweaks
- replacement <- c(replacement, 'if (!is.null(user)) opts <- c(opts, paste("-l", user))')
- }
-
- ## Reverse SSH tunneling (avoids all NAT / firewall issues)
- ## https://github.com/HenrikBengtsson/Wishlist-for-R/issues/32
- if (revtunnel) {
- defaultClusterOptions$revtunnel <- FALSE
-
- ## newPSOCKnode() tweaks
- replacement <- c(replacement, 'revtunnel <- getClusterOption("revtunnel", options)')
- replacement <- c(replacement, 'if (revtunnel) opts <- c(opts, sprintf("-R %d:%s:%d", port + (rank - 1L), master, port))')
- }
-
- ## Support for any type of command-line options (advanced usage)
- ## https://github.com/HenrikBengtsson/Wishlist-for-R/issues/32
- if (rshopts) {
- defaultClusterOptions$rshopts <- NULL
-
- ## newPSOCKnode() tweaks
- replacement <- c(replacement, 'opts <- c(opts, getClusterOption("rshopts", options))')
- }
-
- replacement <- c(replacement, 'opts <- paste(opts, collapse = " ")')
- replacement <- c(replacement, 'cmd <- paste(rshcmd, opts, machine, cmd)')
- replacement <- paste(replacement, collapse="\n")
-
- newPSOCKnode <- gsub_body(pattern, replacement, fun=newPSOCKnode, fixed=TRUE)
-
- ## Use '127.0.0.1' instead of 'localhost'
- if (use127.0.0.1) {
- newPSOCKnode <- gsub_body('socketConnection("localhost"', 'socketConnection("127.0.0.1"', fun=newPSOCKnode, fixed=TRUE)
- }
- }
-
- ## Update newPSOCKnode() and defaultClusterOptions of parallel
- .assignInNamespace("newPSOCKnode", newPSOCKnode, ns=parallel)
- .assignInNamespace("defaultClusterOptions", defaultClusterOptions, ns=parallel)
- } ## tweak_parallel_PSOCK()
-}) ## local()
diff --git a/R/uniprocess.R b/R/uniprocess.R
new file mode 100644
index 0000000..8b6107e
--- /dev/null
+++ b/R/uniprocess.R
@@ -0,0 +1,81 @@
+#' Create a uniprocess future whose value will be in the current R session
+#'
+#' A uniprocess future is a future that is evaluated sequentially in the
+#' current R session. The default is to resolve it eagerly, which means
+#' that its \emph{value is computed and resolved immediately}, which is
+#' how regular expressions are evaluated in R.
+#' The only difference to R itself is that globals are validated
+#' by default just as for all other types of futures in this package.
+#'
+#'
+#' @inheritParams future
+#' @inheritParams multiprocess
+#' @param local If TRUE, the expression is evaluated such that
+#' all assignments are done to local temporary environment, otherwise
+#' the assignments are done in the calling environment.
+#' @param lazy If \code{FALSE} (default), the future is resolved eagerly
+#' (immediately), otherwise not.
+#'
+#' @return A \link{UniprocessFuture}.
+#'
+#' @example incl/uniprocess.R
+#'
+#' @details
+#' The preferred way to create a uniprocess future is not to call these functions
+#' directly, but to register them via \code{\link{plan}(eager)} such that it
+#' becomes the default mechanism for all futures. After this
+#' \code{\link{future}()} and \code{\link{\%<-\%}} will create
+#' \emph{eager uniprocess futures}.
+#'
+#' @section transparent futures:
+#' Transparent futures are eager uniprocess futures configured to emulate how R
+#' evaluates expressions as far as possible. For instance, errors and
+#' warnings are signaled immediately and assignments are done to the
+#' calling environment (without \code{local()} as default for all other
+#' types of futures). This makes transparent futures ideal for
+#' troubleshooting, especially when there are errors.
+#'
+#' @aliases uniprocess
+#' @export
+eager <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE, local=TRUE, earlySignal=FALSE, label=NULL, lazy=FALSE, ...) {
+ if (substitute) expr <- substitute(expr)
+ local <- as.logical(local)
+
+ future <- EagerFuture(expr=expr, envir=envir, substitute=FALSE, globals=globals, local=local, earlySignal=earlySignal, label=label, lazy=lazy, ...)
+ run(future)
+}
+class(eager) <- c("eager", "uniprocess", "future", "function")
+
+#' @rdname eager
+#' @export
+transparent <- function(expr, envir=parent.frame(), substitute=TRUE, globals=FALSE, local=FALSE, earlySignal=TRUE, label=NULL, lazy=FALSE, ...) {
+ if (substitute) expr <- substitute(expr)
+ uniprocess(expr, envir=envir, substitute=FALSE, globals=globals, local=local, earlySignal=earlySignal, label=label, lazy=lazy, ...)
+}
+class(transparent) <- c("transparent", "uniprocess", "future", "function")
+
+#' @rdname eager
+#' @export
+lazy <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE, local=TRUE, earlySignal=FALSE, label=NULL, lazy=TRUE, ...) {
+ if (substitute) expr <- substitute(expr)
+ local <- as.logical(local)
+
+ LazyFuture(expr=expr, envir=envir, local=local, globals=globals, earlySignal=earlySignal, label=label, lazy=lazy, ...)
+}
+class(lazy) <- c("lazy", "uniprocess", "future", "function")
+
+## WORKAROUND:
+## Avoid lazyeval::print.lazy() being called with print(lazy())
+## https://github.com/HenrikBengtsson/future/issues/52
+class(lazy) <- c("function", class(lazy))
+
+
+## Keep private for now until name has been decided, cf.
+## https://github.com/HenrikBengtsson/future/issues/109
+uniprocess <- function(expr, envir=parent.frame(), substitute=TRUE, globals=TRUE, local=TRUE, earlySignal=FALSE, label=NULL, lazy=FALSE, ...) {
+ if (substitute) expr <- substitute(expr)
+ future <- UniprocessFuture(expr=expr, envir=envir, substitute=FALSE, globals=globals, local=local, earlySignal=earlySignal, label=label, lazy=lazy, ...)
+ if (!lazy) future <- run(future)
+ invisible(future)
+}
+class(uniprocess) <- c("uniprocess", "future", "function")
diff --git a/R/utils.R b/R/utils.R
index e96c75c..a1b9abf 100644
--- a/R/utils.R
+++ b/R/utils.R
@@ -67,14 +67,45 @@ mdebug <- function(...) {
} ## mdebug()
+## Create a universally unique identifier (UUID) for an R object
+#' @importFrom digest digest
+uuid <- function(source, keep_source = FALSE) {
+ uuid <- digest(source)
+ uuid <- strsplit(uuid, split="")[[1]]
+ uuid <- paste(c(uuid[1:8], "-", uuid[9:12], "-", uuid[13:16], "-", uuid[17:20], "-", uuid[21:32]), collapse="")
+ if (keep_source) attr(uuid, "source") <- source
+ uuid
+} ## uuid()
+
+uuid_of_connection <- function(con, ..., must_work = TRUE) {
+ stopifnot(inherits(con, "connection"))
+ if (must_work) {
+ info <- summary(con)
+ info$opened <- NULL
+ uuid <- uuid(info, ...)
+ } else {
+ uuid <- tryCatch({
+ info <- summary(con)
+ info$opened <- NULL
+ uuid(info, ...)
+ }, error = function(ex) {
+ attr(con, "uuid")
+ })
+ }
+ uuid
+} ## uuid_of_connection()
+
## A universally unique identifier (UUID) for the current
## R process. Generated only once.
#' @importFrom digest digest
-uuid <- local({
+session_uuid <- local({
value <- NULL
- function() {
+ function(attributes = FALSE) {
uuid <- value
- if (!is.null(uuid)) return(uuid)
+ if (!is.null(uuid)) {
+ if (!attributes) attr(uuid, "source") <- NULL
+ return(uuid)
+ }
info <- Sys.info()
host <- Sys.getenv(c("HOST", "HOSTNAME", "COMPUTERNAME"))
host <- host[nzchar(host)][1]
@@ -85,11 +116,9 @@ uuid <- local({
time=Sys.time(),
random=sample.int(.Machine$integer.max, size=1L)
)
- uuid <- digest(info)
- uuid <- strsplit(uuid, split="")[[1]]
- uuid <- paste(c(uuid[1:8], "-", uuid[9:12], "-", uuid[13:16], "-", uuid[17:20], "-", uuid[21:32]), collapse="")
- attr(uuid, "info") <- info
+ uuid <- uuid(info, keep_source = TRUE)
value <<- uuid
+ if (!attributes) attr(uuid, "source") <- NULL
uuid
}
})
@@ -172,7 +201,7 @@ detectCores <- local({
value <- getOption("future.availableCores.system")
if (!is.null(value)) {
value <- as.integer(value)
- return(value)
+ return(value)
}
value <- parallel::detectCores()
@@ -184,7 +213,7 @@ detectCores <- local({
## Assert positive integer
stopifnot(length(value) == 1L, is.numeric(value),
is.finite(value), value >= 1L)
-
+
res <<- value
}
res
@@ -328,10 +357,33 @@ myInternalIP <- local({
os <- R.version$os
pattern <- "[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+"
if (grepl("^linux", os)) {
- res <- system2("hostname", args="-I", stdout=TRUE)
+ ## (i) Try command 'hostname -I'
+ res <- tryCatch({
+ system2("hostname", args="-I", stdout=TRUE)
+ }, error = identity)
+
+ ## (ii) Try commands 'ifconfig'
+ if (inherits(res, "simpleError")) {
+ res <- tryCatch({
+ system2("ifconfig", stdout=TRUE)
+ }, error = identity)
+ }
+
+ ## (ii) Try command '/sbin/ifconfig'
+ if (inherits(res, "simpleError")) {
+ res <- tryCatch({
+ system2("/sbin/ifconfig", stdout=TRUE)
+ }, error = identity)
+ }
+
+ ## Failed?
+ if (inherits(res, "simpleError")) res <- NA_character_
+
res <- grep(pattern, res, value=TRUE)
res <- unlist(strsplit(res, split="[ ]+", fixed=FALSE), use.names=FALSE)
res <- grep(pattern, res, value=TRUE)
+ res <- unlist(strsplit(res, split=":", fixed=FALSE), use.names=FALSE)
+ res <- grep(pattern, res, value=TRUE)
res <- unique(trim(res))
## Keep private network IPs only (just in case)
value <- res[isPrivateIP(res)]
diff --git a/R/values.R b/R/values.R
index 27faad3..0404eaf 100644
--- a/R/values.R
+++ b/R/values.R
@@ -22,7 +22,15 @@ values.list <- function(x, ...) {
for (ii in seq_along(y)) {
tmp <- y[[ii]]
- if (inherits(tmp, "Future")) y[[ii]] <- value(tmp, ...)
+ if (inherits(tmp, "Future")) {
+ v <- value(tmp, ...)
+ if (is.null(v)) {
+ y[ii] <- list(NULL)
+ } else {
+ y[[ii]] <- v
+ v <- NULL
+ }
+ }
}
y
}
@@ -45,7 +53,15 @@ values.listenv <- function(x, ...) {
y <- resolve(y)
for (ii in seq_along(y)) {
tmp <- y[[ii]]
- if (inherits(tmp, "Future")) y[[ii]] <- value(tmp, ...)
+ if (inherits(tmp, "Future")) {
+ v <- value(tmp, ...)
+ if (is.null(v)) {
+ y[ii] <- list(NULL)
+ } else {
+ y[[ii]] <- v
+ v <- NULL
+ }
+ }
}
y
}
diff --git a/R/zzz.R b/R/zzz.R
index 57c0914..b8daaea 100644
--- a/R/zzz.R
+++ b/R/zzz.R
@@ -63,7 +63,7 @@
}
## Create UUID for this process
- id <- uuid()
+ id <- session_uuid(attributes = TRUE)
mdebug("R process uuid: %s", id)
mdebug("Setting plan('default')")
diff --git a/R/plan.R b/R/zzz.plan.R
similarity index 80%
rename from R/plan.R
rename to R/zzz.plan.R
index 59b963f..5d38c25 100644
--- a/R/plan.R
+++ b/R/zzz.plan.R
@@ -72,9 +72,10 @@
#' @export
plan <- local({
defaultStrategy <- structure(eager, call=substitute(plan(eager)))
-
+ defaultStack <- structure(list(defaultStrategy), class = c("FutureStrategyList", "list"))
+
## Stack of type of futures to use
- stack <- list(defaultStrategy)
+ stack <- defaultStack
## Main function
function(strategy=NULL, ..., substitute=TRUE, .call=TRUE) {
@@ -84,7 +85,9 @@ plan <- local({
## Predefined "actions":
if (is.null(strategy) || identical(strategy, "next")) {
## Next future strategy?
- return(stack[[1L]])
+ strategy <- stack[[1L]]
+ class(strategy) <- c("FutureStrategy", class(strategy))
+ return(strategy)
} else if (identical(strategy, "default")) {
strategy <- getOption("future.plan", eager)
} else if (identical(strategy, "list")) {
@@ -92,13 +95,13 @@ plan <- local({
return(stack)
} else if (identical(strategy, "reset")) {
## Rest stack of future strategies?
- stack <<- list(defaultStrategy)
+ stack <<- defaultStack
return(stack)
} else if (identical(strategy, "pop")) {
## Pop strategy stack and return old stack (so it can be pushed back later)
oldStack <- stack
stack <<- stack[-1L]
- if (length(stack) == 0L) stack <<- list(defaultStrategy)
+ if (length(stack) == 0L) stack <<- defaultStack
return(oldStack)
}
@@ -112,6 +115,7 @@ plan <- local({
for (ii in seq_along(strategy)) {
stopifnot(is.function(strategy[[ii]]))
}
+ class(strategy) <- unique(c("FutureStrategyList", class(strategy)))
stack <<- strategy
return(invisible(oldStack[[1L]]))
}
@@ -194,6 +198,7 @@ plan <- local({
}
## Set new strategy for futures
+ class(newStack) <- c("FutureStrategyList", class(newStack))
stack <<- newStack
stopifnot(is.list(stack), length(stack) >= 1L)
@@ -206,3 +211,52 @@ supportedStrategies <- function(strategies=c("lazy", "eager", "multicore", "mult
if (!supportsMulticore()) strategies <- setdiff(strategies, "multicore")
strategies
}
+
+
+#' @export
+print.future <- function(x, ...) {
+ class <- setdiff(class(x), c("FutureStrategy", "tweaked", "function"))
+ s <- sprintf("%s:", class[1])
+ specs <- list()
+ args <- deparse(args(x))
+ args <- args[-length(args)]
+ args <- gsub("(^[ ]+|[ ]+$)", "", args)
+ args <- paste(args, collapse = " ")
+ specs$args <- args
+ specs$tweaked <- inherits(x, "tweaked")
+ specs$call <- deparse(attr(x, "call"))
+ specs <- sprintf("- %s: %s", names(specs), unlist(specs))
+ s <- c(s, specs)
+ s <- paste(s, collapse = "\n")
+ cat(s, "\n", sep="")
+ invisible(x)
+}
+
+#' @export
+print.FutureStrategy <- print.future
+
+
+#' @export
+print.FutureStrategyList <- function(x, ...) {
+ s <- "List of future strategies:"
+
+ for (kk in seq_along(x)) {
+ x_kk <- x[[kk]]
+ class <- setdiff(class(x_kk), c("tweaked", "function"))
+ s_kk <- sprintf("%d. %s:", kk, class[1])
+ specs <- list()
+ args <- deparse(args(x_kk))
+ args <- args[-length(args)]
+ args <- gsub("(^[ ]+|[ ]+$)", "", args)
+ args <- paste(args, collapse = " ")
+ specs$args <- args
+ specs$tweaked <- inherits(x_kk, "tweaked")
+ specs$call <- deparse(attr(x_kk, "call"))
+ specs <- sprintf(" - %s: %s", names(specs), unlist(specs))
+ s <- c(s, s_kk, specs)
+ }
+
+ s <- paste(s, collapse = "\n")
+ cat(s, "\n", sep="")
+ invisible(x)
+}
diff --git a/README.md b/README.md
index 39a5fbf..db35c56 100644
--- a/README.md
+++ b/README.md
@@ -3,7 +3,7 @@
## Introduction
The purpose of the [future] package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
-In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
+In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
```r
@@ -91,7 +91,7 @@ The future package implements the following types of futures:
| _synchronous:_ | | _non-parallel:_
| `eager` | all |
| `lazy` | all | lazy evaluation - happens only if the value is requested
-| `transparent` | all | for debugging (eager w/ early signaling and w/out local)
+| `transparent` | all | as eager w/ early signaling and w/out local (for debugging)
| _asynchronous:_ | | _parallel_:
| `multiprocess` | all | multicore iff supported, otherwise multisession
| `multisession` | all | background R sessions (on current machine)
@@ -101,7 +101,7 @@ The future package implements the following types of futures:
The future package is designed such that support for additional strategies can be implemented as well. For instance, the [future.BatchJobs] package provides futures for all types of _cluster functions_ ("backends") that the [BatchJobs] package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available.
-By default, future expressions are evaluated instantaneously and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
+By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
### Consistent Behavior Across Futures
@@ -146,7 +146,7 @@ Eager futures are the default unless otherwise specified. They were designed to
> plan(eager)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -165,13 +165,13 @@ Resolving 'b' ...
+ }
Resolving 'c' ...
> b
-[1] 14641
+[1] 10708
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
Since eager evaluation is taking place, each of the three futures is resolved instantaneously in the moment it is created. Note also how `pid` in the calling environment, which was assigned the process ID of the current process, is neither overwritten nor removed. This is because futures are evaluated in a local environment. Since synchronous (uni-)processing is used, future `b` is resolved by the main R process (still in a local environment), which is why the value of `b` and `pid` a [...]
@@ -182,7 +182,7 @@ A lazy future evaluates its expression only if its value is queried. Evaluation
> plan(lazy)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -200,14 +200,14 @@ A lazy future evaluates its expression only if its value is queried. Evaluation
Resolving 'a' ...
> b
Resolving 'b' ...
-[1] 14641
+[1] 10708
> c
Resolving 'c' ...
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
As previously, variable `pid` is unaffected because all evaluation is done in a local environment. More interestingly, future `a` is no longer evaluated in the moment it is created, but instead when it is needed the first time, which happens when future `c` is created. This is because `a` is identified as a global variable that needs to be captured ("frozen" to `a == 3.14`) in order to set up future `c`. Later when `c` (the value of future `c`) is queried, `a` has already been resolve [...]
@@ -229,7 +229,7 @@ We start with multisession futures because they are supported by all operating s
> plan(multisession)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -245,13 +245,13 @@ We start with multisession futures because they are supported by all operating s
+ 2 * a
+ }
> b
-[1] 14662
+[1] 10729
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
The first thing we observe is that the values of `a`, `c` and `pid` are the same as previously. However, we notice that `b` is different from before. This is because future `b` is evaluated in a different R process and therefore it returns a different process ID. Another difference is that the messages, generated by `cat()`, are no longer displayed. This is because they are outputted to the background sessions and not the calling session.
@@ -291,7 +291,7 @@ Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the
> plan(cluster, workers = c("n1", "n2", "n3"))
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -307,13 +307,13 @@ Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the
+ 2 * a
+ }
> b
-[1] 14684
+[1] 10751
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
Just as for the other asynchronous evaluation strategies, the output from `cat()` is not displayed on the current/calling machine.
@@ -325,7 +325,7 @@ plan(cluster, workers=cl)
```
Also, it is considered good style to shut down the cluster when it is no longer needed, that is, calling `parallel::stopCluster(cl)`. However, it will shut itself down if the main process is terminated, which will happen in the first example where the cluster in created internally. For more information on how to set up and manage such clusters, see `help("makeCluster", package="parallel")`.
-Note that with proper firewall and router configurations (e.g. port forwarding) and with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
+Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
@@ -335,7 +335,7 @@ Sometimes one may want to use an alternative evaluation strategy for a specific
> plan(eager)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ Sys.getpid()
+ }
@@ -346,11 +346,11 @@ Sometimes one may want to use an alternative evaluation strategy for a specific
+ Sys.getpid()
+ } %plan% multiprocess
> a
-[1] 14641
+[1] 10708
> b
-[1] 14701
+[1] 10769
> c
-[1] 14702
+[1] 10770
```
As seen by the different process IDs, future `a` is evaluated eagerly using the same process as the calling environment whereas the other two are evaluated using multiprocess futures.
@@ -384,14 +384,14 @@ For instance, here is an example of two "top" futures (`a` and `b`) that uses mu
+ c(b.pid = Sys.getpid(), b1.pid = b1, b2.pid = b2)
+ }
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14703
+[1] 10771
> b
b.pid b1.pid b2.pid
- 14704 14704 14704
+ 10772 10772 10772
```
-By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 14641), and there are the two processes used by `a` (pid 14703) and `b` (pid 14704). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use eager evaluation unless otherwise specified. There are a few reasons for this, but the main reason is th [...]
+By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 10708), and there are the two processes used by `a` (pid 10771) and `b` (pid 10772). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use eager evaluation unless otherwise specified. There are a few reasons for this, but the main reason is th [...]
@@ -404,12 +404,12 @@ We would actually get the same behavior if we try with multiple levels of multip
> plan(list(multiprocess, multiprocess))
[...]
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14705
+[1] 10773
> b
b.pid b1.pid b2.pid
- 14706 14706 14706
+ 10774 10774 10774
```
The reason for this is, also here, to protect us from launching more processes than what the machine can support. Internally, this is done by setting `mc.cores` to zero ([sic!](https://github.com/HenrikBengtsson/Wishlist-for-R/issues/7)) such that no _additional_ parallel processes can be launched. This is the case for both multisession and multicore evaluation.
@@ -421,31 +421,31 @@ Continuing, if we start off by eager evaluation and then use multiprocess evalua
Resolving 'a' ...
Resolving 'b' ...
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14641
+[1] 10708
> b
b.pid b1.pid b2.pid
- 14641 14707 14708
+ 10708 10775 10776
```
-which clearly show that `a` and `b` are resolved in the calling process (pid 14641) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids 14707 and 14708).
+which clearly show that `a` and `b` are resolved in the calling process (pid 10708) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids 10775 and 10776).
Having said this, it is indeed possible to use nested multiprocess evaluation strategies, if we explicitly specify (read _force_) the number of cores available at each level. In order to do this we need to "tweak" the default settings, which can be done as follows:
```r
-> plan(list(tweak(multiprocess, workers = 3), tweak(multiprocess,
-+ workers = 3)))
+> plan(list(tweak(multiprocess, workers = 3L), tweak(multiprocess,
++ workers = 3L)))
[...]
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14709
+[1] 10777
> b
b.pid b1.pid b2.pid
- 14710 14711 14712
+ 10778 10779 10840
```
-First, we see that both `a` and `b` are resolved in different processes (pids 14709 and 14710) than the calling process (pid 14641). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids 14711 and 14712).
+First, we see that both `a` and `b` are resolved in different processes (pids 10777 and 10778) than the calling process (pid 10708). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids 10779 and 10840).
To clarify, when we set up the two levels of multiprocess evaluation, we specified that in total 3 processes may be used at each level. We choose three parallel processes, not just two, because one is always consumed by the calling process leaving two to be used for the asynchronous futures. This is why we see that `pid`, `a` and `b` are all resolved by the same process. If we had allowed only two cores at the top level, `a` and `b` would have been resolved by the same background proc [...]
@@ -480,7 +480,7 @@ Waiting for 'a' to be resolved ...
> cat("Waiting for 'a' to be resolved ... DONE\n")
Waiting for 'a' to be resolved ... DONE
> a
-[1] 14713
+[1] 10841
```
@@ -547,9 +547,9 @@ There is one limitation with implicit futures that does not exist for explicit o
> v <- lapply(f, FUN = value)
> str(v)
List of 3
- $ : int 14714
- $ : int 14715
- $ : int 14716
+ $ : int 10921
+ $ : int 10922
+ $ : int 10924
```
This is _not_ possible to do when using implicit futures. This is because the `%<-%` assignment operator _cannot_ be used in all cases where the regular `<-` assignment operator can be used. It can only be used to assign future values to _environments_ (including the calling environment) much like how `assign(name, value, envir)` works. However, we can assign implicit futures to environments using _named indices_, e.g.
```r
@@ -563,9 +563,9 @@ This is _not_ possible to do when using implicit futures. This is because the `
> v <- as.list(v)
> str(v)
List of 3
- $ a: int 14717
- $ b: int 14718
- $ c: int 14719
+ $ a: int 10925
+ $ b: int 10928
+ $ c: int 10931
```
Here `as.list(v)` blocks until all futures in the environment `v` have been resolved. Then their values are collected and returned as a regular list.
@@ -582,9 +582,9 @@ If _numeric indices_ are required, then _list environments_ can be used. List e
> v <- as.list(v)
> str(v)
List of 3
- $ : int 14720
- $ : int 14721
- $ : int 14722
+ $ : int 10932
+ $ : int 10933
+ $ : int 10934
```
As previously, `as.list(v)` blocks until all futures are resolved.
@@ -631,7 +631,7 @@ The goal of this package is to provide a standardized and unified API for using
[Futures in R: Future Topologies]: future-3-topologies.html
## Installation
-R package future is available on [CRAN](http://cran.r-project.org/package=future) and can be installed in R as:
+R package future is available on [CRAN](https://cran.r-project.org/package=future) and can be installed in R as:
```r
install.packages('future')
```
@@ -651,5 +651,5 @@ This will install the package from source.
| Resource: | CRAN | Travis CI | Appveyor |
| ------------- | ------------------- | --------------- | ---------------- |
| _Platforms:_ | _Multiple_ | _Linux & macOS_ | _Windows_ |
-| R CMD check | <a href="http://cran.r-project.org/web/checks/check_results_future.html"><img border="0" src="http://www.r-pkg.org/badges/version/future" alt="CRAN version"></a> | <a href="https://travis-ci.org/HenrikBengtsson/future"><img src="https://travis-ci.org/HenrikBengtsson/future.svg" alt="Build status"></a> | <a href="https://ci.appveyor.com/project/HenrikBengtsson/future"><img src="https://ci.appveyor.com/api/projects/status/github/HenrikBengtsson/future?svg=true" alt="Bui [...]
+| R CMD check | <a href="https://cran.r-project.org/web/checks/check_results_future.html"><img border="0" src="http://www.r-pkg.org/badges/version/future" alt="CRAN version"></a> | <a href="https://travis-ci.org/HenrikBengtsson/future"><img src="https://travis-ci.org/HenrikBengtsson/future.svg" alt="Build status"></a> | <a href="https://ci.appveyor.com/project/HenrikBengtsson/future"><img src="https://ci.appveyor.com/api/projects/status/github/HenrikBengtsson/future?svg=true" alt="Bu [...]
| Test coverage | | <a href="https://codecov.io/gh/HenrikBengtsson/future"><img src="https://codecov.io/gh/HenrikBengtsson/future/branch/develop/graph/badge.svg" alt="Coverage Status"/></a> | |
diff --git a/cran-comments.md b/cran-comments.md
index a89c92d..7bbfbcb 100644
--- a/cran-comments.md
+++ b/cran-comments.md
@@ -1,13 +1,9 @@
-# CRAN submission future 1.1.1
-on 2015-10-10
+# CRAN submission future 1.2.0
+on 2015-11-12
-Fixing R CMD check test error on CRAN r-devel-macos-x86_64-clang server that was (re-)introduced in future 1.1.0, which was submitted yesterday.
-
-Sorry about that and thanks in advance
-
-
-# CRAN submission future 1.1.0
-on 2015-10-09
+I've verified that this submission causes *no* issues for
+any of the 19 reverse (non-recursive) package dependencies
+available on CRAN and Bioconductor.
Thanks in advance
@@ -17,35 +13,37 @@ The package has been verified using `R CMD check --as-cran` on:
* Platform x86_64-apple-darwin13.4.0 (64-bit) [Travis CI]:
- R 3.2.4 Revised (2016-03-16)
- - R version 3.3.1 (2016-06-21)
+ - R version 3.3.2 (2016-10-31)
* Platform x86_64-unknown-linux-gnu (64-bit) [Travis CI]:
- R version 3.2.5 (2016-04-14)
- R version 3.3.1 (2016-06-21)
- - R Under development (unstable) (2016-10-07 r71466)
+ - R Under development (unstable) (2016-11-11 r71647)
* Platform x86_64-pc-linux-gnu (64-bit):
- - R version 3.0.3 (2014-03-06)
- R version 3.1.2 (2014-10-31)
- - R version 3.1.3 (2015-03-09)
- - R version 3.2.5 (2016-04-14)
- R version 3.3.0 (2016-05-03)
+ - R version 3.3.2 (2016-10-31)
+ - R version 3.3.2 Patched (2016-11-09 r71642)
+
+* Platform x86_64-pc-linux-gnu (64-bit) [r-hub]:
- R version 3.3.1 (2016-06-21)
- - R version 3.3.1 Patched (2016-10-05 r71468)
+ - R Under development (unstable) (2016-10-30 r71610)
* Platform i686-pc-linux-gnu (32-bit):
- - R version 3.3.1 (2016-06-21)
-
+ - R version 3.2.3 (2015-12-10)
+ - R version 3.3.2 (2016-10-31)
+
* Platform i386-w64-mingw32 (32-bit) [Appveyor CI]:
- - R Under development (unstable) (2016-10-07 r71466)
+ - R Under development (unstable) (2016-11-11 r71647)
* Platform x86_64-w64-mingw32/x64 (64-bit) [Appveyor CI]:
- - R version 3.3.1 (2016-06-21)
- - R Under development (unstable) (2016-10-07 r71466)
+ - R version 3.3.2 (2016-10-31)
+ - R Under development (unstable) (2016-11-11 r71647)
-* Platform x86_64-w64-mingw32/x64 (64-bit) [win-builder]:
- - R version 3.3.1 (2016-06-21)
- - R Under development (unstable) (2016-10-08 r71472)
+* Platform x86_64-w64-mingw32 (64-bit) [r-hub]:
+ - R version 3.2.5 (2016-04-14)
-The submitted updates cause no issues for any of the
-19 reverse dependencies on CRAN and Bioconductor.
+* Platform x86_64-w64-mingw32/x64 (64-bit) [win-builder]:
+ - R version 3.3.2 (2016-10-31)
+ - R Under development (unstable) (2016-11-09 r71641)
diff --git a/incl/eager.R b/incl/eager.R
deleted file mode 100644
index 3f7db4a..0000000
--- a/incl/eager.R
+++ /dev/null
@@ -1,22 +0,0 @@
-## Use eager futures
-plan(eager)
-
-## A global variable
-a <- 0
-
-## Create eager future (explicitly)
-f <- future({
- b <- 3
- c <- 2
- a * b * c
-})
-
-## Since 'a' is a global variable in _eager_ future 'f',
-## it already has been resolved, and any changes to 'a'
-## at this point will _not_ affect the value of 'f'.
-a <- 7
-print(a)
-
-v <- value(f)
-print(v)
-stopifnot(v == 0)
diff --git a/incl/makeClusterPSOCK.R b/incl/makeClusterPSOCK.R
new file mode 100644
index 0000000..3d4727e
--- /dev/null
+++ b/incl/makeClusterPSOCK.R
@@ -0,0 +1,119 @@
+## Setup of three R workers on two remote machines are set up
+workers <- c("n1.remote.org", "n2.remote.org", "n1.remote.org")
+cl <- makeClusterPSOCK(workers, dryrun = TRUE)
+
+## Same setup when the two machines are on the local network and
+## have identical software setups
+cl <- makeClusterPSOCK(
+ workers,
+ revtunnel = FALSE, homogeneous = TRUE,
+ dryrun = TRUE
+)
+
+## Setup of remote worker with more detailed control on
+## authentication and reverse SSH tunnelling
+cl <- makeClusterPSOCK(
+ "remote.server.org", user = "johnny",
+ ## Manual configuration of reverse SSH tunnelling
+ revtunnel = FALSE,
+ rshopts = c("-v", "-R 11000:gateway:11942"),
+ master = "gateway", port = 11942,
+ ## Run Rscript nicely and skip any startup scripts
+ rscript = c("nice", "/path/to/Rscript"),
+ rscript_args = c("--vanilla"),
+ dryrun = TRUE
+)
+
+## Setup of Docker worker running rocker/r-base
+## (requires installation of future package)
+cl <- makeClusterPSOCK(
+ "localhost",
+ ## Launch Rscript inside Docker container
+ rscript = c(
+ "docker", "run", "--net=host", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package
+ rscript_args = c(
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
+
+
+## Setup of udocker.py worker running rocker/r-base
+## (requires installation of future package and extra quoting)
+cl <- makeClusterPSOCK(
+ "localhost",
+ ## Launch Rscript inside Docker container (using udocker)
+ rscript = c(
+ "udocker.py", "run", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package and manually launch parallel workers
+ ## (need double shQuote():s because udocker.py drops one level)
+ rscript_args = c(
+ "-e", shQuote(shQuote("install.packages('future')")),
+ "-e", shQuote(shQuote("parallel:::.slaveRSOCK()"))
+ ),
+ dryrun = TRUE
+)
+
+
+## Launching worker on Amazon AWS EC2 running one of the
+## Amazon Machine Images (AMI) provided by RStudio
+## (http://www.louisaslett.com/RStudio_AMI/)
+public_ip <- "1.2.3.4"
+ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"
+cl <- makeClusterPSOCK(
+ ## Public IP number of EC2 instance
+ public_ip,
+ ## User name (always 'ubuntu')
+ user = "ubuntu",
+ ## Use private SSH key registered with AWS
+ rshopts = c(
+ "-o", "StrictHostKeyChecking=no",
+ "-o", "IdentitiesOnly=yes",
+ "-i", ssh_private_key_file
+ ),
+ ## Set up .libPaths() for the 'ubuntu' user and
+ ## install future package
+ rscript_args = c(
+ "-e", shQuote("local({
+ p <- Sys.getenv('R_LIBS_USER')
+ dir.create(p, recursive = TRUE, showWarnings = FALSE)
+ .libPaths(p)
+ })"),
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
+
+
+## Launching worker on Google Cloud Engine (GCE) running a
+## container based VM (with a #cloud-config specification)
+public_ip <- "1.2.3.4"
+user <- "johnny"
+ssh_private_key_file <- "~/.ssh/google_compute_engine"
+cl <- makeClusterPSOCK(
+ ## Public IP number of GCE instance
+ public_ip,
+ ## User name (== SSH key label (sic!))
+ user = user,
+ ## Use private SSH key registered with GCE
+ rshopts = c(
+ "-o", "StrictHostKeyChecking=no",
+ "-o", "IdentitiesOnly=yes",
+ "-i", ssh_private_key_file
+ ),
+ ## Launch Rscript inside Docker container
+ rscript = c(
+ "docker", "run", "--net=host", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package
+ rscript_args = c(
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
diff --git a/incl/plan.R b/incl/plan.R
index 541fef3..b0e23aa 100644
--- a/incl/plan.R
+++ b/incl/plan.R
@@ -1,6 +1,6 @@
a <- b <- c <- NA_real_
-# A lazy future
+# A lazy uniprocess future
plan(lazy)
f <- future({
a <- 7
@@ -13,7 +13,7 @@ print(y)
str(list(a=a, b=b, c=c)) ## All NAs
-# An eager future
+# An eager uniprocess future
plan(eager)
f <- future({
a <- 7
diff --git a/incl/uniprocess.R b/incl/uniprocess.R
new file mode 100644
index 0000000..a4b78ea
--- /dev/null
+++ b/incl/uniprocess.R
@@ -0,0 +1,23 @@
+## Use an eager uniprocess futures
+plan(eager)
+
+## A global variable
+a <- 0
+
+## Create an eager uniprocess future (explicitly)
+f <- future({
+ b <- 3
+ c <- 2
+ a * b * c
+})
+
+## Since 'a' is a global variable in future 'f' which
+## is eagerly resolved (default), this global has already
+## been resolved / incorporated, and any changes to 'a'
+## at this point will _not_ affect the value of 'f'.
+a <- 7
+print(a)
+
+v <- value(f)
+print(v)
+stopifnot(v == 0)
diff --git a/inst/vignettes-static/future-1-overview.md.rsp.rsp b/inst/vignettes-static/future-1-overview.md.rsp.rsp
index d5d7dec..b7c68ce 100644
--- a/inst/vignettes-static/future-1-overview.md.rsp.rsp
+++ b/inst/vignettes-static/future-1-overview.md.rsp.rsp
@@ -53,7 +53,7 @@ options(mc.cores=2L)
## Introduction
The purpose of the [future] package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
-In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
+In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
```r
@@ -141,7 +141,7 @@ The future package implements the following types of futures:
| _synchronous:_ | | _non-parallel:_
| `eager` | all |
| `lazy` | all | lazy evaluation - happens only if the value is requested
-| `transparent` | all | for debugging (eager w/ early signaling and w/out local)
+| `transparent` | all | as eager w/ early signaling and w/out local (for debugging)
| _asynchronous:_ | | _parallel_:
| `multiprocess` | all | multicore iff supported, otherwise multisession
| `multisession` | all | background R sessions (on current machine)
@@ -151,7 +151,7 @@ The future package implements the following types of futures:
The future package is designed such that support for additional strategies can be implemented as well. For instance, the [future.BatchJobs] package provides futures for all types of _cluster functions_ ("backends") that the [BatchJobs] package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available.
-By default, future expressions are evaluated instantaneously and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
+By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
### Consistent Behavior Across Futures
@@ -237,7 +237,7 @@ stopifnot(b == pid)
<%---
#### Transparent Futures
-For troubleshooting, there is also a _transparent_ future, which can be specified as `plan(transparent)`. A transparent future is technically an eager future with instant signaling of conditions (including errors and warnings) and where evaluation, and therefore also assignments, take place in the calling environment. Transparent futures are particularly useful for troubleshooting errors.
+For troubleshooting, there is also a _transparent_ future, which can be specified as `plan(transparent)`. A transparent future is technically a eager future with instant signaling of conditions (including errors and warnings) and where evaluation, and therefore also assignments, take place in the calling environment. Transparent futures are particularly useful for troubleshooting errors.
---%>
@@ -342,7 +342,7 @@ plan(cluster, workers=cl)
```
Also, it is considered good style to shut down the cluster when it is no longer needed, that is, calling `parallel::stopCluster(cl)`. However, it will shut itself down if the main process is terminated, which will happen in the first example where the cluster in created internally. For more information on how to set up and manage such clusters, see `help("makeCluster", package="parallel")`.
-Note that with proper firewall and router configurations (e.g. port forwarding) and with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
+Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
diff --git a/man/EagerFuture-class.Rd b/man/EagerFuture-class.Rd
deleted file mode 100644
index 4da57f3..0000000
--- a/man/EagerFuture-class.Rd
+++ /dev/null
@@ -1,25 +0,0 @@
-% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/EagerFuture-class.R
-\name{EagerFuture-class}
-\alias{EagerFuture}
-\alias{EagerFuture-class}
-\title{An eager future is a future whose value will be resolved immediately}
-\usage{
-EagerFuture(expr = NULL, envir = parent.frame(), substitute = FALSE,
- globals = TRUE, local = TRUE, ...)
-}
-\arguments{
-\item{...}{Additional named elements passed to \code{\link{Future}()}.}
-}
-\value{
-An object of class \code{EagerFuture}.
-}
-\description{
-An eager future is a future whose value will be resolved immediately
-}
-\seealso{
-To evaluate an expression using "eager future", see function
-\code{\link{eager}()}.
-}
-\keyword{internal}
-
diff --git a/man/LazyFuture-class.Rd b/man/LazyFuture-class.Rd
deleted file mode 100644
index e24d1c4..0000000
--- a/man/LazyFuture-class.Rd
+++ /dev/null
@@ -1,25 +0,0 @@
-% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/LazyFuture-class.R
-\name{LazyFuture-class}
-\alias{LazyFuture}
-\alias{LazyFuture-class}
-\title{A lazy future is a future whose value will be resolved at the time when it is requested}
-\usage{
-LazyFuture(expr = NULL, envir = parent.frame(), substitute = FALSE,
- globals = TRUE, local = TRUE, ...)
-}
-\arguments{
-\item{...}{Additional named elements passed to \code{\link{Future}()}.}
-}
-\value{
-An object of class \code{LazyFuture}.
-}
-\description{
-A lazy future is a future whose value will be resolved at the time when it is requested
-}
-\seealso{
-To evaluate an expression using "lazy future", see function
-\code{\link{lazy}()}.
-}
-\keyword{internal}
-
diff --git a/man/UniprocessFuture-class.Rd b/man/UniprocessFuture-class.Rd
index 2c35072..e595388 100644
--- a/man/UniprocessFuture-class.Rd
+++ b/man/UniprocessFuture-class.Rd
@@ -1,12 +1,20 @@
% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/UniprocessFuture-class.R
\name{UniprocessFuture-class}
+\alias{EagerFuture}
+\alias{LazyFuture}
\alias{UniprocessFuture}
\alias{UniprocessFuture-class}
\title{An uniprocess future is a future whose value will be resolved synchroneously in the current process}
\usage{
UniprocessFuture(expr = NULL, envir = parent.frame(), substitute = FALSE,
- local = TRUE, ...)
+ globals = TRUE, local = TRUE, lazy = FALSE, ...)
+
+EagerFuture(expr = NULL, envir = parent.frame(), substitute = FALSE,
+ globals = TRUE, local = TRUE, lazy = FALSE, ...)
+
+LazyFuture(expr = NULL, envir = parent.frame(), substitute = FALSE,
+ globals = TRUE, local = TRUE, lazy = FALSE, ...)
}
\arguments{
\item{expr}{An R \link[base]{expression}.}
@@ -21,6 +29,9 @@ is done (or inherits from if \code{local} is TRUE).}
all assignments are done to local temporary environment, otherwise
the assignments are done in the calling environment.}
+\item{lazy}{If \code{FALSE} (default), then the setup and validation of
+global variables are done for eager evaluation, otherwise not.}
+
\item{...}{Additional named elements of the future.}
\item{\dots}{Additional named elements passed to \code{\link{Future}()}.}
@@ -33,7 +44,7 @@ An uniprocess future is a future whose value will be resolved synchroneously in
}
\seealso{
To evaluate an expression using "uniprocess future", see functions
-\code{\link{eager}()} and \code{\link{lazy}()}.
+\code{\link{uniprocess}()}.
}
\keyword{internal}
diff --git a/man/as.cluster.Rd b/man/as.cluster.Rd
new file mode 100644
index 0000000..5016f50
--- /dev/null
+++ b/man/as.cluster.Rd
@@ -0,0 +1,40 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/as.cluster.R
+\name{as.cluster}
+\alias{as.cluster}
+\alias{as.cluster.SOCK0node}
+\alias{as.cluster.SOCKnode}
+\alias{as.cluster.cluster}
+\alias{as.cluster.list}
+\alias{c.cluster}
+\title{Coerce an object to a cluster object}
+\usage{
+as.cluster(x, ...)
+
+\method{as.cluster}{cluster}(x, ...)
+
+\method{as.cluster}{list}(x, ...)
+
+\method{as.cluster}{SOCKnode}(x, ...)
+
+\method{as.cluster}{SOCK0node}(x, ...)
+
+\method{c}{cluster}(..., recursive = FALSE)
+}
+\arguments{
+\item{x}{An object to be coerced.}
+
+\item{...}{Additional arguments passed to the underlying coercion method.
+For \code{c(...)}, the clusters and cluster nodes to be combined.}
+
+\item{recursive}{Not used.}
+}
+\value{
+An object of class \code{cluster}.
+
+\code{c(...)} combine multiple clusters and / or cluster nodes into one cluster returned as an of class \code{cluster}.
+}
+\description{
+Coerce an object to a cluster object
+}
+
diff --git a/man/eager.Rd b/man/eager.Rd
index e3f5817..7d7d6df 100644
--- a/man/eager.Rd
+++ b/man/eager.Rd
@@ -1,12 +1,21 @@
% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/eager.R
+% Please edit documentation in R/uniprocess.R
\name{eager}
\alias{eager}
+\alias{lazy}
\alias{transparent}
-\title{Create an eager future whose value will be resolved immediately}
+\alias{uniprocess}
+\title{Create a uniprocess future whose value will be in the current R session}
\usage{
eager(expr, envir = parent.frame(), substitute = TRUE, globals = TRUE,
- local = TRUE, earlySignal = FALSE, label = NULL, ...)
+ local = TRUE, earlySignal = FALSE, label = NULL, lazy = FALSE, ...)
+
+transparent(expr, envir = parent.frame(), substitute = TRUE,
+ globals = FALSE, local = FALSE, earlySignal = TRUE, label = NULL,
+ lazy = FALSE, ...)
+
+lazy(expr, envir = parent.frame(), substitute = TRUE, globals = TRUE,
+ local = TRUE, earlySignal = FALSE, label = NULL, lazy = TRUE, ...)
}
\arguments{
\item{expr}{An \R \link[base]{expression}.}
@@ -34,28 +43,32 @@ as soon as possible or not.}
\item{label}{An optional character string label attached to the future.}
+\item{lazy}{If \code{FALSE} (default), the future is resolved eagerly
+(immediately), otherwise not.}
+
\item{...}{Additional arguments passed to the "evaluator".}
}
\value{
-An \link{EagerFuture}.
+A \link{UniprocessFuture}.
}
\description{
-An eager future is a future that uses eager evaluation, which means
+A uniprocess future is a future that is evaluated sequentially in the
+current R session. The default is to resolve it eagerly, which means
that its \emph{value is computed and resolved immediately}, which is
how regular expressions are evaluated in R.
The only difference to R itself is that globals are validated
by default just as for all other types of futures in this package.
}
\details{
-The preferred way to create an eager future is not to call this function
-directly, but to register it via \code{\link{plan}(eager)} such that it
+The preferred way to create a uniprocess future is not to call these functions
+directly, but to register them via \code{\link{plan}(eager)} such that it
becomes the default mechanism for all futures. After this
\code{\link{future}()} and \code{\link{\%<-\%}} will create
-\emph{eager futures}.
+\emph{eager uniprocess futures}.
}
\section{transparent futures}{
-Transparent futures are eager futures configured to emulate how R
+Transparent futures are eager uniprocess futures configured to emulate how R
evaluates expressions as far as possible. For instance, errors and
warnings are signaled immediately and assignments are done to the
calling environment (without \code{local()} as default for all other
@@ -63,21 +76,22 @@ types of futures). This makes transparent futures ideal for
troubleshooting, especially when there are errors.
}
\examples{
-## Use eager futures
+## Use an eager uniprocess futures
plan(eager)
## A global variable
a <- 0
-## Create eager future (explicitly)
+## Create an eager uniprocess future (explicitly)
f <- future({
b <- 3
c <- 2
a * b * c
})
-## Since 'a' is a global variable in _eager_ future 'f',
-## it already has been resolved, and any changes to 'a'
+## Since 'a' is a global variable in future 'f' which
+## is eagerly resolved (default), this global has already
+## been resolved / incorporated, and any changes to 'a'
## at this point will _not_ affect the value of 'f'.
a <- 7
print(a)
diff --git a/man/future.Rd b/man/future.Rd
index 18da2c6..fae73b7 100644
--- a/man/future.Rd
+++ b/man/future.Rd
@@ -11,7 +11,7 @@
\title{Create a future}
\usage{
future(expr, envir = parent.frame(), substitute = TRUE,
- evaluator = plan(), ...)
+ evaluator = plan("next"), ...)
futureAssign(x, value, envir = parent.frame(), assign.env = envir,
substitute = TRUE)
@@ -19,7 +19,7 @@ futureAssign(x, value, envir = parent.frame(), assign.env = envir,
x \%<-\% value
futureCall(FUN, args = NULL, envir = parent.frame(), globals = TRUE,
- evaluator = plan(), ...)
+ evaluator = plan("next"), ...)
}
\arguments{
\item{expr}{An \R \link[base]{expression}.}
diff --git a/man/lazy.Rd b/man/lazy.Rd
deleted file mode 100644
index 4ff2597..0000000
--- a/man/lazy.Rd
+++ /dev/null
@@ -1,114 +0,0 @@
-% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/lazy.R
-\name{lazy}
-\alias{lazy}
-\title{Create a lazy future whose value will be resolved at the time when requested}
-\usage{
-lazy(expr, envir = parent.frame(), substitute = TRUE, globals = TRUE,
- local = TRUE, earlySignal = FALSE, label = NULL, ...)
-}
-\arguments{
-\item{expr}{An \R \link[base]{expression}.}
-
-\item{envir}{The \link{environment} from where global
-objects should be identified. Depending on "evaluator",
-it may also be the environment in which the expression
-is evaluated.}
-
-\item{substitute}{If TRUE, argument \code{expr} is
-\code{\link[base]{substitute}()}:ed, otherwise not.}
-
-\item{globals}{If TRUE, global objects are resolved ("frozen") at
-the point of time when the future is created, otherwise they are
-resolved when the future is resolved.}
-
-\item{local}{If TRUE, the expression is evaluated such that
-all assignments are done to local temporary environment, otherwise
-the assignments are done in the calling environment.}
-
-\item{earlySignal}{Specified whether conditions should be signaled
-as soon as possible or not.}
-
-\item{label}{An optional character string label attached to the future.}
-
-\item{...}{Additional arguments passed to the "evaluator".}
-}
-\value{
-A \link{LazyFuture}.
-}
-\description{
-A lazy future is a future that uses lazy evaluation, which means
-that its \emph{value is only computed and resolved at the time when the
-value is requested}. This means that the future will not be resolved
-if the value is never requested.
-}
-\details{
-The preferred way to create a lazy future is not to call this function
-directly, but to register it via \code{\link{plan}(lazy)} such that it
-becomes the default mechanism for all futures. After this
-\code{\link{future}()} and \code{\link{\%<-\%}} will create
-\emph{lazy futures}.
-}
-\examples{
-## Use lazy futures
-plan(lazy)
-
-## A global variable
-a <- 0
-
-## Create lazy future (explicitly)
-f <- future({
- b <- 3
- c <- 2
- a * b * c
-})
-
-## Although 'f' is a _lazy_ future and therefore
-## resolved/evaluates the future expression only
-## when the value is requested, any global variables
-## identified in the expression (here 'a') are
-## "frozen" at the time point when the future is
-## created. Because of this, the 'a' in the
-## the future expression preserved the zero value
-## although we reassign it in the global environment
-a <- 7
-print(a)
-
-v <- value(f)
-print(v)
-stopifnot(v == 0)
-
-
-## Another example illustrating that lazy futures go
-## hand-in-hand with lazy evaluation of arguments
-
-## A function that may or may not touch it's argument
-foo <- function(a, use=FALSE) {
- cat("foo() called\\n")
- if (use) cat("a=", a, "\\n", sep="")
-}
-
-## Create a future
-x \%<-\% { cat("Pow!\\n"); 1 }
-
-## Lazy evaluation where argument is not used
-foo(x, use=FALSE)
-# Outputs:
-# foo() called
-
-## Lazy evaluation where argument is used
-## Hint: 'x' will be resolved
-foo(x, use=TRUE)
-# Outputs:
-# foo() called
-# Pow!
-# a=1
-
-## Lazy evaluation where argument is used (again)
-## Hint: 'x' is already resolved
-foo(x, use=TRUE)
-# Outputs:
-# foo() called
-# a=1
-}
-
diff --git a/man/makeClusterPSOCK.Rd b/man/makeClusterPSOCK.Rd
new file mode 100644
index 0000000..c9646df
--- /dev/null
+++ b/man/makeClusterPSOCK.Rd
@@ -0,0 +1,225 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/makeClusterPSOCK.R
+\name{makeClusterPSOCK}
+\alias{makeClusterPSOCK}
+\alias{makeNodePSOCK}
+\title{Create a Parallel Socket Cluster}
+\usage{
+makeClusterPSOCK(workers, makeNode = makeNodePSOCK, port = c("auto",
+ "random"), ..., verbose = getOption("future.debug", FALSE))
+
+makeNodePSOCK(worker = "localhost", master = NULL, port,
+ connectTimeout = 2 * 60, timeout = 30 * 24 * 60 * 60, rscript = NULL,
+ homogeneous = NULL, rscript_args = NULL, methods = TRUE,
+ useXDR = TRUE, outfile = "/dev/null", renice = NA_integer_,
+ rshcmd = "ssh", user = NULL, revtunnel = TRUE, rshopts = NULL,
+ rank = 1L, manual = FALSE, dryrun = FALSE, verbose = FALSE)
+}
+\arguments{
+\item{workers}{The host names of workers (as a character vector) or
+the number of localhost workers (as a positive integer).}
+
+\item{makeNode}{A function that creates a \code{"SOCKnode"}
+or \code{"SOCK0node"} object, which represents a connection
+to a worker.}
+
+\item{port}{The port number of the master used to for communicating with all the workers (via socket connections). If an integer vector of ports, then a random one among those is chosen. If \code{"random"}, then a random port in \code{11000:11999} is chosen. If \code{"auto"} (default), then the default is taken from environment variable \env{R_PARALLEL_PORT}, otherwise \code{"random"} is used.}
+
+\item{...}{Optional arguments passed to \code{makeNode(workers[i], ..., rank=i)} where \code{i = seq_along{workers}}.}
+
+\item{verbose}{If TRUE, informative messages are outputted.}
+
+\item{worker}{The host name or IP number of the machine where the worker should run.}
+
+\item{master}{The host name or IP number of the master / calling machine, as known to the workers. If NULL (default), then the default is \code{Sys.info()[["nodename"]]} unless \code{worker} is the localhost (\code{"localhost"} or \code{"127.0.0.1"}) or \code{revtunnel = TRUE} in case it is \code{"localhost"}.}
+
+\item{connectTimeout}{The maximum time (in seconds) allowed for each sockect connection between the master and a worker to be established (defaults to 2 minutes). \emph{See note below on current lack of support on Linux and macOS systems.}}
+
+\item{timeout}{The maximum time (in seconds) allowed to pass without the master and a worker communicate with each other (defaults to 30 days).}
+
+\item{rscript, homogeneous}{The system command for launching Rscript on the worker. If \code{NULL} (default), the default is \code{"Rscript"} unless \code{homogenenous} is TRUE, which in case it is \code{file.path(R.home("bin"), "Rscript")}. Argument \code{homogenenous} defaults to FALSE, unless \code{master} is the localhost (\code{"localhost"} or \code{"127.0.0.1"}).}
+
+\item{rscript_args}{Additional arguments to \code{Rscript} (as a character vector).}
+
+\item{methods}{If TRUE, then the \pkg{methods} package is also loaded.}
+
+\item{useXDR}{If TRUE, the communication between master and workers, which is binary, will be use big-endian (XDR).}
+
+\item{outfile}{Where to direct the \link[base:stdout]{stdout} and \link[base:stderr]{stderr} connection output from the workers.}
+
+\item{renice}{A numerical 'niceness' (priority) to set for the worker processes.}
+
+\item{rshcmd}{The command to be run on the master to launch a process on another host. Only applicable if \code{machine} is not localhost.}
+
+\item{user}{(optional) The user name to be used when communicating with another host.}
+
+\item{revtunnel}{If TRUE, a reverse SSH tunneling is set up for each worker such that the worker R process sets up a socket connect to its local port \code{(port - rank + 1)} which then reaches the master on port \code{port}. If FALSE, then the worker will try to connect directly to port \code{port} on \code{master}.}
+
+\item{rshopts}{Additional arguments to \code{rshcmd} (as a character vector).}
+
+\item{rank}{A unique one-based index for each worker (automatically set).}
+
+\item{manual}{If TRUE the workers will need to be run manually.}
+
+\item{dryrun}{If TRUE, nothing is set up, but a message suggesting how to launch the worker from the terminal is outputted. This is useful for troubleshooting.}
+}
+\value{
+An object of class \code{c("SOCKcluster", "cluster")} consisting
+ of a list of \code{"SOCKnode"} or \code{"SOCK0node"} workers.
+
+\code{makeNodePSOCK()} returns a
+ \code{"SOCKnode"} or \code{"SOCK0node"} object
+ representing an established connection to a worker.
+}
+\description{
+Create a Parallel Socket Cluster
+}
+\details{
+The \code{makeClusterPSOCK()} function is similar to \code{\link[parallel:makePSOCKcluster]{makePSOCKcluster}} of the \pkg{parallel} package, but provides more flexibility in controlling the setup of the system calls that launch the background R workers and how to connect to external machines.
+
+The default is to use reverse SSH tunnelling for workers
+running on other machines. This avoids the complication of
+otherwise having to configure port forwarding in firewalls,
+which often requires static IP address but which also most
+users don't have priviligies to do themselves.
+It also has the advantage of not having to know the internal
+and / or the public IP address / host name of the master.
+
+If there is no communication between the master and a
+worker within the \code{timeout} limit, then the corresponding
+socket connection will be closed automatically. This will
+eventually result in an error in code trying to access the
+connection.
+}
+\section{Connection time out}{
+
+Argument \code{connectTimeout} does \emph{not} work properly
+on Unix and macOS due to limitation in \R itself. For more details
+on this, please R devel thread 'BUG?: On Linux setTimeLimit() fails
+to propagate timeout error when it occurs (works on Windows)' on
+2016-10-26 (\url{https://stat.ethz.ch/pipermail/r-devel/2016-October/073309.html}). When used, the timeout will eventually trigger an error, but
+it won't happen until the socket connection timeout \code{timeout}
+itself happens.
+}
+\examples{
+## Setup of three R workers on two remote machines are set up
+workers <- c("n1.remote.org", "n2.remote.org", "n1.remote.org")
+cl <- makeClusterPSOCK(workers, dryrun = TRUE)
+
+## Same setup when the two machines are on the local network and
+## have identical software setups
+cl <- makeClusterPSOCK(
+ workers,
+ revtunnel = FALSE, homogeneous = TRUE,
+ dryrun = TRUE
+)
+
+## Setup of remote worker with more detailed control on
+## authentication and reverse SSH tunnelling
+cl <- makeClusterPSOCK(
+ "remote.server.org", user = "johnny",
+ ## Manual configuration of reverse SSH tunnelling
+ revtunnel = FALSE,
+ rshopts = c("-v", "-R 11000:gateway:11942"),
+ master = "gateway", port = 11942,
+ ## Run Rscript nicely and skip any startup scripts
+ rscript = c("nice", "/path/to/Rscript"),
+ rscript_args = c("--vanilla"),
+ dryrun = TRUE
+)
+
+## Setup of Docker worker running rocker/r-base
+## (requires installation of future package)
+cl <- makeClusterPSOCK(
+ "localhost",
+ ## Launch Rscript inside Docker container
+ rscript = c(
+ "docker", "run", "--net=host", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package
+ rscript_args = c(
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
+
+
+## Setup of udocker.py worker running rocker/r-base
+## (requires installation of future package and extra quoting)
+cl <- makeClusterPSOCK(
+ "localhost",
+ ## Launch Rscript inside Docker container (using udocker)
+ rscript = c(
+ "udocker.py", "run", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package and manually launch parallel workers
+ ## (need double shQuote():s because udocker.py drops one level)
+ rscript_args = c(
+ "-e", shQuote(shQuote("install.packages('future')")),
+ "-e", shQuote(shQuote("parallel:::.slaveRSOCK()"))
+ ),
+ dryrun = TRUE
+)
+
+
+## Launching worker on Amazon AWS EC2 running one of the
+## Amazon Machine Images (AMI) provided by RStudio
+## (http://www.louisaslett.com/RStudio_AMI/)
+public_ip <- "1.2.3.4"
+ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"
+cl <- makeClusterPSOCK(
+ ## Public IP number of EC2 instance
+ public_ip,
+ ## User name (always 'ubuntu')
+ user = "ubuntu",
+ ## Use private SSH key registered with AWS
+ rshopts = c(
+ "-o", "StrictHostKeyChecking=no",
+ "-o", "IdentitiesOnly=yes",
+ "-i", ssh_private_key_file
+ ),
+ ## Set up .libPaths() for the 'ubuntu' user and
+ ## install future package
+ rscript_args = c(
+ "-e", shQuote("local({
+ p <- Sys.getenv('R_LIBS_USER')
+ dir.create(p, recursive = TRUE, showWarnings = FALSE)
+ .libPaths(p)
+ })"),
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
+
+
+## Launching worker on Google Cloud Engine (GCE) running a
+## container based VM (with a #cloud-config specification)
+public_ip <- "1.2.3.4"
+user <- "johnny"
+ssh_private_key_file <- "~/.ssh/google_compute_engine"
+cl <- makeClusterPSOCK(
+ ## Public IP number of GCE instance
+ public_ip,
+ ## User name (== SSH key label (sic!))
+ user = user,
+ ## Use private SSH key registered with GCE
+ rshopts = c(
+ "-o", "StrictHostKeyChecking=no",
+ "-o", "IdentitiesOnly=yes",
+ "-i", ssh_private_key_file
+ ),
+ ## Launch Rscript inside Docker container
+ rscript = c(
+ "docker", "run", "--net=host", "rocker/r-base",
+ "Rscript"
+ ),
+ ## Install future package
+ rscript_args = c(
+ "-e", shQuote("install.packages('future')")
+ ),
+ dryrun = TRUE
+)
+}
+
diff --git a/man/multicore.Rd b/man/multicore.Rd
index 7599186..8746d57 100644
--- a/man/multicore.Rd
+++ b/man/multicore.Rd
@@ -39,7 +39,7 @@ as soon as possible or not.}
A \link{MulticoreFuture}
If \code{workers == 1}, then all processing using done in the
current/main R session and we therefore fall back to using
-a eager future. This is also the case whenever multicore
+an eager future. This is also the case whenever multicore
processing is not supported, e.g. on Windows.
}
\description{
diff --git a/man/plan.Rd b/man/plan.Rd
index 096a752..64f186a 100644
--- a/man/plan.Rd
+++ b/man/plan.Rd
@@ -1,5 +1,5 @@
% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/plan.R
+% Please edit documentation in R/zzz.plan.R
\name{plan}
\alias{plan}
\title{Plan how to resolve a future}
@@ -82,7 +82,7 @@ and OpenLava.
\examples{
a <- b <- c <- NA_real_
-# A lazy future
+# A lazy uniprocess future
plan(lazy)
f <- future({
a <- 7
@@ -95,7 +95,7 @@ print(y)
str(list(a=a, b=b, c=c)) ## All NAs
-# An eager future
+# An eager uniprocess future
plan(eager)
f <- future({
a <- 7
diff --git a/man/sessionDetails.Rd b/man/sessionDetails.Rd
new file mode 100644
index 0000000..9258062
--- /dev/null
+++ b/man/sessionDetails.Rd
@@ -0,0 +1,22 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/sessionDetails.R
+\name{sessionDetails}
+\alias{sessionDetails}
+\title{Outputs details on the current R session}
+\usage{
+sessionDetails(env = FALSE)
+}
+\arguments{
+\item{env}{If TRUE, \code{Sys.getenv()} information is returned.}
+}
+\value{
+Invisibly a list of all details.
+}
+\description{
+Outputs details on the current R session
+}
+\details{
+Note that remote futures use \code{persistent=TRUE} by default.
+}
+\keyword{internal}
+
diff --git a/man/tweak_parallel_PSOCK.Rd b/man/tweak_parallel_PSOCK.Rd
deleted file mode 100644
index b742fdd..0000000
--- a/man/tweak_parallel_PSOCK.Rd
+++ /dev/null
@@ -1,68 +0,0 @@
-% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/tweak_parallel_PSOCK.R
-\name{tweak_parallel_PSOCK}
-\alias{tweak_parallel_PSOCK}
-\title{Tweak PSOCK backend of the parallel package}
-\usage{
-tweak_parallel_PSOCK(user = TRUE, revtunnel = TRUE, rshopts = TRUE,
- use127.0.0.1 = FALSE, reset = FALSE)
-}
-\arguments{
-\item{user}{If TRUE, parallel is tweaked to only pass username to SSH if it is specified via argument \code{user}.}
-
-\item{revtunnel}{If TRUE, parallel is tweaked to make use of reverse SSH tunneling.}
-
-\item{rshopts}{If TRUE, parallel is tweaked so it is possible to specify additional command-line options to the \code{rshcmd} executable.}
-
-\item{use127.0.0.1}{If TRUE, \code{127.0.0.1} is used instead of \code{localhost}.}
-
-\item{reset}{If TRUE, all tweaks are undone.}
-}
-\value{
-Nothing.
-}
-\description{
-Tweak PSOCK backend of the parallel package
-}
-\examples{
-\donttest{\dontrun{
-trace(system, tracer = quote(stop("Command: ", command)), print = FALSE)
-
-## Without tweaks
-try(cl <- parallel::makeCluster("remote.myserver.org", revtunnel=TRUE,
- master="localhost", homogeneous=FALSE))
-### Error in eval(expr, envir, enclos) :
-### Command:
-### ssh
-### -l hb
-### remote.myserver.org
-### "Rscript
-### --default-packages=datasets,utils,grDevices,graphics,stats,methods
-### -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11099
-### OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
-
-## With tweaks
-future:::tweak_parallel_PSOCK()
-try(cl <- parallel::makeCluster("remote.myserver.org", revtunnel=TRUE,
- master="localhost", homogeneous=FALSE))
-### Error in eval(expr, envir, enclos) :
-### Command:
-### ssh
-### -R 11624:localhost:11624
-### remote.myserver.org
-### "Rscript
-### --default-packages=datasets,utils,grDevices,graphics,stats,methods
-### -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11099
-### OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
-
-## Undo tweaks
-untrace(system)
-future:::tweak_parallel_PSOCK(reset=TRUE)
-}}
-
-}
-\references{
-\url{https://github.com/HenrikBengtsson/Wishlist-for-R/issues/32}\cr
-}
-\keyword{internal}
-
diff --git a/revdep/README.md b/revdep/README.md
index 8277f64..8745595 100644
--- a/revdep/README.md
+++ b/revdep/README.md
@@ -4,24 +4,24 @@
|setting |value |
|:--------|:----------------------------|
-|version |R version 3.3.1 (2016-06-21) |
+|version |R version 3.3.2 (2016-10-31) |
|system |x86_64, linux-gnu |
|ui |X11 |
|language |en |
|collate |en_US.UTF-8 |
-|tz |SystemV/PST8PDT |
-|date |2016-10-09 |
+|tz |Europe/Copenhagen |
+|date |2016-11-12 |
## Packages
-|package |* |version |date |source |
-|:--------|:--|:-------|:----------|:--------------|
-|digest | |0.6.10 |2016-08-02 |cran (@0.6.10) |
-|future | |1.0.1 |2016-07-04 |cran (@1.0.1) |
-|globals | |0.7.0 |2016-09-09 |cran (@0.7.0) |
-|listenv | |0.6.0 |2015-12-28 |CRAN (R 3.3.1) |
-|markdown | |0.7.7 |2015-04-22 |CRAN (R 3.3.1) |
-|R.rsp | |0.30.0 |2016-05-15 |CRAN (R 3.3.1) |
+|package |* |version |date |source |
+|:--------|:--|:----------|:----------|:---------------------------------|
+|digest | |0.6.10 |2016-08-02 |CRAN (R 3.3.1) |
+|future | |1.1.1-9000 |2016-11-12 |local (HenrikBengtsson/future at NA) |
+|globals | |0.7.1 |2016-10-14 |local |
+|listenv | |0.6.0 |2015-12-28 |CRAN (R 3.3.1) |
+|markdown | |0.7.7 |2015-04-22 |cran (@0.7.7) |
+|R.rsp | |0.30.0 |2016-05-15 |cran (@0.30.0) |
# Check results
@@ -34,20 +34,20 @@
|aroma.cn |1.6.1 | 0| 0| 0|
|aroma.core |3.0.0 | 0| 0| 1|
|calmate |0.12.1 | 0| 0| 0|
-|doFuture |0.2.1 | 0| 0| 0|
+|doFuture |0.3.0 | 0| 0| 0|
|fiery |0.2.1 | 0| 0| 0|
-|future.BatchJobs |0.13.0 | 0| 0| 0|
-|GeneBreak |1.3.0 | 0| 0| 1|
+|future.BatchJobs |0.13.1 | 0| 0| 0|
+|GeneBreak |1.4.0 | 0| 0| 1|
|MPAgenomics |1.1.2 | 0| 0| 2|
|NSA |0.0.32 | 0| 0| 6|
|pbmcapply |1.1.1 | 0| 0| 0|
-|PECA |1.9.0 | 0| 0| 1|
-|PSCBS |0.61.0 | 0| 0| 0|
-|PureCN |1.1.54 | 0| 0| 1|
-|QDNAseq |1.9.2 | 0| 0| 0|
-|Repitools |1.19.3 | 0| 0| 2|
+|PECA |1.10.0 | 0| 0| 1|
+|PSCBS |0.62.0 | 0| 0| 0|
+|PureCN |1.2.3 | 0| 1| 1|
+|QDNAseq |1.10.0 | 0| 0| 0|
+|Repitools |1.20.0 | 0| 0| 2|
|R.filesets |2.10.0 | 0| 0| 0|
-|TIN |1.5.2 | 0| 0| 2|
+|TIN |1.6.0 | 0| 0| 2|
## ACNE (0.8.1)
Maintainer: Henrik Bengtsson <henrikb at braju.com>
@@ -85,7 +85,7 @@ Bug reports: https://github.com/HenrikBengtsson/calmate/issues
0 errors | 0 warnings | 0 notes
-## doFuture (0.2.1)
+## doFuture (0.3.0)
Maintainer: Henrik Bengtsson <henrikb at braju.com>
Bug reports: https://github.com/HenrikBengtsson/doFuture/issues
@@ -97,13 +97,13 @@ Bug reports: https://github.com/thomasp85/fiery/issues
0 errors | 0 warnings | 0 notes
-## future.BatchJobs (0.13.0)
+## future.BatchJobs (0.13.1)
Maintainer: Henrik Bengtsson <henrikb at braju.com>
Bug reports: https://github.com/HenrikBengtsson/future.BatchJobs/issues
0 errors | 0 warnings | 0 notes
-## GeneBreak (1.3.0)
+## GeneBreak (1.4.0)
Maintainer: Evert van den Broek <vandenbroek.evert at gmail.com>
0 errors | 0 warnings | 1 note
@@ -249,7 +249,7 @@ Bug reports: https://github.com/kvnkuang/pbmcapply/issues
0 errors | 0 warnings | 0 notes
-## PECA (1.9.0)
+## PECA (1.10.0)
Maintainer: Tomi Suomi <tomi.suomi at utu.fi>
0 errors | 0 warnings | 1 note
@@ -263,18 +263,23 @@ Rd file 'PECA.Rd':
These lines will be truncated in the PDF manual.
```
-## PSCBS (0.61.0)
+## PSCBS (0.62.0)
Maintainer: Henrik Bengtsson <henrikb at braju.com>
Bug reports: https://github.com/HenrikBengtsson/PSCBS/issues
0 errors | 0 warnings | 0 notes
-## PureCN (1.1.54)
+## PureCN (1.2.3)
Maintainer: Markus Riester <markus.riester at novartis.com>
-0 errors | 0 warnings | 1 note
+0 errors | 1 warning | 1 note
```
+checking whether package ‘PureCN’ can be installed ... WARNING
+Found the following significant warnings:
+ Warning: namespace ‘Matrix’ is not available and has been replaced
+See ‘/home/hb/repositories/future/revdep/checks/PureCN.Rcheck/00install.out’ for details.
+
checking R code for possible problems ... NOTE
Found the following calls to data() loading into the global environment:
File ‘PureCN/R/bootstrapResults.R’:
@@ -298,13 +303,13 @@ File ‘PureCN/R/readCurationFile.R’:
See section ‘Good practice’ in ‘?data’.
```
-## QDNAseq (1.9.2)
+## QDNAseq (1.10.0)
Maintainer: Daoud Sie <d.sie at vumc.nl>
Bug reports: https://github.com/ccagc/QDNAseq/issues
0 errors | 0 warnings | 0 notes
-## Repitools (1.19.3)
+## Repitools (1.20.0)
Maintainer: Mark Robinson <mark.robinson at imls.uzh.ch>
0 errors | 0 warnings | 2 notes
@@ -363,7 +368,7 @@ Bug reports: https://github.com/HenrikBengtsson/R.filesets/issues
0 errors | 0 warnings | 0 notes
-## TIN (1.5.2)
+## TIN (1.6.0)
Maintainer: Bjarne Johannessen <bjajoh at rr-research.no>
0 errors | 0 warnings | 2 notes
diff --git a/revdep/checks.rds b/revdep/checks.rds
index 6693cae..325050b 100644
Binary files a/revdep/checks.rds and b/revdep/checks.rds differ
diff --git a/revdep/problems.md b/revdep/problems.md
index 29435d4..cde0fc7 100644
--- a/revdep/problems.md
+++ b/revdep/problems.md
@@ -4,29 +4,64 @@
|setting |value |
|:--------|:----------------------------|
-|version |R version 3.3.1 (2016-06-21) |
+|version |R version 3.3.2 (2016-10-31) |
|system |x86_64, linux-gnu |
|ui |X11 |
|language |en |
|collate |en_US.UTF-8 |
-|tz |SystemV/PST8PDT |
-|date |2016-10-09 |
+|tz |Europe/Copenhagen |
+|date |2016-11-12 |
## Packages
-|package |* |version |date |source |
-|:--------|:--|:-------|:----------|:--------------|
-|digest | |0.6.10 |2016-08-02 |cran (@0.6.10) |
-|future | |1.0.1 |2016-07-04 |cran (@1.0.1) |
-|globals | |0.7.0 |2016-09-09 |cran (@0.7.0) |
-|listenv | |0.6.0 |2015-12-28 |CRAN (R 3.3.1) |
-|markdown | |0.7.7 |2015-04-22 |CRAN (R 3.3.1) |
-|R.rsp | |0.30.0 |2016-05-15 |CRAN (R 3.3.1) |
+|package |* |version |date |source |
+|:--------|:--|:----------|:----------|:---------------------------------|
+|digest | |0.6.10 |2016-08-02 |CRAN (R 3.3.1) |
+|future | |1.1.1-9000 |2016-11-12 |local (HenrikBengtsson/future at NA) |
+|globals | |0.7.1 |2016-10-14 |local |
+|listenv | |0.6.0 |2015-12-28 |CRAN (R 3.3.1) |
+|markdown | |0.7.7 |2015-04-22 |cran (@0.7.7) |
+|R.rsp | |0.30.0 |2016-05-15 |cran (@0.30.0) |
# Check results
-0 packages with problems
+1 packages with problems
+|package |version | errors| warnings| notes|
+|:-------|:-------|------:|--------:|-----:|
+|PureCN |1.2.3 | 0| 1| 1|
+## PureCN (1.2.3)
+Maintainer: Markus Riester <markus.riester at novartis.com>
+0 errors | 1 warning | 1 note
+
+```
+checking whether package ‘PureCN’ can be installed ... WARNING
+Found the following significant warnings:
+ Warning: namespace ‘Matrix’ is not available and has been replaced
+See ‘/home/hb/repositories/future/revdep/checks/PureCN.Rcheck/00install.out’ for details.
+
+checking R code for possible problems ... NOTE
+Found the following calls to data() loading into the global environment:
+File ‘PureCN/R/bootstrapResults.R’:
+ data(purecn.example.output)
+File ‘PureCN/R/callAlterations.R’:
+ data(purecn.example.output)
+ data(purecn.example.output)
+File ‘PureCN/R/callLOH.R’:
+ data(purecn.example.output)
+File ‘PureCN/R/createCurationFile.R’:
+ data(purecn.example.output)
+File ‘PureCN/R/curateResults.R’:
+ data(purecn.example.output)
+ data(purecn.example.output)
+File ‘PureCN/R/plotAbs.R’:
+ data(purecn.example.output)
+File ‘PureCN/R/predictSomatic.R’:
+ data(purecn.example.output)
+File ‘PureCN/R/readCurationFile.R’:
+ data(purecn.example.output)
+See section ‘Good practice’ in ‘?data’.
+```
diff --git a/revdep/timing.md b/revdep/timing.md
index f37edd1..8681cd1 100644
--- a/revdep/timing.md
+++ b/revdep/timing.md
@@ -2,24 +2,24 @@
| |package |version | check_time|
|:--|:----------------|:-------|----------:|
-|15 |PureCN |1.1.54 | 519.5|
-|17 |Repitools |1.19.3 | 324.9|
-|7 |fiery |0.2.1 | 252|
-|14 |PSCBS |0.61.0 | 192.5|
-|2 |aroma.affymetrix |3.0.0 | 156.5|
-|16 |QDNAseq |1.9.2 | 144.6|
-|19 |TIN |1.5.2 | 140.7|
-|9 |GeneBreak |1.3.0 | 121.1|
-|6 |doFuture |0.2.1 | 118.2|
-|13 |PECA |1.9.0 | 100.5|
-|8 |future.BatchJobs |0.13.0 | 92.7|
-|18 |R.filesets |2.10.0 | 91.1|
-|4 |aroma.core |3.0.0 | 90|
-|10 |MPAgenomics |1.1.2 | 46|
-|3 |aroma.cn |1.6.1 | 39.1|
-|5 |calmate |0.12.1 | 30.5|
-|1 |ACNE |0.8.1 | 28|
-|11 |NSA |0.0.32 | 25.8|
-|12 |pbmcapply |1.1.1 | 10.1|
+|15 |PureCN |1.2.3 | 654|
+|17 |Repitools |1.20.0 | 441.5|
+|14 |PSCBS |0.62.0 | 281.6|
+|7 |fiery |0.2.1 | 259.8|
+|16 |QDNAseq |1.10.0 | 205.1|
+|19 |TIN |1.6.0 | 199.8|
+|13 |PECA |1.10.0 | 193.1|
+|9 |GeneBreak |1.4.0 | 178.4|
+|2 |aroma.affymetrix |3.0.0 | 170.2|
+|8 |future.BatchJobs |0.13.1 | 149.5|
+|6 |doFuture |0.3.0 | 135.7|
+|18 |R.filesets |2.10.0 | 131.4|
+|4 |aroma.core |3.0.0 | 124.7|
+|10 |MPAgenomics |1.1.2 | 73|
+|3 |aroma.cn |1.6.1 | 72.5|
+|1 |ACNE |0.8.1 | 58.6|
+|5 |calmate |0.12.1 | 49.4|
+|11 |NSA |0.0.32 | 36.8|
+|12 |pbmcapply |1.1.1 | 24|
diff --git a/tests/Future-class.R b/tests/Future-class.R
index 4e9c311..3c26ea5 100644
--- a/tests/Future-class.R
+++ b/tests/Future-class.R
@@ -23,9 +23,10 @@ print(expr)
stopifnot(is.call(expr))
clazzes <- list(
+ uniprocess = UniprocessFuture,
+ multisession = function(...) MultisessionFuture(..., workers=2L),
eager = EagerFuture,
- lazy = LazyFuture,
- multisession = function(...) MultisessionFuture(..., workers=2L)
+ lazy = LazyFuture
)
if (supportsMulticore()) clazzes$multicore = function(...) MulticoreFuture(..., workers=2L)
diff --git a/tests/as.cluster.R b/tests/as.cluster.R
new file mode 100644
index 0000000..e5f90e5
--- /dev/null
+++ b/tests/as.cluster.R
@@ -0,0 +1,64 @@
+source("incl/start.R")
+stopCluster <- parallel::stopCluster
+
+message("*** cluster operations ...")
+
+message("*** cluster operations - as.cluster() ...")
+
+local({
+ cl <- makeClusterPSOCK(1L)
+ on.exit(stopCluster(cl))
+ print(cl)
+ cl1 <- as.cluster(cl)
+ print(cl1)
+ stopifnot(inherits(cl1, "cluster"), identical(cl1, cl))
+
+ node <- cl[[1]]
+ print(node)
+ cl2 <- as.cluster(cl)
+ stopifnot(inherits(cl2, "cluster"), length(cl2) == 1L,
+ identical(cl2[[1]], node))
+})
+
+message("*** cluster operations - as.cluster() ... DONE")
+
+
+message("*** cluster operations - c(...) ...")
+
+local({
+ cl1 <- makeClusterPSOCK(1L)
+ on.exit(stopCluster(cl1))
+ print(cl1)
+
+ cl2 <- makeClusterPSOCK(rep("localhost", times = 2L))
+ on.exit(stopCluster(cl2), add = TRUE)
+ print(cl2)
+
+ cl <- c(cl1, cl2)
+ print(cl)
+
+ stopifnot(inherits(cl, "cluster"), length(cl) == 3L)
+ stopifnot(identical(cl[1], cl1),
+ identical(cl[2], cl2[1]), identical(cl[3], cl2[2]))
+})
+
+message("*** cluster operations - c(...) ... DONE")
+
+
+message("*** cluster operations - makeClusterPSOCK(remotes) ...")
+
+remotes <- Sys.getenv("R_FUTURE_TESTS_REMOTES", "localhost,localhost")
+remotes <- gsub(" ", "", unlist(strsplit(remotes, split=",")))
+remotes <- remotes[nzchar(remotes)]
+if (length(remotes) > 0) {
+ message("Remotes: ", paste(sQuote(remotes), collapse = ", "))
+ cl <- makeClusterPSOCK(remotes, verbose = TRUE)
+ print(cl)
+ stopCluster(cl)
+}
+
+message("*** cluster operations - makeClusterPSOCK(remotes) ... DONE")
+
+message("*** cluster operations ... DONE")
+
+source("incl/end.R")
diff --git a/tests/cluster.R b/tests/cluster.R
index f06521d..e0abff5 100644
--- a/tests/cluster.R
+++ b/tests/cluster.R
@@ -179,11 +179,43 @@ message("*** cluster() - setDefaultCluster() ... DONE")
message("*** cluster() - exceptions ...")
res <- try(cluster(42L, workers=NA), silent=TRUE)
+print(res)
stopifnot(inherits(res, "try-error"))
message("*** cluster() - exceptions ... DONE")
+message("*** cluster() - crashed worker ...")
+
+cl <- parallel::makeCluster("localhost")
+plan(cluster, workers = cl)
+x %<-% 42L
+stopifnot(x == 42L)
+
+## Force R worker to quit
+x %<-% quit(save = "no")
+res <- tryCatch(y <- x, error = identity)
+print(res)
+stopifnot(
+ inherits(res, "simpleError"),
+ inherits(res, "FutureError")
+)
+
+## This is needed in order to reset the ClusterRegistry
+future:::ClusterRegistry("stop")
+
+## An alternative is to do:
+## plan(uniprocess); x %<-% NULL; print(x)
+
+## Verify that the reset worked
+cl <- parallel::makeCluster("localhost")
+plan(cluster, workers = cl)
+x %<-% 42L
+stopifnot(x == 42L)
+
+message("*** cluster() - crashed worker ... DONE")
+
+
message("*** cluster() ... DONE")
## Cleanup
diff --git a/tests/eager.R b/tests/eager.R
index 451278e..7050175 100644
--- a/tests/eager.R
+++ b/tests/eager.R
@@ -9,7 +9,7 @@ message(sprintf("*** eager(..., globals=%s) without globals", globals))
f <- eager({
42L
}, globals=globals)
-stopifnot(inherits(f, "EagerFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), !f$lazy, inherits(f, "EagerFuture"))
print(resolved(f))
stopifnot(resolved(f))
@@ -44,7 +44,7 @@ f <- eager({
1
}, globals=globals)
print(f)
-stopifnot(inherits(f, "EagerFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), !f$lazy, inherits(f, "EagerFuture"))
res <- try(value(f), silent=TRUE)
print(res)
diff --git a/tests/early-signaling.R b/tests/early-signaling.R
index d89bd2a..8086152 100644
--- a/tests/early-signaling.R
+++ b/tests/early-signaling.R
@@ -4,9 +4,9 @@ options(future.debug=FALSE)
message("*** Early signaling of conditions ...")
-message("*** Early signaling of conditions with eager futures ...")
+message("*** Early signaling of conditions with uniprocess futures ...")
-plan(eager)
+plan(uniprocess)
f <- future({ stop("bang!") })
Sys.sleep(1.0)
r <- resolved(f)
@@ -14,11 +14,11 @@ stopifnot(r)
v <- try(value(f), silent=TRUE)
stopifnot(inherits(v, "try-error"))
-plan(eager, earlySignal=TRUE)
+plan(uniprocess, earlySignal=TRUE)
f <- try(future({ stop("bang!") }), silent=TRUE)
stopifnot(inherits(f, "try-error"))
-message("*** Early signaling of conditions with eager futures ... DONE")
+message("*** Early signaling of conditions with uniprocess futures ... DONE")
message("*** Early signaling of conditions with lazy futures ...")
diff --git a/tests/futures.R b/tests/futures.R
index 9753102..915e7fa 100644
--- a/tests/futures.R
+++ b/tests/futures.R
@@ -47,7 +47,7 @@ for (cores in 1:min(3L, availableCores())) {
x$a <- 1
x$b <- future(2)
- x$c <- 3
+ x$c <- future(NULL)
if (type != "list") x$d %<-% { 4 }
if (type != "environment") x[[6]] <- 6
str(x)
diff --git a/tests/globals,manual.R b/tests/globals,manual.R
index 97aa54c..c40e150 100644
--- a/tests/globals,manual.R
+++ b/tests/globals,manual.R
@@ -61,7 +61,7 @@ for (strategy in supportedStrategies()) {
print(f)
rm(list=names(globals))
y <- tryCatch(value(f), error = identity)
- if (!inherits(f, c("EagerFuture", "MulticoreFuture"))) {
+ if (!inherits(f, c("UniprocessFuture", "MulticoreFuture"))) {
stopifnot(inherits(y, "simpleError"))
}
diff --git a/tests/globals,tricky.R b/tests/globals,tricky.R
index f7c96fd..144ce42 100644
--- a/tests/globals,tricky.R
+++ b/tests/globals,tricky.R
@@ -88,7 +88,7 @@ for (cores in 1:min(3L, availableCores())) {
## overwritten by the name of the last package attached
## by the future.
pkg <- "foo"
- f <- eager({ pkg })
+ f <- uniprocess({ pkg })
v <- value(f)
message(sprintf("value(f)=%s", sQuote(v)))
stopifnot(pkg == "foo", v == "foo")
diff --git a/tests/incl/start,load-only.R b/tests/incl/start,load-only.R
index 19f7ec8..acf9d8f 100644
--- a/tests/incl/start,load-only.R
+++ b/tests/incl/start,load-only.R
@@ -18,6 +18,7 @@ future::plan("eager")
asIEC <- future:::asIEC
ClusterRegistry <- future:::ClusterRegistry
constant <- future:::constant
+uniprocess <- future:::uniprocess ## To become public
detectCores <- future:::detectCores
flapply <- future:::flapply
FutureRegistry <- future:::FutureRegistry
diff --git a/tests/invalid-owner.R b/tests/invalid-owner.R
index cb3a922..5575f07 100644
--- a/tests/invalid-owner.R
+++ b/tests/invalid-owner.R
@@ -14,26 +14,26 @@ plan(multisession, workers=3L)
message("*** future() - invalid ownership ...")
## This R process
-uuid <- future:::uuid()
-cat(sprintf("Main R process: %s\n", uuid))
+session_uuid <- future:::session_uuid(attributes = TRUE)
+cat(sprintf("Main R process: %s\n", session_uuid))
message("- Asserting ownership ...")
message("Creating future #1:")
-f1 <- future({ future:::uuid() })
+f1 <- future({ future:::session_uuid(attributes = TRUE) })
stopifnot(inherits(f1, "MultisessionFuture"))
cat(sprintf("Future #1 session: %d\n", f1$node))
v1 <- value(f1)
cat(sprintf("Future #1 R process: %s\n", v1))
-stopifnot(v1 != uuid)
+stopifnot(v1 != session_uuid)
message("Creating future #2:")
-f2 <- future({ future:::uuid() })
+f2 <- future({ future:::session_uuid(attributes = TRUE) })
stopifnot(inherits(f2, "MultisessionFuture"))
cat(sprintf("Future #2 session: %d\n", f2$node))
v2 <- value(f2)
cat(sprintf("Future #2 R process: %s\n", v2))
-stopifnot(v2 != uuid)
+stopifnot(v2 != session_uuid)
message("Creating future #3:")
f3 <- future({ f1$owner })
@@ -41,7 +41,7 @@ stopifnot(inherits(f3, "MultisessionFuture"))
cat(sprintf("Future #3 session: %d\n", f3$node))
v3 <- value(f3)
cat(sprintf("Future #3 owner: %s\n", v3))
-stopifnot(v3 == uuid)
+stopifnot(v3 == session_uuid)
message("Creating future #4:")
f4 <- future({ f1$owner })
@@ -49,10 +49,10 @@ stopifnot(inherits(f4, "MultisessionFuture"))
cat(sprintf("Future #4 session: %d\n", f4$node))
v4 <- value(f4)
cat(sprintf("Future #4 owner: %s\n", v4))
-stopifnot(v4 == uuid)
+stopifnot(v4 == session_uuid)
message("Creating future #5:")
-f5 <- future({ stopifnot(f1$owner != future:::uuid()); "not-owner" })
+f5 <- future({ stopifnot(f1$owner != future:::session_uuid(attributes = TRUE)); "not-owner" })
stopifnot(inherits(f5, "MultisessionFuture"))
v5 <- value(f5)
stopifnot(v5 == "not-owner")
@@ -66,14 +66,14 @@ message("Creating future #1:")
f1 <- future({ Sys.sleep(5); 42L })
print(f1)
cat(sprintf("Future #1 session: %d\n", f1$node))
-stopifnot(identical(f1$owner, uuid))
+stopifnot(identical(f1$owner, session_uuid))
print(usedNodes(f1))
message("Creating future #2:")
f2 <- future({ value(f1) })
print(f2)
cat(sprintf("Future #2 session: %d\n", f2$node))
-stopifnot(identical(f2$owner, uuid))
+stopifnot(identical(f2$owner, session_uuid))
print(usedNodes(f2))
message("Getting value of future #2:")
diff --git a/tests/lazy.R b/tests/lazy.R
index cc7b24e..c92a104 100644
--- a/tests/lazy.R
+++ b/tests/lazy.R
@@ -9,7 +9,7 @@ message("*** lazy() without globals")
f <- lazy({
42L
})
-stopifnot(inherits(f, "LazyFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), f$lazy)
## Check whether a lazy future is resolved
## or not will force evaluation
diff --git a/tests/multicore.R b/tests/multicore.R
index 41322a5..b514895 100644
--- a/tests/multicore.R
+++ b/tests/multicore.R
@@ -9,7 +9,7 @@ for (cores in 1:min(3L, availableCores("multicore"))) {
options(mc.cores=cores-1L)
if (!supportsMulticore()) {
- message(sprintf("Multicore futures are not supporting on '%s'. Falling back to use synchroneous eager futures", .Platform$OS.type))
+ message(sprintf("Multicore futures are not supporting on '%s'. Falling back to use synchronous uniprocess futures", .Platform$OS.type))
}
for (globals in c(FALSE, TRUE)) {
@@ -19,7 +19,7 @@ for (cores in 1:min(3L, availableCores("multicore"))) {
f <- multicore({
42L
}, globals=globals)
- stopifnot(inherits(f, "MulticoreFuture") || ((cores ==1 || !supportsMulticore()) && inherits(f, "EagerFuture")))
+ stopifnot(inherits(f, "MulticoreFuture") || ((cores ==1 || !supportsMulticore()) && inherits(f, "UniprocessFuture")))
print(resolved(f))
y <- value(f)
diff --git a/tests/multisession.R b/tests/multisession.R
index 142b278..48b98c1 100644
--- a/tests/multisession.R
+++ b/tests/multisession.R
@@ -13,7 +13,7 @@ for (cores in 1:min(3L, availableCores())) {
42L
})
print(f)
- stopifnot(inherits(f, "ClusterFuture") || inherits(f, "LazyFuture"))
+ stopifnot(inherits(f, "ClusterFuture") || (inherits(f, "UniprocessFuture") && f$lazy))
print(resolved(f))
y <- value(f)
diff --git a/tests/nbrOfWorkers.R b/tests/nbrOfWorkers.R
index b4acfa1..b0cbf82 100644
--- a/tests/nbrOfWorkers.R
+++ b/tests/nbrOfWorkers.R
@@ -2,7 +2,7 @@ source("incl/start.R")
message("*** nbrOfWorkers() ...")
-strategies <- c("eager", "lazy", "transparent")
+strategies <- c("uniprocess", "transparent", "eager", "lazy")
for (strategy in strategies) {
message("Type of future: ", strategy)
diff --git a/tests/nested_futures,mc.cores.R b/tests/nested_futures,mc.cores.R
index ab2a77b..5c61a2a 100644
--- a/tests/nested_futures,mc.cores.R
+++ b/tests/nested_futures,mc.cores.R
@@ -19,8 +19,8 @@ for (mc in 0:3) {
mc2 <- min(mc, cores)
for (strategy in strategies) {
- message(sprintf("plan(list('eager', '%s')):", strategy))
- plan(list('eager', strategy))
+ message(sprintf("plan(list('uniprocess', '%s')):", strategy))
+ plan(list('uniprocess', strategy))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
@@ -32,8 +32,8 @@ for (mc in 0:3) {
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot(((mc2 <= 1 || a$cores <= 2) && a$pid2 == a$pid1) || (a$pid2 != a$pid1))
- message(sprintf("plan(list('eager', '%s':3)):", strategy))
- plan(list('eager', tweak(strategy, workers=3)))
+ message(sprintf("plan(list('uniprocess', '%s':3)):", strategy))
+ plan(list('uniprocess', tweak(strategy, workers=3)))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
@@ -45,8 +45,8 @@ for (mc in 0:3) {
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot((mc2 <= 1 && a$pid2 == a$pid1) || (a$pid2 != a$pid1))
- message(sprintf("plan(list('%s', 'eager')):", strategy))
- plan(list(strategy, 'eager'))
+ message(sprintf("plan(list('%s', 'uniprocess')):", strategy))
+ plan(list(strategy, 'uniprocess'))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
diff --git a/tests/nested_futures.R b/tests/nested_futures.R
index a57ea5e..6ec56b9 100644
--- a/tests/nested_futures.R
+++ b/tests/nested_futures.R
@@ -16,26 +16,26 @@ for (strategy1 in strategies) {
all(names(nested) == c("a", "b")),
inherits(plan(), strategy1)
)
-
+
x %<-% {
a <- 1L
## IMPORTANT: Use future::plan() - not just plan() - otherwise
## we're exporting the plan() function including its local stack!
- plan_a <- future::plan("list")
+ plan_a <- unclass(future::plan("list"))
nested_a <- nested[-1]
-
+
stopifnot(
length(nested_a) == 1L,
length(plan_a) == 1L,
- inherits(plan_a[[1]], "future"),
+ inherits(plan_a[[1]], "future"),
all.equal(plan_a, nested_a),
inherits(future::plan(), strategy2)
)
-
+
y %<-% {
b <- 2L
-
+
## IMPORTANT: Use future::plan() - not just plan() - otherwise
## we're exporting the plan() function including its local stack!
plan_b <- future::plan("list")
@@ -43,36 +43,36 @@ for (strategy1 in strategies) {
stopifnot(
length(nested_b) == 0L,
length(plan_b) == 1L,
- inherits(plan_b[[1]], "future"),
- inherits(future::plan(), getOption("future.default", "eager"))
+ inherits(plan_b[[1]], "future"),
+ inherits(future::plan(), getOption("future.default", "uniprocess"))
)
- list(a = a, nested_a = nested_a, plan_a = plan_a,
- b = b, nested_b = nested_b, plan_b = plan_b)
+ list(a = a, nested_a = nested_a, plan_a = plan_a,
+ b = b, nested_b = nested_b, plan_b = plan_b)
}
y
}
-
+
str(x)
stopifnot(
length(x) == 3 * length(nested),
all(names(x) == c("a", "nested_a", "plan_a",
"b", "nested_b", "plan_b")),
-
+
x$a == 1L,
length(x$nested_a) == 1L,
is.list(x$plan_a),
length(x$plan_a) == 1L,
inherits(x$plan_a[[1]], "future"),
all.equal(x$plan_a, nested[-1L]),
-
+
x$b == 2L,
length(x$nested_b) == 0L,
is.list(x$plan_b),
length(x$plan_b) == 1L,
inherits(x$plan_b[[1]], "future"),
- inherits(x$plan_b[[1]], getOption("future.default", "eager"))
+ inherits(x$plan_b[[1]], getOption("future.default", "uniprocess"))
)
rm(list=c("nested", "x"))
diff --git a/tests/plan.R b/tests/plan.R
index fa03e02..c37ee0c 100644
--- a/tests/plan.R
+++ b/tests/plan.R
@@ -57,8 +57,8 @@ print(v)
stopifnot(v == 0)
-message("*** plan(eager)")
-plan(eager)
+message("*** plan(uniprocess)")
+plan(uniprocess)
a <- 0
f <- future({
b <- 3
@@ -71,15 +71,15 @@ print(v)
stopifnot(v == 0)
-message("*** plan('eager')")
+message("*** plan('uniprocess')")
## Setting strategy by name
plan("lazy")
print(plan())
message("*** plan() and overriding defaults")
-message("*** plan(eager)")
-plan(eager)
+message("*** plan(uniprocess)")
+plan(uniprocess)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == TRUE)
@@ -88,8 +88,8 @@ f <- future({ x <- 1 })
print(value(f))
stopifnot(x == 0)
-message("*** plan(eager, local=FALSE)")
-plan(eager, local=FALSE)
+message("*** plan(uniprocess, local=FALSE)")
+plan(uniprocess, local=FALSE)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
@@ -98,22 +98,22 @@ f <- future({ x <- 1 })
print(value(f))
stopifnot(x == 1)
-message("*** plan(eager, local=FALSE, abc=1, def=TRUE)")
-plan(eager, local=FALSE, abc=1, def=TRUE)
+message("*** plan(uniprocess, local=FALSE, abc=1, def=TRUE)")
+plan(uniprocess, local=FALSE, abc=1, def=TRUE)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
-message("*** plan(eager(local=FALSE))")
+message("*** plan(uniprocess(local=FALSE))")
plan(lazy)
-plan(eager(local=FALSE))
+plan(uniprocess(local=FALSE))
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
-message("*** plan(tweak(eager, local=FALSE))")
+message("*** plan(tweak(uniprocess, local=FALSE))")
plan(lazy)
-plan(tweak(eager, local=FALSE))
+plan(tweak(uniprocess, local=FALSE))
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
@@ -122,37 +122,37 @@ stopifnot(formals(fcn)$local == FALSE)
message("*** old <- plan(new)")
truth <- plan()
old <- plan(lazy, local=FALSE)
-stopifnot(identical(old, truth))
+stopifnot(identical(unclass(old), unclass(truth)))
curr <- plan() ## curr == lazy(local=FALSE)
prev <- plan(old) ## prev == lazy(local=FALSE)
-stopifnot(identical(curr, prev))
+stopifnot(identical(unclass(curr), unclass(prev)))
curr <- plan() ## curr == old
-stopifnot(identical(curr, old))
-stopifnot(identical(curr, truth))
+stopifnot(identical(unclass(curr), unclass(old)))
+stopifnot(identical(unclass(curr), unclass(truth)))
-message("*** %plan% 'eager'")
+message("*** %plan% 'uniprocess'")
plan(lazy)
-x %<-% { a <- 1 } %plan% "eager"
+x %<-% { a <- 1 } %plan% "uniprocess"
stopifnot(identical(body(plan()), body(lazy)))
-message("*** %plan% eager")
+message("*** %plan% uniprocess")
plan(lazy)
## %plan% can operate on any expression, so it
## works just as an withPlan({ ... }, plan=...)
-fun <- { plan() } %plan% eager
+fun <- { plan() } %plan% uniprocess
f <- fun(1)
-stopifnot(inherits(f, "EagerFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), !f$lazy, inherits(f, "UniprocessFuture"))
-x %<-% { a <- 1 } %plan% eager
+x %<-% { a <- 1 } %plan% uniprocess
stopifnot(identical(body(plan()), body(lazy)))
-message("*** %plan% eager(local=FALSE) ")
+message("*** %plan% uniprocess(local=FALSE) ")
plan(lazy)
a <- 0
-x %<-% { a } %plan% eager(local=FALSE)
+x %<-% { a } %plan% uniprocess(local=FALSE)
a <- 42
print(x)
stopifnot(x == 0)
@@ -165,7 +165,7 @@ c %<-% {
a %<-% {
message("Resolving 'a'")
2
- } %plan% eager
+ } %plan% uniprocess
b %<-% {
message("Resolving 'b'")
-9 * a
@@ -181,35 +181,35 @@ stopifnot(c == 6)
message("*** plan() by functions and character names ... ")
-plan(eager)
+plan(uniprocess)
a %<-% 42
stopifnot(a == 42)
-plan("eager")
+plan("uniprocess")
a %<-% 42
stopifnot(a == 42)
-plan(list(eager))
+plan(list(uniprocess))
a %<-% 42
stopifnot(a == 42)
-plan(list("eager"))
+plan(list("uniprocess"))
a %<-% 42
stopifnot(a == 42)
-plan(list(eager, lazy))
+plan(list(uniprocess, lazy))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
-plan(list("eager", lazy))
+plan(list("uniprocess", lazy))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
-plan(list(eager, "lazy"))
+plan(list(uniprocess, "lazy"))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
-plan(list("eager", "lazy"))
+plan(list("uniprocess", "lazy"))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
@@ -218,7 +218,7 @@ message("*** plan() by functions and character names ... DONE")
message("*** plan() w/ commands ...")
-plan(list(eager, eager))
+plan(list(uniprocess, uniprocess))
res <- plan("list")
print(res)
stopifnot(length(res) == 2)
diff --git a/tests/rng.R b/tests/rng.R
index ab739ae..c49ee93 100644
--- a/tests/rng.R
+++ b/tests/rng.R
@@ -45,7 +45,7 @@ dummy <- sample(0:9, size=1L)
seed0 <- .Random.seed
## Reference sample with fixed random seed
-plan("eager")
+plan("uniprocess")
y0 <- fsample(0:9, seed=42L)
## Assert that random seed is reset
diff --git a/tests/sessionDetails.R b/tests/sessionDetails.R
new file mode 100644
index 0000000..f8faaa0
--- /dev/null
+++ b/tests/sessionDetails.R
@@ -0,0 +1,20 @@
+source("incl/start.R")
+
+message("*** Session details ...")
+
+x <- sessionDetails()
+print(x)
+
+x <- sessionDetails(env = FALSE)
+print(x)
+
+x <- sessionDetails(env = TRUE)
+print(x)
+
+y <- x[1:3]
+print(y)
+stopifnot(all(class(y) == class(x)))
+
+message("*** Session details ... DONE")
+
+source("incl/end.R")
diff --git a/tests/startup.R b/tests/startup.R
index 92a61b2..e6a102e 100644
--- a/tests/startup.R
+++ b/tests/startup.R
@@ -62,7 +62,7 @@ options(future.plan=NULL, future.cmdargs=c("-p", 1))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
-stopifnot(inherits(strategy, "eager"))
+stopifnot(inherits(strategy, "uniprocess"))
plan("default")
message("- .onLoad() w/ -p 1 ... DONE")
@@ -72,7 +72,7 @@ options(future.plan=NULL, future.cmdargs="-parallel=1")
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
-stopifnot(inherits(strategy, "eager"))
+stopifnot(inherits(strategy, "uniprocess"))
plan("default")
message("- .onLoad() w/ --parallel=1 ... DONE")
diff --git a/tests/transparent.R b/tests/transparent.R
index 5269842..3e8bcd8 100644
--- a/tests/transparent.R
+++ b/tests/transparent.R
@@ -8,7 +8,7 @@ f <- try(transparent({
42L
}), silent=FALSE)
print(f)
-stopifnot(inherits(f, "EagerFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), !f$lazy)
print(resolved(f))
y <- value(f)
@@ -22,7 +22,7 @@ f <- try(future({
42L
}), silent=FALSE)
print(f)
-stopifnot(inherits(f, "EagerFuture"))
+stopifnot(inherits(f, "UniprocessFuture"), !f$lazy)
print(resolved(f))
y <- value(f)
diff --git a/tests/tweak.R b/tests/tweak.R
index 2ad177d..4f644ef 100644
--- a/tests/tweak.R
+++ b/tests/tweak.R
@@ -57,7 +57,7 @@ stopifnot(identical(formals(lazy2)$local, FALSE))
message("*** y %<-% { expr } %tweak% tweaks ...")
-plan(eager)
+plan(uniprocess)
a <- 0
x %<-% { a <- 1; a }
@@ -69,7 +69,7 @@ print(x)
stopifnot(a == 2, x == 2)
-plan(eager, local=FALSE)
+plan(uniprocess, local=FALSE)
a <- 0
x %<-% { a <- 1; a }
@@ -82,7 +82,7 @@ stopifnot(a == 1, x == 2)
# Preserve nested futures
-plan(list(A=eager, B=tweak(eager, local=FALSE)))
+plan(list(A=uniprocess, B=tweak(uniprocess, local=FALSE)))
a <- 0
x %<-% {
@@ -111,7 +111,7 @@ res <- tryCatch(tweak(multisession, gc=TRUE), condition=identity)
stopifnot(inherits(res, "tweaked"))
## Argument 'gc' is unknown
-res <- tryCatch(tweak(eager, gc=TRUE), condition=identity)
+res <- tryCatch(tweak(uniprocess, gc=TRUE), condition=identity)
stopifnot(inherits(res, "warning"))
res <- tryCatch(tweak(multicore, gc=TRUE), condition=identity)
@@ -126,7 +126,7 @@ message("*** tweak() - exceptions ...")
res <- try(tweak("<unknown-future-strategy>"), silent=TRUE)
stopifnot(inherits(res, "try-error"))
-res <- try(tweak(eager, "unnamed-argument"), silent=TRUE)
+res <- try(tweak(uniprocess, "unnamed-argument"), silent=TRUE)
stopifnot(inherits(res, "try-error"))
message("*** tweak() - exceptions ... DONE")
diff --git a/tests/uuid.R b/tests/uuid.R
index a59b0f9..67cbb0d 100644
--- a/tests/uuid.R
+++ b/tests/uuid.R
@@ -1,18 +1,18 @@
source("incl/start.R")
-uuid <- future:::uuid
+session_uuid <- future:::session_uuid
-message("*** uuid() ...")
+message("*** session_uuid() ...")
-id0 <- uuid()
+id0 <- session_uuid()
print(id0)
-## Reset UUID (hack)
-environment(uuid)$value <- NULL
+## Reset session UUID (hack)
+environment(session_uuid)$value <- NULL
-id <- uuid()
+id <- session_uuid()
print(id)
stopifnot(id != id0)
-message("*** uuid() ... DONE")
+message("*** session_uuid() ... DONE")
source("incl/end.R")
diff --git a/vignettes/future-1-overview.md.rsp b/vignettes/future-1-overview.md.rsp
index 6b21a6c..40c9e01 100644
--- a/vignettes/future-1-overview.md.rsp
+++ b/vignettes/future-1-overview.md.rsp
@@ -22,7 +22,7 @@ file found under inst/vignettes-static/ of the source package.
## Introduction
The purpose of the [future] package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
-In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
+In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved d [...]
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
```r
@@ -110,7 +110,7 @@ The future package implements the following types of futures:
| _synchronous:_ | | _non-parallel:_
| `eager` | all |
| `lazy` | all | lazy evaluation - happens only if the value is requested
-| `transparent` | all | for debugging (eager w/ early signaling and w/out local)
+| `transparent` | all | as eager w/ early signaling and w/out local (for debugging)
| _asynchronous:_ | | _parallel_:
| `multiprocess` | all | multicore iff supported, otherwise multisession
| `multisession` | all | background R sessions (on current machine)
@@ -120,7 +120,7 @@ The future package implements the following types of futures:
The future package is designed such that support for additional strategies can be implemented as well. For instance, the [future.BatchJobs] package provides futures for all types of _cluster functions_ ("backends") that the [BatchJobs] package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available.
-By default, future expressions are evaluated instantaneously and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
+By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as "eager" and we refer to futures using this strategy as "eager futures". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
### Consistent Behavior Across Futures
@@ -165,7 +165,7 @@ Eager futures are the default unless otherwise specified. They were designed to
> plan(eager)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -184,13 +184,13 @@ Resolving 'b' ...
+ }
Resolving 'c' ...
> b
-[1] 14641
+[1] 10708
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
Since eager evaluation is taking place, each of the three futures is resolved instantaneously in the moment it is created. Note also how `pid` in the calling environment, which was assigned the process ID of the current process, is neither overwritten nor removed. This is because futures are evaluated in a local environment. Since synchronous (uni-)processing is used, future `b` is resolved by the main R process (still in a local environment), which is why the value of `b` and `pid` a [...]
@@ -201,7 +201,7 @@ A lazy future evaluates its expression only if its value is queried. Evaluation
> plan(lazy)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -219,14 +219,14 @@ A lazy future evaluates its expression only if its value is queried. Evaluation
Resolving 'a' ...
> b
Resolving 'b' ...
-[1] 14641
+[1] 10708
> c
Resolving 'c' ...
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
As previously, variable `pid` is unaffected because all evaluation is done in a local environment. More interestingly, future `a` is no longer evaluated in the moment it is created, but instead when it is needed the first time, which happens when future `c` is created. This is because `a` is identified as a global variable that needs to be captured ("frozen" to `a == 3.14`) in order to set up future `c`. Later when `c` (the value of future `c`) is queried, `a` has already been resolve [...]
@@ -248,7 +248,7 @@ We start with multisession futures because they are supported by all operating s
> plan(multisession)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -264,13 +264,13 @@ We start with multisession futures because they are supported by all operating s
+ 2 * a
+ }
> b
-[1] 14662
+[1] 10729
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
The first thing we observe is that the values of `a`, `c` and `pid` are the same as previously. However, we notice that `b` is different from before. This is because future `b` is evaluated in a different R process and therefore it returns a different process ID. Another difference is that the messages, generated by `cat()`, are no longer displayed. This is because they are outputted to the background sessions and not the calling session.
@@ -310,7 +310,7 @@ Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the
> plan(cluster, workers = c("n1", "n2", "n3"))
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
@@ -326,13 +326,13 @@ Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the
+ 2 * a
+ }
> b
-[1] 14684
+[1] 10751
> c
[1] 6.28
> a
[1] 3.14
> pid
-[1] 14641
+[1] 10708
```
Just as for the other asynchronous evaluation strategies, the output from `cat()` is not displayed on the current/calling machine.
@@ -344,7 +344,7 @@ plan(cluster, workers=cl)
```
Also, it is considered good style to shut down the cluster when it is no longer needed, that is, calling `parallel::stopCluster(cl)`. However, it will shut itself down if the main process is terminated, which will happen in the first example where the cluster in created internally. For more information on how to set up and manage such clusters, see `help("makeCluster", package="parallel")`.
-Note that with proper firewall and router configurations (e.g. port forwarding) and with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
+Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
@@ -354,7 +354,7 @@ Sometimes one may want to use an alternative evaluation strategy for a specific
> plan(eager)
> pid <- Sys.getpid()
> pid
-[1] 14641
+[1] 10708
> a %<-% {
+ Sys.getpid()
+ }
@@ -365,11 +365,11 @@ Sometimes one may want to use an alternative evaluation strategy for a specific
+ Sys.getpid()
+ } %plan% multiprocess
> a
-[1] 14641
+[1] 10708
> b
-[1] 14701
+[1] 10769
> c
-[1] 14702
+[1] 10770
```
As seen by the different process IDs, future `a` is evaluated eagerly using the same process as the calling environment whereas the other two are evaluated using multiprocess futures.
@@ -403,14 +403,14 @@ For instance, here is an example of two "top" futures (`a` and `b`) that uses mu
+ c(b.pid = Sys.getpid(), b1.pid = b1, b2.pid = b2)
+ }
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14703
+[1] 10771
> b
b.pid b1.pid b2.pid
- 14704 14704 14704
+ 10772 10772 10772
```
-By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 14641), and there are the two processes used by `a` (pid 14703) and `b` (pid 14704). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use eager evaluation unless otherwise specified. There are a few reasons for this, but the main reason is th [...]
+By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 10708), and there are the two processes used by `a` (pid 10771) and `b` (pid 10772). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use eager evaluation unless otherwise specified. There are a few reasons for this, but the main reason is th [...]
@@ -423,12 +423,12 @@ We would actually get the same behavior if we try with multiple levels of multip
> plan(list(multiprocess, multiprocess))
[...]
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14705
+[1] 10773
> b
b.pid b1.pid b2.pid
- 14706 14706 14706
+ 10774 10774 10774
```
The reason for this is, also here, to protect us from launching more processes than what the machine can support. Internally, this is done by setting `mc.cores` to zero ([sic!](https://github.com/HenrikBengtsson/Wishlist-for-R/issues/7)) such that no _additional_ parallel processes can be launched. This is the case for both multisession and multicore evaluation.
@@ -440,31 +440,31 @@ Continuing, if we start off by eager evaluation and then use multiprocess evalua
Resolving 'a' ...
Resolving 'b' ...
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14641
+[1] 10708
> b
b.pid b1.pid b2.pid
- 14641 14707 14708
+ 10708 10775 10776
```
-which clearly show that `a` and `b` are resolved in the calling process (pid 14641) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids 14707 and 14708).
+which clearly show that `a` and `b` are resolved in the calling process (pid 10708) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids 10775 and 10776).
Having said this, it is indeed possible to use nested multiprocess evaluation strategies, if we explicitly specify (read _force_) the number of cores available at each level. In order to do this we need to "tweak" the default settings, which can be done as follows:
```r
-> plan(list(tweak(multiprocess, workers = 3), tweak(multiprocess,
-+ workers = 3)))
+> plan(list(tweak(multiprocess, workers = 3L), tweak(multiprocess,
++ workers = 3L)))
[...]
> pid
-[1] 14641
+[1] 10708
> a
-[1] 14709
+[1] 10777
> b
b.pid b1.pid b2.pid
- 14710 14711 14712
+ 10778 10779 10840
```
-First, we see that both `a` and `b` are resolved in different processes (pids 14709 and 14710) than the calling process (pid 14641). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids 14711 and 14712).
+First, we see that both `a` and `b` are resolved in different processes (pids 10777 and 10778) than the calling process (pid 10708). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids 10779 and 10840).
To clarify, when we set up the two levels of multiprocess evaluation, we specified that in total 3 processes may be used at each level. We choose three parallel processes, not just two, because one is always consumed by the calling process leaving two to be used for the asynchronous futures. This is why we see that `pid`, `a` and `b` are all resolved by the same process. If we had allowed only two cores at the top level, `a` and `b` would have been resolved by the same background proc [...]
@@ -499,7 +499,7 @@ Waiting for 'a' to be resolved ...
> cat("Waiting for 'a' to be resolved ... DONE\n")
Waiting for 'a' to be resolved ... DONE
> a
-[1] 14713
+[1] 10841
```
@@ -566,9 +566,9 @@ There is one limitation with implicit futures that does not exist for explicit o
> v <- lapply(f, FUN = value)
> str(v)
List of 3
- $ : int 14714
- $ : int 14715
- $ : int 14716
+ $ : int 10921
+ $ : int 10922
+ $ : int 10924
```
This is _not_ possible to do when using implicit futures. This is because the `%<-%` assignment operator _cannot_ be used in all cases where the regular `<-` assignment operator can be used. It can only be used to assign future values to _environments_ (including the calling environment) much like how `assign(name, value, envir)` works. However, we can assign implicit futures to environments using _named indices_, e.g.
```r
@@ -582,9 +582,9 @@ This is _not_ possible to do when using implicit futures. This is because the `
> v <- as.list(v)
> str(v)
List of 3
- $ a: int 14717
- $ b: int 14718
- $ c: int 14719
+ $ a: int 10925
+ $ b: int 10928
+ $ c: int 10931
```
Here `as.list(v)` blocks until all futures in the environment `v` have been resolved. Then their values are collected and returned as a regular list.
@@ -601,9 +601,9 @@ If _numeric indices_ are required, then _list environments_ can be used. List e
> v <- as.list(v)
> str(v)
List of 3
- $ : int 14720
- $ : int 14721
- $ : int 14722
+ $ : int 10932
+ $ : int 10933
+ $ : int 10934
```
As previously, `as.list(v)` blocks until all futures are resolved.
diff --git a/vignettes/future-2-issues.md.rsp b/vignettes/future-2-issues.md.rsp
index 6316ffc..d45beaf 100644
--- a/vignettes/future-2-issues.md.rsp
+++ b/vignettes/future-2-issues.md.rsp
@@ -60,7 +60,7 @@ instead of
do.call("file_ext", list("foo.txt"))
```
so that `file_ext()` is properly located and exported. Although you may not notice a difference when evaluating futures in the same R session, it may become a problem if you use a character string instead of a function object when futures are evaluated in external R sessions, such as on a cluster.
-It may also become a problem with lazy uniprocess futures if the intended function is redefined after the future is resolved. For example,
+It may also become a problem with lazy futures if the intended function is redefined after the future is resolved. For example,
```r
> library("future")
> library("listenv")
@@ -133,7 +133,7 @@ Error: Invalid usage of futures: A future whose value has not yet been collected
```
As previously, this can be avoided by making sure `x$a` is resolved first, which can be one in various ways, e.g. `dummy <- x$a`, `resolve(x$a)` and `force(x$a)`.
-_Footnote_: (*) Although uniprocess futures could be passed on to other futures part of the same R process and be resolved there because they share the same evaluation process, by definition of the Future API it is invalid to do so regardless of future type. This conservative approach is taken in order to make future expressions behave consistently regardless of the type of future used.
+_Footnote_: (*) Although eager futures could be passed on to other futures part of the same R process and be resolved there because they share the same evaluation process, by definition of the Future API it is invalid to do so regardless of future type. This conservative approach is taken in order to make future expressions behave consistently regardless of the type of future used.
## Miscellaneous
diff --git a/vignettes/future-3-topologies.md.rsp b/vignettes/future-3-topologies.md.rsp
index c2bd7c5..c8962f2 100644
--- a/vignettes/future-3-topologies.md.rsp
+++ b/vignettes/future-3-topologies.md.rsp
@@ -74,11 +74,11 @@ plan(list(multiprocess, eager))
```
The internals for processing multiprocess future queries `availableCores()` to infer how many cores can be used simultaneously, so there is no need to explicitly specify that there are 8 cores available.
-_Comment_: Since synchronous is the default future, we could skip trailing synchronous eager futures in the setup, e.g. `plan(list(multiprocess))` or just `plan(uniprocess)`. However, it does not hurt to be explicit.
+_Comment_: Since synchronous is the default future, we could skip trailing synchronous eager futures in the setup, e.g. `plan(list(multiprocess))` or just `plan(eager)`. However, it does not hurt to be explicit.
If we instead would like to process the sample sequentially and the chromosomes in parallel, we can use:
```r
-plan(list(uniprocess, multiprocess))
+plan(list(eager, multiprocess))
```
We could also process the data such that we allocate two cores for processing two samples in parallel each using four cores for processing four chromosomes in parallel:
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/r-cran-future.git
More information about the debian-med-commit
mailing list