,
Nima Hejazi [ctb] | Title: | A Framework for Parallelizing Dependent Tasks |
|---|---|
| Description: | Mechanisms to parallelize dependent tasks in a manner that optimizes the compute resources available. It provides access to "delayed" computations, which may be parallelized using futures. It is, to an extent, a facsimile of the 'Dask' library (<https://www.dask.org/>), for the 'Python' language. |
| Authors: | Jeremy Coyle [aut, cre, cph] ,
Nima Hejazi [ctb] |
| Maintainer: | Jeremy Coyle <[email protected]> |
| License: | GPL-3 |
| Version: | 0.5.0 |
| Built: | 2024-11-11 05:15:59 UTC |
| Source: | https://github.com/tlverse/delayed |
A Delayed version of a function may be called to generate Delayed objects
delayed(expr, sequential = FALSE, expect_error = FALSE, timeout = NULL) delayed_fun(fun, sequential = FALSE, expect_error = FALSE)delayed(expr, sequential = FALSE, expect_error = FALSE, timeout = NULL) delayed_fun(fun, sequential = FALSE, expect_error = FALSE)
expr |
expression to delay |
sequential |
if TRUE, never parallelize this task |
expect_error |
if TRUE, pass error to downstream tasks instead of |
timeout |
specify a time limit for computation halting computation |
fun |
function to delay |
d <- delayed(3 + 4) d$compute() adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z$compute()d <- delayed(3 + 4) d$compute() adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z$compute()
Delayed class that manages dependencies and computes when necessary
d <- delayed(3 + 4) methods::is(d, "Delayed") d$compute()d <- delayed(3 + 4) methods::is(d, "Delayed") d$compute()
Helper Function to Evaluate Delayed
eval_delayed(to_eval, timeout = Inf)eval_delayed(to_eval, timeout = Inf)
to_eval |
a list as generated from Delayed$prepare_eval() |
timeout |
a timeout indicating when to terminate the job |
Searches through a network of delayed objects for the first object with state "error"
find_delayed_error(delayed_object)find_delayed_error(delayed_object)
delayed_object |
the object in which an error occured |
delayed_error <- delayed_fun(stop) error_message <- "this is an error" broken_delayed <- delayed_error(error_message) broken_delayed$expect_error <- TRUE result <- broken_delayed$compute()delayed_error <- delayed_fun(stop) error_message <- "this is an error" broken_delayed <- delayed_error(error_message) broken_delayed$expect_error <- TRUE result <- broken_delayed$compute()
A Job that leverages the future framework to
evaluate asynchronously.
library(future) plan(multicore, workers = 1) d <- delayed(3 + 4) sched <- Scheduler$new(d, FutureJob, nworkers = 1)library(future) plan(multicore, workers = 1) d <- delayed(3 + 4) sched <- Scheduler$new(d, FutureJob, nworkers = 1)
uses shiny
plot_delayed_shiny(scheduler)plot_delayed_shiny(scheduler)
scheduler |
the scheduler to animate |
## Not run: adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z2 <- delayed_adder(z, 4) z2$sequential <- TRUE z3 <- delayed_adder(z2, z) plot_delayed_shiny(z3) ## End(Not run)## Not run: adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z2 <- delayed_adder(z, 4) z2$sequential <- TRUE z3 <- delayed_adder(z2, z) plot_delayed_shiny(z3) ## End(Not run)
Plot Method for Delayed Objects
## S3 method for class 'Delayed' plot(x, color = TRUE, height = "500px", width = "100%", ...)## S3 method for class 'Delayed' plot(x, color = TRUE, height = "500px", width = "100%", ...)
x |
An object of class |
color |
If |
height |
passed to visNetwork |
width |
passed to visNetwork |
... |
Additional arugments (passed to visNetwork). |
adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z2 <- delayed_adder(z, 4) z2$sequential <- TRUE z3 <- delayed_adder(z2, z) plot(z3)adder <- function(x, y) { x + y } delayed_adder <- delayed_fun(adder) z <- delayed_adder(3, 4) z2 <- delayed_adder(z, 4) z2$sequential <- TRUE z3 <- delayed_adder(z2, z) plot(z3)
Scheduler class that orders compute tasks and dispatches tasks to workers
d <- delayed(3 + 4) sched <- Scheduler$new(d, SequentialJob) sched$compute()d <- delayed(3 + 4) sched <- Scheduler$new(d, SequentialJob) sched$compute()
A Job that will evaluate immediately (i.e., in a
sequential fashion), blocking the current process until it completes.
d <- delayed(3 + 4) sched <- Scheduler$new(d, SequentialJob)d <- delayed(3 + 4) sched <- Scheduler$new(d, SequentialJob)