Lab: Futureverse 1

Speed up your code through parallel computing

Note

This is the first of two parallelization labs. It will take you through some basic steps to parallelize your code using Futureverse. It focuses on core functions future() and value() for the purpose of illustrating what happens behind the scenes when we parallelize R code.

You are highly encouraged to test things out yourself and tweak things to figure out how these methods behave.

Slides: You’ll find the slides in the menus above.

Install

We will start out by installing common Futureverse packages part of the Futureverse. We will not need them all in this lab, but it is convenient to have them all installed already now.

install.packages("futureverse")

Exercises

In order to illustrate parallelization, we need two things: (i) a way to measure time, and (ii) something that takes at least a few seconds to run.

Task 1:

Copy and paste the following two code blocks.

Create functions tic() and toc() to measure time:

tic <- function() {
  tic_start <<- base::Sys.time()
}

toc <- function() {
  dt <- base::difftime(base::Sys.time(), tic_start)
  dt <- round(dt, digits = 1L)
  message(paste(format(dt), "since tic()"))
}

These functions can be used as a timer, e.g.

tic()
Sys.sleep(1.5)
toc()

1.5 secs since tic()

Sys.sleep(4.0)
toc()

5.5 secs since tic()

Next, create toy function slow_sum() for calculating the sum of a vector really slowly:

slow_sum <- function(x) {
  sum <- 0
  for (value in x) {
    Sys.sleep(1.0)     ## one-second slowdown per value
    sum <- sum + value
  }
  sum
}

This function works just like sum(), but it is very slow. If we use it to calculate \(1 + 2 + \ldots + 10\), it will takes us ten seconds to get the result;

tic()
y <- slow_sum(1:10)
y
toc()

Make sure you can run the latter, that it takes ten seconds to complete and that it returns the correct value.

We are now ready to get rolling!

Simple parallel tasks

At the very core of Futureverse is the future package. Let us start out by loading this core package:

library(future)

It provides us with the fundamental building blocks for running R code in parallel; functions future(), value(), and resolved(). Other Futureverse packages, such as future.apply, furrr, and doFuture, rely on these three functions to build up more feature-rich functions. We will return to those later, but for now we will focus on future() and value().

Task 2:

Let’s start by writing our initial example using futures:

tic()
f <- future(slow_sum(1:10))
y <- value(f)
toc()

Confirm that you get the correct result. Did it run faster?

Task 3:

Add another toc() just after the future() call;

tic()
f <- future(slow_sum(1:10))
toc()
y <- value(f)
toc()
y
toc()

How long did the creation of the future take?

Task 4:

By design, Futureverse runs everything sequentially by default. We can configure it run code in parallel using two background workers as:

plan(multisession, workers = 2)

Make this change, and rerun the above example. Did the different steps take as long as you expected? What do you think the reason is for the change?

Task 5:

Let’s calculate \(1 + 2 + \ldots + 10\) in two steps: (a) \(1 + 2 + \ldots + 5\) and (b) \(6 + 7 + \ldots + 10\), and then sum the two results.

fa <- future(slow_sum(1:5))
fb <- future(slow_sum(6:10))
y <- value(fa) + value(fb)
y

But first, make sure to add toc() after each statement to better understand how long each step takes;

tic()
fa <- future(slow_sum(1:5))
toc()
fb <- future(slow_sum(6:10))
toc()
y <- value(fa) + value(fb)
toc()
y
toc()

Make sure you get the expected result. Did it finish sooner? Which step takes the longest? Why do you think that is?

Create many parallel tasks via a for loop

Task 6:

Here is a very complicated way of calculating the sum \(1 + 2 + \ldots + 20\) in four chunks and outputting messages to show the progress:

tic()
xs <- list(1:5, 6:10, 11:15, 16:20)
ys <- list()
for (ii in seq_along(xs)) {
  message(paste0("Iteration ", ii))
  ys[[ii]] <- slow_sum(xs[[ii]])
}
message("Done")
print(ys)

ys <- unlist(ys)
ys

y <- sum(ys)
y
toc()

Rewrite it such that each iteration is parallelized via a future. Use four parallel workers as in:

library(future)
plan(multisession, workers = 4)

Task 7:

Retry with three parallel workers as in:

library(future)
plan(multisession, workers = 3)

Did you notice something? What do you think happened?

Our own parallel lapply

Task 8:

Above, you used a for-loop to parallelize tasks. See if you can achieve the same using lapply() instead.

Task 9:

Take your parallel lapply() code and wrap it up in a function parallel_lapply() that takes two arguments X and FUN so that we can call:

library(future)
plan(multisession)

xs <- list(1:5, 6:10, 11:15, 16:20)

ys <- parallel_lapply(xs, slow_sum)
ys <- unlist(ys)
y <- sum(ys)

Solution

parallel_lapply <- function(X, FUN) {
  ## Create futures that calls FUN(X[[1]]), FUN(X[[2]]), ...
  fs <- lapply(X, function(x) {
    ## For element 'x', create future that calls FUN(x)
    future(FUN(x))
  })
  
  ## Collect the values from all futures
  value(fs)
}