Explore your data: the tidyverse

The R Statistical Language · Session 2

1 Welcome back

1.1 Where we left off

In Session 1 you built the core structures of R:

  • the console, scripts, and R as a calculator;
  • vectors: the workhorse, and the four ways to subset them;
  • lists: containers of anything, reached with [[/$;
  • data frames & tibbles: the tables you analyse;
  • reading & writing data files.

1.2 Today’s map

1Foundations
2Explore your data
3Repeat & package

Session 2 is the “explore your data” move of the course: take rows and columns you stored, and turn them into understanding.

  • shape it: the pipe |> and dplyr/tidyr: filter, select, mutate, arrange, group_by/summarise, pivot_*;
  • read it: base-R workhorses, then ggplot2’s layered grammar: histograms, scatters, facets, boxplots;
  • understand it: the transform → plot → interpret loop that is Exploratory Data Analysis.

The thread. We keep the penguins data the whole session: shape it, then plot it, then read the answers back off the plots. The EDA pipeline you build here is what the repeat & package session and the Reproducible Workflows course will build on.

2 Tidyverse data transformation

2.1 The tidyverse at a glance

A family of R packages that work as one: consistent syntax, a shared pipe, and the same data frame in → data frame out flow.

  • readr: bring data in (read_csv);
  • tibble: the modern data frame;
  • dplyr: transform rows, columns, groups;
  • tidyr: reshape: keep data tidy (pivot_*);
  • ggplot2: visualise;
  • type-focused: purrr (iterate), stringr (strings), forcats (factors), lubridate (dates).

Today we master dplyr + tidyr: the transform engine that feeds every ggplot and every analysis. The long tail (purrr, stringr, …) you pick up on demand.

2.2 The pipe: |>

The pipe |> sends the value on its left into the first argument of the function on its right:

The pipe: top-to-bottom

penguins |>                                        # raw data set
  dplyr::filter(species == "Adelie") |>            # keep only Adelie penguins
  dplyr::mutate(body_mass_kg = body_mass_g / 1000) # compute a new column

Reads: penguins, THEN filter(…). One line per data processing step: easy to comment/read.

The nested equivalent: inside-out

filtered_penguins <- dplyr::filter(penguins, species == "Adelie")
dplyr::mutate(filtered_penguins, body_mass_kg = body_mass_g / 1000)

The pipe keeps your script readable top-to-bottom; the nested alternative reads inside-out. The backbone of data work: one rule, the data always goes first.

2.3 Rows and columns: filter & select

penguins |> 
  # keep penguins with body mass ≥ 4000 g
  dplyr::filter(body_mass_g >= 4000) |> 
  # show the first 4 penguins that meet this condition
  dplyr::slice(1:4)
# A tibble: 4 × 8
  species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
  <chr>   <chr>              <dbl>         <dbl>             <dbl>       <dbl>
1 Adelie  Torgersen           39.2          19.6               195        4675
2 Adelie  Torgersen           42            20.2               190        4250
3 Adelie  Torgersen           34.6          21.1               198        4400
4 Adelie  Torgersen           42.5          20.7               197        4500
# ℹ 2 more variables: sex <chr>, year <dbl>
penguins |> 
  # conditions AND together
  dplyr::filter(
    species != "Chinstrap", 
    !is.na(sex)
  ) |>
  # show first 4 rows
  dplyr::slice(1:4)
# A tibble: 4 × 8
  species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
  <chr>   <chr>              <dbl>         <dbl>             <dbl>       <dbl>
1 Adelie  Torgersen           39.1          18.7               181        3750
2 Adelie  Torgersen           39.5          17.4               186        3800
3 Adelie  Torgersen           40.3          18                 195        3250
4 Adelie  Torgersen           36.7          19.3               193        3450
# ℹ 2 more variables: sex <chr>, year <dbl>
  • filter() keeps rows;
  • Multiple conditions in filter are AND-ed by default; use other logical operators (|, !, %in%) to combine them.
penguins |> 
  # keep these columns
  dplyr::select(species, island, body_mass_g)
# A tibble: 344 × 3
   species island    body_mass_g
   <chr>   <chr>           <dbl>
 1 Adelie  Torgersen        3750
 2 Adelie  Torgersen        3800
 3 Adelie  Torgersen        3250
 4 Adelie  Torgersen          NA
 5 Adelie  Torgersen        3450
 6 Adelie  Torgersen        3650
 7 Adelie  Torgersen        3625
 8 Adelie  Torgersen        4675
 9 Adelie  Torgersen        3475
10 Adelie  Torgersen        4250
# ℹ 334 more rows
penguins |> 
  # drop columns whose names end with "_mm"
  dplyr::select(-dplyr::ends_with("_mm"))
# A tibble: 344 × 5
   species island    body_mass_g sex     year
   <chr>   <chr>           <dbl> <chr>  <dbl>
 1 Adelie  Torgersen        3750 male    2007
 2 Adelie  Torgersen        3800 female  2007
 3 Adelie  Torgersen        3250 female  2007
 4 Adelie  Torgersen          NA <NA>    2007
 5 Adelie  Torgersen        3450 female  2007
 6 Adelie  Torgersen        3650 male    2007
 7 Adelie  Torgersen        3625 female  2007
 8 Adelie  Torgersen        4675 male    2007
 9 Adelie  Torgersen        3475 <NA>    2007
10 Adelie  Torgersen        4250 <NA>    2007
# ℹ 334 more rows
  • select() keeps columns;
  • helpful helpers: starts_with(), ends_with(), contains(), matches(), everything().

2.4 New columns: mutate

mutate.R

penguins |>
  # compute the body-mass index (BMI) in kg/m²
  dplyr::mutate(bmi = body_mass_g / (flipper_length_mm / 100)^2) |>
  # keep only the columns we want to see
  dplyr::select(species, flipper_length_mm, body_mass_g, bmi) |> 
  # show first 5 rows
  dplyr::slice(1:5)
# A tibble: 5 × 4
  species flipper_length_mm body_mass_g   bmi
  <chr>               <dbl>       <dbl> <dbl>
1 Adelie                181        3750 1145.
2 Adelie                186        3800 1098.
3 Adelie                195        3250  855.
4 Adelie                 NA          NA   NA 
5 Adelie                193        3450  926.
  • mutate() computes new columns from existing ones, keeping everything else;
  • New columns are available to later steps in the same pipe;
  • You can mutate() several columns at once.

2.5 Recode with case_when

case-when.R

penguins |>
  # create a new column "size" based on body mass
  dplyr::mutate(size = dplyr::case_when(
    body_mass_g >= 5000 ~ "large",
    body_mass_g >= 3500 ~ "medium",
    .default = "small"
  )) |> 
  # keep only the columns we want to see
  dplyr::select(species, body_mass_g, size) |>
  # show first 10 rows
  dplyr::slice(1:10)
# A tibble: 10 × 3
   species body_mass_g size  
   <chr>         <dbl> <chr> 
 1 Adelie         3750 medium
 2 Adelie         3800 medium
 3 Adelie         3250 small 
 4 Adelie           NA small 
 5 Adelie         3450 small 
 6 Adelie         3650 medium
 7 Adelie         3625 medium
 8 Adelie         4675 medium
 9 Adelie         3475 small 
10 Adelie         4250 medium
  • case_when() turns a sequence of if…else conditions into a new column: a tidyverse-friendly replacement for nested ifelse;
  • case_when() reads like a look-up table: each condition ~ value row is evaluated in order, and .default catches everything else;
  • Perfect for binning or relabelling inside a mutate().

2.6 Strings with {stringr}

penguins |>
  # species contains "inst" (Chinstrap…)
  dplyr::filter(stringr::str_detect(species, "inst")) |> 
  dplyr::select(species, island, body_mass_g) |>
  dplyr::slice(1:14)
# A tibble: 14 × 3
   species   island body_mass_g
   <chr>     <chr>        <dbl>
 1 Chinstrap Dream         3500
 2 Chinstrap Dream         3900
 3 Chinstrap Dream         3650
 4 Chinstrap Dream         3525
 5 Chinstrap Dream         3725
 6 Chinstrap Dream         3950
 7 Chinstrap Dream         3250
 8 Chinstrap Dream         3750
 9 Chinstrap Dream         4150
10 Chinstrap Dream         3700
11 Chinstrap Dream         3800
12 Chinstrap Dream         3775
13 Chinstrap Dream         3700
14 Chinstrap Dream         4050
species_clean <- penguins |>
  # "adelie" → "Adelie"
  dplyr::mutate(
    species = stringr::str_to_title(species)
  )
species_clean |>
  # "Adelie" → "Adélie"
  dplyr::mutate(
    species = stringr::str_replace(
      species, 
      pattern = "Adelie", 
      replacement = "Adélie"
    )
  ) |>
  dplyr::select(species, island, body_mass_g) |>
  dplyr::slice(1:14)
# A tibble: 14 × 3
   species island    body_mass_g
   <chr>   <chr>           <dbl>
 1 Adélie  Torgersen        3750
 2 Adélie  Torgersen        3800
 3 Adélie  Torgersen        3250
 4 Adélie  Torgersen          NA
 5 Adélie  Torgersen        3450
 6 Adélie  Torgersen        3650
 7 Adélie  Torgersen        3625
 8 Adélie  Torgersen        4675
 9 Adélie  Torgersen        3475
10 Adélie  Torgersen        4250
11 Adélie  Torgersen        3300
12 Adélie  Torgersen        3700
13 Adélie  Torgersen        3200
14 Adélie  Torgersen        3800
  • {stringr} gives you one consistent family for text: every function is str_* with the string first, so they all pipe;
  • str_detect pairs with filter,
  • str_replace/str_to_* with mutate: strings behave exactly like any tidyverse column once you use str_*.

2.7 Sort: arrange

penguins |> 
  dplyr::arrange(body_mass_g) # lightest first
# A tibble: 344 × 8
   species   island   bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
   <chr>     <chr>             <dbl>         <dbl>             <dbl>       <dbl>
 1 Chinstrap Dream              46.9          16.6               192        2700
 2 Adelie    Biscoe             36.5          16.6               181        2850
 3 Adelie    Biscoe             36.4          17.1               184        2850
 4 Adelie    Biscoe             34.5          18.1               187        2900
 5 Adelie    Dream              33.1          16.1               178        2900
 6 Adelie    Torgers…           38.6          17                 188        2900
 7 Chinstrap Dream              43.2          16.6               187        2900
 8 Adelie    Biscoe             37.9          18.6               193        2925
 9 Adelie    Dream              37.5          18.9               179        2975
10 Adelie    Dream              37            16.9               185        3000
# ℹ 334 more rows
# ℹ 2 more variables: sex <chr>, year <dbl>
penguins |> 
  dplyr::arrange(dplyr::desc(body_mass_g), bill_length_mm) # heaviest first, ties by bill length
# A tibble: 344 × 8
   species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
   <chr>   <chr>           <dbl>         <dbl>             <dbl>       <dbl>
 1 Gentoo  Biscoe           49.2          15.2               221        6300
 2 Gentoo  Biscoe           59.6          17                 230        6050
 3 Gentoo  Biscoe           48.8          16.2               222        6000
 4 Gentoo  Biscoe           51.1          16.3               220        6000
 5 Gentoo  Biscoe           45.2          16.4               223        5950
 6 Gentoo  Biscoe           49.8          15.9               229        5950
 7 Gentoo  Biscoe           48.4          14.6               213        5850
 8 Gentoo  Biscoe           49.3          15.7               217        5850
 9 Gentoo  Biscoe           55.1          16                 230        5850
10 Gentoo  Biscoe           48.6          16                 230        5800
# ℹ 334 more rows
# ℹ 2 more variables: sex <chr>, year <dbl>

2.8 Summarise by group

data · ungrouped
speciesmass (g)
Adelie
3400
Adelie
3700
Gentoo
4700
Gentoo
5200
group_by→
grouped · 4 rows
speciesmass (g)
Adelie
3400
Adelie
3700
Gentoo
4700
Gentoo
5200
summarise→
summary · one row per group
speciesnmean
Adelie
2
3550
Gentoo
2
4950
penguins |>
  dplyr::group_by(species) |>
  dplyr::summarise(
    n = dplyr::n(), 
    avg_mass = mean(body_mass_g, na.rm = TRUE)
  )
# A tibble: 3 × 3
  species       n avg_mass
  <chr>     <int>    <dbl>
1 Adelie      152    3701.
2 Chinstrap    68    3733.
3 Gentoo      124    5076.
  • group_by() splits the data;
  • summarise() collapses rows into one row per group;
  • n() counts rows per group; na.rm = TRUE skips missing values.

2.9 across(): apply to many columns

across.R

penguins |>
  dplyr::group_by(species) |>
  dplyr::summarise(dplyr::across(
    .cols = dplyr::where(is.numeric), 
    .fns = \(.x) {
      mean(.x, na.rm = TRUE)
    }
  ))
# A tibble: 3 × 6
  species   bill_length_mm bill_depth_mm flipper_length_mm body_mass_g  year
  <chr>              <dbl>         <dbl>             <dbl>       <dbl> <dbl>
1 Adelie              38.8          18.3              190.       3701. 2008.
2 Chinstrap           48.8          18.4              196.       3733. 2008.
3 Gentoo              47.5          15.0              217.       5076. 2008.
  • across() applies one expression to every column that matches a selector: no copy-pasting verbs per column;
  • across(where(is.numeric), ...) targets a whole family of columns at once: the bridge from “many similar lines” to programmatic thinking you’ll meet again in Session 3’s functions;
  • \() {} is the anonymous (lambda) function syntax to create a custom function.

2.10 Reshape with {tidyr}: pivot_longer & pivot_wider

wide · 2 rows
penguinbill_lbill_dflipper
A
39.1
18.7
181
B
46.5
17.9
195
pivot_longer→
←pivot_wider
long · 6 rows
penguinmeasuremm
A
bill_l
39.1
A
bill_d
18.7
A
flipper
181
B
bill_l
46.5
B
bill_d
17.9
B
flipper
195

reshape.R

# wide → long (melt several columns back into one key/value pair)
df |> tidyr::pivot_longer(cols = -penguin, names_to = "measure", values_to = "mm")
# long → wide (spread key/value columns back out)
df |> tidyr::pivot_wider(names_from = measure, values_from = mm)
  • Tidy data: one row per observation, one column per variable. When it isn’t, reshape;
  • Every tidyverse tool expects tidy data: reshape first, analyse after.

2.11 Your turn: tidyverse

  1. Filter Gentoo penguins; select three columns → gentoo.
  2. Add mass_kg = body_mass_g / 1000.
  3. Group by island; summarise mean & sd of bill_length_mm.
  4. Heaviest median body mass per species (arrange + desc).
  5. pivot_longer a wide table (names_to, values_to).

3 Plotting with base R

3.1 Why plot?

A plot is the fastest way to feel your data. Anscombe’s quartet is the classic proof: four data sets with nearly identical means, variances and correlations, yet plots that look nothing alike.

The four data sets share the same slope, intercept, correlation and means; only the plots tell them apart. A table could not, a hist() of a single column would not, an illustration does.

The lesson. Look at your data before you summarise it: a single well-chosen plot often tells you more than ten summary statistics. This is the heart of Exploratory Data Analysis.

3.2 Histograms in base R

The simplest first look at one variable: hist().

hist.R

par(bg = "white")                # opaque white figure: dark slide would hide the default black ink
hist(
  penguins$body_mass_g,          # the variable
  breaks = 30,                   # how many bins
  col = "steelblue", 
  border = "white",
  main = "Penguin body mass (g)" # title
)

Positron draws the plot in the Plots pane. hist() is one call, but its defaults are utilitarian.

3.3 Other base-R workhorses

The rest of the base-R drawer

par(bg = "white")   # opaque white figure so the default black ink shows on the dark slide
x <- rnorm(10)
y <- rnorm(10)
plot(x, y)          # scatter plot

The rest of the base-R drawer

par(bg = "white")   # opaque white figure so the default black ink shows on the dark slide
g <- rpois(n = 10L, lambda = 1)
barplot(table(g))   # counts of categories

The rest of the base-R drawer

par(bg = "white")   # opaque white figure so the default black ink shows on the dark slide
boxplot(y ~ g)      # one box per group

The rest of the base-R drawer

par(bg = "white")   # opaque white figure so the default black ink shows on the dark slide
curve(sin, 0, 2*pi) # plot a function

Base R can already do a lot, but its look is dated and customisation is verbose: why we meet ggplot2 next.

3.4 The limit of base-R plots

Base plot() is fine for a quick peek. For publication-grade graphics (faceting, legends, themes, layered encodings), it gets unwieldy fast.

That’s the gap ggplot2 fills: a single coherent grammar, rather than a grab-bag of functions.

You’ll implement several of these base-R plots by hand in lab 2, then rebuild one in ggplot2 to feel the difference.

4 ggplot2

4.1 The grammar of graphics

ggplot2 (part of the tidyverse) builds every plot from a few composable pieces:

  • data: a data frame (usually a tibble);
  • aesthetics (aes): which variables map to which visual channel (x, y, colour, size, fill, …);
  • geometries (geom_*): how the data are drawn (points, lines, bars, histograms, …);
  • facets: small multiples, splitting by a variable;
  • scales/themes: the polish.

4.2 The first ggplot

Every plot starts with ggplot(data = , mapping = aes(...)), then adds layers with +:

ggplot(
    data = penguins, 
    mapping = aes(x = body_mass_g)
  ) + 
  geom_histogram(bins = 30, color = "white")

This is the same histogram we drew in base R, but now each piece is explicit and easy to extend.

4.3 Scatter: two continuous variables

Mapping variables to x and y, and a third to color:

ggplot(
    data = penguins,
    mapping = aes(x = flipper_length_mm, y = body_mass_g)
  ) + 
  geom_point(aes(color = species), alpha = 0.7) + 
  theme_dark() + 
  labs(
    title = "Body mass variation with flipper length",
    caption = "Data from the {palmerpengins} package.",
    x = "Flipper Length (mm)", 
    y = "Body Mass (g)", 
    color = "Species"
  )

Read it aloud. “Body mass grows with flipper length”, and each species clusters in its own region of this space. That’s a finding you’d never see from summary().

4.4 One variable per facet

Faceting splits the plot into small multiples: one per level of a variable:

ggplot(
    data = penguins, 
    mapping = aes(x = body_mass_g)
  ) + 
  geom_histogram(
    bins = 25, 
    fill = "#2E7D6B", 
    color = "#17251F"
  ) + 
  facet_wrap(
    facets = vars(species), 
    ncol = 1
  ) + 
  theme_dark()

Different species have different distributions of body mass: a comparison that’s hard to see in one pooled histogram.

4.5 Boxplots for groups

Categorical x + continuous y:

ggplot(
    data = penguins, 
    mapping = aes(x = species, y = bill_length_mm, fill = species)
  ) + 
  geom_boxplot(alpha = 0.8) + 
  theme_dark() + 
  theme(legend.position = "none")

A boxplot shows the median, the box (IQR) and outliers at a glance: ideal for comparing a numeric variable across groups.

4.6 Order factors with {forcats}

{forcats} lets you reorder those levels so graphs tell the story you want:

penguins |>
  dplyr::mutate(species = forcats::fct_reorder(
    .f = species, 
    .x = bill_length_mm, 
    .fun = median, 
    na.rm = TRUE
  )) |>
  ggplot(mapping = aes(
    x = species, 
    y = bill_length_mm, 
    fill = species
  )) +
  geom_boxplot()

fct_reorder() puts the highest median first on the axis: order becomes part of your message. fct_relevel/fct_lump fine-tune and collapse rare levels.

4.7 Layers compose

An important ggplot2 habit: plots are built from many layers. For example, scatter + a smooth trend:

ggplot(
    data = penguins, 
    mapping = aes(x = flipper_length_mm, y = body_mass_g)
  ) + 
  geom_point(aes(color = species), alpha = 0.6) + 
  geom_smooth(
    method = "lm", 
    se = FALSE, 
    color = "#E8A33D", 
    linewidth = 1
  ) + 
  theme_dark()

geom_point() and geom_smooth() layer on the same plot, each with its own mapping. This layering is what makes ggplot2 so expressive.

4.8 A grammar, not a menu

The power is consistency: you learn a few verbs (geom_point, geom_bar, geom_histogram, facet_wrap, aes) and combine them, instead of memorising a different function per plot type.

There’s a gallery of extensions (e.g. {ggpubr}, {patchwork}, {plotly} for interactivity): https://exts.ggplot2.tidyverse.org

We’ll run a ggplot2 build-up live: from a bare histogram to a faceted, titled, themed figure in a handful of added lines. Watch how small edits compound.

4.9 Your turn: plotting

  1. Load penguins with read_csv("penguins.csv").
  2. Base R: scatter body_mass_g vs flipper_length_mm, by species.
  3. ggplot2: same plot with color = species.
  4. hist() of body_mass_g + facet_wrap(~ species).
  5. Which approach would you reach for?

5 Wrap up

5.1 The EDA loop

Shape, read, understand.

5.2 Shape, read, understand

Exploring data is one loop.

Shape it

The pipe and dplyr/tidyr: filter, select, mutate, arrange, group_by/summarise, pivot_*, to get the data the shape you need.

Read it

Base-R workhorses (hist, plot, boxplot), then ggplot2’s layered grammar: mapping, geoms, facets, themes.

Understand it

Always look before you summarise: a well-chosen plot answers questions ten summary statistics can’t.

5.3 Understand: the thread

Transform → plot → interpret.

The EDA loop

Shape the data until it’s tidy, plot it to see it, and read the answer back off the plot; then reshape and replot to chase the next question. This transform → plot → interpret loop is the heart of data science: repeat it often, and the repeat & package session shows you the loops underneath and turns the moves you repeat into reusable functions you keep.

5.4 Next time

Repeat & package: loops, matrices, functions.

Between sessions: finish any outstanding labs, and skim R for Data Science, ch. 3 (visualisation) to reinforce the ggplot2 grammar you’ll reuse everywhere.