Strings and regular expressions

Reference material — not covered in the taught sessions

Author

Gabriel Mateus Bernardo Harrington

This page is reference material. It is not taught in a session and you will not be assessed on the details of regular expressions.

It is here because real data is full of messy text — inconsistent column names, identifiers with mixed separators, categories typed five different ways — and stringr is how you deal with that in the tidyverse. Come back to it when you hit that problem in the report.

Setup

library(stringr)
library(dplyr)
library(tidyr)
library(gapminder)
library(nycflights13)

words and sentences used below are small example datasets that ship with stringr, so they are available as soon as it is loaded.

What is a string?

  • Strings are a sequence of characters, which has to be represented in memory in binary.
  • This was first widely done using ASCII (American Standard Code for Information Interchange).
  • But ASCII only allows for 128 characters, so nowhere near enough for all languages.
  • We now represent strings using Unicode (which didn’t appear in its current form until the 90s), which allows for a lot — more than a million — characters.
  • UTF-8 is the most common encoding for Unicode. It uses a variable number of 8-bit units to represent characters.
  • R uses UTF-8 by default (though it will report the label “unknown” unless non-ASCII characters appear), and stringr fully supports Unicode.

stringr

  • Removes inconsistencies found in base R.
  • Built on top of the stringi package.
  • Starts all functions with str_, so autocomplete is genuinely useful.
  • More advanced use requires regular expressions.

Regular expressions

As quoted by R for Data Science, 1st edition:

“When you first look at a regexp, you’ll think a cat walked across your keyboard, but as your understanding improves they will soon start to make sense.”

If you’ve read about regex before, you will also have come across:

“Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems.”

The short version

  • They allow you to match patterns in strings.
  • The most basic form matches an actual chunk of text, e.g. hag in hagrid.
  • Match classes of character with \s, \d, \w, [abc] and [^abc].
  • Because we’re in R, you have to write \\ instead of \ — so \\w, \\d. This catches everybody at least once.
  • Match any character at all with ..
  • Expand a match with ?, +, * or {n,m}.
  • Anchor it to the start with ^ or the end with $.
  • So ^\\w+_\\d{4}$ matches "hagrid_2020" but not "hagrid_120".

The rest of this section works through each of those on one example vector.

Matching literal text

my_string <- c(
  'Hagrid',
  'Hermione',
  'Harry.Potter',
  'Ronald_Weasley',
  '24xHouse Elves'
)
str_detect(my_string, 'Hag')
[1]  TRUE FALSE FALSE FALSE FALSE

Character classes

Match types of character with \\w (word), \\d (digit) and \\s (whitespace).

my_string
[1] "Hagrid"         "Hermione"       "Harry.Potter"   "Ronald_Weasley"
[5] "24xHouse Elves"
str_detect(my_string, '\\w')
[1] TRUE TRUE TRUE TRUE TRUE
str_detect(my_string, '\\d')
[1] FALSE FALSE FALSE FALSE  TRUE
str_detect(my_string, '\\s')
[1] FALSE FALSE FALSE FALSE  TRUE

Matching any character

. matches anything, which is why it matches every element here.

str_detect(my_string, '.')
[1] TRUE TRUE TRUE TRUE TRUE

Quantifiers

?, +, * and {n,m} control how many times the preceding pattern may repeat.

str_detect(my_string, '\\w+')
[1] TRUE TRUE TRUE TRUE TRUE

Note the difference [xyz] versus [^xyz] makes — the second negates the class.

str_detect(my_string, '\\d{2}[xyz]\\w+\\s?\\w+')
[1] FALSE FALSE FALSE FALSE  TRUE
str_detect(my_string, '\\d{2}[^xyz]\\w+\\s?\\w+')
[1] FALSE FALSE FALSE FALSE FALSE

Anchors

^ anchors to the start of the string, $ to the end. Anchoring is usually what turns a pattern that nearly works into one that does.

str_detect(my_string, '^H\\w+$')
[1]  TRUE  TRUE FALSE FALSE FALSE
str_detect(my_string, '^Hag$')
[1] FALSE FALSE FALSE FALSE FALSE

Basic stringr operations

String length and counts

# number of characters in each string
str_length(my_string)
[1]  6  8 12 14 14
# number of matches within each string
str_count(my_string, 'Ha')
[1] 1 0 1 0 0

Concatenating strings

str_c('this', 'that', sep = ', ')
[1] "this, that"

Manipulating strings

Extracting and replacing

# extract substrings by position
str_sub(my_string, 1, 3)
[1] "Hag" "Her" "Har" "Ron" "24x"
# replace the first match in each string
str_replace(my_string, '[\\._x]', ' ')
[1] "Hagrid"         "Hermione"       "Harry Potter"   "Ronald Weasley"
[5] "24 House Elves"

str_replace() replaces only the first match in each string. str_replace_all() replaces every match. Picking the wrong one is a quiet bug of exactly the kind this module keeps warning you about — it will not error.

Detecting and viewing

# returns a logical vector - the tidyverse alternative to grepl()
str_detect(my_string, 'Hagrid')
[1]  TRUE FALSE FALSE FALSE FALSE
# shows you what your pattern actually matched
str_view(my_string, '^\\w')
[1] │ <H>agrid
[2] │ <H>ermione
[3] │ <H>arry.Potter
[4] │ <R>onald_Weasley
[5] │ <2>4xHouse Elves

str_view() is the single most useful debugging tool here. When a pattern isn’t working, look at what it is matching before changing it.

Splitting and sorting

sentences |>
  head(1) |>
  str_split(" ")
[[1]]
[1] "The"     "birch"   "canoe"   "slid"    "on"      "the"     "smooth" 
[8] "planks."
str_sort(words[1:10], locale = 'en')
 [1] "a"        "able"     "about"    "absolute" "accept"   "account" 
 [7] "achieve"  "across"   "act"      "active"

Always set locale when sorting. Sort order is language-dependent, and leaving it implicit makes your results depend on the machine they ran on — which is exactly what reproducibility is meant to rule out.

Changing case

str_to_upper(words[1:10])
 [1] "A"        "ABLE"     "ABOUT"    "ABSOLUTE" "ACCEPT"   "ACCOUNT" 
 [7] "ACHIEVE"  "ACROSS"   "ACT"      "ACTIVE"  
str_to_lower(words[1:10])
 [1] "a"        "able"     "about"    "absolute" "accept"   "account" 
 [7] "achieve"  "across"   "act"      "active"  
str_to_sentence(words[1:10])
 [1] "A"        "Able"     "About"    "Absolute" "Accept"   "Account" 
 [7] "Achieve"  "Across"   "Act"      "Active"  
str_to_title(words[1:10])
 [1] "A"        "Able"     "About"    "Absolute" "Accept"   "Account" 
 [7] "Achieve"  "Across"   "Act"      "Active"  
sentences |>
  head(1) |>
  str_to_sentence()
[1] "The birch canoe slid on the smooth planks."
sentences |>
  head(1) |>
  str_to_title()
[1] "The Birch Canoe Slid On The Smooth Planks."

Practice

If you want to work through this material rather than just read it:

Concatenating and replacing

  • Concatenate the strings "day to" and "day", separated by a hyphen.
  • Using starwars: select the hair_color column and replace ', ' with '/'. Hint: pipe a column into pull() to get a vector for stringr to work on.
  • Using flights: select the columns ending in delay and remove the underscore from those column names.

Splitting

  • Split "Harry, did you put your name in the Goblet of Fire?" into its components.
  • Use boundary("word") instead of " " and compare the results.
  • Using the fifth line of sentences: split by word boundary, convert to lowercase and sort. Hint: unlist().

Case

  • Using starwars: convert hair_color values to Sentence case, and all column names to Title Case.
  • Using gapminder: remove any camelCase from the column names (all to lower case), and convert the continent values to upper case.

Further reading

  • R for Data Science 2e, chapters 14 and 15
  • The stringr documentation, and vignette("regular-expressions", package = "stringr")
  • This blog post by Joel Spolsky on Unicode and character sets
  • regex101.com — build and test a pattern interactively, with an explanation of each part. Remember to double your backslashes when you bring the pattern back into R.