How to switch between data Structures

Introduction

If you read this, perhaps it’s because you followed some tutorial about R, either here or elsewhere and probably, not being a professional programmer, you found the official documentation somewhat obscure at times. After reading this you should be familiar with the different data structures that R offers: vectors, lists, matrices (or arrays with higher dimension) and you may want to investigate Tables as well or understand how R works together with SQL type databases.

But how should we use those structures — e.g. when is it that a certain structure is more or less appropriate than another and, having information ‘stored’ in one of these, how would you switch to another, and why? Data handling is central for any type of analysis you wish to carry out, whether as a future Data Scientist or ‘just’ to win the latest exciting Kaggle competition.

We’ll cover the basic data structures before looking at some typical cases of handling data, then delve into the most important of them all, at least in data science — the data frame — using a slightly more comprehensive example.

Data structures: elementary definitions

With “data structures” we mean non-elementary (non-atomic) data, in contrast to the well-known Data Types that R offers, as all other languages do: integer, long integer, float, factor, character, etc.

The basic compound data structures in R are the vector, the matrix, and the array (a generalized matrix with a dimension equal to or larger than 3). And below that, the list, as the basic constructive element of the data frame. Let’s start with the most important of them all — the vector.

Vectors

A vector can be defined in a number of ways, some simpler than others:

v1 <- c(1,2,3,4,5,1,2,3,4,5) # use concatenate operator 'c'
# or this:
v2 <- rep(1:5,2)             # use function replicate to repeat the sequence 1 to 5 twice
# or like this
v3 <- rep(seq(1, 5, by=1),2) # replicate twice the generating sequence (seq)

v1, v2, v3 are all identical. Vectors can be accessed or selected in several ways:

v1[5:10]        # select from the 5th to the tenth element
v1[c(5,7:10)]   # select the 5th and then from the 7th to the tenth
v4 <- v1[-(7:9)] # remove all elements of v1 whose index lies between 7 and 9

We can also assign names to vector elements:

names(v1) <- letters[1:length(v1)]
str(v1) # check type

This produces a “named vector,” which could also be defined directly:

v6 <- c(a=1, b=6.1, c=9.0, d=0.7)

A few more useful operations:

sort(v2)       # sort in ascending order (the default)
rev(sort(v2))  # or in reverse order
order(v2)      # similar, but acts on the indexes

When sorting, we change the index of the data, not the data itself — like taking note of the numbers on a shuffled deck of cards and sorting the notes rather than the cards. unique() gives us all the distinct elements, dropping repetitions:

unique(v4)

And is.vector() tells us whether an object is a vector. Note that R performs automatic type conversion when mixed types are combined:

v5 <- c(v4, "blah")
v5
## [1] "1" "2" "3" "4" "5" "1" "5" "blah"

The data type has changed from integer to character — homogeneity within a vector must be preserved.

Lists

Lists are a generalized vector that allow mixed data types, whereas vectors can only contain one type. Lists may store other vectors, lists, matrices, data frames, and even functions. They’re similar to Python’s dictionary. You create lists with list() and access elements with the [[ operator:

MyList <- list()
MyList[["Rockerduck"]] <- 1
MyList[["uncle scrooge"]] <- c("a", "b", "33")
str(MyList)

Accessing elements:

MyList[["Rockerduck"]]
MyList[2]      # the second element of the list
MyList[[2]]    # the "content" of the second element
MyList[1:2]    # from the first to the second element
MyList[[2]][2] # the second element of the character vector inside the second list element

The difference between [ and [[ for lists: [ returns a sub-list, [[ returns the actual element/content. You can convert a list to a vector explicitly with unlist(). Lists are used throughout R — many functions dealing with data analysis (e.g. a linear fit or regression) return their results as a list.

Factors

Factors are “special vectors” that let us handle non-numerical categorical data (gender, color, etc.) with a finite number of possible values, called “levels”:

levs <- c("Red", "Green", "Blue")
x <- factor(sample(c("Red", "Green", "Blue"), 7, replace=T))
str(x)

Note the levels are ordered alphabetically internally (Blue, Green, Red = 1,2,3), regardless of entry order. A summary/contingency table:

table(x)

Factors also arise automatically when reading files into data frames.

Matrices

Matrices are two-dimensional tables of rows and columns, with elements of the same type throughout. Some ways to create one:

A <- matrix(1:4, nrow=4, ncol=4)
A <- matrix(1:4, 4, 4) # identical

# using rbind() and cbind()
A2 <- cbind(c(1,2,3,4), c(1,2,3,4), c(1,2,3,4), c(1,2,3,4))
A3 <- rbind(c(1,5), c(7,2))

Transposition swaps rows and columns:

D <- matrix(3:6, 4, 4, byrow=FALSE)
D2 <- t(D)

Matrices are, in effect, “vectors of vectors” — a plain vector has no dim(), only a length(), whereas a matrix has both. Mixing types when binding vectors into a matrix coerces everything to a common type (e.g. numeric + character → character).

Element/cell access works with two indexes:

a <- A[2,3]  # cell at row 2, column 3
v <- A[,3]   # entire column 3
u <- A[1,]   # entire row 1

Higher-dimensional matrices are arrays

An “array” of dimension 1 is what we call a vector; a two-dimensional array is a matrix; the word “array” is reserved here for dimension > 2:

Y <- array(rnorm(2*2*2), dim=c(2,2,2))

Data frames

The most used R data structure — a list of vectors, where rows are observations (“records”) and columns are variables (“features”). Columns may be numeric, character, categorical, or date, and all can coexist in the same data frame.

n <- 20
MyData <- data.frame(v1=c(1:20), x=rnorm(n), y=sample(c(TRUE,FALSE), n, replace=TRUE))

Basic exploration:

str(MyData)     # data types and a peek at content
names(MyData)   # column names
rownames(MyData)
head(MyData, 3) # first 3 rows
tail(MyData, 6) # last 6 rows
summary(MyData) # per-column summary stats

Accessing columns:

MyData[,2]    # all rows, second column
MyData$x      # equivalent, by name
MyData[["x"]] # equivalent, list-style access

Removing a column:

MyData[1] <- NULL          # drop the first column entirely
MyNewData <- MyData[,-(1:2)]                 # drop columns 1 and 2
MyNewData2 <- MyData[,-(1:2), drop = FALSE]  # ...but preserve as a data.frame even if only one column remains

Adding columns:

MyData$Types <- LETTERS[1:20]
MyData$Descript <- rep(letters[1:2], 10)

Changing a column’s type (e.g. character → factor):

MyData$Descript <- as.factor(MyData$Descript)
colnames(MyData)[4] <- "Coding" # rename for clarity

In general: “inspection functions” (is.*()) return TRUE/FALSE, and “conversion functions” (as.*()) return a converted object — for both atomic and compound types, e.g. as.data.frame(MyList) converts a list into a data frame.

More operations on data frames

A couple of common tasks: extraction based on a condition, sorting, and a quick plot.

# sort descending by x
MyData <- MyData[order(MyData$x, decreasing = TRUE),]

# get the sort indexes directly, if needed
OrderedNdx <- order(MyData$x, decreasing = TRUE)

# extract rows where x is positive
MyDataPos <- MyData[MyData$x > 0,]

# further refine: positive x AND y is FALSE
MyDataPosFalse <- MyDataPos[MyDataPos$y == FALSE,]

A quick bar plot with ggplot2, coloring by the Coding factor:

library(ggplot2)
ggplot(MyDataPosFalse, aes(x, Types)) +
  ggtitle("All positive x with false y\nper type") +
  geom_bar(stat="identity", aes(fill=Coding)) +
  theme(legend.text = element_text(colour="blue", size = 12))

Summary

We covered the basic compound data structures in R — vectors, lists, matrices, and factors — then moved to selecting and filtering data from a data frame using the base package, including the type-conversion functions (is.*() / as.*()) that let you move between structures. This is a taster of the kind of handling you’ll do constantly with real-world datasets; packages like plyr and dplyr extend this considerably, and are worth a dedicated post of their own.