Data exploration¶
After reading data from a file (see the previous chapter), the next thing to do is to look at some summary statistics. This is in the first place just to check your data. Very often you discover some strange values that are not quite right. Careful inspection of your data after you read it is very important. It can help avoid a lot of trouble later on when you are trying to explain odd results!
Sometimes you can correct errors in R but in other cases you first need to fix the data file. Fixing a data file can be the way to go if you are dealing with your own primary data. However when you are working with a file provided by someone else it is generally better to not change the file, but to correct things via R code, if possible. That leaves an exact trail of what you have done, and allows you to apply the same corrections to a new version of the data.
Inspecting your data is also important to understand it better, and such “exploratory data analysis” can be an important step before doing the specific analyses of interest. There are many general and specialized tools for this, and here we only cover some of the basics.
Summary and table¶
Consider data.frame d
d <- data.frame(id=1:10,
name=c('Bob', 'Bobby', '???', 'Bob', 'Bab', 'Jim', 'Jim', 'jim', '', 'Jim'),
score1=c(8, 10, 7, 9, 2, 5, 1, 6, 3, 4),
score2=c(3,4,5,-999,5,5,-999,2,3,4), stringsAsFactors=FALSE)
d
## id name score1 score2
## 1 1 Bob 8 3
## 2 2 Bobby 10 4
## 3 3 ??? 7 5
## 4 4 Bob 9 -999
## 5 5 Bab 2 5
## 6 6 Jim 5 5
## 7 7 Jim 1 -999
## 8 8 jim 6 2
## 9 9 3 3
## 10 10 Jim 4 4
d is very small and we can simply look at the values in d to see
if they look all-right. But with real data you may have 100s or 1000s of
rows and many columns, making it very hard, if not impossible, to spot
errors by eye-balling.
The summary function is an easy way to start, at least for numeric variables.
summary(d)
## id name score1 score2
## Min. : 1.00 Length:10 Min. : 1.00 Min. :-999.00
## 1st Qu.: 3.25 Class :character 1st Qu.: 3.25 1st Qu.: 2.25
## Median : 5.50 Mode :character Median : 5.50 Median : 3.50
## Mean : 5.50 Mean : 5.50 Mean :-196.70
## 3rd Qu.: 7.75 3rd Qu.: 7.75 3rd Qu.: 4.75
## Max. :10.00 Max. :10.00 Max. : 5.00
The minimum value of variable score2 is -999. That was probably used
in data entry to indicate a missing value. These should be changed to
NA.
# which values in score2 are -999?
i <- d$score2 == -999
# set these to NA
d$score2[i] <- NA
summary(d)
## id name score1 score2
## Min. : 1.00 Length:10 Min. : 1.00 Min. :2.000
## 1st Qu.: 3.25 Class :character 1st Qu.: 3.25 1st Qu.:3.000
## Median : 5.50 Mode :character Median : 5.50 Median :4.000
## Mean : 5.50 Mean : 5.50 Mean :3.875
## 3rd Qu.: 7.75 3rd Qu.: 7.75 3rd Qu.:5.000
## Max. :10.00 Max. :10.00 Max. :5.000
## NA's :2
The two steps used above: i <- d$score2 == -999 and
d$score2[i] <- NA are usually done in a single line:
d$score2[d$score2 == -999] <- NA.
For character (and integer) variables it can be useful to use unique
and table:
unique(d$name)
## [1] "Bob" "Bobby" "???" "Bab" "Jim" "jim" ""
table(d$name)
##
## ??? Bab Bob Bobby jim Jim
## 1 1 1 2 1 1 3
Often you will discover slight variations in spelling that need to be corrected. In this case, let’s assume that “Bobby” and “Bab” should both be “Bob”. We replace “Bab” and “Bobby” with “Bob”.
d$name[d$name %in% c("Bab", "Bobby")] <- "Bob"
table(d$name)
##
## ??? Bob jim Jim
## 1 1 4 1 3
“jim” should be “Jim”. It is easy enough to replace as done above. But
what if there were many cases like that? It would be easy to make all
character values lower- or uppercase with d$name <- toupper(d$name)
but I want to keep the normal name capitalization of the first letter
only. So let’s assure that all names start with an uppercase letter.
# get the first letters
first <- substr(d$name, 1, 1)
# get the remainder
remainder <- substr(d$name, 2, nchar(d$name))
# assure that the first letter is upper case
first <- toupper(first)
# combine
name <- paste0(first, remainder)
# assign back to the variable
d$name <- name
table(d$name)
##
## ??? Bob Jim
## 1 1 4 4
The question marks in name should probably also be replaced with
NA.
d$name[d$name == "???"] <- NA
table(d$name)
##
## Bob Jim
## 1 4 4
You can force table to also count the NA values:
table(d$name, useNA="ifany")
##
## Bob Jim <NA>
## 1 4 4 1
Note that there is one “empty” value.
d$name[9]
## [1] ""
That should also be a missing value in this case.
d$name[d$name == ""] <- NA
table(d$name, useNA="ifany")
##
## Bob Jim <NA>
## 4 4 2
You can also use table to make a contingency table of two variables.
table(d[ c("name", "score2")])
## score2
## name 2 3 4 5
## Bob 0 1 1 1
## Jim 1 0 1 1
Quantile, range, and mean¶
Other useful functions include quantile, range, and mean.
quantile(d$score1)
## 0% 25% 50% 75% 100%
## 1.00 3.25 5.50 7.75 10.00
range(d$score1)
## [1] 1 10
mean(d$score1)
## [1] 5.5
Note that in some functions you may need to use na.rm=TRUE if there
are NA values. Otherwise, as soon as there is a single NA value
in a computation, the results becomes NA. This is very common in R
— so keep that in mind if all your results are NA.
quantile(d$score2)
## Error in quantile.default(d$score2): missing values and NaN's not allowed if 'na.rm' is FALSE
range(d$score2)
## [1] NA NA
quantile(d$score2, na.rm=TRUE)
## 0% 25% 50% 75% 100%
## 2 3 4 5 5
range(d$score2, na.rm=TRUE)
## [1] 2 5
In this data exploration phase it is very useful to make plots. We’ll
discuss plotting in a later chapter, but here are four example plots.
Note how par(mfrow=c(2,2)) sets up the canvas for two rows and
columns, that is for four plots.
par(mfrow=c(2,2))
plot(d$score1, d$score2)
boxplot(d[, c("score1", "score2")])
plot(sort(d$score1))
hist(d$score2)
