Functions to get to know your data.

Let’s read the data into R and give these a whirl.

library(dplyr)
library(readr)

scrap <- read_csv("https://itep-r.netlify.com/data/starwars_scrap_jakku.csv")

# View your whole dataframe and the types of data it contains
glimpse(scrap)

## Observations: 1,132
## Variables: 7
## $ receipt_date    <chr> "4/1/2013", "4/2/2013", "4/3/2013", "4/4/2013"...
## $ item            <chr> "Flight recorder", "Proximity sensor", "Vitus-...
## $ origin          <chr> "Outskirts", "Outskirts", "Reestki", "Tuanul",...
## $ destination     <chr> "Niima Outpost", "Raiders", "Raiders", "Raider...
## $ amount          <dbl> 887, 7081, 4901, 707, 107, 32109, 862, 13944, ...
## $ units           <chr> "Tons", "Tons", "Tons", "Tons", "Tons", "Tons"...
## $ price_per_pound <dbl> 590.93, 1229.03, 225.54, 145.27, 188.28, 1229....

# Use the summary function to get a quick of idea of means and maxima for your numeric data
summary(scrap)

##  receipt_date           item              origin         
##  Length:1132        Length:1132        Length:1132       
##  Class :character   Class :character   Class :character  
##  Mode  :character   Mode  :character   Mode  :character  
##                                                          
##                                                          
##                                                          
##                                                          
##  destination            amount           units          
##  Length:1132        Min.   :     65   Length:1132       
##  Class :character   1st Qu.:   1544   Class :character  
##  Mode  :character   Median :   4099   Mode  :character  
##                     Mean   :  18751                     
##                     3rd Qu.:   7475                     
##                     Max.   :2971601                     
##                     NA's   :910                         
##  price_per_pound   
##  Min.   :   145.3  
##  1st Qu.:   259.6  
##  Median :   790.7  
##  Mean   :  3811.7  
##  3rd Qu.:  1496.7  
##  Max.   :579215.3  
##  NA's   :910

# Try the rest on your own, I dare you!

nrow()

ncol()

names()

More dplyr

Dplyr is our go-to package for most analysis tasks. With the six functions below you can accomplish just about anything you’d want to do with data.

Your new analysis toolbox

Function	Job
select()	Select individual columns to drop or keep
arrange()	Sort a table top-to-bottom based on the values of a column
filter()	Keep only a subset of rows depending on the values of a column
mutate()	Add new columns or update existing columns
summarize()	Calculate a single summary for an entire table
group_by()	Sort data into groups based on the values of a column

Drop columns with a minus sign: -column_name

library(dplyr)
library(readr)

scrap <- read_csv("https://itep-r.netlify.com/data/starwars_scrap_jakku.csv")

# Drop the destination column
select(scrap, -destination)

## # A tibble: 1,132 x 6
##    receipt_date item                  origin   amount units price_per_pound
##    <chr>        <chr>                 <chr>     <dbl> <chr>           <dbl>
##  1 4/1/2013     Flight recorder       Outskir~    887 Tons             591.
##  2 4/2/2013     Proximity sensor      Outskir~   7081 Tons            1229.
##  3 4/3/2013     Vitus-Series Attitud~ Reestki    4901 Tons             226.
##  4 4/4/2013     Aural sensor          Tuanul      707 Tons             145.
##  5 4/5/2013     Electromagnetic disc~ Tuanul      107 Tons             188.
##  6 4/6/2013     Proximity sensor      Tuanul    32109 Tons            1229.
##  7 4/7/2013     Hyperdrive motivator  Tuanul      862 Tons            1485.
##  8 4/8/2013     Landing jet           Reestki   13944 Tons            1497.
##  9 4/9/2013     Electromagnetic disc~ Cratert~   7788 Tons             188.
## 10 4/10/2013    Sublight engine       Outskir~  10642 Tons            7211.
## # ... with 1,122 more rows

Keep only three columns

# Keep the item, amount and price_per_pound columns
select(scrap, c(item, amount, price_per_pound))

## # A tibble: 1,132 x 3
##    item                             amount price_per_pound
##    <chr>                             <dbl>           <dbl>
##  1 Flight recorder                     887            591.
##  2 Proximity sensor                   7081           1229.
##  3 Vitus-Series Attitude Thrusters    4901            226.
##  4 Aural sensor                        707            145.
##  5 Electromagnetic discharge filter    107            188.
##  6 Proximity sensor                  32109           1229.
##  7 Hyperdrive motivator                862           1485.
##  8 Landing jet                       13944           1497.
##  9 Electromagnetic discharge filter   7788            188.
## 10 Sublight engine                   10642           7211.
## # ... with 1,122 more rows

everything()

select() also works to change the order of columns. The code below puts the item column first and then moves the units and amount columns directly after item. We then keep everything() else the same.

# Make the `item`, `units`, and `amount` columns the first three columns
# Leave `everything()` else in the same order
select(scrap, item, units, amount, everything()) %>% head()

## # A tibble: 6 x 7
##   item        units amount receipt_date origin  destination price_per_pound
##   <chr>       <chr>  <dbl> <chr>        <chr>   <chr>                 <dbl>
## 1 Flight rec~ Tons     887 4/1/2013     Outski~ Niima Outp~            591.
## 2 Proximity ~ Tons    7081 4/2/2013     Outski~ Raiders               1229.
## 3 Vitus-Seri~ Tons    4901 4/3/2013     Reestki Raiders                226.
## 4 Aural sens~ Tons     707 4/4/2013     Tuanul  Raiders                145.
## 5 Electromag~ Tons     107 4/5/2013     Tuanul  Niima Outp~            188.
## 6 Proximity ~ Tons   32109 4/6/2013     Tuanul  Trade cara~           1229.

Only 145 credits! That’s not very much at all, oh wait…

Big things first: desc()

To arrange a column in descending order with the biggest numbers on top, we use: desc(price_per_pound)

# Put most expensive items on top
scrap <- arrange(scrap, desc(price_per_pound))

# View the top 8 rows
head(scrap, 8)

## # A tibble: 8 x 7
##   receipt_date item      origin    destination amount units price_per_pound
##   <chr>        <chr>     <chr>     <chr>        <dbl> <chr>           <dbl>
## 1 12/31/2016   Total     All       All         2.97e6 Tons          579215.
## 2 4/10/2013    Sublight~ Outskirts Niima Outp~ 1.06e4 Tons            7211.
## 3 4/14/2013    Sublight~ Outskirts Raiders     2.38e3 Tons            7211.
## 4 4/15/2013    Sublight~ Craterto~ Raiders     6.31e3 Tons            7211.
## 5 4/16/2013    Sublight~ Tuanul    Trade cara~ 3.98e3 Tons            7211.
## 6 5/14/2013    Sublight~ Craterto~ Raiders     2.99e2 Tons            7211.
## 7 6/14/2013    Sublight~ Blowback~ Raiders     8.58e3 Tons            7211.
## 8 8/6/2013     Sublight~ Craterto~ Raiders     1.77e3 Tons            7211.

Pro-tip!

Comparisons

Processing data requires many types of filtering. You’ll want to know how to select observations in your table by making various comparisons.

Key comparison operators

Exercise

Try comparing some things in the console and see if you get what you’d expect. R doesn’t always think like we do.

4 != 5

4 == 4

4 < 3

4 > c(1, 3, 5)

5 == c(1, 3, 5)

5 %in% c(1, 3, 5)

2 %in% c(1, 3, 5)

2 == NA

Dropping rows

Let’s look at the data without that pesky All category. Look in the comparison table above to find the NOT operator. We’re going to filter the data to keep only the origins that are NOT equal to “All”.

scrap <- filter(scrap, origin != "All")

We can arrange the data in ascending order by item to confirm that the “All” category is gone.

# Arrange data
scrap <- arrange(scrap, item)

head(scrap)

## # A tibble: 6 x 7
##   receipt_date item      origin    destination amount units price_per_pound
##   <chr>        <chr>     <chr>     <chr>        <dbl> <chr>           <dbl>
## 1 4/4/2013     Aural se~ Tuanul    Raiders        707 Tons             145.
## 2 5/22/2013    Aural se~ Outskirts Niima Outp~   3005 Tons             145.
## 3 5/23/2013    Aural se~ Tuanul    Raiders       6204 Tons             145.
## 4 6/4/2013     Aural se~ Tuanul    Raiders       3120 Tons             145.
## 5 6/10/2013    Aural se~ Blowback~ Niima Outp~   2312 Tons             145.
## 6 6/20/2013    Aural se~ Outskirts Trade cara~   6272 Tons             145.

Now let’s take another look at that bar chart. Is there anything else that is less than perfect with our data?

library(ggplot2)

ggplot(scrap, aes(x = origin, y = amount)) + geom_col()

Add a column

First, let’s add a column with our names. That way Rey can thank us personally when her ship is finally up and running.

# Add your name as a column
scrap <- mutate(scrap, scrap_finder = "BB8")

Add several columns

Let’s also add a new column to document the data measurement method.

#  Add your name as a column and
#  some information about the method
scrap <- mutate(scrap, 
                scrap_finder    = "BB8",
                measure_method  = "REM-24")

## REM = Republic Equivalent Method

Change a column

Remember how the units of Tons was written two ways: “TONS” and “Tons”? We can use mutate() together with tolower() to make sure all of the scrap units are written in lower case. This will help prevent them from getting grouped separately in our plots and summaries.

# Set units to all lower-case
scrap <- mutate(scrap, units = tolower(units))

# toupper() changes all of the letters to upper-case.

Add calculated columns

In our work we often use mutate to convert units for measurements. In this case, let’s estimate the total pounds for the scrap items that are reported in tons.

scrap_pounds <- mutate(scrap, amount_lbs = amount * 2000)

Final stretch!

It’s time to get off this dusty planet and Rey needs an Ion engine. Let’s filter the data to only that type of item.

# Grab only the items named "Ion engine"
scrap_pounds <- filter(scrap_pounds, item == "Ion engine")

Arrange the data in descending order of pounds so Rey knows where the highest amount of scrap comes from. Then she can go there, trade for some discounted parts and we’re free to FLY TO ENDOR!

# Arrange data
scrap_pounds <- arrange(scrap_pounds, desc(amount_lbs))

# Return the origin of the highest amount_lbs of scrap
head(scrap_pounds, 1)

# Plot the total amount_lbs by origin
ggplot(scrap_pounds, aes(x = origin, y = amount_lbs)) + 
  geom_col()

First mission complete!

Time to get off this dusty planet, we’re flying to Endor!

BB8 has a new mission

BB8 just received new data suggesting there was a large magnetic storm right before Site 1 burned down on Endor. Sounds like we’re going to be STORM CHASERS!

Get the dataset

While we’re relaxing and flying to Endor let’s get set up with our new data.

DOWNLOAD - Endor Air data

1. Download the data above.
2. Create a new folder called “data” in your project folder.
3. Move the Endor data to there.
4. Open a new R script called endor_air.R

Now we can read in the data from the data folder with our favorite read_csv function.

library(readr)
library(dplyr)
library(janitor)

air_endor <- read_csv("data/air_endor.csv")

names(air_endor)

# Clean the column names
air_endor <- clean_names(air_endor)

names(air_endor)

## [1] "analyte"        "lab_code"       "units"          "start_run_date"
## [5] "site_name"      "long_site_name" "planet"         "result"

The lubridate package

It’s about time! Lubridate makes working with dates easier.

We can find how much time has elapsed, add or subtract days, and find seasonal and day of the week averages.

Convert text to a DATE

Get date parts

Install lubridate

First, type or copy and paste install.packages("lubridate") into RStudio.

Clean the dates

Let’s use the mdy() function to turn the start_run_date column into a nicely formatted date.

library(lubridate)

mag_endor <- mutate(mag_endor, date = mdy(start_run_date),
                               year = year(date))

According to the request we received, the Resistance is only interested in observations from the year 2017. So let’s filter the data down to only dates within that year.

mag_endor <- filter(mag_endor, year == "2017")

Time series

ggplot(mag_endor, aes(x = date, y = result)) + 
  geom_line(size = 2, color = "tomato") + 
  geom_point(size = 5, alpha = 0.5)  # alpha changes transparency

Looks like the measurements definitely picked up a signal towards the end of the year. Let’s make a different chart to show when this occurred.

Heatmap or calendar plot

Sometimes with air data we want to know if there is seasonality in the data, this is also true for magnetic storm data from Endor. Calendar plots are great for this.

We will use ggplot() and bring out the geom_tile() to organize our data by dates and find when and if that storm happened.

First, we’ll pull more information out of the dates.

mag_endor <- mutate(mag_endor, 
                    day_of_week   = wday(date, label = TRUE),
                    day_of_month  = mday(date),
                    week_of_month = day(date)/7, 
                    month         = month(date, label = TRUE))

ggplot(mag_endor, aes(x = day_of_week, y = desc(week_of_month), fill = result)) +
  geom_tile(color = "gray") +
  geom_label(aes(label = day_of_month), size = 8, color = "white") +
  facet_wrap("month")

…I wish I could type the name of the data frame less often.

Now you can!

We use the %>% to chain functions together and make our scripts more streamlined. We call it the pipe.

Here are 2 examples of how the %>% operator is helpful.

#1: It eliminates the need for nested parentheses.

Say you wanted to take the sum of 3 numbers and then take the log and then round the final result.

round(log(sum(c(10,20,30,40))))

The code doesn’t look much like the words we used to describe it. Let’s pipe it so we can read the code from left to right.

c(10,20,30,50) %>% sum() %>% log10() %>% round()

#2: We can combine many processing steps into one cohesive chunk.

Here are some of the functions we’ve applied to the scrap data so far:

scrap <- arrange(scrap, desc(price_per_pound))

scrap <- filter(scrap, origin != "All")

scrap <- mutate(scrap, 
                scrap_finder    = "BB8",
                measure_method  = "REM-24")

We can use the %>% operator and write it this way.

scrap <- scrap %>%
           arrange(desc(price_per_pound)) %>%
           filter(origin != "All") %>%
           mutate(scrap_finder    = "BB8",
                  measure_method  = "REM-24")

mag_endor <- filter(air_endor, analyte == "magnetic_field")

mag_endor <- mutate(mag_endor, date = mdy(start_run_date))