I. BASICS: Introduction to R, and R Commander (Rcmdr) and R functions for basic statistical models

Statistics is everywhere! Do you see it even in this news item?

Example: Nanos Poll (Justin T.) (.pdf)

(a) Topic 1: Installations / Introduction to R Graphics

2018 Cohort, please note! I moved the insallation info files to new locations to make it easier for the new cohort. Please adjust your bookmarks.

a.1 Installations

As of 2018-07-19, current version for R is 3.5.1, and for Rcmdr 2.4-4.

a.1.1 Software Installation Videos

If you want to see and hear videos where I explain how to install the R and R Commander software, please visit the following link:

The Install Videos . These videos show the installations of R and Rcmdr on PC and Mac computers.

a.1.2 Software Installation Documentation

If you want to follow written instructions, please see below.

(I would advise consulting John Fox's Windows and Mac installation instructions for Rcmdr for futher details.)

Instructions for downloading and installing R and Rcmdr on Windows. [Link for downloading R.]

Here are the screenshots for the steps to install R and Rcmdr:

First, uninstall earlier versions of R (if applicable):

Screenshots of step-by-step instructions to uninstall earlier version of R

Install R:

Screenshots of step-by-step instructions to install R

Note 1: After the "Select Additional Tasks" window, R will install several files on your computer.

Note 2: After "Completing the R for Windows ..." window, R is installed on your computer. Now go to your desktop and choose "Run as adminstrator" on the R icon.

Install Rcmdr (from within R):

Screenshot of step-by-step instructions to install Rcmdr

Note: For R's Mac OS X and Linux/Unix installation instructions, please click here.

Note: For Rcmdr's Mac OS X and Linux/Unix installation instructions, please click here.

• Before downloading any of the datasets, you should create a folder (preferably under "My Documents") and give it a name similar to the dataset.
• Save your dataset to the folder you created.
• After starting Rcmdr, click "File > Change Working Directory..." and point at the folder you have created for that dataset.
• Any files you save from within Rcmdr will then appear in the folder you have created.

*

Wolfgang Jank's book Business Analytics for Managers (Use R!) is our main text and it is available free-of-charge as a .pdf file from McMaster's online library.

The "theory" (i.e., background material) behind the techniques described below will be given after each example.

a.2 Graphics in R

Example: Let's use the Direct marketing data set [Table 2.6 DirectMarketing.csv] to plot some amazing graphs via Rcmdr. Graphics obtained from Rcmdr in this dataset are here as a .pdf file.

(b) Topic 2: Descriptive Statistics and Probability Calculations

b.1 Symmetry, positively-skewed and negatively-skewed

Symmetric distribution

Example: Here is an Excel file [IQScores-1000.csv] of the IQ scores of 1,000 individuals. Plotting the histogram reveals that these scores are distributed in a symmetric manner. ¶

Note: It is claimed that Marilyn vos Savant has an IQ score of almost 190. We will have more to say about her when we do the "Car and the Goats" problem.

Mac salaries: Skewed or symmetric?

Example: Here's the distribution of the incomes of McMaster employees who earned above $100,000 in 2017.  [McMaster-Sunshine-NoNames-Sorted-2017.csv] Is the distribution symmetric, positively-skewed or negatively-skewed?

Here's the Excel version of the same file: McMaster-Sunshine-NoNames-Sorted-2017.xlsx

This information is public and the most recent data (2017) are available on the Ontario Government web site.

*

...and finally, an article which would interest almost everyone! The following link is excerpted from the book "Who We Are," by C. Rudder, 2014 (Random House). Mr. Rudder was one of the founders of the online dating site OkCupid.com. This is similar to other online dating sites for singles such as eHarmony.com and Match.com.

His article in the National Post:

Dataclysm: The data guru for a popular dating site explains what men and women want from a mate

CAVEAT: The histogram in the above link for a "50-year old woman" would certainly look different if we had data from dating sites such as OurTime.com which caters to people over 50.

An "outlier": Here is a news item about an older man and his wife who was one-third of his age. (A rather sad story.) [.pdf]

b.2 Mean and variance of a dataset

Example: I will motivate these concepts with the help of hot and cold water buckets!

Scenario 1: One bucket (BLUE) has freezing water at 0C, other (RED) has boiling water at 100C. Average is 50C. Why am I so uncomfortable?

Scenario 2: One bucket (GREEN) has lukewarm water at 50C, other (GREEN) has also lukewarm water at 50C. Average is 50C. So nice!

Both means are the same but what distinguishes the two scenarios? The Variance!

Example: (Exam scores in a small MBA class) Here is an Excel file of the calculations for mean and variance. If we have a population of N items, then division for variance is performed using N. If we have a sample of n items, then division for variance is performed using n-1.

Example/Exercise: Here is a .csv file of the same data. Use R to analyze it.

b.3 Probability calculations

Example: Coin Toss  : The fraction of heads obtained in a series of coin tosses approaches 0.5.

Example: Lotto 6/49 from Ontario Lottery Corporation. This is how they pick the six lucky numbers.

• You have a better chance of being struck by lightning, but someone wins it. Look at this former DSB employee (.pdf)
• What is the probability of winning in "Baby Lotto 2/4"?
• Two important R functions: choose(4,2) and combn(4,2)

Example: Birthdays : In a set of 50 randomly chosen people, what is the probability that any two will have the same birthday? Let's see what happened in our four sections.

• Birthdays of the Fall 2018 EMBA class. We have 31 people (including me) and four matches!
• Birthdays in all four MBA sections (Fall 2015)

The result may seem paradixocal; so, here's a link that explains the birthday problem.

Here, I explain this problem for the case of finding two matching birthdays as days of the week (M, T, W, Th, F, S, Su).

Example: The Monty Hall Problem and Monty's show "Let's Make a Deal". 

Monty asks: "Do you want to switch the door?"

Here are some explanations of this problem.

• Ron Clarke's explanation
• Numb3rs Math
• And here's a simulation page for this problem that works with Internet Explorer, only.
• Here's my solution using a probability tree.

♦ We will talk about two important concepts (independence of events and mutually exclusive events) before the next Example.

Example: One final example: Psy's Gangnam Style on my iPod. See notes.

• Would you believe it? Even the Mac president did the Gangnam  Style dance (see 1:40).

Top of the Page

(c) Topic 3: Random Variables

c.1 Discrete random variables

Example: A gamble based on a coin toss. Fair gamble vs. Unfair gamble. What is the "average" gain in each gamble? Did you just discover the formula for the expected (mean) gain in these gambles?

Example: Pierik's bikes. We will calculate the expected (mean) value of demand, E(X); and variance of demand Var(X).

Here's the Excel spreadsheet  for calculating these quantities for the bicycle shop data.

• R documentation for binomial distribution.
• Background material for binomial probabilities. Wikipedia.

Exercise: Here is a more challenging problem from healthcare area involving the testing of a new drug. Find the solution using Rcmdr. (Answer: 0.74)

c.3 Expected value and variance

Example: Roulette is a board game with a large "house edge." 

Here is the board for American roulette:

The roulette wheel looks like this:

Payout amounts if you win your bet in roulette.

This link simulates the roulette game.

We can simulate a roulette roll using Excel's =RANDBETWEEN(1,38)function. Here's an example: RouletteWithExcel

But please note: I am not advocating gambling; in fact, I am very much against playing such games as they eventually ruin the gambler. The purpose of this example is to illustrate that roulette is an unfair game and you shouldn't play it with real money!

(d) Topic 4: Confidence Intervals and Hypothesis Testing

d.1 Confidence intervals

Poll Results

Nanos Poll (again!)

Confidence Intervals for the Proportion p

We will do this with the participation of the class and we will use an inflatable globe to estimate the proportion of the water surface to the total surface of the globe.

Here is the video of this experiment I recorded in Section C02 (November 4, 2010, Thursday).

*****

Example: (Population proportion) The CI for population proportion is easy to obtain. Suppose you poll 1000 people and 340 of them state that they would vote Liberal, if the election were held today. Here is what we do to find a 95% CI:

> prop.test(340,1000)

1-sample proportions test with continuity correction

data: 340 out of 1000, null probability 0.5
X-squared = 101.761, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
0.3108142 0.3704312
sample estimates:
p
0.34

So, the sample proportion is 0.34, with a 95% CI of [0.3108,0.3704], i.e., a margin of error of about 3%. ¶

d.2 Hypothesis testing

(What is the meaning of the word "hypo" in "hypo-thesis"?

Hypo-allergenic as in ?

Hypo-thermia as in ?

"Hypo-potamus" (??) as in ?                                             Tricked you! This is a hippo-potamus. :-)

"Hypo" means "below, under" in Greek.

"Thesis", is something that is proven to be true.

So, "hypo-thesis" is something that is yet to be proven to be true.

Now, what does a lady tasting tea have to do with hypothesis testing?

"The Lady Tasting Tea" : Can tea poured into milk taste differently than that of milk poured into tea? This experiment was originally designed by Professor Ronald Fisher in the 1920s, and it will help us motivate the discussion of hypothesis testing. We will, however, use Coke and Pepsi in our experiment. The "lady" in the story is Dr. Muriel Bristol of Cambridge University.

She claims that she knows the difference. Here, my null hypothesis is "H0: She is guessing". Now, if she is purely guessing, there is a 0.014 probability of getting all 8 cups correct. This is such an unlikely outcome but if it happens, I am willing to change my mind and reject my null H0 and believe that she can tell the difference. But what if she just guessed and got all correct? Then I made a mistake in changing my mind, but the probability of me making this mistake is only 0.014. This is the p-value.

In case you were wondering, here is the mathematics behind the calculations. In this link you can find the probaibilities of 0, 2, 4, 6 or 8 correct identifications which uses the hypergeometric probabilities (which we did not discuss).

Type I and Type II errors

Type I error : In 1959, Steven Truscott was found guilty of murdering his classmate even though he did not commit any crime. In 2007, he was formally acquitted of the crime. In 2008, the government of Ontario awarded him $6.50 million in compensation

Type II error : Many people believe that O. J. Simpson had murdered his wife and he should have been found guilty. But after a lenghty trial, he was acquitted in 1995.

• R documentation for  t.test.
• Background material for hypothesis testing. Wikipedia.

Exercise: For the Atkins problem test the null hypothesis that Atkins users lose, on average, 17 pounds after 6 months. (This is now a two-sided test.)

Exercise: Use the following data values to test the hypothesis that true mean is 750 vs. the hypothesis that it differs from 750: (801,814,784,836,820) What is the p-value? (Answer: p = .0023; so reject the null)