I. INSTALLATIONS and BASICS: Installations for R, and R Commander (Rcmdr) and R functions for basic statistical models

 

 

Statistics is everywhere! Do you see it even in this news item?

 

Example: Nanos Poll (Justin T.) (.pdf)

 

"Dewey Defeats Truman" : The picture and the story behind the Harry Truman vs. Thomas Dewey 1948 presidential election.  (Did Dewey win?)

 

Approximate formula for the margin of error E of a 95% confidence interval for such things:

 

E = ±100/sqrt(n) % where n is the sample size.

 

So, if n = 900, then E = 100/sqrt(900) = 3.3%

 

 

(a) Topic 1: Installations / Introduction to R Graphics

 

IMPORTANT: If you are using a company computer, please make sure that you can install R and Rcmdr on that computer. This may involve removing the company firewall! Please consult your company’s IT personnel to find out how that can be done.

 

You will receive a bonus of 1.0 mark if you can install the software correctly before the first class and report it to me. (Yup!) Just send me a screenshot of the Rcmdr screen on your laptop.

 

NOTE: Once you install R, and install Rcmdr and then load Rcmdr, your screenshot should look like this. If you have Dropbox, clicking the PrintScreen key will place your screenshot in Dropbox's Screenshots folder. If you don't have Dropbox, you can take a picture of the screen with your phone and send it to me.

 

a.1 Installations for R and Rcmdr

 

As of 2019-08-16, current version for R is 3.6.1, and for Rcmdr 2.5-3.

 

a.1.1 Software Installation (Written Documentation) for current version of R and Rcmdr

 

The simplest way to install these software is by following my written instructions below. (I would advise consulting John Fox's Windows and Mac installation instructions for Rcmdr for futher details.) You can also check out my videos and screenshots; see Section a.1.2 and Section a.1.3, below.

 

Written instructions for downloading and installing R and Rcmdr on Windows.

 

Link for downloading R.

 

 

a.1.2 Software Installation (Videos) for an earlier version of R v3.4.1 and Rcmdr v2.3-2

 

If you want to see and hear videos where I explain how to install the R and R Commander software, please see below. These videos show the installations of R and Rcmdr on PC and Mac computers.

 

a.1.2.i : EMBA T711: R, and R Commander (Rcmdr) on Windows (and corrplot)

 

Uninstall R (Video) : Uninstall older versions of R, if applicable.

Install R (Video) : Install R. < Download link for R >

Install Rcmdr (Video) : Install R Commander (Rcmdr). This is done from within R.

 

corrplot (An R package for graphing correlation matrices)

Install corrplot (Video) : Use InstallRun-corrplot for instructions and Table2.1HousePrices.csv for data.

 

Graphics with Rcmdr

Instructions for step-by-step guide to plot graphics with Rcmdr (Video)

 

vLab (If needed)

Instructions for accessing R/Rcmdr on our vLab (Video) (This uses an older version of R, but that's OK.)

 

a.1.2.ii : EMBA T711: R, and R Commander (Rcmdr) on Mac

 

Install R and Rcmdr on Mac (Video): This is a YouTube video prepared by one of our IT personnel. (Thanks, Alex Wang!, <wangal@mcmaster.ca>) It shows the installations of XQuartz, R and then Rcmdr. I hope it works. (I was told that, as an alternative, Mac users can open a Windows partition and install R there, too.)

 

 

 

a.1.3 Software Installation (Screenshots) for an earlier version of R v3.4.0 and Rcmdr v2.3-2

 

If you want to see the screenshots for the steps to install R and Rcmdr, they are here:

 

First, uninstall earlier versions of R (if applicable)

 

Install R:

Note 1: After the "Select Additional Tasks" window, R will install several files on your computer.

Note 2: After "Completing the R for Windows ..." window, R is installed on your computer. Now go to your desktop and choose "Run as adminstrator" on the R icon.

 

 

Install Rcmdr (from within R):

Screenshot of step-by-step instructions to install Rcmdr

 

Note: For R's Mac OS X and Linux/Unix installation instructions, please click here.

 

Note: For Rcmdr's Mac OS X and Linux/Unix installation instructions, please click here.

 

a.1.4 Comments

 

Important! When you start using R, if you want to save your datasets and other files, please follow these instructions:

  • Before downloading any of the datasets, you should create a folder (preferably under "My Documents") and give it a name similar to the dataset.
  • Save your dataset to the folder you created.
  • After starting Rcmdr, click "File > Change Working Directory..." and point at the folder you have created for that dataset.
  • Any files you save from within Rcmdr will then appear in the folder you have created.

 

*

 

Wolfgang Jank's book Business Analytics for Managers (Use R!) is our main text and it is available free-of-charge as a .pdf file from McMaster's online library.

 

 

 

 

The "theory" (i.e., background material) behind the techniques described below will be given after each example.

 

a.2 Graphics in R

 

 

Example: Let's use the Direct marketing data set [Table 2.6 DirectMarketing.csv] to plot some amazing graphs via Rcmdr. Graphics obtained from Rcmdr in this dataset are here as a .pdf file.

 

Let me also show you how we can use corrplot to plot the correlation matrix of this dataset. We will do this step-by-step.

 

But first, some examples of correlation (which measures the degree of connection between two variables). Height and weight are positively correlated. Taller people tend to be heavier. Interest rates and inflation are usually negatively correlated. Check this out.

 

 

Take a look at this for the variety of options you can use with corrplot.

 

 

Exercise: Now use this House price data set [Table 2.1 HousePrices.csv] and generate graphics as we did above. Graphics obtained from Rcmdr in this dataset are here.

 

Exercise: The problem statement for this Education Level/Gender/Income problem is here. You will need this Excel data file [Education-Gender-Income.xlsx] to import into Rcmdr and do the calculations.

 

Top of the Page

 

 

(b) Topic 2: Descriptive Statistics and Probability Calculations

 

b.1 Symmetry, positively-skewed and negatively-skewed

 

Symmetric distribution

 

 

Example: Here is an Excel file [IQScores-1000.csv] of the IQ scores of 1,000 individuals. Plotting the histogram reveals that these scores are distributed in a symmetric manner. ¶

 

Note: It is claimed that Marilyn vos Savant has an IQ score of almost 190. We will have more to say about her when we do the "Car and the Goats" problem.

 

Mac salaries: Skewed or symmetric?

McMaster

Example: Here's the distribution of the incomes (as an Excel file) of McMaster employees who earned above $100,000 in 2018.  [MacSalaries-2018.xlsx] Is the distribution symmetric, positively-skewed or negatively-skewed?

This information is public and the most recent data (2018) are available on the Ontario Government web site.

*

...and finally, an article which would interest almost everyone! The following link is excerpted from the book "Who We Are," by C. Rudder, 2014 (Random House). Mr. Rudder was one of the founders of the online dating site OkCupid.com. This is similar to other online dating sites for singles such as eHarmony.com and Match.com.

 

His article in the National Post:

Dataclysm: The data guru for a popular dating site explains what men and women want from a mate

 

CAVEAT: The histogram in the above link for a "50-year old woman" would certainly look different if we had data from dating sites such as OurTime.com which caters to people over 50.

An "outlier": Here is a news item about an older man and his wife who was one-third of his age. (A rather sad story.) [.pdf]

 

Here's a Wikipedia article on age disparity between husband and wife. Interesting histogram!

 

 

b.2 Mean and variance of a dataset

 

 

Example: I will motivate these concepts with the help of hot and cold water buckets!

 

Scenario 1: One bucket (BLUE) has freezing water at 0C, other (RED) has boiling water at 100C. Average is 50C. Why am I so uncomfortable?

Scenario 2: One bucket (GREEN) has lukewarm water at 50C, other (GREEN) has also lukewarm water at 50C. Average is 50C. So nice!

 

Both means are the same but what distinguishes the two scenarios? The Variance!

 

Example: (Exam scores in a small MBA class) Here is an Excel file of the calculations for mean and variance. If we have a population of N items, then division for variance is performed using N. If we have a sample of n items, then division for variance is performed using n-1.

 

Example/Exercise: Here is a .csv file of the same data. Use R to analyze it.

 

 

b.3 Probability calculations

 

coin toss

Example: Coin Toss  : The fraction of heads obtained in a series of coin tosses approaches 0.5.

6-49

Example: Lotto 6/49 from Ontario Lottery Corporation. This is how they pick the six lucky numbers.

 

    • You have a better chance of being struck by lightning, but someone wins it. Look at this former DSB employee (.pdf)
  • What is the probability of winning in "Baby Lotto 2/4"?
  • Two important R functions: choose(4,2) and combn(4,2)

> choose(4,2)
[1] 6

 

> combn(4,2)
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] 1 1 1 2 2 3
[2,] 2 3 4 3 4 4

 

Now try these functions with (49,6).

Example: Birthdays : In a set of 50 randomly chosen people, what is the probability that any two will have the same birthday?

 

Birthday

 

 

 

 

The result may seem paradixocal; so, here's a link that explains the birthday problem.

Here, I explain this problem for the case of finding two matching birthdays as days of the week (M, T, W, Th, F, S, Su).

Example: The Monty Hall Problem and Monty's show "Let's Make a Deal"

Monty asks: "Do you want to switch the door?"

Here are some explanations of this problem.

car and goats

♦ We will talk about two important concepts (independence of events and mutually exclusive events) before the next Example.

Independent events

 

I am holding a fair coin (two sides) and a fair die (six sides). For the coin, the probabiilty of getting any side is 1/2. For the die it is 1/6.

 

Scenario 1: I roll the die and I get a 4. Now I will flip the coin. What is the probability of getting a Heads? It is still Pr(H)  = 1/2. The result of the die roll does not affect the coin's outcome.

 

Scenario 2; Now I flip the coin and get a Heads (H). I will roll the die now. What is the probability of getting a 4? It is still Pr(4) = 1/6. The result of the coin toss does not affect the die's outcome.

 

So, here coin toss and die roll outcomes are independent.

 

Now I roll the die and toss the coin at the same time. What is the probabiltiy of getting a 4 and H? There are a total of 12 outcomes, so it must be 1/12.

 

We can do it more easily: Pr(4 and H) = Pr(4) x Pr(H) = (1/6) x (1/2) = 1/12. So, for independent events, probability of their joint outcome is simply the product of the individual probabilities.

 

We note that the 12 outcomes {(1,H), (2,H), ...,(6,H),(1,T), (2,T), ...,(6,T)} are mutually exclusive and collectively exhaustive; i.e., they are all different and they cover all possibilities.

 

Example: One final and important example: Psy's Gangnam Style on my iPod which has about 500 songs. (I wasted so much money on these amd made iTunes very rich.) See notes.

 

Two important questions: (1) What is the probability of hitting Psy's song if I shuffle once? (2) What is the probability of hitting Psy's song at least once if I shuffle three times?

Top of the Page

 

(c) Topic 3: Random Variables

 

c.1 Discrete random variables

 

Example: A gamble based on a coin toss. Fair gamble vs. Unfair gamble. What is the "average" gain in each gamble? Did you just discover the formula for the expected (mean) gain in these gambles?

 

Example: Pierik's bikes. We will calculate the expected (mean) value of demand, E(X); and variance of demand Var(X).

 

Here's the Excel spreadsheet  for calculating these quantities for the bicycle shop data.

bicycle pierik

 

 

c.2 Binomial distribution (random variable)

BinomialHere is my handwritten notes on the binomial distribution.

ballsThree tennis balls  : My success probability at each throw is p = 0.6. What is the probability that I will have all three balls in the bucket? Two balls? One ball? Zero? 

 

 

Exercise: Here is a more challenging problem from healthcare area involving the testing of a new drug. Find the solution using Rcmdr. (Answer: 0.74)

 

c.3 Normal (symmetric) distribution (random variable)

 

 

Example: Here is again the Excel file [IQScores-1000.csv] of the IQ scores of 1,000 individuals. Plotting the histogram and other related graphs, especially the Quantile comparison plot, reveal that these scores are distributed normally with a mean of about 100 and a standard deviation of about 15. The probability that someone picked at random from this group has an IQ of at least 145 is 0.0013. Here are the results. ¶

 

More Examples:

Normal distribution

People

Heights (in meters) and handspans (in centimeters) of students.

Actual heights and handspans of the 2019 EMBA cohort. [xlsx] Handspan distribuiton looks nice and almost normal.

Exam

 

Binomial and normal : When n is large and p is around 0.5, binomial looks like normal. Let's see it first with Rcmdr.

The next few links illustrate this.

Galton

Galton's Board: Here's what happens if you drop a large number of marbles in the board and p = 1/2. (Binomial turns into normal.) This link does it in real time. But here is a cool animation.

Check this out to see how the shape of the normal distribution changes if we vary the mean and standard deviation.

Visual check for normality: This involves Rcmdr's "Quantile comparison plot". Try it with this

Here is a Wikipedia article on the normal distribution.

 

c.3 Expected value and variance

Example: Roulette is a board game with a large "house edge." 

Here is the board for American roulette: American roulette

The roulette wheel looks like this: Wheel

Payout amounts if you win your bet in roulette.

This link simulates the roulette game.

We can simulate a roulette roll using Excel's =RANDBETWEEN(1,38)function. Here's an example: RouletteWithExcel

But please note: I am not advocating gambling; in fact, I am very much against playing such games as they eventually ruin the gambler. The purpose of this example is to illustrate that roulette is an unfair game and you shouldn't play it with real money!

  • What is the probability of winning if you bet,
    • On Red? (If you bet $1 and you win, you get $1)
    • On 1st 12? (If you bet $1 and you win, you get $2)
    • On 17? (If you bet $1 and you win, you get $35)
  • We will do a few examples.
  • Warning! In the long run, your winnings are always less than the amount you bet. SO, IF YOU KEEP PLAYING, YOU WILL LOSE EVERYTHING YOU HAVE!
  • Why would you want to buy a stock which will surely lose you money in the long run?

The expected value in American Roulette is -5.2%. That is, every time you bet $100, on average you LOSE $5.2.

 

Top of the Page

 

 

(d) Topic 4: Confidence Intervals and Hypothesis Testing

 

d.1 Confidence intervals

 

Poll Results

Poll

Nanos Poll (again!)

  • As of mid-June 2013, Liberals had the support of 34.2% of voters, Conservatives 29.4%, and NDP 25.3%.
  • The article states that Nanos surveyed 816 committed voters and the poll is accurate plus or minus 3.5 percentage points, 19 times out of 20.
  • So, we are 95% sure that the true proportion of Liberal support is approximately somewhere between 30.7% (34.2 - 3.5%) and 37.7% (34.2 + 3.5%).
Confidence Intervals for the Proportion p

Globe

We will do this with the participation of the class and we will use an inflatable globe to estimate the proportion of the water surface to the total surface of the globe.

Here is the video of this experiment I recorded in Section C02 (November 4, 2010, Thursday).

 

 

*****

 

Example: (Population proportion) The CI for population proportion is easy to obtain. Suppose you poll 1000 people and 340 of them state that they would vote Liberal, if the election were held today. Here is what we do to find a 95% CI:

 

> prop.test(340,1000)

 

1-sample proportions test with continuity correction

data: 340 out of 1000, null probability 0.5
X-squared = 101.761, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
0.3108142 0.3704312

sample estimates:
p
0.34

So, the sample proportion is 0.34, with a 95% CI of [0.3108,0.3704], i.e., a margin of error of about 3%. ¶

 

 

Example: Here is an hypothetical problem. One thousand US citizens were asked who they would vote for; Trump or Clinton? The sample results are in this Excel file [Trump-vs-Clinton.xlsx]. What is the CI for Clinton supporters? We use Rcmdr's single-sample proportion test, and obtain these results. Note that this test works with text data as "factors," only. ¶

 

Exercise: Find a 99% CI for the population proportion problem (Liberal supporters) discussed above. You will need to refer to the R documentation for prop.test to do this.

 

 

d.2 Hypothesis testing

 

(What is the meaning of the word "hypo" in "hypo-thesis"?

Hypo-allergenic as in Hypo-allergenic?

 

Hypo-thermia as in hypothermia?

 

"Hypo-potamus" (??) as in Hippopotamus?                                             Tricked you! This is a hippo-potamus. :-)

"Hypo" means "below, under" in Greek.

"Thesis", is something that is proven to be true.

So, "hypo-thesis" is something that is yet to be proven to be true.

Now, what does a lady tasting tea have to do with hypothesis testing?

Lady

"The Lady Tasting Tea" : Can tea poured into milk taste differently than that of milk poured into tea? This experiment was originally designed by Professor Ronald Fisher in the 1920s, and it will help us motivate the discussion of hypothesis testing. We will, however, use Coke and Pepsi in our experiment. The "lady" in the story is Dr. Muriel Bristol of Cambridge University.

She claims that she knows the difference. Here, my null hypothesis is "H0: She is guessing". Now, if she is purely guessing, there is a 0.014 probability of getting all 8 cups correct. This is such an unlikely outcome but if it happens, I am willing to change my mind and reject my null H0 and believe that she can tell the difference. But what if she just guessed and got all correct? Then I made a mistake in changing my mind, but the probability of me making this mistake is only 0.014. This is the p-value.

In case you were wondering, here is the mathematics behind the calculations. In this link you can find the probaibilities of 0, 2, 4, 6 or 8 correct identifications which uses the hypergeometric probabilities (which we did not discuss).

Type I and Type II errors

Steven

Type I error : In 1959, Steven Truscott was found guilty of murdering his classmate even though he did not commit any crime. In 2007, he was formally acquitted of the crime. In 2008, the government of Ontario awarded him $6.50 million in compensation

OJ

Type II error : Many people believe that O. J. Simpson had murdered his wife and he should have been found guilty. But after a lenghty trial, he was acquitted in 1995.

Examples

 

Hypothesis testing in R (with one or two populations) still uses the t.test function described above. We now discuss a problem with one population.

 

 

Example: The data set [Atkins-Diet.csv] concerns the weight losses experienced by dieters using the Atkins diet. We want to test Atkins's hypothesis that people who use their method lose, on average, at least 20 pounds in 6 months. The p-value is about 0.03 so we reject this hypothesis. However, if the claim is at least 10 pounds in 6 months, we find p-value as 0.98, so we don't have enough evidence to reject this claim. Here are the results from Rcmdr. ¶

 

 

Exercise: For the Atkins problem test the null hypothesis that Atkins users lose, on average, 17 pounds after 6 months. (This is now a two-sided test.)

 

Exercise: Use the following data values to test the hypothesis that true mean is 750 vs. the hypothesis that it differs from 750: (801,814,784,836,820) What is the p-value? (Answer: p = .0023; so reject the null)