advance statistical learning

NANI1012

advancestatisticallearningbook.pdf

Home >Mathematics homework help >Statistics homework help >advance statistical learning

Gareth James • Daniela Witten • Trevor Hastie • Robert Tibshirani

An Introduction to Statistical Learning with Applications in R

Second Edition

123

Daniela Witten

First Printing: August 4, 2021

To our parents:

Alison and Michael James

Chiara Nappi and Edward Witten

Valerie and Patrick Hastie

Vera and Sami Tibshirani

and to our families:

Michael, Daniel, and Catherine

Tessa, Theo, Otto, and Ari

Samantha, Timothy, and Lynda

Charlie, Ryan, Julie, and Cheryl

Preface

Statistical learning refers to a set of tools for making sense of complex datasets. In recent years, we have seen a staggering increase in the scale and scope of data collection across virtually all areas of science and industry. As a result, statistical learning has become a critical toolkit for anyone who wishes to understand data — and as more and more of today’s jobs involve data, this means that statistical learning is fast becoming a critical toolkit for everyone. One of the first books on statistical learning — The Elements of Statisti-

cal Learning (ESL, by Hastie, Tibshirani, and Friedman) — was published in 2001, with a second edition in 2009. ESL has become a popular text not only in statistics but also in related fields. One of the reasons for ESL’s popularity is its relatively accessible style. But ESL is best-suited for indi- viduals with advanced training in the mathematical sciences. An Introduction to Statistical Learning (ISL) arose from the clear need

for a broader and less technical treatment of the key topics in statistical learning. The intention behind ISL is to concentrate more on the applica- tions of the methods and less on the mathematical details. Beginning with Chapter 2, each chapter in ISL contains a lab illustrating how to implement the statistical learning methods seen in that chapter using the popular sta- tistical software package R. These labs provide the reader with valuable hands-on experience. ISL is appropriate for advanced undergraduates or master’s students in

Statistics or related quantitative fields, or for individuals in other disciplines who wish to use statistical learning tools to analyze their data. It can be used as a textbook for a course spanning two semesters.

vii

The first edition of ISL covered a number of important topics, including sparse methods for classification and regression, decision trees, boosting, support vector machines, and clustering. Since it was published in 2013, it has become a mainstay of undergraduate and graduate classrooms across the United States and worldwide, as well as a key reference book for data scientists. In this second edition of ISL, we have greatly expanded the set of topics

covered. In particular, the second edition includes new chapters on deep learning (Chapter 10), survival analysis (Chapter 11), and multiple testing (Chapter 13). We have also substantially expanded some chapters that were part of the first edition: among other updates, we now include treatments of naive Bayes and generalized linear models in Chapter 4, Bayesian addi- tive regression trees in Chapter 8, and matrix completion in Chapter 12. Furthermore, we have updated the R code throughout the labs to ensure that the results that they produce agree with recent R releases. We are grateful to these readers for providing valuable comments on the

first edition of this book: Pallavi Basu, Alexandra Chouldechova, Patrick Danaher, Will Fithian, Luella Fu, Sam Gross, Max Grazier G’Sell, Court- ney Paulson, Xinghao Qiao, Elisa Sheng, Noah Simon, Kean Ming Tan, Xin Lu Tan. We thank these readers for helpful input on the second edi- tion of this book: Alan Agresti, Iain Carmichael, Yiqun Chen, Erin Craig, Daisy Ding, Lucy Gao, Ismael Lemhadri, Bryan Martin, Anna Neufeld, Ge- off Tims, Carsten Voelkmann, Steve Yadlowsky, and James Zou. We also thank Anna Neufeld for her assistance in reformatting the R code through- out this book. We are immensely grateful to Balasubramanian “Naras” Narasimhan for his assistance on both editions of this textbook. It has been an honor and a privilege for us to see the considerable impact

that the first edition of ISL has had on the way in which statistical learning is practiced, both in and out of the academic setting. We hope that this new edition will continue to give today’s and tomorrow’s applied statisticians and data scientists the tools they need for success in a data-driven world.

It’s tough to make predictions, especially about the future.

-Yogi Berra

viii Preface

Contents

Preface vii

1 Introduction 1

2 Statistical Learning 15 2.1 What Is Statistical Learning? . . . . . . . . . . . . . . . . . 15

2.1.1 Why Estimate f? . . . . . . . . . . . . . . . . . . . 17 2.1.2 How Do We Estimate f? . . . . . . . . . . . . . . . 21 2.1.3 The Trade-Off Between Prediction Accuracy

and Model Interpretability . . . . . . . . . . . . . . 24 2.1.4 Supervised Versus Unsupervised Learning . . . . . 26 2.1.5 Regression Versus Classification Problems . . . . . 28

2.2 Assessing Model Accuracy . . . . . . . . . . . . . . . . . . 29 2.2.1 Measuring the Quality of Fit . . . . . . . . . . . . 29 2.2.2 The Bias-Variance Trade-Off . . . . . . . . . . . . . 33 2.2.3 The Classification Setting . . . . . . . . . . . . . . 37

2.3 Lab: Introduction to R . . . . . . . . . . . . . . . . . . . . 42 2.3.1 Basic Commands . . . . . . . . . . . . . . . . . . . 43 2.3.2 Graphics . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3.3 Indexing Data . . . . . . . . . . . . . . . . . . . . . 47 2.3.4 Loading Data . . . . . . . . . . . . . . . . . . . . . 48 2.3.5 Additional Graphical and Numerical Summaries . . 50

2.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3 Linear Regression 59 3.1 Simple Linear Regression . . . . . . . . . . . . . . . . . . . 60

3.1.1 Estimating the Coefficients . . . . . . . . . . . . . 61 3.1.2 Assessing the Accuracy of the Coefficient

Estimates . . . . . . . . . . . . . . . . . . . . . . . 63 3.1.3 Assessing the Accuracy of the Model . . . . . . . . 68

3.2 Multiple Linear Regression . . . . . . . . . . . . . . . . . . 71 3.2.1 Estimating the Regression Coefficients . . . . . . . 72 3.2.2 Some Important Questions . . . . . . . . . . . . . . 75

3.3 Other Considerations in the Regression Model . . . . . . . 83

3.3.1 Qualitative Predictors . . . . . . . . . . . . . . . . 83 3.3.2 Extensions of the Linear Model . . . . . . . . . . . 87 3.3.3 Potential Problems . . . . . . . . . . . . . . . . . . 92

3.4 The Marketing Plan . . . . . . . . . . . . . . . . . . . . . . 103 3.5 Comparison of Linear Regression with K-Nearest

Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.6 Lab: Linear Regression . . . . . . . . . . . . . . . . . . . . 110

3.6.1 Libraries . . . . . . . . . . . . . . . . . . . . . . . . 110 3.6.2 Simple Linear Regression . . . . . . . . . . . . . . . 111 3.6.3 Multiple Linear Regression . . . . . . . . . . . . . . 114 3.6.4 Interaction Terms . . . . . . . . . . . . . . . . . . . 116 3.6.5 Non-linear Transformations of the Predictors . . . 116 3.6.6 Qualitative Predictors . . . . . . . . . . . . . . . . 119 3.6.7 Writing Functions . . . . . . . . . . . . . . . . . . . 120

3.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

4 Classification 129 4.1 An Overview of Classification . . . . . . . . . . . . . . . . . 130 4.2 Why Not Linear Regression? . . . . . . . . . . . . . . . . . 131 4.3 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . 133

4.3.1 The Logistic Model . . . . . . . . . . . . . . . . . . 133 4.3.2 Estimating the Regression Coefficients . . . . . . . 135 4.3.3 Making Predictions . . . . . . . . . . . . . . . . . . 136 4.3.4 Multiple Logistic Regression . . . . . . . . . . . . . 137 4.3.5 Multinomial Logistic Regression . . . . . . . . . . . 140

4.4 Generative Models for Classification . . . . . . . . . . . . . 141 4.4.1 Linear Discriminant Analysis for p = 1 . . . . . . . 142 4.4.2 Linear Discriminant Analysis for p >1 . . . . . . . 145 4.4.3 Quadratic Discriminant Analysis . . . . . . . . . . 152 4.4.4 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . 153

4.5 A Comparison of Classification Methods . . . . . . . . . . 158 4.5.1 An Analytical Comparison . . . . . . . . . . . . . . 158 4.5.2 An Empirical Comparison . . . . . . . . . . . . . . 161

4.6 Generalized Linear Models . . . . . . . . . . . . . . . . . . 164 4.6.1 Linear Regression on the Bikeshare Data . . . . . . 164 4.6.2 Poisson Regression on the Bikeshare Data . . . . . 167 4.6.3 Generalized Linear Models in Greater Generality . 170

4.7 Lab: Classification Methods . . . . . . . . . . . . . . . . . . 171 4.7.1 The Stock Market Data . . . . . . . . . . . . . . . 171 4.7.2 Logistic Regression . . . . . . . . . . . . . . . . . . 172 4.7.3 Linear Discriminant Analysis . . . . . . . . . . . . 177 4.7.4 Quadratic Discriminant Analysis . . . . . . . . . . 179 4.7.5 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . 180 4.7.6 K-Nearest Neighbors . . . . . . . . . . . . . . . . . 181 4.7.7 Poisson Regression . . . . . . . . . . . . . . . . . . 185

x CONTENTS

CONTENTS xi

4.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

5 Resampling Methods 197 5.1 Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . 198

5.1.1 The Validation Set Approach . . . . . . . . . . . . 198 5.1.2 Leave-One-Out Cross-Validation . . . . . . . . . . 200 5.1.3 k-Fold Cross-Validation . . . . . . . . . . . . . . . 203 5.1.4 Bias-Variance Trade-Off for k-Fold

Cross-Validation . . . . . . . . . . . . . . . . . . . 205 5.1.5 Cross-Validation on Classification Problems . . . . 206

5.2 The Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . 209 5.3 Lab: Cross-Validation and the Bootstrap . . . . . . . . . . 212

5.3.1 The Validation Set Approach . . . . . . . . . . . . 213 5.3.2 Leave-One-Out Cross-Validation . . . . . . . . . . 214 5.3.3 k-Fold Cross-Validation . . . . . . . . . . . . . . . 215 5.3.4 The Bootstrap . . . . . . . . . . . . . . . . . . . . 216

5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

6 Linear Model Selection and Regularization 225 6.1 Subset Selection . . . . . . . . . . . . . . . . . . . . . . . . 227

6.1.1 Best Subset Selection . . . . . . . . . . . . . . . . . 227 6.1.2 Stepwise Selection . . . . . . . . . . . . . . . . . . 229 6.1.3 Choosing the Optimal Model . . . . . . . . . . . . 232

6.2 Shrinkage Methods . . . . . . . . . . . . . . . . . . . . . . 237 6.2.1 Ridge Regression . . . . . . . . . . . . . . . . . . . 237 6.2.2 The Lasso . . . . . . . . . . . . . . . . . . . . . . . 241 6.2.3 Selecting the Tuning Parameter . . . . . . . . . . . 250

6.3 Dimension Reduction Methods . . . . . . . . . . . . . . . . 251 6.3.1 Principal Components Regression . . . . . . . . . . 252 6.3.2 Partial Least Squares . . . . . . . . . . . . . . . . . 259

6.4 Considerations in High Dimensions . . . . . . . . . . . . . 261 6.4.1 High-Dimensional Data . . . . . . . . . . . . . . . . 261 6.4.2 What Goes Wrong in High Dimensions? . . . . . . 262 6.4.3 Regression in High Dimensions . . . . . . . . . . . 264 6.4.4 Interpreting Results in High Dimensions . . . . . . 266

6.5 Lab: Linear Models and Regularization Methods . . . . . . 267 6.5.1 Subset Selection Methods . . . . . . . . . . . . . . 267 6.5.2 Ridge Regression and the Lasso . . . . . . . . . . . 274 6.5.3 PCR and PLS Regression . . . . . . . . . . . . . . 279

6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

7 Moving Beyond Linearity 289 7.1 Polynomial Regression . . . . . . . . . . . . . . . . . . . . . 290 7.2 Step Functions . . . . . . . . . . . . . . . . . . . . . . . . . 292 7.3 Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . 294

7.4 Regression Splines . . . . . . . . . . . . . . . . . . . . . . . 295 7.4.1 Piecewise Polynomials . . . . . . . . . . . . . . . . 295 7.4.2 Constraints and Splines . . . . . . . . . . . . . . . 295 7.4.3 The Spline Basis Representation . . . . . . . . . . 297 7.4.4 Choosing the Number and Locations

of the Knots . . . . . . . . . . . . . . . . . . . . . . 298 7.4.5 Comparison to Polynomial Regression . . . . . . . 300

7.5 Smoothing Splines . . . . . . . . . . . . . . . . . . . . . . . 301 7.5.1 An Overview of Smoothing Splines . . . . . . . . . 301 7.5.2 Choosing the Smoothing Parameter λ . . . . . . . 302

7.6 Local Regression . . . . . . . . . . . . . . . . . . . . . . . . 304 7.7 Generalized Additive Models . . . . . . . . . . . . . . . . . 306

7.7.1 GAMs for Regression Problems . . . . . . . . . . . 307 7.7.2 GAMs for Classification Problems . . . . . . . . . . 310

7.8 Lab: Non-linear Modeling . . . . . . . . . . . . . . . . . . . 311 7.8.1 Polynomial Regression and Step Functions . . . . . 312 7.8.2 Splines . . . . . . . . . . . . . . . . . . . . . . . . . 317 7.8.3 GAMs . . . . . . . . . . . . . . . . . . . . . . . . . 318

7.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321

8 Tree-Based Methods 327 8.1 The Basics of Decision Trees . . . . . . . . . . . . . . . . . 327

8.1.1 Regression Trees . . . . . . . . . . . . . . . . . . . 328 8.1.2 Classification Trees . . . . . . . . . . . . . . . . . . 335 8.1.3 Trees Versus Linear Models . . . . . . . . . . . . . 338 8.1.4 Advantages and Disadvantages of Trees . . . . . . 339

8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees . . . . . . . . . . . . . . . . . . . . . . . . 340 8.2.1 Bagging . . . . . . . . . . . . . . . . . . . . . . . . 340 8.2.2 Random Forests . . . . . . . . . . . . . . . . . . . . 343 8.2.3 Boosting . . . . . . . . . . . . . . . . . . . . . . . . 345 8.2.4 Bayesian Additive Regression Trees . . . . . . . . . 348 8.2.5 Summary of Tree Ensemble Methods . . . . . . . . 351

8.3 Lab: Decision Trees . . . . . . . . . . . . . . . . . . . . . . 353 8.3.1 Fitting Classification Trees . . . . . . . . . . . . . . 353 8.3.2 Fitting Regression Trees . . . . . . . . . . . . . . . 356 8.3.3 Bagging and Random Forests . . . . . . . . . . . . 357 8.3.4 Boosting . . . . . . . . . . . . . . . . . . . . . . . . 359 8.3.5 Bayesian Additive Regression Trees . . . . . . . . . 360

8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361

9 Support Vector Machines 367 9.1 Maximal Margin Classifier . . . . . . . . . . . . . . . . . . 368

9.1.1 What Is a Hyperplane? . . . . . . . . . . . . . . . . 368 9.1.2 Classification Using a Separating Hyperplane . . . 369

xii CONTENTS

CONTENTS xiii

9.1.3 The Maximal Margin Classifier . . . . . . . . . . . 371 9.1.4 Construction of the Maximal Margin Classifier . . 372 9.1.5 The Non-separable Case . . . . . . . . . . . . . . . 373

9.2 Support Vector Classifiers . . . . . . . . . . . . . . . . . . . 373 9.2.1 Overview of the Support Vector Classifier . . . . . 373 9.2.2 Details of the Support Vector Classifier . . . . . . . 375

9.3 Support Vector Machines . . . . . . . . . . . . . . . . . . . 379 9.3.1 Classification with Non-Linear Decision

Boundaries . . . . . . . . . . . . . . . . . . . . . . 379 9.3.2 The Support Vector Machine . . . . . . . . . . . . 380 9.3.3 An Application to the Heart Disease Data . . . . . 383

9.4 SVMs with More than Two Classes . . . . . . . . . . . . . 385 9.4.1 One-Versus-One Classification . . . . . . . . . . . . 385 9.4.2 One-Versus-All Classification . . . . . . . . . . . . 385

9.5 Relationship to Logistic Regression . . . . . . . . . . . . . 386 9.6 Lab: Support Vector Machines . . . . . . . . . . . . . . . . 388

9.6.1 Support Vector Classifier . . . . . . . . . . . . . . . 389 9.6.2 Support Vector Machine . . . . . . . . . . . . . . . 392 9.6.3 ROC Curves . . . . . . . . . . . . . . . . . . . . . . 394 9.6.4 SVM with Multiple Classes . . . . . . . . . . . . . 396 9.6.5 Application to Gene Expression Data . . . . . . . . 396

9.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398

10 Deep Learning 403 10.1 Single Layer Neural Networks . . . . . . . . . . . . . . . . 404 10.2 Multilayer Neural Networks . . . . . . . . . . . . . . . . . . 407 10.3 Convolutional Neural Networks . . . . . . . . . . . . . . . . 411

10.3.1 Convolution Layers . . . . . . . . . . . . . . . . . . 412 10.3.2 Pooling Layers . . . . . . . . . . . . . . . . . . . . 415 10.3.3 Architecture of a Convolutional Neural Network . . 415 10.3.4 Data Augmentation . . . . . . . . . . . . . . . . . . 417 10.3.5 Results Using a Pretrained Classifier . . . . . . . . 417

10.4 Document Classification . . . . . . . . . . . . . . . . . . . . 419 10.5 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . 421

10.5.1 Sequential Models for Document Classification . . 424 10.5.2 Time Series Forecasting . . . . . . . . . . . . . . . 427 10.5.3 Summary of RNNs . . . . . . . . . . . . . . . . . . 431

10.6 When to Use Deep Learning . . . . . . . . . . . . . . . . . 432 10.7 Fitting a Neural Network . . . . . . . . . . . . . . . . . . . 434

10.7.1 Backpropagation . . . . . . . . . . . . . . . . . . . 435 10.7.2 Regularization and Stochastic Gradient Descent . . 436 10.7.3 Dropout Learning . . . . . . . . . . . . . . . . . . . 438 10.7.4 Network Tuning . . . . . . . . . . . . . . . . . . . . 438

10.8 Interpolation and Double Descent . . . . . . . . . . . . . . 439 10.9 Lab: Deep Learning . . . . . . . . . . . . . . . . . . . . . . 443

10.9.1 A Single Layer Network on the Hitters Data . . . . 443 10.9.2 A Multilayer Network on the MNIST Digit Data . 445 10.9.3 Convolutional Neural Networks . . . . . . . . . . . 448 10.9.4 Using Pretrained CNN Models . . . . . . . . . . . 451 10.9.5 IMDb Document Classification . . . . . . . . . . . 452 10.9.6 Recurrent Neural Networks . . . . . . . . . . . . . 454

10.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458

11 Survival Analysis and Censored Data 461 11.1 Survival and Censoring Times . . . . . . . . . . . . . . . . 462 11.2 A Closer Look at Censoring . . . . . . . . . . . . . . . . . . 463 11.3 The Kaplan-Meier Survival Curve . . . . . . . . . . . . . . 464 11.4 The Log-Rank Test . . . . . . . . . . . . . . . . . . . . . . 466 11.5 Regression Models With a Survival Response . . . . . . . . 469

11.5.1 The Hazard Function . . . . . . . . . . . . . . . . . 469 11.5.2 Proportional Hazards . . . . . . . . . . . . . . . . . 471 11.5.3 Example: Brain Cancer Data . . . . . . . . . . . . 475 11.5.4 Example: Publication Data . . . . . . . . . . . . . 475

11.6 Shrinkage for the Cox Model . . . . . . . . . . . . . . . . . 478 11.7 Additional Topics . . . . . . . . . . . . . . . . . . . . . . . 480

11.7.1 Area Under the Curve for Survival Analysis . . . . 480 11.7.2 Choice of Time Scale . . . . . . . . . . . . . . . . . 481 11.7.3 Time-Dependent Covariates . . . . . . . . . . . . . 481 11.7.4 Checking the Proportional Hazards Assumption . . 482 11.7.5 Survival Trees . . . . . . . . . . . . . . . . . . . . . 482

11.8 Lab: Survival Analysis . . . . . . . . . . . . . . . . . . . . . 483 11.8.1 Brain Cancer Data . . . . . . . . . . . . . . . . . . 483 11.8.2 Publication Data . . . . . . . . . . . . . . . . . . . 486 11.8.3 Call Center Data . . . . . . . . . . . . . . . . . . . 487

11.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490

12 Unsupervised Learning 497 12.1 The Challenge of Unsupervised Learning . . . . . . . . . . 497 12.2 Principal Components Analysis . . . . . . . . . . . . . . . . 498

12.2.1 What Are Principal Components? . . . . . . . . . . 499 12.2.2 Another Interpretation of Principal Components . 503 12.2.3 The Proportion of Variance Explained . . . . . . . 505 12.2.4 More on PCA . . . . . . . . . . . . . . . . . . . . . 507 12.2.5 Other Uses for Principal Components . . . . . . . . 510

12.3 Missing Values and Matrix Completion . . . . . . . . . . . 510 12.4 Clustering Methods . . . . . . . . . . . . . . . . . . . . . . 516

12.4.1 K-Means Clustering . . . . . . . . . . . . . . . . . 517 12.4.2 Hierarchical Clustering . . . . . . . . . . . . . . . . 521 12.4.3 Practical Issues in Clustering . . . . . . . . . . . . 530

12.5 Lab: Unsupervised Learning . . . . . . . . . . . . . . . . . 532

xiv CONTENTS

CONTENTS xv

12.5.1 Principal Components Analysis . . . . . . . . . . . 532 12.5.2 Matrix Completion . . . . . . . . . . . . . . . . . . 535 12.5.3 Clustering . . . . . . . . . . . . . . . . . . . . . . . 538 12.5.4 NCI60 Data Example . . . . . . . . . . . . . . . . . 542

12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548

13 Multiple Testing 553 13.1 A Quick Review of Hypothesis Testing . . . . . . . . . . . 554

13.1.1 Testing a Hypothesis . . . . . . . . . . . . . . . . . 555 13.1.2 Type I and Type II Errors . . . . . . . . . . . . . . 559

13.2 The Challenge of Multiple Testing . . . . . . . . . . . . . . 560 13.3 The Family-Wise Error Rate . . . . . . . . . . . . . . . . . 561

13.3.1 What is the Family-Wise Error Rate? . . . . . . . 562 13.3.2 Approaches to Control the Family-Wise Error Rate 564 13.3.3 Trade-Off Between the FWER and Power . . . . . 570

13.4 The False Discovery Rate . . . . . . . . . . . . . . . . . . . 571 13.4.1 Intuition for the False Discovery Rate . . . . . . . 571 13.4.2 The Benjamini-Hochberg Procedure . . . . . . . . 573

13.5 A Re-Sampling Approach to p-Values and False Discovery Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 13.5.1 A Re-Sampling Approach to the p-Value . . . . . . 576 13.5.2 A Re-Sampling Approach to the False Discovery Rate578 13.5.3 When Are Re-Sampling Approaches Useful? . . . . 581

13.6 Lab: Multiple Testing . . . . . . . . . . . . . . . . . . . . . 582 13.6.1 Review of Hypothesis Tests . . . . . . . . . . . . . 582 13.6.2 The Family-Wise Error Rate . . . . . . . . . . . . . 583 13.6.3 The False Discovery Rate . . . . . . . . . . . . . . 586 13.6.4 A Re-Sampling Approach . . . . . . . . . . . . . . 588

13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591

Index 597

1 Introduction

An Overview of Statistical Learning

Statistical learning refers to a vast set of tools for understanding data. These tools can be classified as supervised or unsupervised. Broadly speaking, supervised statistical learning involves building a statistical model for pre- dicting, or estimating, an output based on one or more inputs. Problems of this nature occur in fields as diverse as business, medicine, astrophysics, and public policy. With unsupervised statistical learning, there are inputs but no supervising output; nevertheless we can learn relationships and struc- ture from such data. To provide an illustration of some applications of statistical learning, we briefly discuss three real-world data sets that are considered in this book.

Wage Data

In this application (which we refer to as the Wage data set throughout this book), we examine a number of factors that relate to wages for a group of men from the Atlantic region of the United States. In particular, we wish to understand the association between an employee’s age and education, as well as the calendar year, on his wage. Consider, for example, the left-hand panel of Figure 1.1, which displays wage versus age for each of the individu- als in the data set. There is evidence that wage increases with age but then decreases again after approximately age 60. The blue line, which provides an estimate of the average wage for a given age, makes this trend clearer.

http://crossmark.crossref.org/dialog/?doi=10.1007/978-1-0716-1418-1_1&domain=pdf

2 1. Introduction

20 40 60 80

5 0

1 0

0 2

0 0

3 0

Age

W a

g e

2003 2006 2009 5

0 1

0 0

2 0

0 3

0 0

Year

W a

g e

1 2 3 4 5

5 0

1 0

0 2

0 0

3 0

Education Level

W a

g e

FIGURE 1.1. Wage data, which contains income survey information for men from the central Atlantic region of the United States. Left: wage as a function of age. On average, wage increases with age until about 60 years of age, at which point it begins to decline. Center: wage as a function of year. There is a slow but steady increase of approximately $10,000 in the average wage between 2003 and 2009. Right: Boxplots displaying wage as a function of education, with 1 indicating the lowest level (no high school diploma) and 5 the highest level (an advanced graduate degree). On average, wage increases with the level of education.

Given an employee’s age, we can use this curve to predict his wage. However, it is also clear from Figure 1.1 that there is a significant amount of vari- ability associated with this average value, and so age alone is unlikely to provide an accurate prediction of a particular man’s wage. We also have information regarding each employee’s education level and

the year in which the wage was earned. The center and right-hand panels of Figure 1.1, which display wage as a function of both year and education, in- dicate that both of these factors are associated with wage. Wages increase by approximately $10,000, in a roughly linear (or straight-line) fashion, between 2003 and 2009, though this rise is very slight relative to the vari- ability in the data. Wages are also typically greater for individuals with higher education levels: men with the lowest education level (1) tend to have substantially lower wages than those with the highest education level (5). Clearly, the most accurate prediction of a given man’s wage will be obtained by combining his age, his education, and the year. In Chapter 3, we discuss linear regression, which can be used to predict wage from this data set. Ideally, we should predict wage in a way that accounts for the non-linear relationship between wage and age. In Chapter 7, we discuss a class of approaches for addressing this problem.

1. Introduction 3

Down Up

− 4

− 2

0 2

4 6

Yesterday

Today’s Direction

P e

rc e

n ta

g e

c h

a n

g e

i n

S &

Down Up

− 4

− 2

0 2

4 6

Two Days Previous

Today’s Direction

P e

rc e

n ta

g e

c h

a n

g e

i n

S &

Down Up

− 4

− 2

0 2

4 6

Three Days Previous

Today’s Direction

P e

rc e

n ta

g e

c h

a n

g e

i n

S &

FIGURE 1.2. Left: Boxplots of the previous day’s percentage change in the S&P index for the days for which the market increased or decreased, obtained from the Smarket data. Center and Right: Same as left panel, but the percentage changes for 2 and 3 days previous are shown.

Stock Market Data

The Wage data involves predicting a continuous or quantitative output value. This is often referred to as a regression problem. However, in certain cases we may instead wish to predict a non-numerical value—that is, a categorical or qualitative output. For example, in Chapter 4 we examine a stock market data set that contains the daily movements in the Standard & Poor’s 500 (S&P) stock index over a 5-year period between 2001 and 2005. We refer to this as the Smarket data. The goal is to predict whether the index will increase or decrease on a given day, using the past 5 days’ percentage changes in the index. Here the statistical learning problem does not involve predicting a numerical value. Instead it involves predicting whether a given day’s stock market performance will fall into the Up bucket or the Down bucket. This is known as a classification problem. A model that could accurately predict the direction in which the market will move would be very useful! The left-hand panel of Figure 1.2 displays two boxplots of the previous

day’s percentage changes in the stock index: one for the 648 days for which the market increased on the subsequent day, and one for the 602 days for which the market decreased. The two plots look almost identical, suggest- ing that there is no simple strategy for using yesterday’s movement in the S&P to predict today’s returns. The remaining panels, which display box- plots for the percentage changes 2 and 3 days previous to today, similarly indicate little association between past and present returns. Of course, this lack of pattern is to be expected: in the presence of strong correlations be- tween successive days’ returns, one could adopt a simple trading strategy

4 1. Introduction

Down Up

0 .4

6 0 .4

8 0 .5

0 0 .5

Today’s Direction

P re

d ic

te d P

ro b a b ili

FIGURE 1.3. We fit a quadratic discriminant analysis model to the subset of the Smarket data corresponding to the 2001–2004 time period, and predicted the probability of a stock market decrease using the 2005 data. On average, the predicted probability of decrease is higher for the days in which the market does decrease. Based on these results, we are able to correctly predict the direction of movement in the market 60% of the time.

to generate profits from the market. Nevertheless, in Chapter 4, we explore these data using several different statistical learning methods. Interestingly, there are hints of some weak trends in the data that suggest that, at least for this 5-year period, it is possible to correctly predict the direction of movement in the market approximately 60% of the time (Figure 1.3).

Gene Expression Data

The previous two applications illustrate data sets with both input and output variables. However, another important class of problems involves situations in which we only observe input variables, with no corresponding output. For example, in a marketing setting, we might have demographic information for a number of current or potential customers. We may wish to understand which types of customers are similar to each other by grouping individuals according to their observed characteristics. This is known as a clustering problem. Unlike in the previous examples, here we are not trying to predict an output variable. We devote Chapter 12 to a discussion of statistical learning methods

for problems in which no natural output variable is available. We consider the NCI60 data set, which consists of 6,830 gene expression measurements for each of 64 cancer cell lines. Instead of predicting a particular output variable, we are interested in determining whether there are groups, or clusters, among the cell lines based on their gene expression measurements. This is a difficult question to address, in part because there are thousands of gene expression measurements per cell line, making it hard to visualize the data.

1. Introduction 5

−40 −20 0 20 40 60

− 6

0 −

4 0

− 2

0 0

2 0

−40 −20 0 20 40 60

− 6

0 −

4 0

− 2

0 0

2 0

Z1Z1 Z 2

Z 2

FIGURE 1.4. Left: Representation of the NCI60 gene expression data set in a two-dimensional space, Z1 and Z2. Each point corresponds to one of the 64 cell lines. There appear to be four groups of cell lines, which we have represented using different colors. Right: Same as left panel except that we have represented each of the 14 different types of cancer using a different colored symbol. Cell lines corresponding to the same cancer type tend to be nearby in the two-dimensional space.

The left-hand panel of Figure 1.4 addresses this problem by represent- ing each of the 64 cell lines using just two numbers, Z1 and Z2. These are the first two principal components of the data, which summarize the 6,830 expression measurements for each cell line down to two numbers or dimensions. While it is likely that this dimension reduction has resulted in some loss of information, it is now possible to visually examine the data for evidence of clustering. Deciding on the number of clusters is often a difficult problem. But the left-hand panel of Figure 1.4 suggests at least four groups of cell lines, which we have represented using separate colors. In this particular data set, it turns out that the cell lines correspond

to 14 different types of cancer. (However, this information was not used to create the left-hand panel of Figure 1.4.) The right-hand panel of Fig- ure 1.4 is identical to the left-hand panel, except that the 14 cancer types are shown using distinct colored symbols. There is clear evidence that cell lines with the same cancer type tend to be located near each other in this two-dimensional representation. In addition, even though the cancer infor- mation was not used to produce the left-hand panel, the clustering obtained does bear some resemblance to some of the actual cancer types observed in the right-hand panel. This provides some independent verification of the accuracy of our clustering analysis.

6 1. Introduction

A Brief History of Statistical Learning

Though the term statistical learning is fairly new, many of the concepts that underlie the field were developed long ago. At the beginning of the nine- teenth century, the method of least squares was developed, implementing the earliest form of what is now known as linear regression. The approach was first successfully applied to problems in astronomy. Linear regression is used for predicting quantitative values, such as an individual’s salary. In order to predict qualitative values, such as whether a patient survives or dies, or whether the stock market increases or decreases, linear discrim- inant analysis was proposed in 1936. In the 1940s, various authors put forth an alternative approach, logistic regression. In the early 1970s, the term generalized linear model was developed to describe an entire class of statistical learning methods that include both linear and logistic regression as special cases. By the end of the 1970s, many more techniques for learning from data

were available. However, they were almost exclusively linear methods be- cause fitting non-linear relationships was computationally difficult at the time. By the 1980s, computing technology had finally improved sufficiently that non-linear methods were no longer computationally prohibitive. In the mid 1980s, classification and regression trees were developed, followed shortly by generalized additive models. Neural networks gained popularity in the 1980s, and support vector machines arose in the 1990s. Since that time, statistical learning has emerged as a new subfield in

statistics, focused on supervised and unsupervised modeling and prediction. In recent years, progress in statistical learning has been marked by the increasing availability of powerful and relatively user-friendly software, such as the popular and freely available R system. This has the potential to continue the transformation of the field from a set of techniques used and developed by statisticians and computer scientists to an essential toolkit for a much broader community.

This Book

The Elements of Statistical Learning (ESL) by Hastie, Tibshirani, and Friedman was first published in 2001. Since that time, it has become an important reference on the fundamentals of statistical machine learning. Its success derives from its comprehensive and detailed treatment of many important topics in statistical learning, as well as the fact that (relative to many upper-level statistics textbooks) it is accessible to a wide audience. However, the greatest factor behind the success of ESL has been its topical nature. At the time of its publication, interest in the field of statistical

1. Introduction 7

learning was starting to explode. ESL provided one of the first accessible and comprehensive introductions to the topic. Since ESL was first published, the field of statistical learning has con-

tinued to flourish. The field’s expansion has taken two forms. The most obvious growth has involved the development of new and improved statis- tical learning approaches aimed at answering a range of scientific questions across a number of fields. However, the field of statistical learning has also expanded its audience. In the 1990s, increases in computational power generated a surge of interest in the field from non-statisticians who were eager to use cutting-edge statistical tools to analyze their data. Unfortu- nately, the highly technical nature of these approaches meant that the user community remained primarily restricted to experts in statistics, computer science, and related fields with the training (and time) to understand and implement them. In recent years, new and improved software packages have significantly

eased the implementation burden for many statistical learning methods. At the same time, there has been growing recognition across a number of fields, from business to health care to genetics to the social sciences and beyond, that statistical learning is a powerful tool with important practical applications. As a result, the field has moved from one of primarily academic interest to a mainstream discipline, with an enormous potential audience. This trend will surely continue with the increasing availability of enormous quantities of data and the software to analyze it. The purpose of An Introduction to Statistical Learning (ISL) is to facili-

tate the transition of statistical learning from an academic to a mainstream field. ISL is not intended to replace ESL, which is a far more comprehen- sive text both in terms of the number of approaches considered and the depth to which they are explored. We consider ESL to be an important companion for professionals (with graduate degrees in statistics, machine learning, or related fields) who need to understand the technical details behind statistical learning approaches. However, the community of users of statistical learning techniques has expanded to include individuals with a wider range of interests and backgrounds. Therefore, there is a place for a less technical and more accessible version of ESL. In teaching these topics over the years, we have discovered that they are

of interest to master’s and PhD students in fields as disparate as business administration, biology, and computer science, as well as to quantitatively- oriented upper-division undergraduates. It is important for this diverse group to be able to understand the models, intuitions, and strengths and weaknesses of the various approaches. But for this audience, many of the technical details behind statistical learning methods, such as optimiza- tion algorithms and theoretical properties, are not of primary interest. We believe that these students do not need a deep understanding of these aspects in order to become informed users of the various methodologies, and

8 1. Introduction

in order to contribute to their chosen fields through the use of statistical learning tools. ISL is based on the following four premises.

1. Many statistical learning methods are relevant and useful in a wide range of academic and non-academic disciplines, beyond just the sta- tistical sciences. We believe that many contemporary statistical learn- ing procedures should, and will, become as widely available and used as is currently the case for classical methods such as linear regres- sion. As a result, rather than attempting to consider every possible approach (an impossible task), we have concentrated on presenting the methods that we believe are most widely applicable.

2. Statistical learning should not be viewed as a series of black boxes. No single approach will perform well in all possible applications. With- out understanding all of the cogs inside the box, or the interaction between those cogs, it is impossible to select the best box. Hence, we have attempted to carefully describe the model, intuition, assump- tions, and trade-offs behind each of the methods that we consider.

3. While it is important to know what job is performed by each cog, it is not necessary to have the skills to construct the machine inside the box! Thus, we have minimized discussion of technical details related to fitting procedures and theoretical properties. We assume that the reader is comfortable with basic mathematical concepts, but we do not assume a graduate degree in the mathematical sciences. For in- stance, we have almost completely avoided the use of matrix algebra, and it is possible to understand the entire book without a detailed knowledge of matrices and vectors.

4. We presume that the reader is interested in applying statistical learn- ing methods to real-world problems. In order to facilitate this, as well as to motivate the techniques discussed, we have devoted a section within each chapter to R computer labs. In each lab, we walk the reader through a realistic application of the methods considered in that chapter. When we have taught this material in our courses, we have allocated roughly one-third of classroom time to working through the labs, and we have found them to be extremely useful. Many of the less computationally-oriented students who were ini- tially intimidated by R’s command level interface got the hang of things over the course of the quarter or semester. We have used R because it is freely available and is powerful enough to implement all of the methods discussed in the book. It also has optional packages that can be downloaded to implement literally thousands of addi- tional methods. Most importantly, R is the language of choice for academic statisticians, and new approaches often become available in

1. Introduction 9

R years before they are implemented in commercial packages. How- ever, the labs in ISL are self-contained, and can be skipped if the reader wishes to use a different software package or does not wish to apply the methods discussed to real-world problems.

Who Should Read This Book?

This book is intended for anyone who is interested in using modern statis- tical methods for modeling and prediction from data. This group includes scientists, engineers, data analysts, data scientists, and quants, but also less technical individuals with degrees in non-quantitative fields such as the social sciences or business. We expect that the reader will have had at least one elementary course in statistics. Background in linear regression is also useful, though not required, since we review the key concepts behind linear regression in Chapter 3. The mathematical level of this book is mod- est, and a detailed knowledge of matrix operations is not required. This book provides an introduction to the statistical programming language R. Previous exposure to a programming language, such as MATLAB or Python, is useful but not required. The first edition of this textbook has been used as to teach master’s and

PhD students in business, economics, computer science, biology, earth sci- ences, psychology, and many other areas of the physical and social sciences. It has also been used to teach advanced undergraduates who have already taken a course on linear regression. In the context of a more mathemat- ically rigorous course in which ESL serves as the primary textbook, ISL could be used as a supplementary text for teaching computational aspects of the various approaches.

Notation and Simple Matrix Algebra

Choosing notation for a textbook is always a difficult task. For the most part we adopt the same notational conventions as ESL. We will use n to represent the number of distinct data points, or observa-

tions, in our sample. We will let p denote the number of variables that are available for use in making predictions. For example, the Wage data set con- sists of 11 variables for 3,000 people, so we have n = 3,000 observations and p = 11 variables (such as year, age, race, and more). Note that throughout this book, we indicate variable names using colored font: Variable Name. In some examples, p might be quite large, such as on the order of thou-

sands or even millions; this situation arises quite often, for example, in the analysis of modern biological data or web-based advertising data.

10 1. Introduction

In general, we will let xij represent the value of the jth variable for the ith observation, where i = 1, 2, . . . , n and j = 1, 2, . . . , p. Throughout this book, i will be used to index the samples or observations (from 1 to n) and j will be used to index the variables (from 1 to p). We let X denote an n×p matrix whose (i, j)th element is xij. That is,

X =

⎛

⎜⎜⎜ ⎝

x11 x12 . . . x1p x21 x22 . . . x2p ...

... ...

... xn1 xn2 . . . xnp

⎞

⎟⎟⎟ ⎠

For readers who are unfamiliar with matrices, it is useful to visualize X as a spreadsheet of numbers with n rows and p columns. At times we will be interested in the rows of X, which we write as

x1, x2, . . . , xn. Here xi is a vector of length p, containing the p variable measurements for the ith observation. That is,

xi =

⎛

⎜⎜⎜ ⎝

xi1 xi2 ...

xip

⎞

⎟⎟⎟ ⎠

. (1.1)

(Vectors are by default represented as columns.) For example, for the Wage data, xi is a vector of length 11, consisting of year, age, race, and other values for the ith individual. At other times we will instead be interested in the columns of X, which we write as x1, x2, . . . , xp. Each is a vector of length n. That is,

xj =

⎛

⎜⎜⎜ ⎝

x1j x2j ...

xnj

⎞

⎟⎟⎟ ⎠

For example, for the Wage data, x1 contains the n = 3,000 values for year. Using this notation, the matrix X can be written as

X = ( x1 x2 · · · xp

) ,

X =

⎛

⎜⎜⎜ ⎝

xT1 xT2 ... xTn

⎞

⎟⎟⎟ ⎠

1. Introduction 11

The T notation denotes the transpose of a matrix or vector. So, for example,

XT =

⎛

⎜⎜⎜ ⎝

x11 x21 . . . xn1 x12 x22 . . . xn2 ...

... ...

x1p x2p . . . xnp

⎞

⎟⎟⎟ ⎠

while xTi =

( xi1 xi2 · · · xip

) .

We use yi to denote the ith observation of the variable on which we wish to make predictions, such as wage. Hence, we write the set of all n observations in vector form as

y =

⎛

⎜⎜⎜ ⎝

y1 y2 ... yn

⎞

⎟⎟⎟ ⎠

Then our observed data consists of {(x1, y1), (x2, y2), . . . , (xn, yn)}, where each xi is a vector of length p. (If p = 1, then xi is simply a scalar.) In this text, a vector of length n will always be denoted in lower case

bold; e.g.

a =

⎛

⎜⎜⎜ ⎝

a1 a2 ... an

⎞

⎟⎟⎟ ⎠

However, vectors that are not of length n (such as feature vectors of length p, as in (1.1)) will be denoted in lower case normal font, e.g. a. Scalars will also be denoted in lower case normal font, e.g. a. In the rare cases in which these two uses for lower case normal font lead to ambiguity, we will clarify which use is intended. Matrices will be denoted using bold capitals, such as A. Random variables will be denoted using capital normal font, e.g. A, regardless of their dimensions. Occasionally we will want to indicate the dimension of a particular ob-

ject. To indicate that an object is a scalar, we will use the notation a ∈ R. To indicate that it is a vector of length k, we will use a ∈ Rk (or a ∈ Rn if it is of length n). We will indicate that an object is an r × s matrix using A ∈ Rr×s. We have avoided using matrix algebra whenever possible. However, in

a few instances it becomes too cumbersome to avoid it entirely. In these rare instances it is important to understand the concept of multiplying two matrices. Suppose that A ∈ Rr×d and B ∈ Rd×s. Then the product of A and B is denoted AB. The (i, j)th element of AB is computed by

12 1. Introduction

multiplying each element of the ith row of A by the corresponding element of the jth column of B. That is, (AB)ij =

∑d k=1 aikbkj. As an example,

consider

A =

( 1 2 3 4

) and B =

( 5 6 7 8

) .

Then

AB =

( 1 2 3 4

)( 5 6 7 8

) =

( 1×5 + 2×7 1×6 + 2×8 3×5 + 4×7 3×6 + 4×8

) =

( 19 22 43 50

) .

Note that this operation produces an r × s matrix. It is only possible to compute AB if the number of columns of A is the same as the number of rows of B.

Organization of This Book

Chapter 2 introduces the basic terminology and concepts behind statisti- cal learning. This chapter also presents the K-nearest neighbor classifier, a very simple method that works surprisingly well on many problems. Chap- ters 3 and 4 cover classical linear methods for regression and classification. In particular, Chapter 3 reviews linear regression, the fundamental start- ing point for all regression methods. In Chapter 4 we discuss two of the most important classical classification methods, logistic regression and lin- ear discriminant analysis. A central problem in all statistical learning situations involves choosing

the best method for a given application. Hence, in Chapter 5 we intro- duce cross-validation and the bootstrap, which can be used to estimate the accuracy of a number of different methods in order to choose the best one. Much of the recent research in statistical learning has concentrated on

non-linear methods. However, linear methods often have advantages over their non-linear competitors in terms of interpretability and sometimes also accuracy. Hence, in Chapter 6 we consider a host of linear methods, both classical and more modern, which offer potential improvements over stan- dard linear regression. These include stepwise selection, ridge regression, principal components regression, and the lasso. The remaining chapters move into the world of non-linear statistical

learning. We first introduce in Chapter 7 a number of non-linear meth- ods that work well for problems with a single input variable. We then show how these methods can be used to fit non-linear additive models for which there is more than one input. In Chapter 8, we investigate tree-based methods, including bagging, boosting, and random forests. Support vector machines, a set of approaches for performing both linear and non-linear classification, are discussed in Chapter 9. We cover deep learning, an ap- proach for non-linear regression and classification that has received a lot

1. Introduction 13

of attention in recent years, in Chapter 10. Chapter 11 explores survival analysis, a regression approach that is specialized to the setting in which the output variable is censored, i.e. not fully observed. In Chapter 12, we consider the unsupervised setting in which we have

input variables but no output variable. In particular, we present princi- pal components analysis, K-means clustering, and hierarchical clustering. Finally, in Chapter 13 we cover the very important topic of multiple hy- pothesis testing. At the end of each chapter, we present one or more R lab sections in

which we systematically work through applications of the various meth- ods discussed in that chapter. These labs demonstrate the strengths and weaknesses of the various approaches, and also provide a useful reference for the syntax required to implement the various methods. The reader may choose to work through the labs at his or her own pace, or the labs may be the focus of group sessions as part of a classroom environment. Within each R lab, we present the results that we obtained when we performed the lab at the time of writing this book. However, new versions of R are continuously released, and over time, the packages called in the labs will be updated. Therefore, in the future, it is possible that the results shown in the lab sections may no longer correspond precisely to the results obtained by the reader who performs the labs. As necessary, we will post updates to the labs on the book website.

We use the symbol to denote sections or exercises that contain more challenging concepts. These can be easily skipped by readers who do not wish to delve as deeply into the material, or who lack the mathematical background.

Data Sets Used in Labs and Exercises

In this textbook, we illustrate statistical learning methods using applica- tions from marketing, finance, biology, and other areas. The ISLR2 package available on the book website and CRAN contains a number of data sets that are required in order to perform the labs and exercises associated with this book. One other data set is part of the base R distribution. Table 1.1 contains a summary of the data sets required to perform the labs and ex- ercises. A couple of these data sets are also available as text files on the book website, for use in Chapter 2.

14 1. Introduction

Name Description Auto Gas mileage, horsepower, and other information for cars. Bikeshare Hourly usage of a bike sharing program in Washington, DC. Boston Housing values and other information about Boston census tracts. BrainCancer Survival times for patients diagnosed with brain cancer. Caravan Information about individuals offered caravan insurance. Carseats Information about car seat sales in 400 stores. College Demographic characteristics, tuition, and more for USA colleges. Credit Information about credit card debt for 10,000 customers. Default Customer default records for a credit card company. Fund Returns of 2,000 hedge fund managers over 50 months. Hitters Records and salaries for baseball players. Khan Gene expression measurements for four cancer types. NCI60 Gene expression measurements for 64 cancer cell lines. NYSE Returns, volatility, and volume for the New York Stock Exchange. OJ Sales information for Citrus Hill and Minute Maid orange juice. Portfolio Past values of financial assets, for use in portfolio allocation. Publication Time to publication for 244 clinical trials. Smarket Daily percentage returns for S&P 500 over a 5-year period. USArrests Crime statistics per 100,000 residents in 50 states of USA. Wage Income survey data for men in central Atlantic region of USA. Weekly 1,089 weekly stock market returns for 21 years.

TABLE 1.1. A list of data sets needed to perform the labs and exercises in this textbook. All data sets are available in the ISLR2 library, with the exception of USArrests, which is part of the base R distribution.

Book Website

The website for this book is located at

It contains a number of resources, including the R package associated with this book, and some additional data sets.

Acknowledgements

A few of the plots in this book were taken from ESL: Figures 6.7, 8.3, and 12.14. All other plots are new to this book.

www.statlearning.com

2 Statistical Learning

2.1 What Is Statistical Learning?

In order to motivate our study of statistical learning, we begin with a simple example. Suppose that we are statistical consultants hired by a client to investigate the association between advertising and sales of a particular product. The Advertising data set consists of the sales of that product in 200 different markets, along with advertising budgets for the product in each of those markets for three different media: TV, radio, and newspaper. The data are displayed in Figure 2.1. It is not possible for our client to directly increase sales of the product. On the other hand, they can control the advertising expenditure in each of the three media. Therefore, if we determine that there is an association between advertising and sales, then we can instruct our client to adjust advertising budgets, thereby indirectly increasing sales. In other words, our goal is to develop an accurate model that can be used to predict sales on the basis of the three media budgets. In this setting, the advertising budgets are input variables while sales

input variableis an output variable. The input variables are typically denoted using the

output variable

symbol X, with a subscript to distinguish them. So X1 might be the TV budget, X2 the radio budget, and X3 the newspaper budget. The inputs go by different names, such as predictors, independent variables, features,

predictor

independent variable feature

or sometimes just variables. The output variable—in this case, sales—is

variable

often called the response or dependent variable, and is typically denoted

response

dependent variable

using the symbol Y . Throughout this book, we will use all of these terms interchangeably.

http://crossmark.crossref.org/dialog/?doi=10.1007/978-1-0716-1418-1_2&domain=pdf

16 2. Statistical Learning

0 50 100 200 300

5 1 0

1 5

2 0

2 5

S a le

0 10 20 30 40 50

5 1 0

1 5

2 0

2 5

Radio

S a le

0 20 40 60 80 100

5 1 0

1 5

2 0

2 5

Newspaper

S a le

FIGURE 2.1. The Advertising data set. The plot displays sales, in thousands of units, as a function of TV, radio, and newspaper budgets, in thousands of dollars, for 200 different markets. In each plot we show the simple least squares fit of sales to that variable, as described in Chapter 3. In other words, each blue line represents a simple model that can be used to predict sales using TV, radio, and newspaper, respectively.

More generally, suppose that we observe a quantitative response Y and p different predictors, X1, X2, . . . , Xp. We assume that there is some relationship between Y and X = (X1, X2, . . . , Xp), which can be written in the very general form

Y = f(X) + ϵ. (2.1)

Here f is some fixed but unknown function of X1, . . . , Xp, and ϵ is a random error term, which is independent of X and has mean zero. In this formula-

error term tion, f represents the systematic information that X provides about Y .

systematic As another example, consider the left-hand panel of Figure 2.2, a plot of

income versus years of education for 30 individuals in the Income data set. The plot suggests that one might be able to predict income using years of education. However, the function f that connects the input variable to the output variable is in general unknown. In this situation one must estimate f based on the observed points. Since Income is a simulated data set, f is known and is shown by the blue curve in the right-hand panel of Figure 2.2. The vertical lines represent the error terms ϵ. We note that some of the 30 observations lie above the blue curve and some lie below it; overall, the errors have approximately mean zero. In general, the function f may involve more than one input variable.

In Figure 2.3 we plot income as a function of years of education and seniority. Here f is a two-dimensional surface that must be estimated based on the observed data.

2.1 What Is Statistical Learning? 17

10 12 14 16 18 20 22

2 0

3 0

4 0

5 0

6 0

7 0

8 0

Years of Education

In c o

m e

10 12 14 16 18 20 22

2 0

3 0

4 0

5 0

6 0

7 0

8 0

Years of Education

In c o

m e

FIGURE 2.2. The Income data set. Left: The red dots are the observed values of income (in tens of thousands of dollars) and years of education for 30 indi- viduals. Right: The blue curve represents the true underlying relationship between income and years of education, which is generally unknown (but is known in this case because the data were simulated). The black lines represent the error associated with each observation. Note that some errors are positive (if an ob- servation lies above the blue curve) and some are negative (if an observation lies below the curve). Overall, these errors have approximately mean zero.

In essence, statistical learning refers to a set of approaches for estimating f. In this chapter we outline some of the key theoretical concepts that arise in estimating f, as well as tools for evaluating the estimates obtained.

2.1.1 Why Estimate f?

There are two main reasons that we may wish to estimate f: prediction and inference. We discuss each in turn.

Prediction

In many situations, a set of inputs X are readily available, but the output Y cannot be easily obtained. In this setting, since the error term averages to zero, we can predict Y using

Ŷ = f̂(X), (2.2)

where f̂ represents our estimate for f, and Ŷ represents the resulting pre- diction for Y . In this setting, f̂ is often treated as a black box, in the sense that one is not typically concerned with the exact form of f̂, provided that it yields accurate predictions for Y . As an example, suppose that X1, . . . , Xp are characteristics of a patient’s

blood sample that can be easily measured in a lab, and Y is a variable encoding the patient’s risk for a severe adverse reaction to a particular

18 2. Statistical Learning

Years of Education

S en

io rit

In c o m

FIGURE 2.3. The plot displays income as a function of years of education and seniority in the Income data set. The blue surface represents the true un- derlying relationship between income and years of education and seniority, which is known since the data are simulated. The red dots indicate the observed values of these quantities for 30 individuals.

drug. It is natural to seek to predict Y using X, since we can then avoid giving the drug in question to patients who are at high risk of an adverse reaction—that is, patients for whom the estimate of Y is high. The accuracy of Ŷ as a prediction for Y depends on two quantities,

which we will call the reducible error and the irreducible error. In general, reducible error irreducible error

f̂ will not be a perfect estimate for f, and this inaccuracy will introduce some error. This error is reducible because we can potentially improve the accuracy of f̂ by using the most appropriate statistical learning technique to estimate f. However, even if it were possible to form a perfect estimate for f, so that our estimated response took the form Ŷ = f(X), our prediction would still have some error in it! This is because Y is also a function of ϵ, which, by definition, cannot be predicted using X. Therefore, variability associated with ϵ also affects the accuracy of our predictions. This is known as the irreducible error, because no matter how well we estimate f, we cannot reduce the error introduced by ϵ. Why is the irreducible error larger than zero? The quantity ϵ may con-

tain unmeasured variables that are useful in predicting Y : since we don’t measure them, f cannot use them for its prediction. The quantity ϵ may also contain unmeasurable variation. For example, the risk of an adverse reaction might vary for a given patient on a given day, depending on manufacturing variation in the drug itself or the patient’s general feeling of well-being on that day.

2.1 What Is Statistical Learning? 19

Consider a given estimate f̂ and a set of predictors X, which yields the prediction Ŷ = f̂(X). Assume for a moment that both f̂ and X are fixed, so that the only variability comes from ϵ. Then, it is easy to show that

E(Y − Ŷ )2 = E[f(X) + ϵ− f̂(X)]2

= [f(X)− f̂(X)]2 ︸︷︷︸

Reducible

+ Var(ϵ) ︸︷︷︸

Irreducible

, (2.3)

where E(Y − Ŷ )2 represents the average, or expected value, of the squared expected valuedifference between the predicted and actual value of Y , and Var(ϵ) repre-

sents the variance associated with the error term ϵ. variance

The focus of this book is on techniques for estimating f with the aim of minimizing the reducible error. It is important to keep in mind that the irreducible error will always provide an upper bound on the accuracy of our prediction for Y . This bound is almost always unknown in practice.

Inference

We are often interested in understanding the association between Y and X1, . . . , Xp. In this situation we wish to estimate f, but our goal is not

necessarily to make predictions for Y . Now f̂ cannot be treated as a black box, because we need to know its exact form. In this setting, one may be interested in answering the following questions:

• Which predictors are associated with the response? It is often the case that only a small fraction of the available predictors are substantially associated with Y . Identifying the few important predictors among a large set of possible variables can be extremely useful, depending on the application.

• What is the relationship between the response and each predictor? Some predictors may have a positive relationship with Y , in the sense that larger values of the predictor are associated with larger values of Y . Other predictors may have the opposite relationship. Depending on the complexity of f, the relationship between the response and a given predictor may also depend on the values of the other predictors.

• Can the relationship between Y and each predictor be adequately sum- marized using a linear equation, or is the relationship more compli- cated? Historically, most methods for estimating f have taken a linear form. In some situations, such an assumption is reasonable or even de- sirable. But often the true relationship is more complicated, in which case a linear model may not provide an accurate representation of the relationship between the input and output variables.

In this book, we will see a number of examples that fall into the prediction setting, the inference setting, or a combination of the two.

20 2. Statistical Learning

For instance, consider a company that is interested in conducting a direct-marketing campaign. The goal is to identify individuals who are likely to respond positively to a mailing, based on observations of demo- graphic variables measured on each individual. In this case, the demo- graphic variables serve as predictors, and response to the marketing cam- paign (either positive or negative) serves as the outcome. The company is not interested in obtaining a deep understanding of the relationships be- tween each individual predictor and the response; instead, the company simply wants to accurately predict the response using the predictors. This is an example of modeling for prediction. In contrast, consider the Advertising data illustrated in Figure 2.1. One

may be interested in answering questions such as:

– Which media are associated with sales?

– Which media generate the biggest boost in sales? or

– How large of an increase in sales is associated with a given increase in TV advertising?

This situation falls into the inference paradigm. Another example involves modeling the brand of a product that a customer might purchase based on variables such as price, store location, discount levels, competition price, and so forth. In this situation one might really be most interested in the association between each variable and the probability of purchase. For in- stance, to what extent is the product’s price associated with sales? This is an example of modeling for inference. Finally, some modeling could be conducted both for prediction and in-

ference. For example, in a real estate setting, one may seek to relate values of homes to inputs such as crime rate, zoning, distance from a river, air quality, schools, income level of community, size of houses, and so forth. In this case one might be interested in the association between each individ- ual input variable and housing price—for instance, how much extra will a house be worth if it has a view of the river? This is an inference problem. Alternatively, one may simply be interested in predicting the value of a home given its characteristics: is this house under- or over-valued? This is a prediction problem. Depending on whether our ultimate goal is prediction, inference, or a

combination of the two, different methods for estimating f may be appro- priate. For example, linear models allow for relatively simple and inter-

linear model pretable inference, but may not yield as accurate predictions as some other approaches. In contrast, some of the highly non-linear approaches that we discuss in the later chapters of this book can potentially provide quite accu- rate predictions for Y , but this comes at the expense of a less interpretable model for which inference is more challenging.

2.1 What Is Statistical Learning? 21

2.1.2 How Do We Estimate f?

Throughout this book, we explore many linear and non-linear approaches for estimating f. However, these methods generally share certain charac- teristics. We provide an overview of these shared characteristics in this section. We will always assume that we have observed a set of n different data points. For example in Figure 2.2 we observed n = 30 data points. These observations are called the training data because we will use these

training dataobservations to train, or teach, our method how to estimate f. Let xij

represent the value of the jth predictor, or input, for observation i, where i = 1, 2, . . . , n and j = 1, 2, . . . , p. Correspondingly, let yi represent the response variable for the ith observation. Then our training data consist of {(x1, y1), (x2, y2), . . . , (xn, yn)} where xi = (xi1, xi2, . . . , xip)T . Our goal is to apply a statistical learning method to the training data

in order to estimate the unknown function f. In other words, we want to find a function f̂ such that Y ≈ f̂(X) for any observation (X, Y ). Broadly speaking, most statistical learning methods for this task can be character- ized as either parametric or non-parametric. We now briefly discuss these

parametric non- parametric

two types of approaches.

Parametric Methods

Parametric methods involve a two-step model-based approach.

1. First, we make an assumption about the functional form, or shape, of f. For example, one very simple assumption is that f is linear in X:

f(X) = β0 + β1X1 + β2X2 + · · · + βpXp. (2.4)

This is a linear model, which will be discussed extensively in Chap- ter 3. Once we have assumed that f is linear, the problem of estimat- ing f is greatly simplified. Instead of having to estimate an entirely arbitrary p-dimensional function f(X), one only needs to estimate the p + 1 coefficients β0, β1, . . . , βp.

2. After a model has been selected, we need a procedure that uses the training data to fit or train the model. In the case of the linear model

fit train

(2.4), we need to estimate the parameters β0, β1, . . . , βp. That is, we want to find values of these parameters such that

Y ≈ β0 + β1X1 + β2X2 + · · · + βpXp.

The most common approach to fitting the model (2.4) is referred to as (ordinary) least squares, which we discuss in Chapter 3. However,

least squares least squares is one of many possible ways to fit the linear model. In Chapter 6, we discuss other approaches for estimating the parameters in (2.4).

22 2. Statistical Learning

Years of Education

S en

io rit

In c o m

FIGURE 2.4. A linear model fit by least squares to the Income data from Fig- ure 2.3. The observations are shown in red, and the yellow plane indicates the least squares fit to the data.

The model-based approach just described is referred to as parametric; it reduces the problem of estimating f down to one of estimating a set of parameters. Assuming a parametric form for f simplifies the problem of estimating f because it is generally much easier to estimate a set of pa- rameters, such as β0, β1, . . . , βp in the linear model (2.4), than it is to fit an entirely arbitrary function f. The potential disadvantage of a paramet- ric approach is that the model we choose will usually not match the true unknown form of f. If the chosen model is too far from the true f, then our estimate will be poor. We can try to address this problem by choos- ing flexible models that can fit many different possible functional forms

flexible for f. But in general, fitting a more flexible model requires estimating a greater number of parameters. These more complex models can lead to a phenomenon known as overfitting the data, which essentially means they

overfitting follow the errors, or noise, too closely. These issues are discussed through-

noise out this book. Figure 2.4 shows an example of the parametric approach applied to the

Income data from Figure 2.3. We have fit a linear model of the form

income ≈ β0 + β1 ×education + β2 ×seniority.

Since we have assumed a linear relationship between the response and the two predictors, the entire fitting problem reduces to estimating β0, β1, and β2, which we do using least squares linear regression. Comparing Figure 2.3 to Figure 2.4, we can see that the linear fit given in Figure 2.4 is not quite right: the true f has some curvature that is not captured in the linear fit. However, the linear fit still appears to do a reasonable job of capturing the

2.1 What Is Statistical Learning? 23

Years of Education

S en

io rit

In c o m

FIGURE 2.5. A smooth thin-plate spline fit to the Income data from Figure 2.3 is shown in yellow; the observations are displayed in red. Splines are discussed in Chapter 7.

positive relationship between years of education and income, as well as the slightly less positive relationship between seniority and income. It may be that with such a small number of observations, this is the best we can do.

Non-Parametric Methods

Non-parametric methods do not make explicit assumptions about the func- tional form of f. Instead they seek an estimate of f that gets as close to the data points as possible without being too rough or wiggly. Such approaches can have a major advantage over parametric approaches: by avoiding the assumption of a particular functional form for f, they have the potential to accurately fit a wider range of possible shapes for f. Any parametric approach brings with it the possibility that the functional form used to estimate f is very different from the true f, in which case the resulting model will not fit the data well. In contrast, non-parametric approaches completely avoid this danger, since essentially no assumption about the form of f is made. But non-parametric approaches do suffer from a major disadvantage: since they do not reduce the problem of estimating f to a small number of parameters, a very large number of observations (far more than is typically needed for a parametric approach) is required in order to obtain an accurate estimate for f. An example of a non-parametric approach to fitting the Income data is

shown in Figure 2.5. A thin-plate spline is used to estimate f. This ap- thin-plate splineproach does not impose any pre-specified model on f. It instead attempts

to produce an estimate for f that is as close as possible to the observed data, subject to the fit—that is, the yellow surface in Figure 2.5—being

24 2. Statistical Learning

Years of Education

S en

io rit

In c o m

FIGURE 2.6. A rough thin-plate spline fit to the Income data from Figure 2.3. This fit makes zero errors on the training data.

smooth. In this case, the non-parametric fit has produced a remarkably ac- curate estimate of the true f shown in Figure 2.3. In order to fit a thin-plate spline, the data analyst must select a level of smoothness. Figure 2.6 shows the same thin-plate spline fit using a lower level of smoothness, allowing for a rougher fit. The resulting estimate fits the observed data perfectly! However, the spline fit shown in Figure 2.6 is far more variable than the true function f, from Figure 2.3. This is an example of overfitting the data, which we discussed previously. It is an undesirable situation because the fit obtained will not yield accurate estimates of the response on new observations that were not part of the original training data set. We dis- cuss methods for choosing the correct amount of smoothness in Chapter 5. Splines are discussed in Chapter 7. As we have seen, there are advantages and disadvantages to parametric

and non-parametric methods for statistical learning. We explore both types of methods throughout this book.

2.1.3 The Trade-Off Between Prediction Accuracy and Model Interpretability

Of the many methods that we examine in this book, some are less flexible, or more restrictive, in the sense that they can produce just a relatively small range of shapes to estimate f. For example, linear regression is a relatively inflexible approach, because it can only generate linear functions such as the lines shown in Figure 2.1 or the plane shown in Figure 2.4. Other methods, such as the thin plate splines shown in Figures 2.5 and 2.6,

2.1 What Is Statistical Learning? 25

Flexibility

In te

rp re

ta b ili

Low High

L ow

H ig

h Subset Selection Lasso

Least Squares

Generalized Additive Models Trees

Bagging, Boosting

Support Vector Machines Deep Learning

FIGURE 2.7. A representation of the tradeoff between flexibility and inter- pretability, using different statistical learning methods. In general, as the flexibil- ity of a method increases, its interpretability decreases.

are considerably more flexible because they can generate a much wider range of possible shapes to estimate f. One might reasonably ask the following question: why would we ever

choose to use a more restrictive method instead of a very flexible approach? There are several reasons that we might prefer a more restrictive model. If we are mainly interested in inference, then restrictive models are much more interpretable. For instance, when inference is the goal, the linear model may be a good choice since it will be quite easy to understand the relationship between Y and X1, X2, . . . , Xp. In contrast, very flexible approaches, such as the splines discussed in Chapter 7 and displayed in Figures 2.5 and 2.6, and the boosting methods discussed in Chapter 8, can lead to such complicated estimates of f that it is difficult to understand how any individual predictor is associated with the response. Figure 2.7 provides an illustration of the trade-off between flexibility and

interpretability for some of the methods that we cover in this book. Least squares linear regression, discussed in Chapter 3, is relatively inflexible but is quite interpretable. The lasso, discussed in Chapter 6, relies upon the

lasso linear model (2.4) but uses an alternative fitting procedure for estimating the coefficients β0, β1, . . . , βp. The new procedure is more restrictive in es- timating the coefficients, and sets a number of them to exactly zero. Hence in this sense the lasso is a less flexible approach than linear regression. It is also more interpretable than linear regression, because in the final model the response variable will only be related to a small subset of the predictors—namely, those with nonzero coefficient estimates. Generalized additive models (GAMs), discussed in Chapter 7, instead extend the lin-

generalized additive model

ear model (2.4) to allow for certain non-linear relationships. Consequently,

26 2. Statistical Learning

GAMs are more flexible than linear regression. They are also somewhat less interpretable than linear regression, because the relationship between each predictor and the response is now modeled using a curve. Finally, fully non-linear methods such as bagging, boosting, support vector machines

bagging

boosting with non-linear kernels, and neural networks (deep learning), discussed in

support vector machine

Chapters 8, 9, and 10, are highly flexible approaches that are harder to interpret. We have established that when inference is the goal, there are clear ad-

vantages to using simple and relatively inflexible statistical learning meth- ods. In some settings, however, we are only interested in prediction, and the interpretability of the predictive model is simply not of interest. For instance, if we seek to develop an algorithm to predict the price of a stock, our sole requirement for the algorithm is that it predict accurately— interpretability is not a concern. In this setting, we might expect that it will be best to use the most flexible model available. Surprisingly, this is not always the case! We will often obtain more accurate predictions using a less flexible method. This phenomenon, which may seem counterintuitive at first glance, has to do with the potential for overfitting in highly flexible methods. We saw an example of overfitting in Figure 2.6. We will discuss this very important concept further in Section 2.2 and throughout this book.

2.1.4 Supervised Versus Unsupervised Learning

Most statistical learning problems fall into one of two categories: supervised supervised

or unsupervised. The examples that we have discussed so far in this chap- unsupervised

ter all fall into the supervised learning domain. For each observation of the predictor measurement(s) xi, i = 1, . . . , n there is an associated response measurement yi. We wish to fit a model that relates the response to the predictors, with the aim of accurately predicting the response for future observations (prediction) or better understanding the relationship between the response and the predictors (inference). Many classical statistical learn- ing methods such as linear regression and logistic regression (Chapter 4), as

logistic regressionwell as more modern approaches such as GAM, boosting, and support vec-

tor machines, operate in the supervised learning domain. The vast majority of this book is devoted to this setting. By contrast, unsupervised learning describes the somewhat more chal-

lenging situation in which for every observation i = 1, . . . , n, we observe a vector of measurements xi but no associated response yi. It is not pos- sible to fit a linear regression model, since there is no response variable to predict. In this setting, we are in some sense working blind; the sit- uation is referred to as unsupervised because we lack a response vari- able that can supervise our analysis. What sort of statistical analysis is possible? We can seek to understand the relationships between the variables or between the observations. One statistical learning tool that we may use

2.1 What Is Statistical Learning? 27

0 2 4 6 8 10 12

2 4

6 8

1 0

1 2

0 2 4 6

2 4

6 8

X1X1 X

X 2

FIGURE 2.8. A clustering data set involving three groups. Each group is shown using a different colored symbol. Left: The three groups are well-separated. In this setting, a clustering approach should successfully identify the three groups. Right: There is some overlap among the groups. Now the clustering task is more challenging.

in this setting is cluster analysis, or clustering. The goal of cluster analysis cluster analysisis to ascertain, on the basis of x1, . . . , xn, whether the observations fall into

relatively distinct groups. For example, in a market segmentation study we might observe multiple characteristics (variables) for potential customers, such as zip code, family income, and shopping habits. We might believe that the customers fall into different groups, such as big spenders versus low spenders. If the information about each customer’s spending patterns were available, then a supervised analysis would be possible. However, this information is not available—that is, we do not know whether each poten- tial customer is a big spender or not. In this setting, we can try to cluster the customers on the basis of the variables measured, in order to identify distinct groups of potential customers. Identifying such groups can be of interest because it might be that the groups differ with respect to some property of interest, such as spending habits. Figure 2.8 provides a simple illustration of the clustering problem. We

have plotted 150 observations with measurements on two variables, X1 and X2. Each observation corresponds to one of three distinct groups. For illustrative purposes, we have plotted the members of each group using dif- ferent colors and symbols. However, in practice the group memberships are unknown, and the goal is to determine the group to which each observa- tion belongs. In the left-hand panel of Figure 2.8, this is a relatively easy task because the groups are well-separated. By contrast, the right-hand panel illustrates a more challenging setting in which there is some overlap

28 2. Statistical Learning

between the groups. A clustering method could not be expected to assign all of the overlapping points to their correct group (blue, green, or orange). In the examples shown in Figure 2.8, there are only two variables, and

so one can simply visually inspect the scatterplots of the observations in order to identify clusters. However, in practice, we often encounter data sets that contain many more than two variables. In this case, we cannot easily plot the observations. For instance, if there are p variables in our data set, then p(p − 1)/2 distinct scatterplots can be made, and visual inspection is simply not a viable way to identify clusters. For this reason, automated clustering methods are important. We discuss clustering and other unsupervised learning approaches in Chapter 12. Many problems fall naturally into the supervised or unsupervised learn-

ing paradigms. However, sometimes the question of whether an analysis should be considered supervised or unsupervised is less clear-cut. For in- stance, suppose that we have a set of n observations. For m of the observa- tions, where m < n, we have both predictor measurements and a response measurement. For the remaining n − m observations, we have predictor measurements but no response measurement. Such a scenario can arise if the predictors can be measured relatively cheaply but the corresponding responses are much more expensive to collect. We refer to this setting as a semi-supervised learning problem. In this setting, we wish to use a sta-

semi- supervised learning

tistical learning method that can incorporate the m observations for which response measurements are available as well as the n−m observations for which they are not. Although this is an interesting topic, it is beyond the scope of this book.

2.1.5 Regression Versus Classification Problems

Variables can be characterized as either quantitative or qualitative (also quantitative

qualitative known as categorical). Quantitative variables take on numerical values.

categorical Examples include a person’s age, height, or income, the value of a house, and the price of a stock. In contrast, qualitative variables take on values in one of K different classes, or categories. Examples of qualitative vari-

class ables include a person’s marital status (married or not), the brand of prod- uct purchased (brand A, B, or C), whether a person defaults on a debt (yes or no), or a cancer diagnosis (Acute Myelogenous Leukemia, Acute Lymphoblastic Leukemia, or No Leukemia). We tend to refer to problems with a quantitative response as regression problems, while those involv-

regression ing a qualitative response are often referred to as classification problems.

classification However, the distinction is not always that crisp. Least squares linear re- gression (Chapter 3) is used with a quantitative response, whereas logistic regression (Chapter 4) is typically used with a qualitative (two-class, or binary) response. Thus, despite its name, logistic regression is a classifica-

binary tion method. But since it estimates class probabilities, it can be thought of as a regression method as well. Some statistical methods, such as K-nearest

2.2 Assessing Model Accuracy 29

neighbors (Chapters 2 and 4) and boosting (Chapter 8), can be used in the case of either quantitative or qualitative responses. We tend to select statistical learning methods on the basis of whether

the response is quantitative or qualitative; i.e. we might use linear regres- sion when quantitative and logistic regression when qualitative. However, whether the predictors are qualitative or quantitative is generally consid- ered less important. Most of the statistical learning methods discussed in this book can be applied regardless of the predictor variable type, provided that any qualitative predictors are properly coded before the analysis is performed. This is discussed in Chapter 3.

2.2 Assessing Model Accuracy

One of the key aims of this book is to introduce the reader to a wide range of statistical learning methods that extend far beyond the standard linear regression approach. Why is it necessary to introduce so many different statistical learning approaches, rather than just a single best method? There is no free lunch in statistics: no one method dominates all others over all possible data sets. On a particular data set, one specific method may work best, but some other method may work better on a similar but different data set. Hence it is an important task to decide for any given set of data which method produces the best results. Selecting the best approach can be one of the most challenging parts of performing statistical learning in practice. In this section, we discuss some of the most important concepts that

arise in selecting a statistical learning procedure for a specific data set. As the book progresses, we will explain how the concepts presented here can be applied in practice.

2.2.1 Measuring the Quality of Fit

In order to evaluate the performance of a statistical learning method on a given data set, we need some way to measure how well its predictions actually match the observed data. That is, we need to quantify the extent to which the predicted response value for a given observation is close to the true response value for that observation. In the regression setting, the most commonly-used measure is the mean squared error (MSE), given by mean

squared error

MSE = 1

n∑

i=1

(yi − f̂(xi))2, (2.5)

where f̂(xi) is the prediction that f̂ gives for the ith observation. The MSE will be small if the predicted responses are very close to the true responses,

30 2. Statistical Learning

and will be large if for some of the observations, the predicted and true responses differ substantially. The MSE in (2.5) is computed using the training data that was used to

fit the model, and so should more accurately be referred to as the training MSE. But in general, we do not really care how well the method works

training MSEon the training data. Rather, we are interested in the accuracy of the pre-

dictions that we obtain when we apply our method to previously unseen test data. Why is this what we care about? Suppose that we are interested

test data in developing an algorithm to predict a stock’s price based on previous stock returns. We can train the method using stock returns from the past 6 months. But we don’t really care how well our method predicts last week’s stock price. We instead care about how well it will predict tomorrow’s price or next month’s price. On a similar note, suppose that we have clinical measurements (e.g. weight, blood pressure, height, age, family history of disease) for a number of patients, as well as information about whether each patient has diabetes. We can use these patients to train a statistical learn- ing method to predict risk of diabetes based on clinical measurements. In practice, we want this method to accurately predict diabetes risk for future patients based on their clinical measurements. We are not very interested in whether or not the method accurately predicts diabetes risk for patients used to train the model, since we already know which of those patients have diabetes. To state it more mathematically, suppose that we fit our statistical learn-

ing method on our training observations {(x1, y1), (x2, y2), . . . , (xn, yn)}, and we obtain the estimate f̂. We can then compute f̂(x1), f̂(x2), . . . , f̂(xn). If these are approximately equal to y1, y2, . . . , yn, then the training MSE given by (2.5) is small. However, we are really not interested in whether f̂(xi) ≈ yi; instead, we want to know whether f̂(x0) is approximately equal to y0, where (x0, y0) is a previously unseen test observation not used to train the statistical learning method. We want to choose the method that gives the lowest test MSE, as opposed to the lowest training MSE. In other words,

test MSE if we had a large number of test observations, we could compute

Ave(y0 − f̂(x0))2, (2.6)

the average squared prediction error for these test observations (x0, y0). We’d like to select the model for which this quantity is as small as possible. How can we go about trying to select a method that minimizes the test

MSE? In some settings, we may have a test data set available—that is, we may have access to a set of observations that were not used to train the statistical learning method. We can then simply evaluate (2.6) on the test observations, and select the learning method for which the test MSE is smallest. But what if no test observations are available? In that case, one might imagine simply selecting a statistical learning method that minimizes the training MSE (2.5). This seems like it might be a sensible approach,

2.2 Assessing Model Accuracy 31

0 20 40 60 80 100

2 4

6 8

1 0

1 2

2 5 10 20

0 .0

0 .5

1 .0

1 .5

2 .0

2 .5

Flexibility

M e

a n

S q

u a

re d

E rr

o r

FIGURE 2.9. Left: Data simulated from f, shown in black. Three estimates of f are shown: the linear regression line (orange curve), and two smoothing spline fits (blue and green curves). Right: Training MSE (grey curve), test MSE (red curve), and minimum possible test MSE over all methods (dashed line). Squares represent the training and test MSEs for the three fits shown in the left-hand panel.

since the training MSE and the test MSE appear to be closely related. Unfortunately, there is a fundamental problem with this strategy: there is no guarantee that the method with the lowest training MSE will also have the lowest test MSE. Roughly speaking, the problem is that many statistical methods specifically estimate coefficients so as to minimize the training set MSE. For these methods, the training set MSE can be quite small, but the test MSE is often much larger. Figure 2.9 illustrates this phenomenon on a simple example. In the left-

hand panel of Figure 2.9, we have generated observations from (2.1) with the true f given by the black curve. The orange, blue and green curves illus- trate three possible estimates for f obtained using methods with increasing levels of flexibility. The orange line is the linear regression fit, which is rela- tively inflexible. The blue and green curves were produced using smoothing splines, discussed in Chapter 7, with different levels of smoothness. It is

smoothing splineclear that as the level of flexibility increases, the curves fit the observed

data more closely. The green curve is the most flexible and matches the data very well; however, we observe that it fits the true f (shown in black) poorly because it is too wiggly. By adjusting the level of flexibility of the smoothing spline fit, we can produce many different fits to this data. We now move on to the right-hand panel of Figure 2.9. The grey curve

displays the average training MSE as a function of flexibility, or more for- mally the degrees of freedom, for a number of smoothing splines. The de-

degrees of freedomgrees of freedom is a quantity that summarizes the flexibility of a curve; it

32 2. Statistical Learning

is discussed more fully in Chapter 7. The orange, blue and green squares indicate the MSEs associated with the corresponding curves in the left- hand panel. A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom. The training MSE declines monotonically as flexibility increases. In this example the true f is non-linear, and so the orange linear fit is not flexible enough to estimate f well. The green curve has the lowest training MSE of all three methods, since it corresponds to the most flexible of the three curves fit in the left-hand panel. In this example, we know the true function f, and so we can also com-

pute the test MSE over a very large test set, as a function of flexibility. (Of course, in general f is unknown, so this will not be possible.) The test MSE is displayed using the red curve in the right-hand panel of Figure 2.9. As with the training MSE, the test MSE initially declines as the level of flex- ibility increases. However, at some point the test MSE levels off and then starts to increase again. Consequently, the orange and green curves both have high test MSE. The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best in the left-hand panel of Figure 2.9. The horizontal dashed line indicates Var(ϵ), the irreducible error in (2.3), which corresponds to the lowest achievable test MSE among all possible methods. Hence, the smoothing spline repre- sented by the blue curve is close to optimal. In the right-hand panel of Figure 2.9, as the flexibility of the statistical

learning method increases, we observe a monotone decrease in the training MSE and a U-shape in the test MSE. This is a fundamental property of statistical learning that holds regardless of the particular data set at hand and regardless of the statistical method being used. As model flexibility increases, training MSE will decrease, but the test MSE may not. When a given method yields a small training MSE but a large test MSE, we are said to be overfitting the data. This happens because our statistical learning procedure is working too hard to find patterns in the training data, and may be picking up some patterns that are just caused by random chance rather than by true properties of the unknown function f. When we overfit the training data, the test MSE will be very large because the supposed patterns that the method found in the training data simply don’t exist in the test data. Note that regardless of whether or not overfitting has occurred, we almost always expect the training MSE to be smaller than the test MSE because most statistical learning methods either directly or indirectly seek to minimize the training MSE. Overfitting refers specifically to the case in which a less flexible model would have yielded a smaller test MSE. Figure 2.10 provides another example in which the true f is approxi-

mately linear. Again we observe that the training MSE decreases mono- tonically as the model flexibility increases, and that there is a U-shape in

2.2 Assessing Model Accuracy 33

0 20 40 60 80 100

2 4

6 8

1 0

1 2

2 5 10 20

0 .0

0 .5

1 .0

1 .5

2 .0

2 .5

Flexibility

M e

a n

S q

u a

re d

E rr

o r

FIGURE 2.10. Details are as in Figure 2.9, using a different true f that is much closer to linear. In this setting, linear regression provides a very good fit to the data.

the test MSE. However, because the truth is close to linear, the test MSE only decreases slightly before increasing again, so that the orange least squares fit is substantially better than the highly flexible green curve. Fi- nally, Figure 2.11 displays an example in which f is highly non-linear. The training and test MSE curves still exhibit the same general patterns, but now there is a rapid decrease in both curves before the test MSE starts to increase slowly. In practice, one can usually compute the training MSE with relative

ease, but estimating test MSE is considerably more difficult because usually no test data are available. As the previous three examples illustrate, the flexibility level corresponding to the model with the minimal test MSE can vary considerably among data sets. Throughout this book, we discuss a variety of approaches that can be used in practice to estimate this minimum point. One important method is cross-validation (Chapter 5), which is a cross-

validationmethod for estimating test MSE using the training data.

2.2.2 The Bias-Variance Trade-Off

The U-shape observed in the test MSE curves (Figures 2.9–2.11) turns out to be the result of two competing properties of statistical learning methods. Though the mathematical proof is beyond the scope of this book, it is possible to show that the expected test MSE, for a given value x0, can always be decomposed into the sum of three fundamental quantities: the variance of f̂(x0), the squared bias of f̂(x0) and the variance of the error

variance bias

34 2. Statistical Learning

0 20 40 60 80 100

− 1

0 0

1 0

2 0

2 5 10 20

0 5

1 0

1 5

2 0

Flexibility

M e

a n

S q

u a

re d

E rr

o r

FIGURE 2.11. Details are as in Figure 2.9, using a different f that is far from linear. In this setting, linear regression provides a very poor fit to the data.

terms ϵ. That is,

E ( y0 − f̂(x0)

)2 = Var(f̂(x0)) + [Bias(f̂(x0))]

2 + Var(ϵ). (2.7)

Here the notation E ( y0 − f̂(x0)

)2 defines the expected test MSE at x0,

expected test MSEand refers to the average test MSE that we would obtain if we repeatedly

estimated f using a large number of training sets, and tested each at x0. The

overall expected test MSE can be computed by averaging E ( y0 − f̂(x0)

over all possible values of x0 in the test set. Equation 2.7 tells us that in order to minimize the expected test error,

we need to select a statistical learning method that simultaneously achieves low variance and low bias. Note that variance is inherently a nonnegative quantity, and squared bias is also nonnegative. Hence, we see that the expected test MSE can never lie below Var(ϵ), the irreducible error from (2.3). What do we mean by the variance and bias of a statistical learning

method? Variance refers to the amount by which f̂ would change if we estimated it using a different training data set. Since the training data are used to fit the statistical learning method, different training data sets will result in a different f̂. But ideally the estimate for f should not vary too much between training sets. However, if a method has high variance then small changes in the training data can result in large changes in f̂. In general, more flexible statistical methods have higher variance. Consider the green and orange curves in Figure 2.9. The flexible green curve is following the observations very closely. It has high variance because changing any

2.2 Assessing Model Accuracy 35

one of these data points may cause the estimate f̂ to change considerably. In contrast, the orange least squares line is relatively inflexible and has low variance, because moving any single observation will likely cause only a small shift in the position of the line. On the other hand, bias refers to the error that is introduced by approxi-

mating a real-life problem, which may be extremely complicated, by a much simpler model. For example, linear regression assumes that there is a linear relationship between Y and X1, X2, . . . , Xp. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing lin- ear regression will undoubtedly result in some bias in the estimate of f. In Figure 2.11, the true f is substantially non-linear, so no matter how many training observations we are given, it will not be possible to produce an accurate estimate using linear regression. In other words, linear regression results in high bias in this example. However, in Figure 2.10 the true f is very close to linear, and so given enough data, it should be possible for linear regression to produce an accurate estimate. Generally, more flexible methods result in less bias. As a general rule, as we use more flexible methods, the variance will

increase and the bias will decrease. The relative rate of change of these two quantities determines whether the test MSE increases or decreases. As we increase the flexibility of a class of methods, the bias tends to initially decrease faster than the variance increases. Consequently, the expected test MSE declines. However, at some point increasing flexibility has little impact on the bias but starts to significantly increase the variance. When this happens the test MSE increases. Note that we observed this pattern of decreasing test MSE followed by increasing test MSE in the right-hand panels of Figures 2.9–2.11. The three plots in Figure 2.12 illustrate Equation 2.7 for the examples in

Figures 2.9–2.11. In each case the blue solid curve represents the squared bias, for different levels of flexibility, while the orange curve corresponds to the variance. The horizontal dashed line represents Var(ϵ), the irreducible error. Finally, the red curve, corresponding to the test set MSE, is the sum of these three quantities. In all three cases, the variance increases and the bias decreases as the method’s flexibility increases. However, the flexibility level corresponding to the optimal test MSE differs considerably among the three data sets, because the squared bias and variance change at different rates in each of the data sets. In the left-hand panel of Figure 2.12, the bias initially decreases rapidly, resulting in an initial sharp decrease in the expected test MSE. On the other hand, in the center panel of Figure 2.12 the true f is close to linear, so there is only a small decrease in bias as flex- ibility increases, and the test MSE only declines slightly before increasing rapidly as the variance increases. Finally, in the right-hand panel of Fig- ure 2.12, as flexibility increases, there is a dramatic decline in bias because the true f is very non-linear. There is also very little increase in variance

36 2. Statistical Learning

2 5 10 20

0 .0

0 .5

1 .0

1 .5

2 .0

2 .5

Flexibility

2 5 10 20

0 .0

0 .5

1 .0

1 .5

2 .0

2 .5

Flexibility

2 5 10 20

0 5

1 0

1 5

2 0

Flexibility

MSE Bias Var

FIGURE 2.12. Squared bias (blue curve), variance (orange curve), Var(ϵ) (dashed line), and test MSE (red curve) for the three data sets in Figures 2.9–2.11. The vertical dotted line indicates the flexibility level corresponding to the smallest test MSE.

as flexibility increases. Consequently, the test MSE declines substantially before experiencing a small increase as model flexibility increases. The relationship between bias, variance, and test set MSE given in Equa-

tion 2.7 and displayed in Figure 2.12 is referred to as the bias-variance trade-off. Good test set performance of a statistical learning method re-

bias-variance trade-offquires low variance as well as low squared bias. This is referred to as a

trade-off because it is easy to obtain a method with extremely low bias but high variance (for instance, by drawing a curve that passes through every single training observation) or a method with very low variance but high bias (by fitting a horizontal line to the data). The challenge lies in finding a method for which both the variance and the squared bias are low. This trade-off is one of the most important recurring themes in this book. In a real-life situation in which f is unobserved, it is generally not pos-

sible to explicitly compute the test MSE, bias, or variance for a statistical learning method. Nevertheless, one should always keep the bias-variance trade-off in mind. In this book we explore methods that are extremely flexible and hence can essentially eliminate bias. However, this does not guarantee that they will outperform a much simpler method such as linear regression. To take an extreme example, suppose that the true f is linear. In this situation linear regression will have no bias, making it very hard for a more flexible method to compete. In contrast, if the true f is highly non-linear and we have an ample number of training observations, then we may do better using a highly flexible approach, as in Figure 2.11. In Chapter 5 we discuss cross-validation, which is a way to estimate the test MSE using the training data.

2.2 Assessing Model Accuracy 37

2.2.3 The Classification Setting

Thus far, our discussion of model accuracy has been focused on the regres- sion setting. But many of the concepts that we have encountered, such as the bias-variance trade-off, transfer over to the classification setting with only some modifications due to the fact that yi is no longer quan- titative. Suppose that we seek to estimate f on the basis of training obser- vations {(x1, y1), . . . , (xn, yn)}, where now y1, . . . , yn are qualitative. The most common approach for quantifying the accuracy of our estimate f̂ is the training error rate, the proportion of mistakes that are made if we apply

error rate our estimate f̂ to the training observations:

n∑

i=1

I(yi ̸= ŷi). (2.8)

Here ŷi is the predicted class label for the ith observation using f̂. And I(yi ̸= ŷi) is an indicator variable that equals 1 if yi ̸= ŷi and zero if yi = ŷi.

indicator variableIf I(yi ̸= ŷi) = 0 then the ith observation was classified correctly by our

classification method; otherwise it was misclassified. Hence Equation 2.8 computes the fraction of incorrect classifications. Equation 2.8 is referred to as the training error rate because it is com-

training errorputed based on the data that was used to train our classifier. As in the

regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form

test error (x0, y0) is given by

Ave (I(y0 ̸= ŷ0)) , (2.9)

where ŷ0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest.

The Bayes Classifier

It is possible to show (though the proof is outside of the scope of this book) that the test error rate given in (2.9) is minimized, on average, by a very simple classifier that assigns each observation to the most likely class, given its predictor values. In other words, we should simply assign a test observation with predictor vector x0 to the class j for which

Pr(Y = j|X = x0) (2.10)

is largest. Note that (2.10) is a conditional probability: it is the probability conditional probabilitythat Y = j, given the observed predictor vector x0. This very simple clas-

sifier is called the Bayes classifier. In a two-class problem where there are Bayes classifieronly two possible response values, say class 1 or class 2, the Bayes classifier

38 2. Statistical Learning

o o

o o o

o o

o o o

oo o

o o

X 2

FIGURE 2.13. A simulated data set consisting of 100 observations in each of two groups, indicated in blue and in orange. The purple dashed line represents the Bayes decision boundary. The orange background grid indicates the region in which a test observation will be assigned to the orange class, and the blue background grid indicates the region in which a test observation will be assigned to the blue class.

corresponds to predicting class one if Pr(Y = 1|X = x0) > 0.5, and class two otherwise. Figure 2.13 provides an example using a simulated data set in a two-

dimensional space consisting of predictors X1 and X2. The orange and blue circles correspond to training observations that belong to two different classes. For each value of X1 and X2, there is a different probability of the response being orange or blue. Since this is simulated data, we know how the data were generated and we can calculate the conditional probabilities for each value of X1 and X2. The orange shaded region reflects the set of points for which Pr(Y = orange|X) is greater than 50 %, while the blue shaded region indicates the set of points for which the probability is below 50 %. The purple dashed line represents the points where the probability is exactly 50 %. This is called the Bayes decision boundary. The Bayes

Bayes decision boundary

classifier’s prediction is determined by the Bayes decision boundary; an observation that falls on the orange side of the boundary will be assigned to the orange class, and similarly an observation on the blue side of the boundary will be assigned to the blue class. The Bayes classifier produces the lowest possible test error rate, called

the Bayes error rate. Since the Bayes classifier will always choose the class Bayes error ratefor which (2.10) is largest, the error rate will be 1−maxj Pr(Y = j|X = x0)

2.2 Assessing Model Accuracy 39

at X = x0. In general, the overall Bayes error rate is given by

1−E ( max

j Pr(Y = j|X)

) , (2.11)

where the expectation averages the probability over all possible values of X. For our simulated data, the Bayes error rate is 0.133. It is greater than zero, because the classes overlap in the true population so maxj Pr(Y = j|X = x0) < 1 for some values of x0. The Bayes error rate is analogous to the irreducible error, discussed earlier.

K-Nearest Neighbors

In theory we would always like to predict qualitative responses using the Bayes classifier. But for real data, we do not know the conditional distri- bution of Y given X, and so computing the Bayes classifier is impossi- ble. Therefore, the Bayes classifier serves as an unattainable gold standard against which to compare other methods. Many approaches attempt to estimate the conditional distribution of Y given X, and then classify a given observation to the class with highest estimated probability. One such method is the K-nearest neighbors (KNN) classifier. Given a positive in-

K-nearest neighborsteger K and a test observation x0, the KNN classifier first identifies the

K points in the training data that are closest to x0, represented by N0. It then estimates the conditional probability for class j as the fraction of points in N0 whose response values equal j:

Pr(Y = j|X = x0) = 1

∑

i∈N0

I(yi = j). (2.12)

Finally, KNN classifies the test observation x0 to the class with the largest probability from (2.12). Figure 2.14 provides an illustrative example of the KNN approach. In

the left-hand panel, we have plotted a small training data set consisting of six blue and six orange observations. Our goal is to make a prediction for the point labeled by the black cross. Suppose that we choose K = 3. Then KNN will first identify the three observations that are closest to the cross. This neighborhood is shown as a circle. It consists of two blue points and one orange point, resulting in estimated probabilities of 2/3 for the blue class and 1/3 for the orange class. Hence KNN will predict that the black cross belongs to the blue class. In the right-hand panel of Figure 2.14 we have applied the KNN approach with K = 3 at all of the possible values for X1 and X2, and have drawn in the corresponding KNN decision boundary. Despite the fact that it is a very simple approach, KNN can often pro-

duce classifiers that are surprisingly close to the optimal Bayes classifier. Figure 2.15 displays the KNN decision boundary, using K = 10, when ap- plied to the larger simulated data set from Figure 2.13. Notice that even

40 2. Statistical Learning

o o

FIGURE 2.14. The KNN approach, using K = 3, is illustrated in a simple situation with six blue observations and six orange observations. Left: a test ob- servation at which a predicted class label is desired is shown as a black cross. The three closest points to the test observation are identified, and it is predicted that the test observation belongs to the most commonly-occurring class, in this case blue. Right: The KNN decision boundary for this example is shown in black. The blue grid indicates the region in which a test observation will be assigned to the blue class, and the orange grid indicates the region in which it will be assigned to the orange class.

o o

o o o

o o

o o o

oo o

o o

X 2

KNN: K=10

FIGURE 2.15. The black curve indicates the KNN decision boundary on the data from Figure 2.13, using K = 10. The Bayes decision boundary is shown as a purple dashed line. The KNN and Bayes decision boundaries are very similar.

2.2 Assessing Model Accuracy 41

o o

o o o

o o

o o o

o o

o o o

o o

o o o

o o

KNN: K=1 KNN: K=100

FIGURE 2.16. A comparison of the KNN decision boundaries (solid black curves) obtained using K = 1 and K = 100 on the data from Figure 2.13. With K = 1, the decision boundary is overly flexible, while with K = 100 it is not sufficiently flexible. The Bayes decision boundary is shown as a purple dashed line.

though the true distribution is not known by the KNN classifier, the KNN decision boundary is very close to that of the Bayes classifier. The test error rate using KNN is 0.1363, which is close to the Bayes error rate of 0.1304. The choice of K has a drastic effect on the KNN classifier obtained.

Figure 2.16 displays two KNN fits to the simulated data from Figure 2.13, using K = 1 and K = 100. When K = 1, the decision boundary is overly flexible and finds patterns in the data that don’t correspond to the Bayes decision boundary. This corresponds to a classifier that has low bias but very high variance. As K grows, the method becomes less flexible and produces a decision boundary that is close to linear. This corresponds to a low-variance but high-bias classifier. On this simulated data set, neither K = 1 nor K = 100 give good predictions: they have test error rates of 0.1695 and 0.1925, respectively. Just as in the regression setting, there is not a strong relationship be-

tween the training error rate and the test error rate. With K = 1, the KNN training error rate is 0, but the test error rate may be quite high. In general, as we use more flexible classification methods, the training error rate will decline but the test error rate may not. In Figure 2.17, we have plotted the KNN test and training errors as a function of 1/K. As 1/K in- creases, the method becomes more flexible. As in the regression setting, the training error rate consistently declines as the flexibility increases. However, the test error exhibits a characteristic U-shape, declining at first (with a minimum at approximately K = 10) before increasing again when the method becomes excessively flexible and overfits.

42 2. Statistical Learning

0.01 0.02 0.05 0.10 0.20 0.50 1.00

0 .0

0 0

.0 5

0 .1

0 0

.1 5

0 .2

1/K

E rr

o r

R a

Training Errors Test Errors

FIGURE 2.17. The KNN training error rate (blue, 200 observations) and test error rate (orange, 5,000 observations) on the data from Figure 2.13, as the level of flexibility (assessed using 1/K on the log scale) increases, or equivalently as the number of neighbors K decreases. The black dashed line indicates the Bayes error rate. The jumpiness of the curves is due to the small size of the training data set.

In both the regression and classification settings, choosing the correct level of flexibility is critical to the success of any statistical learning method. The bias-variance tradeoff, and the resulting U-shape in the test error, can make this a difficult task. In Chapter 5, we return to this topic and discuss various methods for estimating test error rates and thereby choosing the optimal level of flexibility for a given statistical learning method.

2.3 Lab: Introduction to R

In this lab, we will introduce some simple R commands. The best way to learn a new language is to try out the commands. R can be downloaded from

http://cran.r-project.org/

We recommend that you run R within an integrated development environ- ment (IDE) such as RStudio, which can be freely downloaded from

http://rstudio.com

http://cran.r-project.org/

http://rstudio.com

2.3 Lab: Introduction to R 43

The RStudio website also provides a cloud-based version of R, which does not require installing any software.

2.3.1 Basic Commands

R uses functions to perform operations. To run a function called funcname, function

we type funcname(input1, input2), where the inputs (or arguments) input1 argument

and input2 tell R how to run the function. A function can have any number of inputs. For example, to create a vector of numbers, we use the function c() (for concatenate). Any numbers inside the parentheses are joined to-

c() gether. The following command instructs R to join together the numbers 1, 3, 2, and 5, and to save them as a vector named x. When we type x, it

vector gives us back the vector.

> x <- c(1, 3, 2, 5)

> x

[1] 1 3 2 5

Note that the > is not part of the command; rather, it is printed by R to indicate that it is ready for another command to be entered. We can also save things using = rather than <-:

> x = c(1, 6, 2)

> x

[1] 1 6 2

> y = c(1, 4, 3)

Hitting the up arrow multiple times will display the previous commands, which can then be edited. This is useful since one often wishes to repeat a similar command. In addition, typing ?funcname will always cause R to open a new help file window with additional information about the function funcname(). We can tell R to add two sets of numbers together. It will then add the

first number from x to the first number from y, and so on. However, x and y should be the same length. We can check their length using the length()

length() function.

> length(x)

[1] 3

> length(y)

[1] 3

> x + y

[1] 2 10 5

The ls() function allows us to look at a list of all of the objects, such ls()

as data and functions, that we have saved so far. The rm() function can be rm()

used to delete any that we don’t want.

> ls()

[1] "x" "y"

> rm(x, y)

44 2. Statistical Learning

> ls()

character (0)

It’s also possible to remove all objects at once:

> rm(list = ls())

The matrix() function can be used to create a matrix of numbers. Before matrix()

we use the matrix() function, we can learn more about it:

> ?matrix

The help file reveals that the matrix() function takes a number of inputs, but for now we focus on the first three: the data (the entries in the matrix), the number of rows, and the number of columns. First, we create a simple matrix.

> x <- matrix(data = c(1, 2, 3, 4), nrow = 2, ncol = 2)

> x

[,1] [,2]

[1,] 1 3

[2,] 2 4

Note that we could just as well omit typing data=, nrow=, and ncol= in the matrix() command above: that is, we could just type

> x <- matrix(c(1, 2, 3, 4), 2, 2)

and this would have the same effect. However, it can sometimes be useful to specify the names of the arguments passed in, since otherwise R will assume that the function arguments are passed into the function in the same order that is given in the function’s help file. As this example illustrates, by default R creates matrices by successively filling in columns. Alternatively, the byrow = TRUE option can be used to populate the matrix in order of the rows.

> matrix(c(1, 2, 3, 4), 2, 2, byrow = TRUE)

[,1] [,2]

[1,] 1 2

[2,] 3 4

Notice that in the above command we did not assign the matrix to a value such as x. In this case the matrix is printed to the screen but is not saved for future calculations. The sqrt() function returns the square root of each

sqrt() element of a vector or matrix. The command x^2 raises each element of x to the power 2; any powers are possible, including fractional or negative powers.

> sqrt(x)

[,1] [,2]

[1,] 1.00 1.73

[2,] 1.41 2.00

> x^2

[,1] [,2]

[1,] 1 9

[2,] 4 16

2.3 Lab: Introduction to R 45

The rnorm() function generates a vector of random normal variables, rnorm()

with first argument n the sample size. Each time we call this function, we will get a different answer. Here we create two correlated sets of numbers, x and y, and use the cor() function to compute the correlation between

cor() them.

> x <- rnorm (50)

> y <- x + rnorm (50, mean = 50, sd = .1)

> cor(x, y)

[1] 0.995

By default, rnorm() creates standard normal random variables with a mean of 0 and a standard deviation of 1. However, the mean and standard devi- ation can be altered using the mean and sd arguments, as illustrated above. Sometimes we want our code to reproduce the exact same set of random numbers; we can use the set.seed() function to do this. The set.seed()

set.seed() function takes an (arbitrary) integer argument.

> set.seed (1303)

> rnorm (50)

[1] -1.1440 1.3421 2.1854 0.5364 0.0632 0.5022 -0.0004

. . .

We use set.seed() throughout the labs whenever we perform calculations involving random quantities. In general this should allow the user to re- produce our results. However, as new versions of R become available, small discrepancies may arise between this book and the output from R. The mean() and var() functions can be used to compute the mean and

mean()

var() variance of a vector of numbers. Applying sqrt() to the output of var() will give the standard deviation. Or we can simply use the sd() function.

sd()

> set.seed (3)

> y <- rnorm (100)

> mean(y)

[1] 0.0110

> var(y)

[1] 0.7329

> sqrt(var(y))

[1] 0.8561

> sd(y)

[1] 0.8561

2.3.2 Graphics

The plot() function is the primary way to plot data in R. For instance, plot()

plot(x, y) produces a scatterplot of the numbers in x versus the numbers in y. There are many additional options that can be passed in to the plot() function. For example, passing in the argument xlab will result in a label on the x-axis. To find out more information about the plot() function, type ?plot.

46 2. Statistical Learning

> x <- rnorm (100)

> y <- rnorm (100)

> plot(x, y)

> plot(x, y, xlab = "this is the x-axis",

ylab = "this is the y-axis",

main = "Plot of X vs Y")

We will often want to save the output of an R plot. The command that we use to do this will depend on the file type that we would like to create. For instance, to create a pdf, we use the pdf() function, and to create a jpeg,

pdf() we use the jpeg() function.

jpeg() > pdf("Figure.pdf")

> plot(x, y, col = "green")

> dev.off ()

null device

The function dev.off() indicates to R that we are done creating the plot. dev.off()

Alternatively, we can simply copy the plot window and paste it into an appropriate file type, such as a Word document. The function seq() can be used to create a sequence of numbers. For

seq() instance, seq(a, b) makes a vector of integers between a and b. There are many other options: for instance, seq(0, 1, length = 10) makes a sequence of 10 numbers that are equally spaced between 0 and 1. Typing 3:11 is a shorthand for seq(3, 11) for integer arguments.

> x <- seq(1, 10)

> x

[1] 1 2 3 4 5 6 7 8 9 10

> x <- 1:10

> x

[1] 1 2 3 4 5 6 7 8 9 10

> x <- seq(-pi , pi , length = 50)

We will now create some more sophisticated plots. The contour() func- contour()

tion produces a contour plot in order to represent three-dimensional data; contour plot

it is like a topographical map. It takes three arguments:

1. A vector of the x values (the first dimension),

2. A vector of the y values (the second dimension), and

3. A matrix whose elements correspond to the z value (the third dimen- sion) for each pair of (x, y) coordinates.

As with the plot() function, there are many other inputs that can be used to fine-tune the output of the contour() function. To learn more about these, take a look at the help file by typing ?contour.

> y <- x

> f <- outer(x, y, function (x, y) cos(y) / (1 + x^2))

> contour(x, y, f)

> contour(x, y, f, nlevels = 45, add = T)

2.3 Lab: Introduction to R 47

> fa <- (f - t(f)) / 2

> contour(x, y, fa , nlevels = 15)

The image() function works the same way as contour(), except that it image()

produces a color-coded plot whose colors depend on the z value. This is known as a heatmap, and is sometimes used to plot temperature in weather

heatmap forecasts. Alternatively, persp() can be used to produce a three-dimensional

persp() plot. The arguments theta and phi control the angles at which the plot is viewed.

> image(x, y, fa)

> persp(x, y, fa)

> persp(x, y, fa , theta = 30)

> persp(x, y, fa , theta = 30, phi = 20)

> persp(x, y, fa , theta = 30, phi = 70)

> persp(x, y, fa , theta = 30, phi = 40)

2.3.3 Indexing Data

We often wish to examine part of a set of data. Suppose that our data is stored in the matrix A.

> A <- matrix (1:16 , 4, 4)

> A

[,1] [,2] [,3] [,4]

[1,] 1 5 9 13

[2,] 2 6 10 14

[3,] 3 7 11 15

[4,] 4 8 12 16

Then, typing

> A[2, 3]

[1] 10

will select the element corresponding to the second row and the third col- umn. The first number after the open-bracket symbol [ always refers to the row, and the second number always refers to the column. We can also select multiple rows and columns at a time, by providing vectors as the indices.

> A[c(1, 3), c(2, 4)]

[,1] [,2]

[1,] 5 13

[2,] 7 15

> A[1:3 , 2:4]

[,1] [,2] [,3]

[1,] 5 9 13

[2,] 6 10 14

[3,] 7 11 15

> A[1:2 , ]

[,1] [,2] [,3] [,4]

[1,] 1 5 9 13

[2,] 2 6 10 14

48 2. Statistical Learning

> A[, 1:2]

[,1] [,2]

[1,] 1 5

[2,] 2 6

[3,] 3 7

[4,] 4 8

The last two examples include either no index for the columns or no index for the rows. These indicate that R should include all columns or all rows, respectively. R treats a single row or column of a matrix as a vector.

> A[1, ]

[1] 1 5 9 13

The use of a negative sign - in the index tells R to keep all rows or columns except those indicated in the index.

> A[-c(1, 3), ]

[,1] [,2] [,3] [,4]

[1,] 2 6 10 14

[2,] 4 8 12 16

> A[-c(1, 3), -c(1, 3, 4)]

[1] 6 8

The dim() function outputs the number of rows followed by the number of dim()

columns of a given matrix.

> dim(A)

[1] 4 4

2.3.4 Loading Data

For most analyses, the first step involves importing a data set into R. The read.table() function is one of the primary ways to do this. The help file

read.table() contains details about how to use this function. We can use the function write.table() to export data.

write.table() Before attempting to load a data set, we must make sure that R knows

to search for the data in the proper directory. For example, on a Windows system one could select the directory using the Change dir. . . option under the File menu. However, the details of how to do this depend on the oper- ating system (e.g. Windows, Mac, Unix) that is being used, and so we do not give further details here. We begin by loading in the Auto data set. This data is part of the ISLR2

library, discussed in Chapter 3. To illustrate the read.table() function, we load it now from a text file, Auto.data, which you can find on the textbook website. The following command will load the Auto.data file into R and store it as an object called Auto, in a format referred to as a data frame.

data frame Once the data has been loaded, the View() function can be used to view

2.3 Lab: Introduction to R 49

it in a spreadsheet-like window.1 The head() function can also be used to view the first few rows of the data.

> Auto <- read.table("Auto.data")

> View(Auto)

> head(Auto)

V1 V2 V3 V4 V5

1 mpg cylinders displacement horsepower weight

2 18.0 8 307.0 130.0 3504.

3 15.0 8 350.0 165.0 3693.

4 18.0 8 318.0 150.0 3436.

5 16.0 8 304.0 150.0 3433.

6 17.0 8 302.0 140.0 3449.

V6 V7 V8 V9

1 acceleration year origin name

2 12.0 70 1 chevrolet chevelle malibu

3 11.5 70 1 buick skylark 320

4 11.0 70 1 plymouth satellite

5 12.0 70 1 amc rebel sst

6 10.5 70 1 ford torino

Note that Auto.data is simply a text file, which you could alternatively open on your computer using a standard text editor. It is often a good idea to view a data set using a text editor or other software such as Excel before loading it into R. This particular data set has not been loaded correctly, because R has

assumed that the variable names are part of the data and so has included them in the first row. The data set also includes a number of missing observations, indicated by a question mark ?. Missing values are a common occurrence in real data sets. Using the option header = T (or header = TRUE) in the read.table() function tells R that the first line of the file contains the variable names, and using the option na.strings tells R that any time it sees a particular character or set of characters (such as a question mark), it should be treated as a missing element of the data matrix.

> Auto <- read.table("Auto.data", header = T, na.strings = "?",

stringsAsFactors = T)

> View(Auto)

The stringsAsFactors = T argument tells R that any variable containing character strings should be interpreted as a qualitative variable, and that each distinct character string represents a distinct level for that qualitative variable. An easy way to load data from Excel into R is to save it as a csv (comma-separated values) file, and then use the read.csv() function.

> Auto <- read.csv("Auto.csv", na.strings = "?",

stringsAsFactors = T)

> View(Auto)

1This function can sometimes be a bit finicky. If you have trouble using it, then try the head() function instead.

50 2. Statistical Learning

> dim(Auto)

[1] 397 9

> Auto [1:4 , ]

The dim() function tells us that the data has 397 observations, or rows, and dim()

nine variables, or columns. There are various ways to deal with the missing data. In this case, only five of the rows contain missing observations, and so we choose to use the na.omit() function to simply remove these rows.

na.omit() > Auto <- na.omit(Auto)

> dim(Auto)

[1] 392 9

Once the data are loaded correctly, we can use names() to check the names()

variable names.

> names(Auto)

[1] "mpg" "cylinders" " displacement " "horsepower"

[5] "weight" " acceleration " "year" "origin"

[9] "name"

2.3.5 Additional Graphical and Numerical Summaries

We can use the plot() function to produce scatterplots of the quantitative scatterplot

variables. However, simply typing the variable names will produce an error message, because R does not know to look in the Auto data set for those variables.

> plot(cylinders , mpg)

Error in plot(cylinders , mpg) : object ‘cylinders ’ not found

To refer to a variable, we must type the data set and the variable name joined with a $ symbol. Alternatively, we can use the attach() function in

attach() order to tell R to make the variables in this data frame available by name.

> plot(Auto$cylinders , Auto$mpg)

> attach(Auto)

> plot(cylinders , mpg)

The cylinders variable is stored as a numeric vector, so R has treated it as quantitative. However, since there are only a small number of possible values for cylinders, one may prefer to treat it as a qualitative variable. The as.factor() function converts quantitative variables into qualitative

as.factor() variables.

> cylinders <- as.factor(cylinders)

If the variable plotted on the x-axis is qualitative, then boxplots will boxplot

automatically be produced by the plot() function. As usual, a number of options can be specified in order to customize the plots.

> plot(cylinders , mpg)

> plot(cylinders , mpg , col = "red")

> plot(cylinders , mpg , col = "red", varwidth = T)

2.3 Lab: Introduction to R 51

> plot(cylinders , mpg , col = "red", varwidth = T,

horizontal = T)

> plot(cylinders , mpg , col = "red", varwidth = T,

xlab = "cylinders ", ylab = "MPG")

The hist() function can be used to plot a histogram. Note that col = 2 hist()

histogram has the same effect as col = "red".

> hist(mpg)

> hist(mpg , col = 2)

> hist(mpg , col = 2, breaks = 15)

The pairs() function creates a scatterplot matrix, i.e. a scatterplot for every pair of variables. We can also produce scatterplots for just a subset of the variables.

> pairs(Auto)

> pairs(

∼ mpg + displacement + horsepower + weight + acceleration , data = Auto

)

In conjunction with the plot() function, identify() provides a useful identify()

interactive method for identifying the value of a particular variable for points on a plot. We pass in three arguments to identify(): the x-axis variable, the y-axis variable, and the variable whose values we would like to see printed for each point. Then clicking one or more points in the plot and hitting Escape will cause R to print the values of the variable of interest. The numbers printed under the identify() function correspond to the rows for the selected points.

> plot(horsepower , mpg)

> identify (horsepower , mpg , name)

The summary() function produces a numerical summary of each variable in summary()

a particular data set.

> summary(Auto)

mpg cylinders displacement

Min. : 9.00 Min. :3.000 Min. : 68.0

1st Qu .:17.00 1st Qu .:4.000 1st Qu .:105.0

Median :22.75 Median :4.000 Median :151.0

Mean :23.45 Mean :5.472 Mean :194.4

3rd Qu .:29.00 3rd Qu .:8.000 3rd Qu .:275.8

Max. :46.60 Max. :8.000 Max. :455.0

horsepower weight acceleration

Min. : 46.0 Min. :1613 Min. : 8.00

1st Qu.: 75.0 1st Qu .:2225 1st Qu .:13.78

Median : 93.5 Median :2804 Median :15.50

Mean :104.5 Mean :2978 Mean :15.54

3rd Qu .:126.0 3rd Qu .:3615 3rd Qu .:17.02

Max. :230.0 Max. :5140 Max. :24.80

year origin name

52 2. Statistical Learning

Min. :70.00 Min. :1.000 amc matador : 5

1st Qu .:73.00 1st Qu .:1.000 ford pinto : 5

Median :76.00 Median :1.000 toyota corolla : 5

Mean :75.98 Mean :1.577 amc gremlin : 4

3rd Qu .:79.00 3rd Qu .:2.000 amc hornet : 4

Max. :82.00 Max. :3.000 chevrolet chevette: 4

(Other) :365

For qualitative variables such as name, R will list the number of observations that fall in each category. We can also produce a summary of just a single variable.

> summary(mpg)

Min. 1st Qu. Median Mean 3rd Qu. Max.

9.00 17.00 22.75 23.45 29.00 46.60

Once we have finished using R, we type q() in order to shut it down, or q()

quit. When exiting R, we have the option to save the current workspace so workspace

that all objects (such as data sets) that we have created in this R session will be available next time. Before exiting R, we may want to save a record of all of the commands that we typed in the most recent session; this can be accomplished using the savehistory() function. Next time we enter R,

savehistory() we can load that history using the loadhistory() function, if we wish.

loadhistory()

2.4 Exercises

Conceptual

1. For each of parts (a) through (d), indicate whether we would generally expect the performance of a flexible statistical learning method to be better or worse than an inflexible method. Justify your answer.

(a) The sample size n is extremely large, and the number of predic- tors p is small.

(b) The number of predictors p is extremely large, and the number of observations n is small.

(d) The variance of the error terms, i.e. σ2 = Var(ϵ), is extremely high.

2. Explain whether each scenario is a classification or regression prob- lem, and indicate whether we are most interested in inference or pre- diction. Finally, provide n and p.

(a) We collect a set of data on the top 500 firms in the US. For each firm we record profit, number of employees, industry and the CEO salary. We are interested in understanding which factors affect CEO salary.

2.4 Exercises 53

(b) We are considering launching a new product and wish to know whether it will be a success or a failure. We collect data on 20 similar products that were previously launched. For each prod- uct we have recorded whether it was a success or failure, price charged for the product, marketing budget, competition price, and ten other variables.

(c) We are interested in predicting the % change in the USD/Euro exchange rate in relation to the weekly changes in the world stock markets. Hence we collect weekly data for all of 2012. For each week we record the % change in the USD/Euro, the % change in the US market, the % change in the British market, and the % change in the German market.

3. We now revisit the bias-variance decomposition.

(a) Provide a sketch of typical (squared) bias, variance, training er- ror, test error, and Bayes (or irreducible) error curves, on a sin- gle plot, as we go from less flexible statistical learning methods towards more flexible approaches. The x-axis should represent the amount of flexibility in the method, and the y-axis should represent the values for each curve. There should be five curves. Make sure to label each one.

(b) Explain why each of the five curves has the shape displayed in part (a).

4. You will now think of some real-life applications for statistical learn- ing.

(a) Describe three real-life applications in which classification might be useful. Describe the response, as well as the predictors. Is the goal of each application inference or prediction? Explain your answer.

(b) Describe three real-life applications in which regression might be useful. Describe the response, as well as the predictors. Is the goal of each application inference or prediction? Explain your answer.

5. What are the advantages and disadvantages of a very flexible (versus a less flexible) approach for regression or classification? Under what circumstances might a more flexible approach be preferred to a less flexible approach? When might a less flexible approach be preferred?

54 2. Statistical Learning

6. Describe the differences between a parametric and a non-parametric statistical learning approach. What are the advantages of a para- metric approach to regression or classification (as opposed to a non- parametric approach)? What are its disadvantages?

7. The table below provides a training data set containing six observa- tions, three predictors, and one qualitative response variable.

Obs. X1 X2 X3 Y 1 0 3 0 Red 2 2 0 0 Red 3 0 1 3 Red 4 0 1 2 Green 5 −1 0 1 Green 6 1 1 1 Red

Suppose we wish to use this data set to make a prediction for Y when X1 = X2 = X3 = 0 using K-nearest neighbors.

(a) Compute the Euclidean distance between each observation and the test point, X1 = X2 = X3 = 0.

(b) What is our prediction with K = 1? Why?

(d) If the Bayes decision boundary in this problem is highly non- linear, then would we expect the best value for K to be large or small? Why?

Applied

8. This exercise relates to the College data set, which can be found in the file College.csv on the book website. It contains a number of variables for 777 different universities and colleges in the US. The variables are

• Private : Public/private indicator • Apps : Number of applications received • Accept : Number of applicants accepted • Enroll : Number of new students enrolled • Top10perc : New students from top 10 % of high school class • Top25perc : New students from top 25 % of high school class • F.Undergrad : Number of full-time undergraduates • P.Undergrad : Number of part-time undergraduates

2.4 Exercises 55

• Outstate : Out-of-state tuition • Room.Board : Room and board costs • Books : Estimated book costs • Personal : Estimated personal spending • PhD : Percent of faculty with Ph.D.’s • Terminal : Percent of faculty with terminal degree • S.F.Ratio : Student/faculty ratio • perc.alumni : Percent of alumni who donate • Expend : Instructional expenditure per student • Grad.Rate : Graduation rate

Before reading the data into R, it can be viewed in Excel or a text editor.

(a) Use the read.csv() function to read the data into R. Call the loaded data college. Make sure that you have the directory set to the correct location for the data.

(b) Look at the data using the View() function. You should notice that the first column is just the name of each university. We don’t really want R to treat this as data. However, it may be handy to have these names for later. Try the following commands:

> rownames (college) <- college[, 1]

> View(college)

You should see that there is now a row.names column with the name of each university recorded. This means that R has given each row a name corresponding to the appropriate university. R will not try to perform calculations on the row names. However, we still need to eliminate the first column in the data where the names are stored. Try

> college <- college[, -1]

> View(college)

Now you should see that the first data column is Private. Note that another column labeled row.names now appears before the Private column. However, this is not a data column but rather the name that R is giving to each row.

ii. Use the pairs() function to produce a scatterplot matrix of the first ten columns or variables of the data. Recall that you can reference the first ten columns of a matrix A using A[,1:10].

56 2. Statistical Learning

iii. Use the plot() function to produce side-by-side boxplots of Outstate versus Private.

iv. Create a new qualitative variable, called Elite, by binning the Top10perc variable. We are going to divide universities into two groups based on whether or not the proportion of students coming from the top 10 % of their high school classes exceeds 50 %.

> Elite <- rep("No", nrow(college))

> Elite[college$Top10perc > 50] <- "Yes"

> Elite <- as.factor(Elite)

> college <- data.frame(college , Elite)

Use the summary() function to see how many elite univer- sities there are. Now use the plot() function to produce side-by-side boxplots of Outstate versus Elite.

v. Use the hist() function to produce some histograms with differing numbers of bins for a few of the quantitative vari- ables. You may find the command par(mfrow = c(2, 2)) useful: it will divide the print window into four regions so that four plots can be made simultaneously. Modifying the arguments to this function will divide the screen in other ways.

vi. Continue exploring the data, and provide a brief summary of what you discover.

9. This exercise involves the Auto data set studied in the lab. Make sure that the missing values have been removed from the data.

(a) Which of the predictors are quantitative, and which are quali- tative?

(b) What is the range of each quantitative predictor? You can an- swer this using the range() function.

range() (c) What is the mean and standard deviation of each quantitative

predictor?

(d) Now remove the 10th through 85th observations. What is the range, mean, and standard deviation of each predictor in the subset of the data that remains?

(e) Using the full data set, investigate the predictors graphically, using scatterplots or other tools of your choice. Create some plots highlighting the relationships among the predictors. Comment on your findings.

(f) Suppose that we wish to predict gas mileage (mpg) on the basis of the other variables. Do your plots suggest that any of the other variables might be useful in predicting mpg? Justify your answer.

2.4 Exercises 57

10. This exercise involves the Boston housing data set.

(a) To begin, load in the Boston data set. The Boston data set is part of the ISLR2 library.

> library(ISLR2)

Now the data set is contained in the object Boston.

> Boston

Read about the data set:

> ?Boston

How many rows are in this data set? How many columns? What do the rows and columns represent?

(b) Make some pairwise scatterplots of the predictors (columns) in this data set. Describe your findings.

(d) Do any of the census tracts of Boston appear to have particularly high crime rates? Tax rates? Pupil-teacher ratios? Comment on the range of each predictor.

(e) How many of the census tracts in this data set bound the Charles river?

(f) What is the median pupil-teacher ratio among the towns in this data set?

(g) Which census tract of Boston has lowest median value of owner- occupied homes? What are the values of the other predictors for that census tract, and how do those values compare to the overall ranges for those predictors? Comment on your findings.

(h) In this data set, how many of the census tracts average more than seven rooms per dwelling? More than eight rooms per dwelling? Comment on the census tracts that average more than eight rooms per dwelling.

3 Linear Regression

This chapter is about linear regression, a very simple approach for super- vised learning. In particular, linear regression is a useful tool for predicting a quantitative response. It has been around for a long time and is the topic of innumerable textbooks. Though it may seem somewhat dull compared to some of the more modern statistical learning approaches described in later chapters of this book, linear regression is still a useful and widely used sta- tistical learning method. Moreover, it serves as a good jumping-off point for newer approaches: as we will see in later chapters, many fancy statistical learning approaches can be seen as generalizations or extensions of linear regression. Consequently, the importance of having a good understanding of linear regression before studying more complex learning methods cannot be overstated. In this chapter, we review some of the key ideas underlying the linear regression model, as well as the least squares approach that is most commonly used to fit this model. Recall the Advertising data from Chapter 2. Figure 2.1 displays sales

(in thousands of units) for a particular product as a function of advertis- ing budgets (in thousands of dollars) for TV, radio, and newspaper media. Suppose that in our role as statistical consultants we are asked to suggest, on the basis of this data, a marketing plan for next year that will result in high product sales. What information would be useful in order to provide such a recommendation? Here are a few important questions that we might seek to address:

1. Is there a relationship between advertising budget and sales? Our first goal should be to determine whether the data provide evi-

https://doi.org/10.1007/978-1-0716-1418-1_3

http://crossmark.crossref.org/dialog/?doi=10.1007/978-1-0716-1418-1_3&domain=pdf

https://doi.org/10.1007/978-1-0716-1418-1_3

60 3. Linear Regression

dence of an association between advertising expenditure and sales. If the evidence is weak, then one might argue that no money should be spent on advertising!

2. How strong is the relationship between advertising budget and sales? Assuming that there is a relationship between advertising and sales, we would like to know the strength of this relationship. Does knowl- edge of the advertising budget provide a lot of information about product sales?

3. Which media are associated with sales? Are all three media—TV, radio, and newspaper—associated with sales, or are just one or two of the media associated? To answer this question, we must find a way to separate out the individual contribu- tion of each medium to sales when we have spent money on all three media.

4. How large is the association between each medium and sales? For every dollar spent on advertising in a particular medium, by what amount will sales increase? How accurately can we predict this amount of increase?

5. How accurately can we predict future sales? For any given level of television, radio, or newspaper advertising, what is our prediction for sales, and what is the accuracy of this prediction?

6. Is the relationship linear? If there is approximately a straight-line relationship between advertis- ing expenditure in the various media and sales, then linear regression is an appropriate tool. If not, then it may still be possible to trans- form the predictor or the response so that linear regression can be used.

7. Is there synergy among the advertising media? Perhaps spending $50,000 on television advertising and $50,000 on ra- dio advertising is associated with higher sales than allocating $100,000 to either television or radio individually. In marketing, this is known as a synergy effect, while in statistics it is called an interaction effect. synergy

interaction It turns out that linear regression can be used to answer each of these questions. We will first discuss all of these questions in a general context, and then return to them in this specific context in Section 3.4.

3.1 Simple Linear Regression

Simple linear regression lives up to its name: it is a very straightforward simple linear regression

3.1 Simple Linear Regression 61

approach for predicting a quantitative response Y on the basis of a sin- gle predictor variable X. It assumes that there is approximately a linear relationship between X and Y . Mathematically, we can write this linear relationship as

Y ≈ β0 + β1X. (3.1)

You might read “≈” as “is approximately modeled as”. We will sometimes describe (3.1) by saying that we are regressing Y on X (or Y onto X). For example, X may represent TV advertising and Y may represent sales. Then we can regress sales onto TV by fitting the model

sales ≈ β0 + β1 ×TV.

In Equation 3.1, β0 and β1 are two unknown constants that represent the intercept and slope terms in the linear model. Together, β0 and β1 are

intercept

slope known as the model coefficients or parameters. Once we have used our

coefficient parameter

training data to produce estimates β̂0 and β̂1 for the model coefficients, we can predict future sales on the basis of a particular value of TV advertising by computing

ŷ = β̂0 + β̂1x, (3.2)

where ŷ indicates a prediction of Y on the basis of X = x. Here we use a hat symbol, ˆ , to denote the estimated value for an unknown parameter or coefficient, or to denote the predicted value of the response.

3.1.1 Estimating the Coefficients

In practice, β0 and β1 are unknown. So before we can use (3.1) to make predictions, we must use data to estimate the coefficients. Let

(x1, y1), (x2, y2), . . . , (xn, yn)

represent n observation pairs, each of which consists of a measurement of X and a measurement of Y . In the Advertising example, this data set consists of the TV advertising budget and product sales in n = 200 different markets. (Recall that the data are displayed in Figure 2.1.) Our goal is to obtain coefficient estimates β̂0 and β̂1 such that the linear model (3.1) fits the available data well—that is, so that yi ≈ β̂0 + β̂1xi for i = 1, . . . , n. In other words, we want to find an intercept β̂0 and a slope β̂1 such that the resulting line is as close as possible to the n = 200 data points. There are a number of ways of measuring closeness. However, by far the most common approach involves minimizing the least squares criterion,

least squares and we take that approach in this chapter. Alternative approaches will be considered in Chapter 6. Let ŷi = β̂0 + β̂1xi be the prediction for Y based on the ith value of X.

Then ei = yi− ŷi represents the ith residual—this is the difference between residual

62 3. Linear Regression

0 50 100 150 200 250 300

5 1

0 1

5 2

0 2

S a

le s

FIGURE 3.1. For the Advertising data, the least squares fit for the regression of sales onto TV is shown. The fit is found by minimizing the residual sum of squares. Each grey line segment represents a residual. In this case a linear fit captures the essence of the relationship, although it overestimates the trend in the left of the plot.

the ith observed response value and the ith response value that is predicted by our linear model. We define the residual sum of squares (RSS) as

residual sum of squaresRSS = e21 + e

2 2 + · · · + e

2 n,

or equivalently as

RSS = (y1−β̂0−β̂1x1)2 +(y2−β̂0−β̂1x2)2 +· · ·+(yn−β̂0−β̂1xn)2. (3.3)

The least squares approach chooses β̂0 and β̂1 to minimize the RSS. Using some calculus, one can show that the minimizers are

β̂1 =

∑n i=1(xi − x̄)(yi − ȳ)∑n

i=1(xi − x̄)2 ,

β̂0 = ȳ − β̂1x̄, (3.4)

where ȳ ≡ 1 n

∑n i=1 yi and x̄ ≡

1 n

∑n i=1 xi are the sample means. In other

words, (3.4) defines the least squares coefficient estimates for simple linear regression. Figure 3.1 displays the simple linear regression fit to the Advertising

data, where β̂0 = 7.03 and β̂1 = 0.0475. In other words, according to this approximation, an additional $1,000 spent on TV advertising is asso- ciated with selling approximately 47.5 additional units of the product. In

3.1 Simple Linear Regression 63

β0

β 1

2.11

2.15

2.2

2.3

2.5

5 6 7 8 9

0 .0

3 0 .0

4 0 .0

5 0 .0

●

R S

β1β0

FIGURE 3.2. Contour and three-dimensional plots of the RSS on the Advertising data, using sales as the response and TV as the predictor. The red dots correspond to the least squares estimates β̂0 and β̂1, given by (3.4).

Figure 3.2, we have computed RSS for a number of values of β0 and β1, using the advertising data with sales as the response and TV as the predic- tor. In each plot, the red dot represents the pair of least squares estimates (β̂0, β̂1) given by (3.4). These values clearly minimize the RSS.

3.1.2 Assessing the Accuracy of the Coefficient Estimates

Recall from (2.1) that we assume that the true relationship between X and Y takes the form Y = f(X) + ϵ for some unknown function f, where ϵ is a mean-zero random error term. If f is to be approximated by a linear function, then we can write this relationship as

Y = β0 + β1X + ϵ. (3.5)

Here β0 is the intercept term—that is, the expected value of Y when X = 0, and β1 is the slope—the average increase in Y associated with a one-unit increase in X. The error term is a catch-all for what we miss with this simple model: the true relationship is probably not linear, there may be other variables that cause variation in Y , and there may be measurement error. We typically assume that the error term is independent of X. The model given by (3.5) defines the population regression line, which

population regression line

is the best linear approximation to the true relationship between X and Y .1 The least squares regression coefficient estimates (3.4) characterize the least squares line (3.2). The left-hand panel of Figure 3.3 displays these

least squares line

1The assumption of linearity is often a useful working model. However, despite what many textbooks might tell us, we seldom believe that the true relationship is linear.

64 3. Linear Regression

−2 −1 0 1 2

− 1

0 −

5 0

5 1

−2 −1 0 1 2

− 1

0 −

5 0

5 1

0 X

FIGURE 3.3. A simulated data set. Left: The red line represents the true rela- tionship, f(X) = 2 + 3X, which is known as the population regression line. The blue line is the least squares line; it is the least squares estimate for f(X) based on the observed data, shown in black. Right: The population regression line is again shown in red, and the least squares line in dark blue. In light blue, ten least squares lines are shown, each computed on the basis of a separate random set of observations. Each least squares line is different, but on average, the least squares lines are quite close to the population regression line.

two lines in a simple simulated example. We created 100 random Xs, and generated 100 corresponding Y s from the model

Y = 2 + 3X + ϵ, (3.6)

where ϵ was generated from a normal distribution with mean zero. The red line in the left-hand panel of Figure 3.3 displays the true relationship, f(X) = 2 + 3X, while the blue line is the least squares estimate based on the observed data. The true relationship is generally not known for real data, but the least squares line can always be computed using the coefficient estimates given in (3.4). In other words, in real applications, we have access to a set of observations from which we can compute the least squares line; however, the population regression line is unobserved. In the right-hand panel of Figure 3.3 we have generated ten different data sets from the model given by (3.6) and plotted the corresponding ten least squares lines. Notice that different data sets generated from the same true model result in slightly different least squares lines, but the unobserved population regression line does not change. At first glance, the difference between the population regression line and

the least squares line may seem subtle and confusing. We only have one data set, and so what does it mean that two different lines describe the relationship between the predictor and the response? Fundamentally, the concept of these two lines is a natural extension of the standard statistical

3.1 Simple Linear Regression 65

approach of using information from a sample to estimate characteristics of a large population. For example, suppose that we are interested in knowing the population mean µ of some random variable Y . Unfortunately, µ is unknown, but we do have access to n observations from Y , y1, . . . , yn, which we can use to estimate µ. A reasonable estimate is µ̂ = ȳ, where ȳ = 1

∑n i=1 yi is the sample mean. The sample mean and the population

mean are different, but in general the sample mean will provide a good estimate of the population mean. In the same way, the unknown coefficients β0 and β1 in linear regression define the population regression line. We seek to estimate these unknown coefficients using β̂0 and β̂1 given in (3.4). These coefficient estimates define the least squares line. The analogy between linear regression and estimation of the mean of a

random variable is an apt one based on the concept of bias. If we use the bias

sample mean µ̂ to estimate µ, this estimate is unbiased, in the sense that unbiased

on average, we expect µ̂ to equal µ. What exactly does this mean? It means that on the basis of one particular set of observations y1, . . . , yn, µ̂ might overestimate µ, and on the basis of another set of observations, µ̂ might underestimate µ. But if we could average a huge number of estimates of µ obtained from a huge number of sets of observations, then this average would exactly equal µ. Hence, an unbiased estimator does not systematically over- or under-estimate the true parameter. The property of unbiasedness holds for the least squares coefficient estimates given by (3.4) as well: if we estimate β0 and β1 on the basis of a particular data set, then our estimates won’t be exactly equal to β0 and β1. But if we could average the estimates obtained over a huge number of data sets, then the average of these estimates would be spot on! In fact, we can see from the right- hand panel of Figure 3.3 that the average of many least squares lines, each estimated from a separate data set, is pretty close to the true population regression line. We continue the analogy with the estimation of the population mean

µ of a random variable Y . A natural question is as follows: how accurate is the sample mean µ̂ as an estimate of µ? We have established that the average of µ̂’s over many data sets will be very close to µ, but that a single estimate µ̂ may be a substantial underestimate or overestimate of µ. How far off will that single estimate of µ̂ be? In general, we answer this question by computing the standard error of µ̂, written as SE(µ̂). We have

standard errorthe well-known formula

Var(µ̂) = SE(µ̂) 2 =

σ2

n , (3.7)

where σ is the standard deviation of each of the realizations yi of Y . 2

Roughly speaking, the standard error tells us the average amount that this estimate µ̂ differs from the actual value of µ. Equation 3.7 also tells us how

2This formula holds provided that the n observations are uncorrelated.

66 3. Linear Regression

this deviation shrinks with n—the more observations we have, the smaller the standard error of µ̂. In a similar vein, we can wonder how close β̂0 and β̂1 are to the true values β0 and β1. To compute the standard errors associated with β̂0 and β̂1, we use the following formulas:

SE(β̂0) 2 = σ2

[ 1

n +

x̄2 ∑n

i=1(xi − x̄)2

] , SE(β̂1)

2 =

σ2 ∑n

i=1(xi − x̄)2 , (3.8)

where σ2 = Var(ϵ). For these formulas to be strictly valid, we need to assume that the errors ϵi for each observation have common variance σ

2 and are uncorrelated. This is clearly not true in Figure 3.1, but the formula still turns out to be a good approximation. Notice in the formula that SE(β̂1) is smaller when the xi are more spread out; intuitively we have more leverage to estimate a slope when this is the case. We also see that SE(β̂0) would be the same as SE(µ̂) if x̄ were zero (in which case β̂0 would be equal to ȳ). In general, σ2 is not known, but can be estimated from the data. This estimate of σ is known as the residual standard error, and is given by the formula

residual standard error

RSE = √ RSS/(n−2). Strictly speaking, when σ2 is estimated from the

data we should write ŜE(β̂1) to indicate that an estimate has been made, but for simplicity of notation we will drop this extra “hat”. Standard errors can be used to compute confidence intervals. A 95 %

confidence intervalconfidence interval is defined as a range of values such that with 95 %

probability, the range will contain the true unknown value of the param- eter. The range is defined in terms of lower and upper limits computed from the sample of data. A 95% confidence interval has the following prop- erty: if we take repeated samples and construct the confidence interval for each sample, 95% of the intervals will contain the true unknown value of the parameter. For linear regression, the 95 % confidence interval for β1 approximately takes the form

β̂1 ± 2 · SE(β̂1). (3.9)

That is, there is approximately a 95 % chance that the interval [ β̂1 −2 · SE(β̂1), β̂1 + 2 · SE(β̂1)

] (3.10)

will contain the true value of β1. 3 Similarly, a confidence interval for β0

approximately takes the form

β̂0 ± 2 · SE(β̂0). (3.11)

3Approximately for several reasons. Equation 3.10 relies on the assumption that the errors are Gaussian. Also, the factor of 2 in front of the SE(β̂1) term will vary slightly depending on the number of observations n in the linear regression. To be precise, rather than the number 2, (3.10) should contain the 97.5 % quantile of a t-distribution with n−2 degrees of freedom. Details of how to compute the 95 % confidence interval precisely in R will be provided later in this chapter.

3.1 Simple Linear Regression 67

In the case of the advertising data, the 95 % confidence interval for β0 is [6.130, 7.935] and the 95 % confidence interval for β1 is [0.042, 0.053]. Therefore, we can conclude that in the absence of any advertising, sales will, on average, fall somewhere between 6,130 and 7,935 units. Furthermore, for each $1,000 increase in television advertising, there will be an average increase in sales of between 42 and 53 units. Standard errors can also be used to perform hypothesis tests on the

hypothesis testcoefficients. The most common hypothesis test involves testing the null

hypothesis of null hypothesis

H0 : There is no relationship between X and Y (3.12)

versus the alternative hypothesis alternative hypothesis

Ha : There is some relationship between X and Y . (3.13)

Mathematically, this corresponds to testing

H0 : β1 = 0

versus Ha : β1 ̸= 0,

since if β1 = 0 then the model (3.5) reduces to Y = β0 + ϵ, and X is not associated with Y . To test the null hypothesis, we need to determine whether β̂1, our estimate for β1, is sufficiently far from zero that we can be confident that β1 is non-zero. How far is far enough? This of course depends on the accuracy of β̂1—that is, it depends on SE(β̂1). If SE(β̂1) is small, then even relatively small values of β̂1 may provide strong evidence that β1 ̸= 0, and hence that there is a relationship between X and Y . In contrast, if SE(β̂1) is large, then β̂1 must be large in absolute value in order for us to reject the null hypothesis. In practice, we compute a t-statistic,

t-statistic given by

t = β̂1 −0 SE(β̂1)

, (3.14)

which measures the number of standard deviations that β̂1 is away from 0. If there really is no relationship between X and Y , then we expect that (3.14) will have a t-distribution with n−2 degrees of freedom. The t-distribution has a bell shape and for values of n greater than approximately 30 it is quite similar to the standard normal distribution. Consequently, it is a simple matter to compute the probability of observing any number equal to |t| or larger in absolute value, assuming β1 = 0. We call this probability the p-value. Roughly speaking, we interpret the p-value as follows: a small

p-value p-value indicates that it is unlikely to observe such a substantial association between the predictor and the response due to chance, in the absence of any real association between the predictor and the response. Hence, if we

68 3. Linear Regression

Coefficient Std. error t-statistic p-value Intercept 7.0325 0.4578 15.36 < 0.0001 TV 0.0475 0.0027 17.67 < 0.0001

TABLE 3.1. For the Advertising data, coefficients of the least squares model for the regression of number of units sold on TV advertising budget. An increase of $1,000 in the TV advertising budget is associated with an increase in sales by around 50 units. (Recall that the sales variable is in thousands of units, and the TV variable is in thousands of dollars.)

see a small p-value, then we can infer that there is an association between the predictor and the response. We reject the null hypothesis—that is, we declare a relationship to exist between X and Y —if the p-value is small enough. Typical p-value cutoffs for rejecting the null hypothesis are 5% or 1%, although this topic will be explored in much greater detail in Chap- ter 13. When n = 30, these correspond to t-statistics (3.14) of around 2 and 2.75, respectively. Table 3.1 provides details of the least squares model for the regression of

number of units sold on TV advertising budget for the Advertising data. Notice that the coefficients for β̂0 and β̂1 are very large relative to their standard errors, so the t-statistics are also large; the probabilities of seeing such values if H0 is true are virtually zero. Hence we can conclude that β0 ̸= 0 and β1 ̸= 0.4

3.1.3 Assessing the Accuracy of the Model

Once we have rejected the null hypothesis (3.12) in favor of the alternative hypothesis (3.13), it is natural to want to quantify the extent to which the model fits the data. The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the R2

R2 statistic. Table 3.2 displays the RSE, the R2 statistic, and the F-statistic (to be

described in Section 3.2.2) for the linear regression of number of units sold on TV advertising budget.

Residual Standard Error

Recall from the model (3.5) that associated with each observation is an error term ϵ. Due to the presence of these error terms, even if we knew the true regression line (i.e. even if β0 and β1 were known), we would not be

4In Table 3.1, a small p-value for the intercept indicates that we can reject the null hypothesis that β0 = 0, and a small p-value for TV indicates that we can reject the null hypothesis that β1 = 0. Rejecting the latter null hypothesis allows us to conclude that there is a relationship between TV and sales. Rejecting the former allows us to conclude that in the absence of TV expenditure, sales are non-zero.

3.1 Simple Linear Regression 69

Quantity Value Residual standard error 3.26 R2 0.612 F-statistic 312.1

TABLE 3.2. For the Advertising data, more information about the least squares model for the regression of number of units sold on TV advertising budget.

able to perfectly predict Y from X. The RSE is an estimate of the standard deviation of ϵ. Roughly speaking, it is the average amount that the response will deviate from the true regression line. It is computed using the formula

RSE =

√ 1

n−2 RSS =

√√√√ 1 n−2

n∑

i=1

(yi − ŷi)2. (3.15)

Note that RSS was defined in Section 3.1.1, and is given by the formula

RSS = n∑

i=1

(yi − ŷi)2. (3.16)

In the case of the advertising data, we see from the linear regression output in Table 3.2 that the RSE is 3.26. In other words, actual sales in each market deviate from the true regression line by approximately 3,260 units, on average. Another way to think about this is that even if the model were correct and the true values of the unknown coefficients β0 and β1 were known exactly, any prediction of sales on the basis of TV advertising would still be off by about 3,260 units on average. Of course, whether or not 3,260 units is an acceptable prediction error depends on the problem context. In the advertising data set, the mean value of sales over all markets is approximately 14,000 units, and so the percentage error is 3,260/14,000 = 23 %. The RSE is considered a measure of the lack of fit of the model (3.5) to

the data. If the predictions obtained using the model are very close to the true outcome values—that is, if ŷi ≈ yi for i = 1, . . . , n—then (3.15) will be small, and we can conclude that the model fits the data very well. On the other hand, if ŷi is very far from yi for one or more observations, then the RSE may be quite large, indicating that the model doesn’t fit the data well.

R2 Statistic

The RSE provides an absolute measure of lack of fit of the model (3.5) to the data. But since it is measured in the units of Y , it is not always clear what constitutes a good RSE. The R2 statistic provides an alternative measure of fit. It takes the form of a proportion—the proportion of variance

70 3. Linear Regression

explained—and so it always takes on a value between 0 and 1, and is independent of the scale of Y . To calculate R2, we use the formula

R2 = TSS−RSS

TSS = 1−

RSS

TSS (3.17)

where TSS = ∑

(yi − ȳ)2 is the total sum of squares, and RSS is defined total sum of squaresin (3.16). TSS measures the total variance in the response Y , and can be

thought of as the amount of variability inherent in the response before the regression is performed. In contrast, RSS measures the amount of variability that is left unexplained after performing the regression. Hence, TSS−RSS measures the amount of variability in the response that is explained (or removed) by performing the regression, and R2 measures the proportion of variability in Y that can be explained using X. An R2 statistic that is close to 1 indicates that a large proportion of the variability in the response is explained by the regression. A number near 0 indicates that the regression does not explain much of the variability in the response; this might occur because the linear model is wrong, or the error variance σ2 is high, or both. In Table 3.2, the R2 was 0.61, and so just under two-thirds of the variability in sales is explained by a linear regression on TV. The R2 statistic (3.17) has an interpretational advantage over the RSE

(3.15), since unlike the RSE, it always lies between 0 and 1. However, it can still be challenging to determine what is a good R2 value, and in general, this will depend on the application. For instance, in certain problems in physics, we may know that the data truly comes from a linear model with a small residual error. In this case, we would expect to see an R2 value that is extremely close to 1, and a substantially smaller R2 value might indicate a serious problem with the experiment in which the data were generated. On the other hand, in typical applications in biology, psychology, marketing, and other domains, the linear model (3.5) is at best an extremely rough approximation to the data, and residual errors due to other unmeasured factors are often very large. In this setting, we would expect only a very small proportion of the variance in the response to be explained by the predictor, and an R2 value well below 0.1 might be more realistic! The R2 statistic is a measure of the linear relationship between X and

Y . Recall that correlation, defined as correlation

Cor(X, Y ) =

∑n i=1(xi −x)(yi −y)√∑n

i=1(xi −x)2 √∑n

i=1(yi −y)2 , (3.18)

is also a measure of the linear relationship between X and Y .5 This sug- gests that we might be able to use r = Cor(X, Y ) instead of R2 in order to

5We note that in fact, the right-hand side of (3.18) is the sample correlation; thus,

it would be more correct to write !Cor(X, Y ); however, we omit the “hat” for ease of notation.

3.2 Multiple Linear Regression 71

assess the fit of the linear model. In fact, it can be shown that in the simple linear regression setting, R2 = r2. In other words, the squared correlation and the R2 statistic are identical. However, in the next section we will discuss the multiple linear regression problem, in which we use several pre- dictors simultaneously to predict the response. The concept of correlation between the predictors and the response does not extend automatically to this setting, since correlation quantifies the association between a single pair of variables rather than between a larger number of variables. We will see that R2 fills this role.

3.2 Multiple Linear Regression

Simple linear regression is a useful approach for predicting a response on the basis of a single predictor variable. However, in practice we often have more than one predictor. For example, in the Advertising data, we have examined the relationship between sales and TV advertising. We also have data for the amount of money spent advertising on the radio and in newspapers, and we may want to know whether either of these two media is associated with sales. How can we extend our analysis of the advertising data in order to accommodate these two additional predictors? One option is to run three separate simple linear regressions, each of

which uses a different advertising medium as a predictor. For instance, we can fit a simple linear regression to predict sales on the basis of the amount spent on radio advertisements. Results are shown in Table 3.3 (top table). We find that a $1,000 increase in spending on radio advertising is associated with an increase in sales of around 203 units. Table 3.3 (bottom table) contains the least squares coefficients for a simple linear regression of sales onto newspaper advertising budget. A $1,000 increase in newspaper advertising budget is associated with an increase in sales of approximately 55 units. However, the approach of fitting a separate simple linear regression model

for each predictor is not entirely satisfactory. First of all, it is unclear how to make a single prediction of sales given the three advertising media budgets, since each of the budgets is associated with a separate regression equation. Second, each of the three regression equations ignores the other two media in forming estimates for the regression coefficients. We will see shortly that if the media budgets are correlated with each other in the 200 markets in our data set, then this can lead to very misleading estimates of the association between each media budget and sales. Instead of fitting a separate simple linear regression model for each pre-

dictor, a better approach is to extend the simple linear regression model (3.5) so that it can directly accommodate multiple predictors. We can do this by giving each predictor a separate slope coefficient in a single model. In general, suppose that we have p distinct predictors. Then the multiple

72 3. Linear Regression

Simple regression of sales on radio Coefficient Std. error t-statistic p-value

Intercept 9.312 0.563 16.54 < 0.0001 radio 0.203 0.020 9.92 < 0.0001

Simple regression of sales on newspaper

Coefficient Std. error t-statistic p-value

Intercept 12.351 0.621 19.88 < 0.0001 newspaper 0.055 0.017 3.30 0.00115

TABLE 3.3. More simple linear regression models for the Advertising data. Co- efficients of the simple linear regression model for number of units sold on Top: radio advertising budget and Bottom: newspaper advertising budget. A $1,000 in- crease in spending on radio advertising is associated with an average increase in sales by around 203 units, while the same increase in spending on newspaper ad- vertising is associated with an average increase in sales by around 55 units. (Note that the sales variable is in thousands of units, and the radio and newspaper variables are in thousands of dollars.)

linear regression model takes the form

Y = β0 + β1X1 + β2X2 + · · · + βpXp + ϵ, (3.19)

where Xj represents the jth predictor and βj quantifies the association between that variable and the response. We interpret βj as the average effect on Y of a one unit increase in Xj, holding all other predictors fixed. In the advertising example, (3.19) becomes

sales = β0 + β1 × TV + β2 × radio + β3 × newspaper + ϵ. (3.20)

3.2.1 Estimating the Regression Coefficients

As was the case in the simple linear regression setting, the regression coef- ficients β0, β1, . . . , βp in (3.19) are unknown, and must be estimated. Given

estimates β̂0, β̂1, . . . , β̂p, we can make predictions using the formula

ŷ = β̂0 + β̂1x1 + β̂2x2 + · · · + β̂pxp. (3.21)

The parameters are estimated using the same least squares approach that we saw in the context of simple linear regression. We choose β0, β1, . . . , βp to minimize the sum of squared residuals

RSS = n∑

i=1

(yi − ŷi)2

= n∑

i=1

(yi − β̂0 − β̂1xi1 − β̂2xi2 − · · ·− β̂pxip)2. (3.22)

3.2 Multiple Linear Regression 73

FIGURE 3.4. In a three-dimensional setting, with two predictors and one re- sponse, the least squares regression line becomes a plane. The plane is chosen to minimize the sum of the squared vertical distances between each observation (shown in red) and the plane.

The values β̂0, β̂1, . . . , β̂p that minimize (3.22) are the multiple least squares regression coefficient estimates. Unlike the simple linear regression esti- mates given in (3.4), the multiple regression coefficient estimates have somewhat complicated forms that are most easily represented using ma- trix algebra. For this reason, we do not provide them here. Any statistical software package can be used to compute these coefficient estimates, and later in this chapter we will show how this can be done in R. Figure 3.4 illustrates an example of the least squares fit to a toy data set with p = 2 predictors. Table 3.4 displays the multiple regression coefficient estimates when TV,

radio, and newspaper advertising budgets are used to predict product sales using the Advertising data. We interpret these results as follows: for a given amount of TV and newspaper advertising, spending an additional $1,000 on radio advertising is associated with approximately 189 units of additional sales. Comparing these coefficient estimates to those displayed in Tables 3.1 and 3.3, we notice that the multiple regression coefficient estimates for TV and radio are pretty similar to the simple linear regression coefficient estimates. However, while the newspaper regression coefficient estimate in Table 3.3 was significantly non-zero, the coefficient estimate for newspaper

74 3. Linear Regression

in the multiple regression model is close to zero, and the corresponding p- value is no longer significant, with a value around 0.86. This illustrates that the simple and multiple regression coefficients can be quite different. This difference stems from the fact that in the simple regression case, the slope term represents the average increase in product sales associated with a $1,000 increase in newspaper advertising, ignoring other predictors such as TV and radio. By contrast, in the multiple regression setting, the coefficient for newspaper represents the average increase in product sales associated with increasing newspaper spending by $1,000 while holding TV and radio fixed.

Coefficient Std. error t-statistic p-value Intercept 2.939 0.3119 9.42 < 0.0001 TV 0.046 0.0014 32.81 < 0.0001 radio 0.189 0.0086 21.89 < 0.0001 newspaper −0.001 0.0059 −0.18 0.8599

TABLE 3.4. For the Advertising data, least squares coefficient estimates of the multiple linear regression of number of units sold on TV, radio, and newspaper advertising budgets.

Does it make sense for the multiple regression to suggest no relationship between sales and newspaper while the simple linear regression implies the opposite? In fact it does. Consider the correlation matrix for the three predictor variables and response variable, displayed in Table 3.5. Notice that the correlation between radio and newspaper is 0.35. This indicates that markets with high newspaper advertising tend to also have high ra- dio advertising. Now suppose that the multiple regression is correct and newspaper advertising is not associated with sales, but radio advertising is associated with sales. Then in markets where we spend more on radio our sales will tend to be higher, and as our correlation matrix shows, we also tend to spend more on newspaper advertising in those same mar- kets. Hence, in a simple linear regression which only examines sales versus newspaper, we will observe that higher values of newspaper tend to be as- sociated with higher values of sales, even though newspaper advertising is not directly associated with sales. So newspaper advertising is a surrogate for radio advertising; newspaper gets “credit” for the association between radio on sales. This slightly counterintuitive result is very common in many real life

situations. Consider an absurd example to illustrate the point. Running a regression of shark attacks versus ice cream sales for data collected at a given beach community over a period of time would show a positive relationship, similar to that seen between sales and newspaper. Of course no one has (yet) suggested that ice creams should be banned at beaches to reduce shark attacks. In reality, higher temperatures cause more people

3.2 Multiple Linear Regression 75

TV radio newspaper sales

TV 1.0000 0.0548 0.0567 0.7822 radio 1.0000 0.3541 0.5762 newspaper 1.0000 0.2283 sales 1.0000

TABLE 3.5. Correlation matrix for TV, radio, newspaper, and sales for the Advertising data.

to visit the beach, which in turn results in more ice cream sales and more shark attacks. A multiple regression of shark attacks onto ice cream sales and temperature reveals that, as intuition implies, ice cream sales is no longer a significant predictor after adjusting for temperature.

3.2.2 Some Important Questions

When we perform multiple linear regression, we usually are interested in answering a few important questions.

1. Is at least one of the predictors X1, X2, . . . , Xp useful in predicting the response?

2. Do all the predictors help to explain Y , or is only a subset of the predictors useful?

3. How well does the model fit the data?

4. Given a set of predictor values, what response value should we predict, and how accurate is our prediction?

We now address each of these questions in turn.

One: Is There a Relationship Between the Response and Predictors?

Recall that in the simple linear regression setting, in order to determine whether there is a relationship between the response and the predictor we can simply check whether β1 = 0. In the multiple regression setting with p predictors, we need to ask whether all of the regression coefficients are zero, i.e. whether β1 = β2 = · · · = βp = 0. As in the simple linear regression setting, we use a hypothesis test to answer this question. We test the null hypothesis,

H0 : β1 = β2 = · · · = βp = 0

versus the alternative

Ha : at least one βj is non-zero.

This hypothesis test is performed by computing the F-statistic, F-statistic

76 3. Linear Regression

Quantity Value Residual standard error 1.69 R2 0.897 F-statistic 570

TABLE 3.6. More information about the least squares model for the regression of number of units sold on TV, newspaper, and radio advertising budgets in the Advertising data. Other information about this model was displayed in Table 3.4.

F = (TSS−RSS)/p RSS/(n−p−1)

, (3.23)

where, as with simple linear regression, TSS = ∑

(yi − ȳ)2 and RSS =∑ (yi−ŷi)2. If the linear model assumptions are correct, one can show that

E{RSS/(n−p−1)} = σ2

and that, provided H0 is true,

E{(TSS−RSS)/p} = σ2.

Hence, when there is no relationship between the response and predictors, one would expect the F-statistic to take on a value close to 1. On the other hand, if Ha is true, then E{(TSS − RSS)/p} > σ2, so we expect F to be greater than 1. The F-statistic for the multiple linear regression model obtained by re-

gressing sales onto radio, TV, and newspaper is shown in Table 3.6. In this example the F-statistic is 570. Since this is far larger than 1, it provides compelling evidence against the null hypothesis H0. In other words, the large F-statistic suggests that at least one of the advertising media must be related to sales. However, what if the F-statistic had been closer to 1? How large does the F-statistic need to be before we can reject H0 and conclude that there is a relationship? It turns out that the answer depends on the values of n and p. When n is large, an F-statistic that is just a little larger than 1 might still provide evidence against H0. In contrast, a larger F-statistic is needed to reject H0 if n is small. When H0 is true and the errors ϵi have a normal distribution, the F-statistic follows an F-distribution.6 For any given value of n and p, any statistical software package can be used to compute the p-value associated with the F-statistic using this distribution. Based on this p-value, we can determine whether or not to reject H0. For the advertising data, the p-value associated with the F-statistic in Table 3.6 is essentially zero, so we have extremely strong evidence that at least one of the media is associated with increased sales.

6Even if the errors are not normally-distributed, the F-statistic approximately follows an F-distribution provided that the sample size n is large.

3.2 Multiple Linear Regression 77

In (3.23) we are testing H0 that all the coefficients are zero. Sometimes we want to test that a particular subset of q of the coefficients are zero. This corresponds to a null hypothesis

H0 : βp−q+1 = βp−q+2 = · · · = βp = 0,

where for convenience we have put the variables chosen for omission at the end of the list. In this case we fit a second model that uses all the variables except those last q. Suppose that the residual sum of squares for that model is RSS0. Then the appropriate F-statistic is

F = (RSS0 −RSS)/q RSS/(n−p−1)

. (3.24)

Notice that in Table 3.4, for each individual predictor a t-statistic and a p-value were reported. These provide information about whether each individual predictor is related to the response, after adjusting for the other predictors. It turns out that each of these is exactly equivalent7 to the F- test that omits that single variable from the model, leaving all the others in—i.e. q=1 in (3.24). So it reports the partial effect of adding that variable to the model. For instance, as we discussed earlier, these p-values indicate that TV and radio are related to sales, but that there is no evidence that newspaper is associated with sales, when TV and radio are held fixed. Given these individual p-values for each variable, why do we need to look

at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large. For instance, consider an example in which p = 100 and H0 : β1 = β2 =

· · · = βp = 0 is true, so no variable is truly associated with the response. In this situation, about 5 % of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0.05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response.8 In fact, it is likely that we will observe at least one p-value below 0.05 by chance! Hence, if we use the individual t-statistics and associated p-values in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship. However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H0 is true, there is only a 5 % chance that the F-statistic will result in a p- value below 0.05, regardless of the number of predictors or the number of observations.

7The square of each t-statistic is the corresponding F-statistic. 8This is related to the important concept of multiple testing, which is the focus of

Chapter 13.

78 3. Linear Regression

The approach of using an F-statistic to test for any association between the predictors and the response works when p is relatively small, and cer- tainly small compared to n. However, sometimes we have a very large num- ber of variables. If p > n then there are more coefficients βj to estimate than observations from which to estimate them. In this case we cannot even fit the multiple linear regression model using least squares, so the F- statistic cannot be used, and neither can most of the other concepts that we have seen so far in this chapter. When p is large, some of the approaches discussed in the next section, such as forward selection, can be used. This high-dimensional setting is discussed in greater detail in Chapter 6.

high- dimensional

Two: Deciding on Important Variables

As discussed in the previous section, the first step in a multiple regression analysis is to compute the F-statistic and to examine the associated p- value. If we conclude on the basis of that p-value that at least one of the predictors is related to the response, then it is natural to wonder which are the guilty ones! We could look at the individual p-values as in Table 3.4, but as discussed (and as further explored in Chapter 13), if p is large we are likely to make some false discoveries. It is possible that all of the predictors are associated with the response,

but it is more often the case that the response is only associated with a subset of the predictors. The task of determining which predictors are associated with the response, in order to fit a single model involving only those predictors, is referred to as variable selection. The variable selection

variable selectionproblem is studied extensively in Chapter 6, and so here we will provide

only a brief outline of some classical approaches. Ideally, we would like to perform variable selection by trying out a lot of

different models, each containing a different subset of the predictors. For instance, if p = 2, then we can consider four models: (1) a model contain- ing no variables, (2) a model containing X1 only, (3) a model containing X2 only, and (4) a model containing both X1 and X2. We can then se- lect the best model out of all of the models that we have considered. How do we determine which model is best? Various statistics can be used to judge the quality of a model. These include Mallow’s Cp, Akaike informa-

Mallow’s Cp tion criterion (AIC), Bayesian information criterion (BIC), and adjusted

Akaike information criterion Bayesian information criterion

R2. These are discussed in more detail in Chapter 6. We can also deter-

adjusted R2

mine which model is best by plotting various model outputs, such as the residuals, in order to search for patterns. Unfortunately, there are a total of 2p models that contain subsets of p

variables. This means that even for moderate p, trying out every possible subset of the predictors is infeasible. For instance, we saw that if p = 2, then there are 22 = 4 models to consider. But if p = 30, then we must consider 230 = 1,073,741,824 models! This is not practical. Therefore, unless p is very small, we cannot consider all 2p models, and instead we need an automated

3.2 Multiple Linear Regression 79

and efficient approach to choose a smaller set of models to consider. There are three classical approaches for this task:

• Forward selection. We begin with the null model—a model that con- forward selection null model

tains an intercept but no predictors. We then fit p simple linear re- gressions and add to the null model the variable that results in the lowest RSS. We then add to that model the variable that results in the lowest RSS for the new two-variable model. This approach is continued until some stopping rule is satisfied.

• Backward selection. We start with all variables in the model, and backward selectionremove the variable with the largest p-value—that is, the variable

that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached. For instance, we may stop when all remaining variables have a p-value below some threshold.

• Mixed selection. This is a combination of forward and backward se- mixed selectionlection. We start with no variables in the model, and as with forward

selection, we add the variable that provides the best fit. We con- tinue to add variables one-by-one. Of course, as we noted with the Advertising example, the p-values for variables can become larger as new predictors are added to the model. Hence, if at any point the p-value for one of the variables in the model rises above a certain threshold, then we remove that variable from the model. We con- tinue to perform these forward and backward steps until all variables in the model have a sufficiently low p-value, and all variables outside the model would have a large p-value if added to the model.

Backward selection cannot be used if p > n, while forward selection can always be used. Forward selection is a greedy approach, and might include variables early that later become redundant. Mixed selection can remedy this.

Three: Model Fit

Two of the most common numerical measures of model fit are the RSE and R2, the fraction of variance explained. These quantities are computed and interpreted in the same fashion as for simple linear regression. Recall that in simple regression, R2 is the square of the correlation of the

response and the variable. In multiple linear regression, it turns out that it equals Cor(Y, Ŷ )2, the square of the correlation between the response and the fitted linear model; in fact one property of the fitted linear model is that it maximizes this correlation among all possible linear models. An R2 value close to 1 indicates that the model explains a large portion

of the variance in the response variable. As an example, we saw in Table 3.6

80 3. Linear Regression

that for the Advertising data, the model that uses all three advertising me- dia to predict sales has an R2 of 0.8972. On the other hand, the model that uses only TV and radio to predict sales has an R2 value of 0.89719. In other words, there is a small increase in R2 if we include newspaper advertising in the model that already contains TV and radio advertising, even though we saw earlier that the p-value for newspaper advertising in Table 3.4 is not significant. It turns out that R2 will always increase when more variables are added to the model, even if those variables are only weakly associated with the response. This is due to the fact that adding another variable always results in a decrease in the residual sum of squares on the training data (though not necessarily the testing data). Thus, the R2 statistic, which is also computed on the training data, must increase. The fact that adding newspaper advertising to the model containing only TV and radio advertising leads to just a tiny increase in R2 provides addi- tional evidence that newspaper can be dropped from the model. Essentially, newspaper provides no real improvement in the model fit to the training samples, and its inclusion will likely lead to poor results on independent test samples due to overfitting. By contrast, the model containing only TV as a predictor had an R2 of

0.61 (Table 3.2). Adding radio to the model leads to a substantial improve- ment in R2. This implies that a model that uses TV and radio expenditures to predict sales is substantially better than one that uses only TV advertis- ing. We could further quantify this improvement by looking at the p-value for the radio coefficient in a model that contains only TV and radio as predictors. The model that contains only TV and radio as predictors has an RSE

of 1.681, and the model that also contains newspaper as a predictor has an RSE of 1.686 (Table 3.6). In contrast, the model that contains only TV has an RSE of 3.26 (Table 3.2). This corroborates our previous conclusion that a model that uses TV and radio expenditures to predict sales is much more accurate (on the training data) than one that only uses TV spending. Furthermore, given that TV and radio expenditures are used as predictors, there is no point in also using newspaper spending as a predictor in the model. The observant reader may wonder how RSE can increase when newspaper is added to the model given that RSS must decrease. In general RSE is defined as

RSE =

√ 1

n−p−1 RSS, (3.25)

which simplifies to (3.15) for a simple linear regression. Thus, models with more variables can have higher RSE if the decrease in RSS is small relative to the increase in p. In addition to looking at the RSE and R2 statistics just discussed, it

can be useful to plot the data. Graphical summaries can reveal problems with a model that are not visible from numerical statistics. For example,

3.2 Multiple Linear Regression 81

Sales

Radio

FIGURE 3.5. For the Advertising data, a linear regression fit to sales using TV and radio as predictors. From the pattern of the residuals, we can see that there is a pronounced non-linear relationship in the data. The positive residuals (those visible above the surface), tend to lie along the 45-degree line, where TV and Radio budgets are split evenly. The negative residuals (most not visible), tend to lie away from this line, where budgets are more lopsided.

Figure 3.5 displays a three-dimensional plot of TV and radio versus sales. We see that some observations lie above and some observations lie below the least squares regression plane. In particular, the linear model seems to overestimate sales for instances in which most of the advertising money was spent exclusively on either TV or radio. It underestimates sales for instances where the budget was split between the two media. This pro- nounced non-linear pattern suggests a synergy or interaction effect between

interaction the advertising media, whereby combining the media together results in a bigger boost to sales than using any single medium. In Section 3.3.2, we will discuss extending the linear model to accommodate such synergistic effects through the use of interaction terms.

Four: Predictions

Once we have fit the multiple regression model, it is straightforward to apply (3.21) in order to predict the response Y on the basis of a set of values for the predictors X1, X2, . . . , Xp. However, there are three sorts of uncertainty associated with this prediction.

1. The coefficient estimates β̂0, β̂1, . . . , β̂p are estimates for β0, β1, . . . , βp. That is, the least squares plane

Ŷ = β̂0 + β̂1X1 + · · · + β̂pXp

82 3. Linear Regression

is only an estimate for the true population regression plane

f(X) = β0 + β1X1 + · · · + βpXp.

The inaccuracy in the coefficient estimates is related to the reducible error from Chapter 2. We can compute a confidence interval in order to determine how close Ŷ will be to f(X).

2. Of course, in practice assuming a linear model for f(X) is almost always an approximation of reality, so there is an additional source of potentially reducible error which we call model bias. So when we use a linear model, we are in fact estimating the best linear approximation to the true surface. However, here we will ignore this discrepancy, and operate as if the linear model were correct.

3. Even if we knew f(X)—that is, even if we knew the true values for β0, β1, . . . , βp—the response value cannot be predicted perfectly because of the random error ϵ in the model (3.20). In Chapter 2, we referred to this as the irreducible error. How much will Y vary from Ŷ ? We use prediction intervals to answer this question. Prediction intervals are always wider than confidence intervals, because they incorporate both the error in the estimate for f(X) (the reducible error) and the uncertainty as to how much an individual point will differ from the population regression plane (the irreducible error).

We use a confidence interval to quantify the uncertainty surrounding confidence intervalthe average sales over a large number of cities. For example, given that

$100,000 is spent on TV advertising and $20,000 is spent on radio advertising in each city, the 95 % confidence interval is [10,985, 11,528]. We interpret this to mean that 95 % of intervals of this form will contain the true value of f(X).9 On the other hand, a prediction interval can be used to quantify the

prediction intervaluncertainty surrounding sales for a particular city. Given that $100,000 is

spent on TV advertising and $20,000 is spent on radio advertising in that city the 95 % prediction interval is [7,930, 14,580]. We interpret this to mean that 95 % of intervals of this form will contain the true value of Y for this city. Note that both intervals are centered at 11,256, but that the prediction interval is substantially wider than the confidence interval, reflecting the increased uncertainty about sales for a given city in comparison to the average sales over many locations.

9In other words, if we collect a large number of data sets like the Advertising data set, and we construct a confidence interval for the average sales on the basis of each data set (given $100,000 in TV and $20,000 in radio advertising), then 95 % of these confidence intervals will contain the true value of average sales.

3.3 Other Considerations in the Regression Model 83

3.3 Other Considerations in the Regression Model

3.3.1 Qualitative Predictors

In our discussion so far, we have assumed that all variables in our linear regression model are quantitative. But in practice, this is not necessarily the case; often some predictors are qualitative. For example, the Credit data set displayed in Figure 3.6 records variables

for a number of credit card holders. The response is balance (average credit card debt for each individual) and there are several quantitative predictors: age, cards (number of credit cards), education (years of education), income (in thousands of dollars), limit (credit limit), and rating (credit rating). Each panel of Figure 3.6 is a scatterplot for a pair of variables whose iden- tities are given by the corresponding row and column labels. For example, the scatterplot directly to the right of the word “Balance” depicts balance versus age, while the plot directly to the right of “Age” corresponds to age versus cards. In addition to these quantitative variables, we also have four qualitative variables: own (house ownership), student (student status), status (marital status), and region (East, West or South).

Predictors with Only Two Levels

Suppose that we wish to investigate differences in credit card balance be- tween those who own a house and those who don’t, ignoring the other vari- ables for the moment. If a qualitative predictor (also known as a factor)

factor only has two levels, or possible values, then incorporating it into a regres-

level sion model is very simple. We simply create an indicator or dummy variable

dummy variablethat takes on two possible numerical values.

10 For example, based on the own variable, we can create a new variable that takes the form

xi =

{ 1 if ith person owns a house

0 if ith person does not own a house, (3.26)

and use this variable as a predictor in the regression equation. This results in the model

yi = β0 + β1xi + ϵi =

{ β0 + β1 + ϵi if ith person owns a house

β0 + ϵi if ith person does not. (3.27)

Now β0 can be interpreted as the average credit card balance among those who do not own, β0 + β1 as the average credit card balance among those who do own their house, and β1 as the average difference in credit card balance between owners and non-owners.

10In the machine learning community, the creation of dummy variables to handle qualitative predictors is known as “one-hot encoding”.

84 3. Linear Regression

Balance

20 40 60 80 100 5 10 15 20 2000 8000 14000

0 5 0 0

1 5 0 0

2 0

4 0

6 0

8 0

1 0 0

Age

Cards

2 4

6 8

5 1 0

1 5

2 0

Education

Income

5 0

1 0 0

1 5 0

2 0 0 0

8 0 0 0

1 4 0 0 0

Limit

0 500 1500 2 4 6 8 50 100 150 200 600 1000

2 0 0

6 0 0

1 0 0 0

Rating

FIGURE 3.6. The Credit data set contains information about balance, age, cards, education, income, limit, and rating for a number of potential cus- tomers.

Table 3.7 displays the coefficient estimates and other information asso- ciated with the model (3.27). The average credit card debt for non-owners is estimated to be $509.80, whereas owners are estimated to carry $19.73 in additional debt for a total of $509.80 + $19.73 = $529.53. However, we notice that the p-value for the dummy variable is very high. This indicates that there is no statistical evidence of a difference in average credit card balance based on house ownership. The decision to code owners as 1 and non-owners as 0 in (3.27) is ar-

bitrary, and has no effect on the regression fit, but does alter the inter- pretation of the coefficients. If we had coded non-owners as 1 and own- ers as 0, then the estimates for β0 and β1 would have been 529.53 and −19.73, respectively, leading once again to a prediction of credit card debt of $529.53 − $19.73 = $509.80 for non-owners and a prediction of $529.53

3.3 Other Considerations in the Regression Model 85

Coefficient Std. error t-statistic p-value Intercept 509.80 33.13 15.389 < 0.0001 own[Yes] 19.73 46.05 0.429 0.6690

TABLE 3.7. Least squares coefficient estimates associated with the regression of balance onto own in the Credit data set. The linear model is given in (3.27). That is, ownership is encoded as a dummy variable, as in (3.26).

for owners. Alternatively, instead of a 0/1 coding scheme, we could create a dummy variable

xi =

{ 1 if ith person owns a house

−1 if ith person does not own a house

and use this variable in the regression equation. This results in the model

yi = β0+β1xi+ϵi =

{ β0 + β1 + ϵi if ith person owns a house

β0 −β1 + ϵi if ith person does not own a house.

Now β0 can be interpreted as the overall average credit card balance (ig- noring the house ownership effect), and β1 is the amount by which house owners and non-owners have credit card balances that are above and below the average, respectively. In this example, the estimate for β0 is $519.665, halfway between the non-owner and owner averages of $509.80 and $529.53. The estimate for β1 is $9.865, which is half of $19.73, the average difference between owners and non-owners. It is important to note that the final pre- dictions for the credit balances of owners and non-owners will be identical regardless of the coding scheme used. The only difference is in the way that the coefficients are interpreted.

Qualitative Predictors with More than Two Levels

When a qualitative predictor has more than two levels, a single dummy variable cannot represent all possible values. In this situation, we can create additional dummy variables. For example, for the region variable we create two dummy variables. The first could be

xi1 =

{ 1 if ith person is from the South

0 if ith person is not from the South, (3.28)

and the second could be

xi2 =

{ 1 if ith person is from the West

0 if ith person is not from the West. (3.29)

86 3. Linear Regression

Coefficient Std. error t-statistic p-value Intercept 531.00 46.32 11.464 < 0.0001 region[South] −18.69 65.02 −0.287 0.7740 region[West] −12.50 56.68 −0.221 0.8260

TABLE 3.8. Least squares coefficient estimates associated with the regression of balance onto region in the Credit data set. The linear model is given in (3.30). That is, region is encoded via two dummy variables (3.28) and (3.29).

Then both of these variables can be used in the regression equation, in order to obtain the model

yi = β0+β1xi1+β2xi2+ϵi =

⎧ ⎪⎨

⎪⎩

β0+β1+ϵi if ith person is from the South

β0+β2+ϵi if ith person is from the West

β0+ϵi if ith person is from the East.

(3.30) Now β0 can be interpreted as the average credit card balance for individuals from the East, β1 can be interpreted as the difference in the average balance between people from the South versus the East, and β2 can be interpreted as the difference in the average balance between those from the West versus the East. There will always be one fewer dummy variable than the number of levels. The level with no dummy variable—East in this example—is known as the baseline.

baseline From Table 3.8, we see that the estimated balance for the baseline, East,

is $531.00. It is estimated that those in the South will have $18.69 less debt than those in the East, and that those in the West will have $12.50 less debt than those in the East. However, the p-values associated with the coefficient estimates for the two dummy variables are very large, suggesting no statistical evidence of a real difference in average credit card balance between South and East or between West and East.11 Once again, the level selected as the baseline category is arbitrary, and the final predictions for each group will be the same regardless of this choice. However, the coefficients and their p-values do depend on the choice of dummy variable coding. Rather than rely on the individual coefficients, we can use an F-test to test H0 : β1 = β2 = 0; this does not depend on the coding. This F-test has a p-value of 0.96, indicating that we cannot reject the null hypothesis that there is no relationship between balance and region. Using this dummy variable approach presents no difficulties when in-

corporating both quantitative and qualitative predictors. For example, to regress balance on both a quantitative variable such as income and a qual- itative variable such as student, we must simply create a dummy variable for student and then fit a multiple regression model using income and the dummy variable as predictors for credit card balance.

11There could still in theory be a difference between South and West, although the data here does not suggest any difference.

3.3 Other Considerations in the Regression Model 87

There are many different ways of coding qualitative variables besides the dummy variable approach taken here. All of these approaches lead to equivalent model fits, but the coefficients are different and have different interpretations, and are designed to measure particular contrasts. This topic

contrast is beyond the scope of the book.

3.3.2 Extensions of the Linear Model

The standard linear regression model (3.19) provides interpretable results and works quite well on many real-world problems. However, it makes sev- eral highly restrictive assumptions that are often violated in practice. Two of the most important assumptions state that the relationship between the predictors and response are additive and linear. The additivity assumption

additive linear

means that the association between a predictor Xj and the response Y does not depend on the values of the other predictors. The linearity assumption states that the change in the response Y associated with a one-unit change in Xj is constant, regardless of the value of Xj. In later chapters of this book, we examine a number of sophisticated methods that relax these two assumptions. Here, we briefly examine some common classical approaches for extending the linear model.

Removing the Additive Assumption

In our previous analysis of the Advertising data, we concluded that both TV and radio seem to be associated with sales. The linear models that formed the basis for this conclusion assumed that the effect on sales of increasing one advertising medium is independent of the amount spent on the other media. For example, the linear model (3.20) states that the average increase in sales associated with a one-unit increase in TV is always β1, regardless of the amount spent on radio. However, this simple model may be incorrect. Suppose that spending

money on radio advertising actually increases the effectiveness of TV ad- vertising, so that the slope term for TV should increase as radio increases. In this situation, given a fixed budget of $100,000, spending half on radio and half on TV may increase sales more than allocating the entire amount to either TV or to radio. In marketing, this is known as a synergy effect, and in statistics it is referred to as an interaction effect. Figure 3.5 sug- gests that such an effect may be present in the advertising data. Notice that when levels of either TV or radio are low, then the true sales are lower than predicted by the linear model. But when advertising is split between the two media, then the model tends to underestimate sales. Consider the standard linear regression model with two variables,

Y = β0 + β1X1 + β2X2 + ϵ.

88 3. Linear Regression

According to this model, a one-unit increase in X1 is associated with an average increase in Y of β1 units. Notice that the presence of X2 does not alter this statement—that is, regardless of the value of X2, a one- unit increase in X1 is associated with a β1-unit increase in Y . One way of extending this model is to include a third predictor, called an interaction term, which is constructed by computing the product of X1 and X2. This results in the model

Y = β0 + β1X1 + β2X2 + β3X1X2 + ϵ. (3.31)

How does inclusion of this interaction term relax the additive assumption? Notice that (3.31) can be rewritten as

Y = β0 + (β1 + β3X2)X1 + β2X2 + ϵ (3.32)

= β0 + β̃1X1 + β2X2 + ϵ

where β̃1 = β1 + β3X2. Since β̃1 is now a function of X2, the association between X1 and Y is no longer constant: a change in the value of X2 will change the association between X1 and Y . A similar argument shows that a change in the value of X1 changes the association between X2 and Y . For example, suppose that we are interested in studying the productiv-

ity of a factory. We wish to predict the number of units produced on the basis of the number of production lines and the total number of workers. It seems likely that the effect of increasing the number of production lines will depend on the number of workers, since if no workers are available to operate the lines, then increasing the number of lines will not increase production. This suggests that it would be appropriate to include an inter- action term between lines and workers in a linear model to predict units. Suppose that when we fit the model, we obtain

units ≈ 1.2 + 3.4× lines + 0.22× workers + 1.4× (lines × workers) = 1.2 + (3.4 + 1.4× workers)× lines + 0.22× workers.

In other words, adding an additional line will increase the number of units produced by 3.4 + 1.4 × workers. Hence the more workers we have, the stronger will be the effect of lines. We now return to the Advertising example. A linear model that uses

radio, TV, and an interaction between the two to predict sales takes the form

sales = β0 + β1 × TV + β2 × radio + β3 × (radio × TV) + ϵ = β0 + (β1 + β3 × radio)× TV + β2 × radio + ϵ. (3.33)

We can interpret β3 as the increase in the effectiveness of TV advertising associated with a one-unit increase in radio advertising (or vice-versa). The coefficients that result from fitting the model (3.33) are given in Table 3.9.

3.3 Other Considerations in the Regression Model 89

Coefficient Std. error t-statistic p-value Intercept 6.7502 0.248 27.23 < 0.0001 TV 0.0191 0.002 12.70 < 0.0001 radio 0.0289 0.009 3.24 0.0014 TV×radio 0.0011 0.000 20.73 < 0.0001

TABLE 3.9. For the Advertising data, least squares coefficient estimates asso- ciated with the regression of sales onto TV and radio, with an interaction term, as in (3.33).

The results in Table 3.9 strongly suggest that the model that includes the interaction term is superior to the model that contains only main effects.

main effect The p-value for the interaction term, TV×radio, is extremely low, indicating that there is strong evidence for Ha : β3 ̸= 0. In other words, it is clear that the true relationship is not additive. The R2 for the model (3.33) is 96.8 %, compared to only 89.7 % for the model that predicts sales using TV and radio without an interaction term. This means that (96.8 − 89.7)/(100 − 89.7) = 69 % of the variability in sales that remains after fitting the ad- ditive model has been explained by the interaction term. The coefficient estimates in Table 3.9 suggest that an increase in TV advertising of $1,000 is associated with increased sales of (β̂1+β̂3×radio)×1,000 = 19+1.1×radio units. And an increase in radio advertising of $1,000 will be associated with an increase in sales of (β̂2 + β̂3 × TV)×1,000 = 29 + 1.1× TV units. In this example, the p-values associated with TV, radio, and the interac-

tion term all are statistically significant (Table 3.9), and so it is obvious that all three variables should be included in the model. However, it is sometimes the case that an interaction term has a very small p-value, but the associated main effects (in this case, TV and radio) do not. The hier- archical principle states that if we include an interaction in a model, we

hierarchical principleshould also include the main effects, even if the p-values associated with

their coefficients are not significant. In other words, if the interaction be- tween X1 and X2 seems important, then we should include both X1 and X2 in the model even if their coefficient estimates have large p-values. The rationale for this principle is that if X1 × X2 is related to the response, then whether or not the coefficients of X1 or X2 are exactly zero is of lit- tle interest. Also X1 × X2 is typically correlated with X1 and X2, and so leaving them out tends to alter the meaning of the interaction. In the previous example, we considered an interaction between TV and

radio, both of which are quantitative variables. However, the concept of interactions applies just as well to qualitative variables, or to a combination of quantitative and qualitative variables. In fact, an interaction between a qualitative variable and a quantitative variable has a particularly nice interpretation. Consider the Credit data set from Section 3.3.1, and suppose that we wish to predict balance using the income (quantitative) and student (qualitative) variables. In the absence of an interaction term, the model

90 3. Linear Regression

0 50 100 150

2 0

0 6

0 0

1 0

0 0

1 4

0 0

Income

B a

la n

c e

0 50 100 150

2 0

0 6

0 0

1 0

0 0

1 4

0 0

Income

B a

la n

c e

student non−student

FIGURE 3.7. For the Credit data, the least squares lines are shown for pre- diction of balance from income for students and non-students. Left: The model (3.34) was fit. There is no interaction between income and student. Right: The model (3.35) was fit. There is an interaction term between income and student.

takes the form

balancei ≈ β0 + β1 × incomei + { β2 if ith person is a student

0 if ith person is not a student

= β1 × incomei + { β0 + β2 if ith person is a student

β0 if ith person is not a student.

(3.34)

Notice that this amounts to fitting two parallel lines to the data, one for students and one for non-students. The lines for students and non-students have different intercepts, β0 + β2 versus β0, but the same slope, β1. This is illustrated in the left-hand panel of Figure 3.7. The fact that the lines are parallel means that the average effect on balance of a one-unit increase in income does not depend on whether or not the individual is a student. This represents a potentially serious limitation of the model, since in fact a change in income may have a very different effect on the credit card balance of a student versus a non-student. This limitation can be addressed by adding an interaction variable, cre-

ated by multiplying income with the dummy variable for student. Our model now becomes

balancei ≈ β0 + β1 × incomei + { β2 + β3 × incomei if student 0 if not student

{ (β0 + β2) + (β1 + β3)× incomei if student β0 + β1 × incomei if not student.

(3.35)

3.3 Other Considerations in the Regression Model 91

50 100 150 200

1 0

2 0

3 0

4 0

5 0

Horsepower

M ile

s p

e r

g a llo

Linear Degree 2 Degree 5

FIGURE 3.8. The Auto data set. For a number of cars, mpg and horsepower are shown. The linear regression fit is shown in orange. The linear regression fit for a model that includes horsepower2 is shown as a blue curve. The linear regression fit for a model that includes all polynomials of horsepower up to fifth-degree is shown in green.

Once again, we have two different regression lines for the students and the non-students. But now those regression lines have different intercepts, β0+β2 versus β0, as well as different slopes, β1+β3 versus β1. This allows for the possibility that changes in income may affect the credit card balances of students and non-students differently. The right-hand panel of Figure 3.7 shows the estimated relationships between income and balance for students and non-students in the model (3.35). We note that the slope for students is lower than the slope for non-students. This suggests that increases in income are associated with smaller increases in credit card balance among students as compared to non-students.

Non-linear Relationships

As discussed previously, the linear regression model (3.19) assumes a linear relationship between the response and predictors. But in some cases, the true relationship between the response and the predictors may be non- linear. Here we present a very simple way to directly extend the linear model to accommodate non-linear relationships, using polynomial regression. In

polynomial regressionlater chapters, we will present more complex approaches for performing

non-linear fits in more general settings.

92 3. Linear Regression

Coefficient Std. error t-statistic p-value Intercept 56.9001 1.8004 31.6 < 0.0001 horsepower −0.4662 0.0311 −15.0 < 0.0001 horsepower2 0.0012 0.0001 10.1 < 0.0001

TABLE 3.10. For the Auto data set, least squares coefficient estimates associated with the regression of mpg onto horsepower and horsepower2.

Consider Figure 3.8, in which the mpg (gas mileage in miles per gallon) versus horsepower is shown for a number of cars in the Auto data set. The orange line represents the linear regression fit. There is a pronounced rela- tionship between mpg and horsepower, but it seems clear that this relation- ship is in fact non-linear: the data suggest a curved relationship. A simple approach for incorporating non-linear associations in a linear model is to include transformed versions of the predictors. For example, the points in Figure 3.8 seem to have a quadratic shape, suggesting that a model of the

quadratic form

mpg = β0 + β1 × horsepower + β2 × horsepower2 + ϵ (3.36) may provide a better fit. Equation 3.36 involves predicting mpg using a non-linear function of horsepower. But it is still a linear model! That is, (3.36) is simply a multiple linear regression model with X1 = horsepower and X2 = horsepower

2. So we can use standard linear regression software to estimate β0, β1, and β2 in order to produce a non-linear fit. The blue curve in Figure 3.8 shows the resulting quadratic fit to the data. The quadratic fit appears to be substantially better than the fit obtained when just the linear term is included. The R2 of the quadratic fit is 0.688, compared to 0.606 for the linear fit, and the p-value in Table 3.10 for the quadratic term is highly significant. If including horsepower2 led to such a big improvement in the model, why

not include horsepower3, horsepower4, or even horsepower5? The green curve in Figure 3.8 displays the fit that results from including all polynomials up to fifth degree in the model (3.36). The resulting fit seems unnecessarily wiggly—that is, it is unclear that including the additional terms really has led to a better fit to the data. The approach that we have just described for extending the linear model

to accommodate non-linear relationships is known as polynomial regres- sion, since we have included polynomial functions of the predictors in the regression model. We further explore this approach and other non-linear extensions of the linear model in Chapter 7.

3.3.3 Potential Problems

When we fit a linear regression model to a particular data set, many prob- lems may occur. Most common among these are the following:

1. Non-linearity of the response-predictor relationships.

3.3 Other Considerations in the Regression Model 93

2. Correlation of error terms.

3. Non-constant variance of error terms.

4. Outliers.

5. High-leverage points.

6. Collinearity.

In practice, identifying and overcoming these problems is as much an art as a science. Many pages in countless books have been written on this topic. Since the linear regression model is not our primary focus here, we will provide only a brief summary of some key points.

1. Non-linearity of the Data

5 10 15 20 25 30

− 1

5 −

1 0

− 5

0 5

1 0

1 5

2 0

Fitted values

R e

s id

u a

Residual Plot for Linear Fit

323 330

334

15 20 25 30 35

− 1

5 −

1 0

− 5

0 5

1 0

1 5

Fitted values

R e

s id

u a

Residual Plot for Quadratic Fit

334 323

155

FIGURE 3.9. Plots of residuals versus predicted (or fitted) values for the Auto data set. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. Left: A linear regression of mpg on horsepower. A strong pattern in the residuals indicates non-linearity in the data. Right: A linear regression of mpg on horsepower and horsepower2. There is little pattern in the residuals.

The linear regression model assumes that there is a straight-line rela- tionship between the predictors and the response. If the true relationship is far from linear, then virtually all of the conclusions that we draw from the fit are suspect. In addition, the prediction accuracy of the model can be significantly reduced. Residual plots are a useful graphical tool for identifying non-linearity.

residual plot Given a simple linear regression model, we can plot the residuals, ei = yi − ŷi, versus the predictor xi. In the case of a multiple regression model,

94 3. Linear Regression

since there are multiple predictors, we instead plot the residuals versus the predicted (or fitted) values ŷi. Ideally, the residual plot will show no

fitted discernible pattern. The presence of a pattern may indicate a problem with some aspect of the linear model. The left panel of Figure 3.9 displays a residual plot from the linear re-

gression of mpg onto horsepower on the Auto data set that was illustrated in Figure 3.8. The red line is a smooth fit to the residuals, which is displayed in order to make it easier to identify any trends. The residuals exhibit a clear U-shape, which provides a strong indication of non-linearity in the data. In contrast, the right-hand panel of Figure 3.9 displays the residual plot that results from the model (3.36), which contains a quadratic term. There appears to be little pattern in the residuals, suggesting that the quadratic term improves the fit to the data. If the residual plot indicates that there are non-linear associations in the

data, then a simple approach is to use non-linear transformations of the predictors, such as log X,

√ X, and X2, in the regression model. In the

later chapters of this book, we will discuss other more advanced non-linear approaches for addressing this issue.

2. Correlation of Error Terms

An important assumption of the linear regression model is that the error terms, ϵ1, ϵ2, . . . , ϵn, are uncorrelated. What does this mean? For instance, if the errors are uncorrelated, then the fact that ϵi is positive provides little or no information about the sign of ϵi+1. The standard errors that are computed for the estimated regression coefficients or the fitted values are based on the assumption of uncorrelated error terms. If in fact there is correlation among the error terms, then the estimated standard errors will tend to underestimate the true standard errors. As a result, confidence and prediction intervals will be narrower than they should be. For example, a 95 % confidence interval may in reality have a much lower probability than 0.95 of containing the true value of the parameter. In addition, p- values associated with the model will be lower than they should be; this could cause us to erroneously conclude that a parameter is statistically significant. In short, if the error terms are correlated, we may have an unwarranted sense of confidence in our model. As an extreme example, suppose we accidentally doubled our data, lead-

ing to observations and error terms identical in pairs. If we ignored this, our standard error calculations would be as if we had a sample of size 2n, when in fact we have only n samples. Our estimated parameters would be the same for the 2n samples as for the n samples, but the confidence intervals would be narrower by a factor of

√ 2!

Why might correlations among the error terms occur? Such correlations frequently occur in the context of time series data, which consists of ob-

time series servations for which measurements are obtained at discrete points in time.

3.3 Other Considerations in the Regression Model 95

0 20 40 60 80 100

− 3

− 1

0 1

2 3

ρ=0.0

R e

s id

u a

0 20 40 60 80 100

− 4

− 2

0 1

ρ=0.5

R e

s id

u a

0 20 40 60 80 100

− 1

.5 −

0 .5

1 .5

ρ=0.9

R e

s id

u a

Observation

FIGURE 3.10. Plots of residuals from simulated time series data sets generated with differing levels of correlation ρ between error terms for adjacent time points.

In many cases, observations that are obtained at adjacent time points will have positively correlated errors. In order to determine if this is the case for a given data set, we can plot the residuals from our model as a function of time. If the errors are uncorrelated, then there should be no discernible pat- tern. On the other hand, if the error terms are positively correlated, then we may see tracking in the residuals—that is, adjacent residuals may have

tracking similar values. Figure 3.10 provides an illustration. In the top panel, we see the residuals from a linear regression fit to data generated with uncorre- lated errors. There is no evidence of a time-related trend in the residuals. In contrast, the residuals in the bottom panel are from a data set in which adjacent errors had a correlation of 0.9. Now there is a clear pattern in the residuals—adjacent residuals tend to take on similar values. Finally, the center panel illustrates a more moderate case in which the residuals had a correlation of 0.5. There is still evidence of tracking, but the pattern is less clear. Many methods have been developed to properly take account of corre-

lations in the error terms in time series data. Correlation among the error

96 3. Linear Regression

10 15 20 25 30

− 1 0

− 5

0 5

1 0

1 5

Fitted values

R e s id

u a ls

Response Y

998 975

845

2.4 2.6 2.8 3.0 3.2 3.4

− 0 .8

− 0 .6

− 0 .4

− 0 .2

0 .0

0 .2

0 .4

Fitted values R

e s id

u a ls

Response log(Y)

437 671

605

FIGURE 3.11. Residual plots. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. The blue lines track the outer quantiles of the residuals, and emphasize patterns. Left: The funnel shape indicates heteroscedasticity. Right: The response has been log transformed, and there is now no evidence of heteroscedasticity.

terms can also occur outside of time series data. For instance, consider a study in which individuals’ heights are predicted from their weights. The assumption of uncorrelated errors could be violated if some of the indi- viduals in the study are members of the same family, eat the same diet, or have been exposed to the same environmental factors. In general, the assumption of uncorrelated errors is extremely important for linear regres- sion as well as for other statistical methods, and good experimental design is crucial in order to mitigate the risk of such correlations.

3. Non-constant Variance of Error Terms

Another important assumption of the linear regression model is that the error terms have a constant variance, Var(ϵi) = σ

2. The standard errors, confidence intervals, and hypothesis tests associated with the linear model rely upon this assumption. Unfortunately, it is often the case that the variances of the error terms are

non-constant. For instance, the variances of the error terms may increase with the value of the response. One can identify non-constant variances in the errors, or heteroscedasticity, from the presence of a funnel shape in

hetero- scedasticitythe residual plot. An example is shown in the left-hand panel of Figure 3.11,

in which the magnitude of the residuals tends to increase with the fitted values. When faced with this problem, one possible solution is to trans- form the response Y using a concave function such as log Y or

√ Y . Such

a transformation results in a greater amount of shrinkage of the larger re- sponses, leading to a reduction in heteroscedasticity. The right-hand panel

3.3 Other Considerations in the Regression Model 97

−2 −1 0 1 2

− 4

− 2

0 2

4 6

−2 0 2 4 6

− 1

0 1

2 3

Fitted Values R

e s id

u a

−2 0 2 4 6

0 2

4 6

Fitted Values

S tu

d e

n ti z e

d R

e s id

u a

FIGURE 3.12. Left: The least squares regression line is shown in red, and the regression line after removing the outlier is shown in blue. Center: The residual plot clearly identifies the outlier. Right: The outlier has a studentized residual of 6; typically we expect values between −3 and 3.

of Figure 3.11 displays the residual plot after transforming the response using log Y . The residuals now appear to have constant variance, though there is some evidence of a slight non-linear relationship in the data. Sometimes we have a good idea of the variance of each response. For

example, the ith response could be an average of ni raw observations. If each of these raw observations is uncorrelated with variance σ2, then their average has variance σ2i = σ

2/ni. In this case a simple remedy is to fit our model by weighted least squares, with weights proportional to the inverse

weighted least squaresvariances—i.e. wi = ni in this case. Most linear regression software allows

for observation weights.

4. Outliers

An outlier is a point for which yi is far from the value predicted by the outlier

model. Outliers can arise for a variety of reasons, such as incorrect recording of an observation during data collection. The red point (observation 20) in the left-hand panel of Figure 3.12

illustrates a typical outlier. The red solid line is the least squares regression fit, while the blue dashed line is the least squares fit after removal of the outlier. In this case, removing the outlier has little effect on the least squares line: it leads to almost no change in the slope, and a miniscule reduction in the intercept. It is typical for an outlier that does not have an unusual predictor value to have little effect on the least squares fit. However, even if an outlier does not have much effect on the least squares fit, it can cause other problems. For instance, in this example, the RSE is 1.09 when the outlier is included in the regression, but it is only 0.77 when the outlier is removed. Since the RSE is used to compute all confidence intervals and p-values, such a dramatic increase caused by a single data point can have implications for the interpretation of the fit. Similarly, inclusion of the outlier causes the R2 to decline from 0.892 to 0.805.

98 3. Linear Regression

−2 −1 0 1 2 3 4

0 5

1 0

−2 −1 0 1 2

− 2

− 1

0 1

0.00 0.05 0.10 0.15 0.20 0.25

− 1

0 1

2 3

4 5

Leverage

S tu

d e n ti z e d R

e s id

u a ls

X1 X

FIGURE 3.13. Left: Observation 41 is a high leverage point, while 20 is not. The red line is the fit to all the data, and the blue line is the fit with observation 41 removed. Center: The red observation is not unusual in terms of its X1 value or its X2 value, but still falls outside the bulk of the data, and hence has high leverage. Right: Observation 41 has a high leverage and a high residual.

Residual plots can be used to identify outliers. In this example, the out- lier is clearly visible in the residual plot illustrated in the center panel of Figure 3.12. But in practice, it can be difficult to decide how large a resid- ual needs to be before we consider the point to be an outlier. To address this problem, instead of plotting the residuals, we can plot the studentized residuals, computed by dividing each residual ei by its estimated standard

studentized residualerror. Observations whose studentized residuals are greater than 3 in abso-

lute value are possible outliers. In the right-hand panel of Figure 3.12, the outlier’s studentized residual exceeds 6, while all other observations have studentized residuals between −2 and 2. If we believe that an outlier has occurred due to an error in data collec-

tion or recording, then one solution is to simply remove the observation. However, care should be taken, since an outlier may instead indicate a deficiency with the model, such as a missing predictor.

5. High Leverage Points

We just saw that outliers are observations for which the response yi is unusual given the predictor xi. In contrast, observations with high leverage

high leveragehave an unusual value for xi. For example, observation 41 in the left-hand

panel of Figure 3.13 has high leverage, in that the predictor value for this observation is large relative to the other observations. (Note that the data displayed in Figure 3.13 are the same as the data displayed in Figure 3.12, but with the addition of a single high leverage observation.) The red solid line is the least squares fit to the data, while the blue dashed line is the fit produced when observation 41 is removed. Comparing the left-hand panels of Figures 3.12 and 3.13, we observe that removing the high leverage observation has a much more substantial impact on the least squares line than removing the outlier. In fact, high leverage observations tend to have a sizable impact on the estimated regression line. It is cause for concern if

3.3 Other Considerations in the Regression Model 99

the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit. For this reason, it is important to identify high leverage observations. In a simple linear regression, high leverage observations are fairly easy to

identify, since we can simply look for observations for which the predictor value is outside of the normal range of the observations. But in a multiple linear regression with many predictors, it is possible to have an observation that is well within the range of each individual predictor’s values, but that is unusual in terms of the full set of predictors. An example is shown in the center panel of Figure 3.13, for a data set with two predictors, X1 and X2. Most of the observations’ predictor values fall within the blue dashed ellipse, but the red observation is well outside of this range. But neither its value for X1 nor its value for X2 is unusual. So if we examine just X1 or just X2, we will fail to notice this high leverage point. This problem is more pronounced in multiple regression settings with more than two predictors, because then there is no simple way to plot all dimensions of the data simultaneously. In order to quantify an observation’s leverage, we compute the leverage

statistic. A large value of this statistic indicates an observation with high leverage statisticleverage. For a simple linear regression,

hi = 1

n +

(xi − x̄)2∑n i′=1(xi′ − x̄)2

. (3.37)

It is clear from this equation that hi increases with the distance of xi from x̄. There is a simple extension of hi to the case of multiple predictors, though we do not provide the formula here. The leverage statistic hi is always between 1/n and 1, and the average leverage for all the observations is always equal to (p + 1)/n. So if a given observation has a leverage statistic that greatly exceeds (p+1)/n, then we may suspect that the corresponding point has high leverage. The right-hand panel of Figure 3.13 provides a plot of the studentized

residuals versus hi for the data in the left-hand panel of Figure 3.13. Ob- servation 41 stands out as having a very high leverage statistic as well as a high studentized residual. In other words, it is an outlier as well as a high leverage observation. This is a particularly dangerous combination! This plot also reveals the reason that observation 20 had relatively little effect on the least squares fit in Figure 3.12: it has low leverage.

6. Collinearity

Collinearity refers to the situation in which two or more predictor variables collinearity

are closely related to one another. The concept of collinearity is illustrated in Figure 3.14 using the Credit data set. In the left-hand panel of Fig- ure 3.14, the two predictors limit and age appear to have no obvious rela- tionship. In contrast, in the right-hand panel of Figure 3.14, the predictors

100 3. Linear Regression

2000 4000 6000 8000 12000

3 0

4 0

5 0

6 0

7 0

8 0

Limit

A g e

2000 4000 6000 8000 12000

2 0 0

4 0 0

6 0

0 8 0 0

Limit R

a ti n g

FIGURE 3.14. Scatterplots of the observations from the Credit data set. Left: A plot of age versus limit. These two variables are not collinear. Right: A plot of rating versus limit. There is high collinearity.

21.25

21.5

21.8

0.16 0.17 0.18 0.19

− 5

− 4

− 3

− 2

− 1

21.5

21.8

−0.1 0.0 0.1 0.2

0 1

2 3

4 5

βLimitβLimit

β A g e

β R a ti n g

FIGURE 3.15. Contour plots for the RSS values as a function of the parameters β for various regressions involving the Credit data set. In each plot, the black dots represent the coefficient values corresponding to the minimum RSS. Left: A contour plot of RSS for the regression of balance onto age and limit. The minimum value is well defined. Right: A contour plot of RSS for the regression of balance onto rating and limit. Because of the collinearity, there are many pairs (βLimit, βRating) with a similar value for RSS.

limit and rating are very highly correlated with each other, and we say that they are collinear. The presence of collinearity can pose problems in the regression context, since it can be difficult to separate out the indi- vidual effects of collinear variables on the response. In other words, since limit and rating tend to increase or decrease together, it can be difficult to determine how each one separately is associated with the response, balance.

3.3 Other Considerations in the Regression Model 101

Figure 3.15 illustrates some of the difficulties that can result from collinear- ity. The left-hand panel of Figure 3.15 is a contour plot of the RSS (3.22) associated with different possible coefficient estimates for the regression of balance on limit and age. Each ellipse represents a set of coefficients that correspond to the same RSS, with ellipses nearest to the center tak- ing on the lowest values of RSS. The black dots and associated dashed lines represent the coefficient estimates that result in the smallest possible RSS—in other words, these are the least squares estimates. The axes for limit and age have been scaled so that the plot includes possible coeffi- cient estimates that are up to four standard errors on either side of the least squares estimates. Thus the plot includes all plausible values for the coefficients. For example, we see that the true limit coefficient is almost certainly somewhere between 0.15 and 0.20. In contrast, the right-hand panel of Figure 3.15 displays contour plots

of the RSS associated with possible coefficient estimates for the regression of balance onto limit and rating, which we know to be highly collinear. Now the contours run along a narrow valley; there is a broad range of values for the coefficient estimates that result in equal values for RSS. Hence a small change in the data could cause the pair of coefficient values that yield the smallest RSS—that is, the least squares estimates—to move anywhere along this valley. This results in a great deal of uncertainty in the coefficient estimates. Notice that the scale for the limit coefficient now runs from roughly −0.2 to 0.2; this is an eight-fold increase over the plausible range of the limit coefficient in the regression with age. Interestingly, even though the limit and rating coefficients now have much more individual uncertainty, they will almost certainly lie somewhere in this contour valley. For example, we would not expect the true value of the limit and rating coefficients to be −0.1 and 1 respectively, even though such a value is plausible for each coefficient individually. Since collinearity reduces the accuracy of the estimates of the regression

coefficients, it causes the standard error for β̂j to grow. Recall that the

t-statistic for each predictor is calculated by dividing β̂j by its standard error. Consequently, collinearity results in a decline in the t-statistic. As a result, in the presence of collinearity, we may fail to reject H0 : βj = 0. This means that the power of the hypothesis test—the probability of correctly power detecting a non-zero coefficient—is reduced by collinearity. Table 3.11 compares the coefficient estimates obtained from two separate

multiple regression models. The first is a regression of balance on age and limit, and the second is a regression of balance on rating and limit. In the first regression, both age and limit are highly significant with very small p- values. In the second, the collinearity between limit and rating has caused the standard error for the limit coefficient estimate to increase by a factor of 12 and the p-value to increase to 0.701. In other words, the importance of the limit variable has been masked due to the presence of collinearity.

102 3. Linear Regression

Coefficient Std. error t-statistic p-value Intercept −173.411 43.828 −3.957 < 0.0001

Model 1 age −2.292 0.672 −3.407 0.0007 limit 0.173 0.005 34.496 < 0.0001 Intercept −377.537 45.254 −8.343 < 0.0001

Model 2 rating 2.202 0.952 2.312 0.0213 limit 0.025 0.064 0.384 0.7012

TABLE 3.11. The results for two multiple regression models involving the Credit data set are shown. Model 1 is a regression of balance on age and limit, and Model 2 a regression of balance on rating and limit. The standard error of β̂limit increases 12-fold in the second regression, due to collinearity.

To avoid such a situation, it is desirable to identify and address potential collinearity problems while fitting the model. A simple way to detect collinearity is to look at the correlation matrix

of the predictors. An element of this matrix that is large in absolute value indicates a pair of highly correlated variables, and therefore a collinearity problem in the data. Unfortunately, not all collinearity problems can be detected by inspection of the correlation matrix: it is possible for collinear- ity to exist between three or more variables even if no pair of variables has a particularly high correlation. We call this situation multicollinearity.

multi- collinearityInstead of inspecting the correlation matrix, a better way to assess multi-

collinearity is to compute the variance inflation factor (VIF). The VIF is variance inflation factor

the ratio of the variance of β̂j when fitting the full model divided by the

variance of β̂j if fit on its own. The smallest possible value for VIF is 1, which indicates the complete absence of collinearity. Typically in practice there is a small amount of collinearity among the predictors. As a rule of thumb, a VIF value that exceeds 5 or 10 indicates a problematic amount of collinearity. The VIF for each variable can be computed using the formula

VIF(β̂j) = 1

1−R2 Xj|X−j

where R2Xj|X−j is the R 2 from a regression of Xj onto all of the other

predictors. If R2Xj|X−j is close to one, then collinearity is present, and so the VIF will be large. In the Credit data, a regression of balance on age, rating, and limit

indicates that the predictors have VIF values of 1.01, 160.67, and 160.59. As we suspected, there is considerable collinearity in the data! When faced with the problem of collinearity, there are two simple solu-

tions. The first is to drop one of the problematic variables from the regres- sion. This can usually be done without much compromise to the regression fit, since the presence of collinearity implies that the information that this variable provides about the response is redundant in the presence of the other variables. For instance, if we regress balance onto age and limit,

3.4 The Marketing Plan 103

without the rating predictor, then the resulting VIF values are close to the minimum possible value of 1, and the R2 drops from 0.754 to 0.75. So dropping rating from the set of predictors has effectively solved the collinearity problem without compromising the fit. The second solution is to combine the collinear variables together into a single predictor. For in- stance, we might take the average of standardized versions of limit and rating in order to create a new variable that measures credit worthiness.

3.4 The Marketing Plan

We now briefly return to the seven questions about the Advertising data that we set out to answer at the beginning of this chapter.

1. Is there a relationship between sales and advertising budget? This question can be answered by fitting a multiple regression model of sales onto TV, radio, and newspaper, as in (3.20), and testing the hypothesis H0 : βTV = βradio = βnewspaper = 0. In Section 3.2.2, we showed that the F-statistic can be used to determine whether or not we should reject this null hypothesis. In this case the p-value corresponding to the F-statistic in Table 3.6 is very low, indicating clear evidence of a relationship between advertising and sales.

2. How strong is the relationship? We discussed two measures of model accuracy in Section 3.1.3. First, the RSE estimates the standard deviation of the response from the population regression line. For the Advertising data, the RSE is 1.69 units while the mean value for the response is 14.022, indicating a percentage error of roughly 12 %. Second, the R2 statistic records the percentage of variability in the response that is explained by the predictors. The predictors explain almost 90 % of the variance in sales. The RSE and R2 statistics are displayed in Table 3.6.

3. Which media are associated with sales? To answer this question, we can examine the p-values associated with each predictor’s t-statistic (Section 3.1.2). In the multiple linear re- gression displayed in Table 3.4, the p-values for TV and radio are low, but the p-value for newspaper is not. This suggests that only TV and radio are related to sales. In Chapter 6 we explore this question in greater detail.

4. How large is the association between each medium and sales? We saw in Section 3.1.2 that the standard error of β̂j can be used to construct confidence intervals for βj. For the Advertising data, we

104 3. Linear Regression

can use the results in Table 3.4 to compute the 95 % confidence inter- vals for the coefficients in a multiple regression model using all three media budgets as predictors. The confidence intervals are as follows: (0.043, 0.049) for TV, (0.172, 0.206) for radio, and (−0.013, 0.011) for newspaper. The confidence intervals for TV and radio are narrow and far from zero, providing evidence that these media are related to sales. But the interval for newspaper includes zero, indicating that the variable is not statistically significant given the values of TV and radio.

We saw in Section 3.3.3 that collinearity can result in very wide stan- dard errors. Could collinearity be the reason that the confidence in- terval associated with newspaper is so wide? The VIF scores are 1.005, 1.145, and 1.145 for TV, radio, and newspaper, suggesting no evidence of collinearity.

In order to assess the association of each medium individually on sales, we can perform three separate simple linear regressions. Re- sults are shown in Tables 3.1 and 3.3. There is evidence of an ex- tremely strong association between TV and sales and between radio and sales. There is evidence of a mild association between newspaper and sales, when the values of TV and radio are ignored.

5. How accurately can we predict future sales? The response can be predicted using (3.21). The accuracy associ- ated with this estimate depends on whether we wish to predict an individual response, Y = f(X) + ϵ, or the average response, f(X) (Section 3.2.2). If the former, we use a prediction interval, and if the latter, we use a confidence interval. Prediction intervals will always be wider than confidence intervals because they account for the un- certainty associated with ϵ, the irreducible error.

6. Is the relationship linear? In Section 3.3.3, we saw that residual plots can be used in order to identify non-linearity. If the relationships are linear, then the residual plots should display no pattern. In the case of the Advertising data, we observe a non-linear effect in Figure 3.5, though this effect could also be observed in a residual plot. In Section 3.3.2, we discussed the inclusion of transformations of the predictors in the linear regression model in order to accommodate non-linear relationships.

7. Is there synergy among the advertising media? The standard linear regression model assumes an additive relation- ship between the predictors and the response. An additive model is easy to interpret because the association between each predictor and the response is unrelated to the values of the other predictors. However, the additive assumption may be unrealistic for certain data

105

sets. In Section 3.3.2, we showed how to include an interaction term in the regression model in order to accommodate non-additive rela- tionships. A small p-value associated with the interaction term indi- cates the presence of such relationships. Figure 3.5 suggested that the Advertising data may not be additive. Including an interaction term in the model results in a substantial increase in R2, from around 90 % to almost 97 %.

3.5 Comparison of Linear Regression with K-Nearest Neighbors

As discussed in Chapter 2, linear regression is an example of a parametric approach because it assumes a linear functional form for f(X). Parametric methods have several advantages. They are often easy to fit, because one need estimate only a small number of coefficients. In the case of linear re- gression, the coefficients have simple interpretations, and tests of statistical significance can be easily performed. But parametric methods do have a disadvantage: by construction, they make strong assumptions about the form of f(X). If the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. For instance, if we assume a linear relationship between X and Y but the true relationship is far from linear, then the resulting model will provide a poor fit to the data, and any conclusions drawn from it will be suspect. In contrast, non-parametric methods do not explicitly assume a para-

metric form for f(X), and thereby provide an alternative and more flexi- ble approach for performing regression. We discuss various non-parametric methods in this book. Here we consider one of the simplest and best-known non-parametric methods, K-nearest neighbors regression (KNN regression).

K-nearest neighbors regression

The KNN regression method is closely related to the KNN classifier dis- cussed in Chapter 2. Given a value for K and a prediction point x0, KNN regression first identifies the K training observations that are closest to x0, represented by N0. It then estimates f(x0) using the average of all the training responses in N0. In other words,

f̂(x0) = 1

∑

xi∈N0

yi.

Figure 3.16 illustrates two KNN fits on a data set with p = 2 predictors. The fit with K = 1 is shown in the left-hand panel, while the right-hand panel corresponds to K = 9. We see that when K = 1, the KNN fit perfectly interpolates the training observations, and consequently takes the form of a step function. When K = 9, the KNN fit still is a step function, but averaging over nine observations results in much smaller regions of constant

3.5 Comparison of Linear Regression with K-Nearest Neighbors

106 3. Linear Regression

x1x1

x 2x 2

y yy

FIGURE 3.16. Plots of f̂(X) using KNN regression on a two-dimensional data set with 64 observations (orange dots). Left: K = 1 results in a rough step func- tion fit. Right: K = 9 produces a much smoother fit.

prediction, and consequently a smoother fit. In general, the optimal value for K will depend on the bias-variance tradeoff, which we introduced in Chapter 2. A small value for K provides the most flexible fit, which will have low bias but high variance. This variance is due to the fact that the prediction in a given region is entirely dependent on just one observation. In contrast, larger values of K provide a smoother and less variable fit; the prediction in a region is an average of several points, and so changing one observation has a smaller effect. However, the smoothing may cause bias by masking some of the structure in f(X). In Chapter 5, we introduce several approaches for estimating test error rates. These methods can be used to identify the optimal value of K in KNN regression. In what setting will a parametric approach such as least squares linear re-

gression outperform a non-parametric approach such as KNN regression? The answer is simple: the parametric approach will outperform the non- parametric approach if the parametric form that has been selected is close to the true form of f. Figure 3.17 provides an example with data generated from a one-dimensional linear regression model. The black solid lines rep- resent f(X), while the blue curves correspond to the KNN fits using K = 1 and K = 9. In this case, the K = 1 predictions are far too variable, while the smoother K = 9 fit is much closer to f(X). However, since the true relationship is linear, it is hard for a non-parametric approach to compete with linear regression: a non-parametric approach incurs a cost in variance that is not offset by a reduction in bias. The blue dashed line in the left- hand panel of Figure 3.18 represents the linear regression fit to the same data. It is almost perfect. The right-hand panel of Figure 3.18 reveals that linear regression outperforms KNN for this data. The green solid line, plot-

3.5 Comparison of Linear Regression with K-Nearest Neighbors 107

ted as a function of 1/K, represents the test set mean squared error (MSE) for KNN. The KNN errors are well above the black dashed line, which is the test MSE for linear regression. When the value of K is large, then KNN performs only a little worse than least squares regression in terms of MSE. It performs far worse when K is small. In practice, the true relationship between X and Y is rarely exactly lin-

ear. Figure 3.19 examines the relative performances of least squares regres- sion and KNN under increasing levels of non-linearity in the relationship between X and Y . In the top row, the true relationship is nearly linear. In this case we see that the test MSE for linear regression is still superior to that of KNN for low values of K. However, for K ≥ 4, KNN out- performs linear regression. The second row illustrates a more substantial deviation from linearity. In this situation, KNN substantially outperforms linear regression for all values of K. Note that as the extent of non-linearity increases, there is little change in the test set MSE for the non-parametric KNN method, but there is a large increase in the test set MSE of linear regression. Figures 3.18 and 3.19 display situations in which KNN performs slightly

worse than linear regression when the relationship is linear, but much bet- ter than linear regression for non-linear situations. In a real life situation in which the true relationship is unknown, one might suspect that KNN should be favored over linear regression because it will at worst be slightly inferior to linear regression if the true relationship is linear, and may give substantially better results if the true relationship is non-linear. But in re- ality, even when the true relationship is highly non-linear, KNN may still provide inferior results to linear regression. In particular, both Figures 3.18 and 3.19 illustrate settings with p = 1 predictor. But in higher dimensions, KNN often performs worse than linear regression. Figure 3.20 considers the same strongly non-linear situation as in the

second row of Figure 3.19, except that we have added additional noise predictors that are not associated with the response. When p = 1 or p = 2, KNN outperforms linear regression. But for p = 3 the results are mixed, and for p ≥ 4 linear regression is superior to KNN. In fact, the increase in dimension has only caused a small deterioration in the linear regression test set MSE, but it has caused more than a ten-fold increase in the MSE for KNN. This decrease in performance as the dimension increases is a common problem for KNN, and results from the fact that in higher dimensions there is effectively a reduction in sample size. In this data set there are 50 training observations; when p = 1, this provides enough information to accurately estimate f(X). However, spreading 50 observations over p = 20 dimensions results in a phenomenon in which a given observation has no nearby neighbors—this is the so-called curse of dimensionality. That is,

curse of di- mensionalitythe K observations that are nearest to a given test observation x0 may be

very far away from x0 in p-dimensional space when p is large, leading to a very poor prediction of f(x0) and hence a poor KNN fit. As a general rule,

108 3. Linear Regression

−1.0 −0.5 0.0 0.5 1.0

1 2

3 4

−1.0 −0.5 0.0 0.5 1.0

1 2

3 4

FIGURE 3.17. Plots of f̂(X) using KNN regression on a one-dimensional data set with 50 observations. The true relationship is given by the black solid line. Left: The blue curve corresponds to K = 1 and interpolates (i.e. passes directly through) the training data. Right: The blue curve corresponds to K = 9, and represents a smoother fit.

−1.0 −0.5 0.0 0.5 1.0

1 2

3 4

0.2 0.5 1.0

0 .0

0 0 .0

5 0 .1

0 0 .1

M e a n S

q u a re

d E

rr o r

x 1/K

FIGURE 3.18. The same data set shown in Figure 3.17 is investigated further. Left: The blue dashed line is the least squares fit to the data. Since f(X) is in fact linear (displayed as the black line), the least squares regression line provides a very good estimate of f(X). Right: The dashed horizontal line represents the least squares test set MSE, while the green solid line corresponds to the MSE for KNN as a function of 1/K (on the log scale). Linear regression achieves a lower test MSE than does KNN regression, since f(X) is in fact linear. For KNN regression, the best results occur with a very large value of K, corresponding to a small value of 1/K.

3.5 Comparison of Linear Regression with K-Nearest Neighbors 109

−1.0 −0.5 0.0 0.5 1.0

0 .5

1 .0

1 .5

2 .0

2 .5

3 .0

3 .5

0.2 0.5 1.0

0 .0

0 0 .0

2 0 .0

4 0 .0

6 0 .0

M e a n S

q u a re

d E

rr o r

−1.0 −0.5 0.0 0.5 1.0

1 .0

1 .5

2 .0

2 .5

3 .0

3 .5

0.2 0.5 1.0

0 .0

0 0 .0

5 0 .1

0 0 .1

M e a n S

q u a re

d E

rr o r

y y

1/K

FIGURE 3.19. Top Left: In a setting with a slightly non-linear relationship between X and Y (solid black line), the KNN fits with K = 1 (blue) and K = 9 (red) are displayed. Top Right: For the slightly non-linear data, the test set MSE for least squares regression (horizontal black) and KNN with various values of 1/K (green) are displayed. Bottom Left and Bottom Right: As in the top panel, but with a strongly non-linear relationship between X and Y .

110 3. Linear Regression

0.2 0.5 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=1

0.2 0.5 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=2

0.2 0.5 1.0 0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=3

0.2 0.5 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=4

0.2 0.5 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=10

0.2 0.5 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

p=20

M e a n S

q u a re

d E

rr o r

1/K

FIGURE 3.20. Test MSE for linear regression (black dashed lines) and KNN (green curves) as the number of variables p increases. The true function is non– linear in the first variable, as in the lower panel in Figure 3.19, and does not depend on the additional variables. The performance of linear regression deteri- orates slowly in the presence of these additional noise variables, whereas KNN’s performance degrades much more quickly as p increases.

parametric methods will tend to outperform non-parametric approaches when there is a small number of observations per predictor. Even when the dimension is small, we might prefer linear regression to

KNN from an interpretability standpoint. If the test MSE of KNN is only slightly lower than that of linear regression, we might be willing to forego a little bit of prediction accuracy for the sake of a simple model that can be described in terms of just a few coefficients, and for which p-values are available.

3.6 Lab: Linear Regression

3.6.1 Libraries

The library() function is used to load libraries, or groups of functions library()

and data sets that are not included in the base R distribution. Basic func- tions that perform least squares linear regression and other simple analyses come standard with the base distribution, but more exotic functions require additional libraries. Here we load the MASS package, which is a very large collection of data sets and functions. We also load the ISLR2 package, which includes the data sets associated with this book.

> library(MASS)

> library(ISLR2)

If you receive an error message when loading any of these libraries, it likely indicates that the corresponding library has not yet been installed on your system. Some libraries, such as MASS, come with R and do not need to be separately installed on your computer. However, other packages, such as

3.6 Lab: Linear Regression 111

ISLR2, must be downloaded the first time they are used. This can be done di- rectly from within R. For example, on a Windows system, select the Install package option under the Packages tab. After you select any mirror site, a list of available packages will appear. Simply select the package you wish to install and R will automatically download the package. Alternatively, this can be done at the R command line via install.packages("ISLR2"). This installation only needs to be done the first time you use a package. However, the library() function must be called within each R session.

3.6.2 Simple Linear Regression

The ISLR2 library contains the Boston data set, which records medv (me- dian house value) for 506 census tracts in Boston. We will seek to predict medv using 12 predictors such as rm (average number of rooms per house), age (average age of houses), and lstat (percent of households with low socioeconomic status).

> head(Boston)

crim zn indus chas nox rm age dis rad tax

1 0.00632 18 2.31 0 0.538 6.575 65.2 4.0900 1 296

2 0.02731 0 7.07 0 0.469 6.421 78.9 4.9671 2 242

3 0.02729 0 7.07 0 0.469 7.185 61.1 4.9671 2 242

4 0.03237 0 2.18 0 0.458 6.998 45.8 6.0622 3 222

5 0.06905 0 2.18 0 0.458 7.147 54.2 6.0622 3 222

6 0.02985 0 2.18 0 0.458 6.430 58.7 6.0622 3 222

ptratio lstat medv

1 15.3 4.98 24.0

2 17.8 9.14 21.6

3 17.8 4.03 34.7

4 18.7 2.94 33.4

5 18.7 5.33 36.2

6 18.7 5.21 28.7

To find out more about the data set, we can type ?Boston. We will start by using the lm() function to fit a simple linear regression

lm() model, with medv as the response and lstat as the predictor. The basic syntax is lm(y ∼ x, data), where y is the response, x is the predictor, and data is the data set in which these two variables are kept.

> lm.fit <- lm(medv ∼ lstat) Error in eval(expr , envir , enclos) : Object "medv" not found

The command causes an error because R does not know where to find the variables medv and lstat. The next line tells R that the variables are in Boston. If we attach Boston, the first line works fine because R now recognizes the variables.

> lm.fit <- lm(medv ∼ lstat , data = Boston) > attach(Boston)

> lm.fit <- lm(medv ∼ lstat)

112 3. Linear Regression

If we type lm.fit, some basic information about the model is output. For more detailed information, we use summary(lm.fit). This gives us p- values and standard errors for the coefficients, as well as the R2 statistic and F-statistic for the model.

> lm.fit

Call:

lm(formula = medv ∼ lstat)

Coefficients :

(Intercept) lstat

34.55 -0.95

> summary(lm.fit)

Call:

lm(formula = medv ∼ lstat)

Residuals:

Min 1Q Median 3Q Max

-15.17 -3.99 -1.32 2.03 24.50

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 34.5538 0.5626 61.4 <2e-16 ***

lstat -0.9500 0.0387 -24.5 <2e -16 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 6.22 on 504 degrees of freedom

Multiple R-squared: 0.544 , Adjusted R-squared: 0.543

F-statistic: 602 on 1 and 504 DF , p-value: < 2e-16

We can use the names() function in order to find out what other pieces names()

of information are stored in lm.fit. Although we can extract these quan- tities by name—e.g. lm.fit$coefficients—it is safer to use the extractor functions like coef() to access them.

coef()

> names(lm.fit)

[1] " coefficients " "residuals" "effects"

[4] "rank" "fitted.values" "assign"

[7] "qr" "df.residual" "xlevels"

[10] "call" "terms" "model"

> coef(lm.fit)

(Intercept) lstat

34.55 -0.95

In order to obtain a confidence interval for the coefficient estimates, we can use the confint() command.

confint()

> confint(lm.fit)

2.5 % 97.5 %

(Intercept) 33.45 35.659

3.6 Lab: Linear Regression 113

lstat -1.03 -0.874

The predict() function can be used to produce confidence intervals and predict()

prediction intervals for the prediction of medv for a given value of lstat.

> predict(lm.fit , data.frame(lstat = (c(5, 10, 15))),

interval = "confidence ")

fit lwr upr

1 29.80 29.01 30.60

2 25.05 24.47 25.63

3 20.30 19.73 20.87

> predict(lm.fit , data.frame(lstat = (c(5, 10, 15))),

interval = "prediction ")

fit lwr upr

1 29.80 17.566 42.04

2 25.05 12.828 37.28

3 20.30 8.078 32.53

For instance, the 95 % confidence interval associated with a lstat value of 10 is (24.47, 25.63), and the 95 % prediction interval is (12.828, 37.28). As expected, the confidence and prediction intervals are centered around the same point (a predicted value of 25.05 for medv when lstat equals 10), but the latter are substantially wider. We will now plot medv and lstat along with the least squares regression

line using the plot() and abline() functions. abline()

> plot(lstat , medv)

> abline(lm.fit)

There is some evidence for non-linearity in the relationship between lstat and medv. We will explore this issue later in this lab. The abline() function can be used to draw any line, not just the least

squares regression line. To draw a line with intercept a and slope b, we type abline(a, b). Below we experiment with some additional settings for plotting lines and points. The lwd = 3 command causes the width of the regression line to be increased by a factor of 3; this works for the plot() and lines() functions also. We can also use the pch option to create different plotting symbols.

> abline(lm.fit , lwd = 3)

> abline(lm.fit , lwd = 3, col = "red")

> plot(lstat , medv , col = "red")

> plot(lstat , medv , pch = 20)

> plot(lstat , medv , pch = "+")

> plot (1:20 , 1:20 , pch = 1:20)

Next we examine some diagnostic plots, several of which were discussed in Section 3.3.3. Four diagnostic plots are automatically produced by ap- plying the plot() function directly to the output from lm(). In general, this command will produce one plot at a time, and hitting Enter will generate the next plot. However, it is often convenient to view all four plots together. We can achieve this by using the par() and mfrow() functions, which tell R

par()

mfrow()

114 3. Linear Regression

to split the display screen into separate panels so that multiple plots can be viewed simultaneously. For example, par(mfrow = c(2, 2)) divides the plotting region into a 2 ×2 grid of panels. > par(mfrow = c(2, 2))

> plot(lm.fit)

Alternatively, we can compute the residuals from a linear regression fit using the residuals() function. The function rstudent() will return the

residuals()

rstudent() studentized residuals, and we can use this function to plot the residuals against the fitted values.

> plot(predict(lm.fit), residuals (lm.fit))

> plot(predict(lm.fit), rstudent (lm.fit))

On the basis of the residual plots, there is some evidence of non-linearity. Leverage statistics can be computed for any number of predictors using the hatvalues() function.

hatvalues()

> plot(hatvalues (lm.fit))

> which.max(hatvalues (lm.fit))

375

The which.max() function identifies the index of the largest element of a which.max()

vector. In this case, it tells us which observation has the largest leverage statistic.

3.6.3 Multiple Linear Regression

In order to fit a multiple linear regression model using least squares, we again use the lm() function. The syntax lm(y ∼ x1 + x2 + x3) is used to fit a model with three predictors, x1, x2, and x3. The summary() function now outputs the regression coefficients for all the predictors.

> lm.fit <- lm(medv ∼ lstat + age , data = Boston) > summary(lm.fit)

Call:

lm(formula = medv ∼ lstat + age , data = Boston)

Residuals:

Min 1Q Median 3Q Max

-15.98 -3.98 -1.28 1.97 23.16

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 33.2228 0.7308 45.46 <2e-16 ***

lstat -1.0321 0.0482 -21.42 <2e-16 ***

age 0.0345 0.0122 2.83 0.0049 **

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 6.17 on 503 degrees of freedom

3.6 Lab: Linear Regression 115

Multiple R-squared: 0.551 , Adjusted R-squared: 0.549

F-statistic: 309 on 2 and 503 DF , p-value: < 2e-16

The Boston data set contains 12 variables, and so it would be cumbersome to have to type all of these in order to perform a regression using all of the predictors. Instead, we can use the following short-hand:

> lm.fit <- lm(medv ∼ ., data = Boston) > summary(lm.fit)

Call:

lm(formula = medv ∼ ., data = Boston)

Residuals:

Min 1Q Median 3Q Max

-15.130 -2.767 -0.581 1.941 26.253

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 41.61727 4.93604 8.43 3.8e-16 ***

crim -0.12139 0.03300 -3.68 0.00026 ***

zn 0.04696 0.01388 3.38 0.00077 ***

indus 0.01347 0.06214 0.22 0.82852

chas 2.83999 0.87001 3.26 0.00117 **

nox -18.75802 3.85135 -4.87 1.5e-06 ***

rm 3.65812 0.42025 8.70 < 2e -16 ***

age 0.00361 0.01333 0.27 0.78659

dis -1.49075 0.20162 -7.39 6.2e -13 ***

rad 0.28940 0.06691 4.33 1.8e-05 ***

tax -0.01268 0.00380 -3.34 0.00091 ***

ptratio -0.93753 0.13221 -7.09 4.6e-12 ***

lstat -0.55202 0.05066 -10.90 < 2e-16 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 4.8 on 493 degrees of freedom

Multiple R-squared: 0.734 , Adjusted R-squared: 0.728

F-statistic: 114 on 12 and 493 DF , p-value: < 2e-16

We can access the individual components of a summary object by name (type ?summary.lm to see what is available). Hence summary(lm.fit)$r.sq gives us the R2, and summary(lm.fit)$sigma gives us the RSE. The vif()

vif() function, part of the car package, can be used to compute variance inflation factors. Most VIF’s are low to moderate for this data. The car package is not part of the base R installation so it must be downloaded the first time you use it via the install.packages() function in R.

> library(car)

> vif(lm.fit)

crim zn indus chas nox rm age dis

1.77 2.30 3.99 1.07 4.37 1.91 3.09 3.95

rad tax ptratio lstat

7.45 9.00 1.80 2.87

116 3. Linear Regression

What if we would like to perform a regression using all of the variables but one? For example, in the above regression output, age has a high p-value. So we may wish to run a regression excluding this predictor. The following syntax results in a regression using all predictors except age.

> lm.fit1 <- lm(medv ∼ . - age , data = Boston) > summary(lm.fit1)

...

Alternatively, the update() function can be used. update()

> lm.fit1 <- update(lm.fit , ∼ . - age)

3.6.4 Interaction Terms

It is easy to include interaction terms in a linear model using the lm() function. The syntax lstat:black tells R to include an interaction term be- tween lstat and black. The syntax lstat * age simultaneously includes lstat, age, and the interaction term lstat×age as predictors; it is a short- hand for lstat + age + lstat:age.

> summary(lm(medv ∼ lstat * age , data = Boston))

Call:

lm(formula = medv ∼ lstat * age , data = Boston)

Residuals:

Min 1Q Median 3Q Max

-15.81 -4.04 -1.33 2.08 27.55

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 36.088536 1.469835 24.55 < 2e-16 ***

lstat -1.392117 0.167456 -8.31 8.8e -16 ***

age -0.000721 0.019879 -0.04 0.971

lstat:age 0.004156 0.001852 2.24 0.025 *

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 6.15 on 502 degrees of freedom

Multiple R-squared: 0.556 , Adjusted R-squared: 0.553

F-statistic: 209 on 3 and 502 DF , p-value: < 2e-16

3.6.5 Non-linear Transformations of the Predictors

The lm() function can also accommodate non-linear transformations of the predictors. For instance, given a predictor X, we can create a predictor X2

using I(X^2). The function I() is needed since the ^ has a special meaning I()

in a formula object; wrapping as we do allows the standard usage in R, which is to raise X to the power 2. We now perform a regression of medv onto lstat and lstat2.

3.6 Lab: Linear Regression 117

> lm.fit2 <- lm(medv ∼ lstat + I(lstat ^2)) > summary(lm.fit2)

Call:

lm(formula = medv ∼ lstat + I(lstat ^2))

Residuals:

Min 1Q Median 3Q Max

-15.28 -3.83 -0.53 2.31 25.41

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 42.86201 0.87208 49.1 <2e -16 ***

lstat -2.33282 0.12380 -18.8 <2e -16 ***

I(lstat ^2) 0.04355 0.00375 11.6 <2e-16 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 5.52 on 503 degrees of freedom

Multiple R-squared: 0.641 , Adjusted R-squared: 0.639

F-statistic: 449 on 2 and 503 DF , p-value: < 2e-16

The near-zero p-value associated with the quadratic term suggests that it leads to an improved model. We use the anova() function to further

anova() quantify the extent to which the quadratic fit is superior to the linear fit.

> lm.fit <- lm(medv ∼ lstat) > anova(lm.fit , lm.fit2)

Analysis of Variance Table

Model 1: medv ∼ lstat Model 2: medv ∼ lstat + I(lstat ^2)

Res.Df RSS Df Sum of Sq F Pr(>F)

1 504 19472

2 503 15347 1 4125 135 <2e -16 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Here Model 1 represents the linear submodel containing only one predictor, lstat, while Model 2 corresponds to the larger quadratic model that has two predictors, lstat and lstat2. The anova() function performs a hypothesis test comparing the two models. The null hypothesis is that the two models fit the data equally well, and the alternative hypothesis is that the full model is superior. Here the F-statistic is 135 and the associated p-value is virtually zero. This provides very clear evidence that the model containing the predictors lstat and lstat2 is far superior to the model that only contains the predictor lstat. This is not surprising, since earlier we saw evidence for non-linearity in the relationship between medv and lstat. If we type

> par(mfrow = c(2, 2))

> plot(lm.fit2)

118 3. Linear Regression

then we see that when the lstat2 term is included in the model, there is little discernible pattern in the residuals. In order to create a cubic fit, we can include a predictor of the form

I(X^3). However, this approach can start to get cumbersome for higher- order polynomials. A better approach involves using the poly() function

poly() to create the polynomial within lm(). For example, the following command produces a fifth-order polynomial fit:

> lm.fit5 <- lm(medv ∼ poly(lstat , 5)) > summary(lm.fit5)

Call:

lm(formula = medv ∼ poly(lstat , 5))

Residuals:

Min 1Q Median 3Q Max

-13.543 -3.104 -0.705 2.084 27.115

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 22.533 0.232 97.20 < 2e -16 ***

poly(lstat , 5)1 -152.460 5.215 -29.24 < 2e-16 ***

poly(lstat , 5)2 64.227 5.215 12.32 < 2e -16 ***

poly(lstat , 5)3 -27.051 5.215 -5.19 3.1e-07 ***

poly(lstat , 5)4 25.452 5.215 4.88 1.4e -06 ***

poly(lstat , 5)5 -19.252 5.215 -3.69 0.00025 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 5.21 on 500 degrees of freedom

Multiple R-squared: 0.682 , Adjusted R-squared: 0.679

F-statistic: 214 on 5 and 500 DF , p-value: < 2e-16

This suggests that including additional polynomial terms, up to fifth order, leads to an improvement in the model fit! However, further investigation of the data reveals that no polynomial terms beyond fifth order have signifi- cant p-values in a regression fit. By default, the poly() function orthogonalizes the predictors: this means

that the features output by this function are not simply a sequence of powers of the argument. However, a linear model applied to the output of the poly() function will have the same fitted values as a linear model applied to the raw polynomials (although the coefficient estimates, standard errors, and p-values will differ). In order to obtain the raw polynomials from the poly() function, the argument raw = TRUE must be used. Of course, we are in no way restricted to using polynomial transforma-

tions of the predictors. Here we try a log transformation.

> summary(lm(medv ∼ log(rm), data = Boston)) ...

3.6 Lab: Linear Regression 119

3.6.6 Qualitative Predictors

We will now examine the Carseats data, which is part of the ISLR2 library. We will attempt to predict Sales (child car seat sales) in 400 locations based on a number of predictors.

> head(Carseats)

Sales CompPrice Income Advertising Population Price

1 9.50 138 73 11 276 120

2 11.22 111 48 16 260 83

3 10.06 113 35 10 269 80

4 7.40 117 100 4 466 97

5 4.15 141 64 3 340 128

6 10.81 124 113 13 501 72

ShelveLoc Age Education Urban US

1 Bad 42 17 Yes Yes

2 Good 65 10 Yes Yes

3 Medium 59 12 Yes Yes

4 Medium 55 14 Yes Yes

5 Bad 38 13 Yes No

6 Bad 78 16 No Yes

The Carseats data includes qualitative predictors such as Shelveloc, an in- dicator of the quality of the shelving location—that is, the space within a store in which the car seat is displayed—at each location. The pre- dictor Shelveloc takes on three possible values: Bad, Medium, and Good. Given a qualitative variable such as Shelveloc, R generates dummy variables automatically. Below we fit a multiple regression model that includes some interaction terms.

> lm.fit <- lm(Sales ∼ . + Income:Advertising + Price:Age , data = Carseats)

> summary(lm.fit)

Call:

lm(formula = Sales ∼ . + Income:Advertising + Price:Age , data = Carseats)

Residuals:

Min 1Q Median 3Q Max

-2.921 -0.750 0.018 0.675 3.341

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 6.575565 1.008747 6.52 2.2e-10 ***

CompPrice 0.092937 0.004118 22.57 < 2e-16 ***

Income 0.010894 0.002604 4.18 3.6e -05 ***

Advertising 0.070246 0.022609 3.11 0.00203 **

Population 0.000159 0.000368 0.43 0.66533

Price -0.100806 0.007440 -13.55 < 2e-16 ***

ShelveLocGood 4.848676 0.152838 31.72 < 2e-16 ***

ShelveLocMedium 1.953262 0.125768 15.53 < 2e -16 ***

Age -0.057947 0.015951 -3.63 0.00032 ***

120 3. Linear Regression

Education -0.020852 0.019613 -1.06 0.28836

UrbanYes 0.140160 0.112402 1.25 0.21317

USYes -0.157557 0.148923 -1.06 0.29073

Income: Advertising 0.000751 0.000278 2.70 0.00729 **

Price:Age 0.000107 0.000133 0.80 0.42381

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 1.01 on 386 degrees of freedom

Multiple R-squared: 0.876 , Adjusted R-squared: 0.872

F-statistic: 210 on 13 and 386 DF , p-value: < 2e-16

The contrasts() function returns the coding that R uses for the dummy contrasts()

variables.

> attach(Carseats)

> contrasts (ShelveLoc)

Good Medium

Bad 0 0

Good 1 0

Medium 0 1

Use ?contrasts to learn about other contrasts, and how to set them. R has created a ShelveLocGood dummy variable that takes on a value of

1 if the shelving location is good, and 0 otherwise. It has also created a ShelveLocMedium dummy variable that equals 1 if the shelving location is medium, and 0 otherwise. A bad shelving location corresponds to a zero for each of the two dummy variables. The fact that the coefficient for ShelveLocGood in the regression output is positive indicates that a good shelving location is associated with high sales (relative to a bad location). And ShelveLocMedium has a smaller positive coefficient, indicating that a medium shelving location is associated with higher sales than a bad shelv- ing location but lower sales than a good shelving location.

3.6.7 Writing Functions

As we have seen, R comes with many useful functions, and still more func- tions are available by way of R libraries. However, we will often be inter- ested in performing an operation for which no function is available. In this setting, we may want to write our own function. For instance, below we provide a simple function that reads in the ISLR2 and MASS libraries, called LoadLibraries(). Before we have created the function, R returns an error if we try to call it.

> LoadLibraries

Error: object ‘LoadLibraries ’ not found

> LoadLibraries ()

Error: could not find function " LoadLibraries "

We now create the function. Note that the + symbols are printed by R and should not be typed in. The { symbol informs R that multiple commands

3.7 Exercises 121

are about to be input. Hitting Enter after typing { will cause R to print the + symbol. We can then input as many commands as we wish, hitting Enter after each one. Finally the } symbol informs R that no further commands will be entered.

> LoadLibraries <- function () {

+ library(ISLR2)

+ library(MASS)

+ print("The libraries have been loaded.")

+ }

Now if we type in LoadLibraries, R will tell us what is in the function.

> LoadLibraries

function () {

library(ISLR2)

library(MASS)

print (" The libraries have been loaded .")

}

If we call the function, the libraries are loaded in and the print statement is output.

> LoadLibraries ()

[1] "The libraries have been loaded ."

3.7 Exercises

Conceptual

1. Describe the null hypotheses to which the p-values given in Table 3.4 correspond. Explain what conclusions you can draw based on these p-values. Your explanation should be phrased in terms of sales, TV, radio, and newspaper, rather than in terms of the coefficients of the linear model.

2. Carefully explain the differences between the KNN classifier and KNN regression methods.

3. Suppose we have a data set with five predictors, X1 = GPA, X2 = IQ, X3 = Level (1 for College and 0 for High School), X4 = Interac- tion between GPA and IQ, and X5 = Interaction between GPA and Level. The response is starting salary after graduation (in thousands of dollars). Suppose we use least squares to fit the model, and get β̂0 = 50, β̂1 = 20, β̂2 = 0.07, β̂3 = 35, β̂4 = 0.01, β̂5 = −10.

(a) Which answer is correct, and why?

i. For a fixed value of IQ and GPA, high school graduates earn more, on average, than college graduates.

122 3. Linear Regression

ii. For a fixed value of IQ and GPA, college graduates earn more, on average, than high school graduates.

iii. For a fixed value of IQ and GPA, high school graduates earn more, on average, than college graduates provided that the GPA is high enough.

iv. For a fixed value of IQ and GPA, college graduates earn more, on average, than high school graduates provided that the GPA is high enough.

(b) Predict the salary of a college graduate with IQ of 110 and a GPA of 4.0.

(c) True or false: Since the coefficient for the GPA/IQ interaction term is very small, there is very little evidence of an interaction effect. Justify your answer.

4. I collect a set of data (n = 100 observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression, i.e. Y = β0 + β1X + β2X

2 + β3X 3 + ϵ.

(a) Suppose that the true relationship between X and Y is linear, i.e. Y = β0 + β1X + ϵ. Consider the training residual sum of squares (RSS) for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

(b) Answer (a) using test rather than training RSS.

(c) Suppose that the true relationship between X and Y is not linear, but we don’t know how far it is from linear. Consider the training RSS for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

(d) Answer (c) using test rather than training RSS.

5. Consider the fitted values that result from performing linear regres- sion without an intercept. In this setting, the ith fitted value takes the form

ŷi = xiβ̂,

where

β̂ =

( n∑

i=1

xiyi

) /

( n∑

i′=1

x2i′

) . (3.38)

3.7 Exercises 123

Show that we can write

ŷi = n∑

i′=1

ai′yi′.

What is ai′?

Note: We interpret this result by saying that the fitted values from linear regression are linear combinations of the response values.

6. Using (3.4), argue that in the case of simple linear regression, the least squares line always passes through the point (x̄, ȳ).

7. It is claimed in the text that in the case of simple linear regression of Y onto X, the R2 statistic (3.17) is equal to the square of the correlation between X and Y (3.18). Prove that this is the case. For simplicity, you may assume that x̄ = ȳ = 0.

Applied

8. This question involves the use of simple linear regression on the Auto data set.

(a) Use the lm() function to perform a simple linear regression with mpg as the response and horsepower as the predictor. Use the summary() function to print the results. Comment on the output. For example:

i. Is there a relationship between the predictor and the re- sponse?

ii. How strong is the relationship between the predictor and the response?

iii. Is the relationship between the predictor and the response positive or negative?

iv. What is the predicted mpg associated with a horsepower of 98? What are the associated 95 % confidence and prediction intervals?

(b) Plot the response and the predictor. Use the abline() function to display the least squares regression line.

(c) Use the plot() function to produce diagnostic plots of the least squares regression fit. Comment on any problems you see with the fit.

9. This question involves the use of multiple linear regression on the Auto data set.

124 3. Linear Regression

(a) Produce a scatterplot matrix which includes all of the variables in the data set.

(b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable,

cor() which is qualitative.

(c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:

i. Is there a relationship between the predictors and the re- sponse?

ii. Which predictors appear to have a statistically significant relationship to the response?

iii. What does the coefficient for the year variable suggest?

(d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?

(e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?

(f) Try a few different transformations of the variables, such as log(X),

√ X, X2. Comment on your findings.

10. This question should be answered using the Carseats data set.

(a) Fit a multiple regression model to predict Sales using Price, Urban, and US.

(b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!

(d) For which of the predictors can you reject the null hypothesis H0 : βj = 0?

(e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.

(f) How well do the models in (a) and (e) fit the data?

(g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).

3.7 Exercises 125

(h) Is there evidence of outliers or high leverage observations in the model from (e)?

11. In this problem we will investigate the t-statistic for the null hypoth- esis H0 : β = 0 in simple linear regression without an intercept. To begin, we generate a predictor x and a response y as follows.

> set.seed (1)

> x <- rnorm (100)

> y <- 2 * x + rnorm (100)

(a) Perform a simple linear regression of y onto x, without an in- tercept. Report the coefficient estimate β̂, the standard error of this coefficient estimate, and the t-statistic and p-value associ- ated with the null hypothesis H0 : β = 0. Comment on these results. (You can perform regression without an intercept using the command lm(y∼x+0).)

(b) Now perform a simple linear regression of x onto y without an intercept, and report the coefficient estimate, its standard error, and the corresponding t-statistic and p-values associated with the null hypothesis H0 : β = 0. Comment on these results.

(d) For the regression of Y onto X without an intercept, the t- statistic for H0 : β = 0 takes the form β̂/SE(β̂), where β̂ is given by (3.38), and where

SE(β̂) =

√∑n i=1(yi −xiβ̂)2

(n−1) ∑n

i′=1 x 2 i′ .

(These formulas are slightly different from those given in Sec- tions 3.1.1 and 3.1.2, since here we are performing regression without an intercept.) Show algebraically, and confirm numeri- cally in R, that the t-statistic can be written as

( √ n−1)

∑n i=1 xiyi√

( ∑n

i=1 x 2 i )( ∑n

i′=1 y 2 i′)− (

∑n i′=1 xi′yi′)

2 .

(e) Using the results from (d), argue that the t-statistic for the re- gression of y onto x is the same as the t-statistic for the regression of x onto y.

(f) In R, show that when regression is performed with an intercept, the t-statistic for H0 : β1 = 0 is the same for the regression of y onto x as it is for the regression of x onto y.

126 3. Linear Regression

12. This problem involves simple linear regression without an intercept.

(a) Recall that the coefficient estimate β̂ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?

(b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.

(c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.

13. In this exercise you will create some simulated data and will fit simple linear regression models to it. Make sure to use set.seed(1) prior to starting part (a) to ensure consistent results.

(a) Using the rnorm() function, create a vector, x, containing 100 observations drawn from a N(0, 1) distribution. This represents a feature, X.

(b) Using the rnorm() function, create a vector, eps, containing 100 observations drawn from a N(0, 0.25) distribution—a normal distribution with mean zero and variance 0.25.

Y = −1 + 0.5X + ϵ. (3.39)

What is the length of the vector y? What are the values of β0 and β1 in this linear model?

(d) Create a scatterplot displaying the relationship between x and y. Comment on what you observe.

(e) Fit a least squares linear model to predict y using x. Comment on the model obtained. How do β̂0 and β̂1 compare to β0 and β1?

(f) Display the least squares line on the scatterplot obtained in (d). Draw the population regression line on the plot, in a different color. Use the legend() command to create an appropriate leg- end.

(g) Now fit a polynomial regression model that predicts y using x and x2. Is there evidence that the quadratic term improves the model fit? Explain your answer.

3.7 Exercises 127

(h) Repeat (a)–(f) after modifying the data generation process in such a way that there is less noise in the data. The model (3.39) should remain the same. You can do this by decreasing the vari- ance of the normal distribution used to generate the error term ϵ in (b). Describe your results.

(i) Repeat (a)–(f) after modifying the data generation process in such a way that there is more noise in the data. The model (3.39) should remain the same. You can do this by increasing the variance of the normal distribution used to generate the error term ϵ in (b). Describe your results.

(j) What are the confidence intervals for β0 and β1 based on the original data set, the noisier data set, and the less noisy data set? Comment on your results.

14. This problem focuses on the collinearity problem.

(a) Perform the following commands in R:

> set.seed (1)

> x1 <- runif (100)

> x2 <- 0.5 * x1 + rnorm (100) / 10

> y <- 2 + 2 * x1 + 0.3 * x2 + rnorm (100)

The last line corresponds to creating a linear model in which y is a function of x1 and x2. Write out the form of the linear model. What are the regression coefficients?

(b) What is the correlation between x1 and x2? Create a scatterplot displaying the relationship between the variables.

(c) Using this data, fit a least squares regression to predict y using x1 and x2. Describe the results obtained. What are β̂0, β̂1, and β̂2? How do these relate to the true β0, β1, and β2? Can you reject the null hypothesis H0 : β1 = 0? How about the null hypothesis H0 : β2 = 0?

(d) Now fit a least squares regression to predict y using only x1. Comment on your results. Can you reject the null hypothesis H0 : β1 = 0?

(e) Now fit a least squares regression to predict y using only x2. Comment on your results. Can you reject the null hypothesis H0 : β1 = 0?

(f) Do the results obtained in (c)–(e) contradict each other? Explain your answer.

(g) Now suppose we obtain one additional observation, which was unfortunately mismeasured.

128 3. Linear Regression

> x1 <- c(x1 , 0.1)

> x2 <- c(x2 , 0.8)

> y <- c(y, 6)

Re-fit the linear models from (c) to (e) using this new data. What effect does this new observation have on the each of the models? In each model, is this observation an outlier? A high-leverage point? Both? Explain your answers.

15. This problem involves the Boston data set, which we saw in the lab for this chapter. We will now try to predict per capita crime rate using the other variables in this data set. In other words, per capita crime rate is the response, and the other variables are the predictors.

(a) For each predictor, fit a simple linear regression model to predict the response. Describe your results. In which of the models is there a statistically significant association between the predictor and the response? Create some plots to back up your assertions.

(b) Fit a multiple regression model to predict the response using all of the predictors. Describe your results. For which predictors can we reject the null hypothesis H0 : βj = 0?

(c) How do your results from (a) compare to your results from (b)? Create a plot displaying the univariate regression coefficients from (a) on the x-axis, and the multiple regression coefficients from (b) on the y-axis. That is, each predictor is displayed as a single point in the plot. Its coefficient in a simple linear regres- sion model is shown on the x-axis, and its coefficient estimate in the multiple linear regression model is shown on the y-axis.

(d) Is there evidence of non-linear association between any of the predictors and the response? To answer this question, for each predictor X, fit a model of the form

Y = β0 + β1X + β2X 2 + β3X

3 + ϵ.

4 Classification

The linear regression model discussed in Chapter 3 assumes that the re- sponse variable Y is quantitative. But in many situations, the response variable is instead qualitative. For example, eye color is qualitative. Of-

qualitative ten qualitative variables are referred to as categorical; we will use these terms interchangeably. In this chapter, we study approaches for predicting qualitative responses, a process that is known as classification. Predicting

classification a qualitative response for an observation can be referred to as classifying that observation, since it involves assigning the observation to a category, or class. On the other hand, often the methods used for classification first predict the probability that the observation belongs to each of the cate- gories of a qualitative variable, as the basis for making the classification. In this sense they also behave like regression methods. There are many possible classification techniques, or classifiers, that one

classifier might use to predict a qualitative response. We touched on some of these in Sections 2.1.5 and 2.2.3. In this chapter we discuss some widely-used classifiers: logistic regression, linear discriminant analysis, quadratic dis-

logistic regression

linear discriminant analysis

criminant analysis, naive Bayes, and K-nearest neighbors. The discussion

quadratic discriminant analysis

naive Bayes

K-nearest neighbors

of logistic regression is used as a jumping-off point for a discussion of gen- eralized linear models, and in particular, Poisson regression. We discuss

generalized linear models Poisson regression

more computer-intensive classification methods in later chapters: these in- clude generalized additive models (Chapter 7); trees, random forests, and boosting (Chapter 8); and support vector machines (Chapter 9).

129

http://crossmark.crossref.org/dialog/?doi=10.1007/978-1-0716-1418-1_4&domain=pdf

130 4. Classification

4.1 An Overview of Classification

Classification problems occur often, perhaps even more so than regression problems. Some examples include:

1. A person arrives at the emergency room with a set of symptoms that could possibly be attributed to one of three medical conditions. Which of the three conditions does the individual have?

2. An online banking service must be able to determine whether or not a transaction being performed on the site is fraudulent, on the basis of the user’s IP address, past transaction history, and so forth.

3. On the basis of DNA sequence data for a number of patients with and without a given disease, a biologist would like to figure out which DNA mutations are deleterious (disease-causing) and which are not.

Just as in the regression setting, in the classification setting we have a set of training observations (x1, y1), . . . , (xn, yn) that we can use to build a classifier. We want our classifier to perform well not only on the training data, but also on test observations that were not used to train the classifier. In this chapter, we will illustrate the concept of classification using the

simulated Default data set. We are interested in predicting whether an individual will default on his or her credit card payment, on the basis of annual income and monthly credit card balance. The data set is displayed in Figure 4.1. In the left-hand panel of Figure 4.1, we have plotted annual income and monthly credit card balance for a subset of 10, 000 individuals. The individuals who defaulted in a given month are shown in orange, and those who did not in blue. (The overall default rate is about 3 %, so we have plotted only a fraction of the individuals who did not default.) It appears that individuals who defaulted tended to have higher credit card balances than those who did not. In the center and right-hand panels of Figure 4.1, two pairs of boxplots are shown. The first shows the distribution of balance split by the binary default variable; the second is a similar plot for income. In this chapter, we learn how to build a model to predict default (Y ) for any given value of balance (X1) and income (X2). Since Y is not quantitative, the simple linear regression model of Chapter 3 is not a good choice: we will elaborate on this further in Section 4.2. It is worth noting that Figure 4.1 displays a very pronounced relation-

ship between the predictor balance and the response default. In most real applications, the relationship between the predictor and the response will not be nearly so strong. However, for the sake of illustrating the classifica- tion procedures discussed in this chapter, we use an example in which the relationship between the predictor and the response is somewhat exagger- ated.

4.2 Why Not Linear Regression? 131

0 500 1000 1500 2000 2500

0 2 0 0 0 0

4 0 0 0 0

6 0 0 0 0

Balance

In c o m

No Yes

0 5 0 0

1 0 0 0

1 5 0 0

2 0 0 0

2 5 0 0

Default B

a la

n c e

No Yes

0 2 0 0 0 0

4 0 0 0 0

6 0 0 0 0

Default

In c o

m e

FIGURE 4.1. The Default data set. Left: The annual incomes and monthly credit card balances of a number of individuals. The individuals who defaulted on their credit card payments are shown in orange, and those who did not are shown in blue. Center: Boxplots of balance as a function of default status. Right: Boxplots of income as a function of default status.

4.2 Why Not Linear Regression?

We have stated that linear regression is not appropriate in the case of a qualitative response. Why not? Suppose that we are trying to predict the medical condition of a patient

in the emergency room on the basis of her symptoms. In this simplified example, there are three possible diagnoses: stroke, drug overdose, and epileptic seizure. We could consider encoding these values as a quantita- tive response variable, Y , as follows:

Y =

⎧ ⎪⎨

⎪⎩

1 if stroke;

2 if drug overdose;

3 if epileptic seizure.

Using this coding, least squares could be used to fit a linear regression model to predict Y on the basis of a set of predictors X1, . . . , Xp. Unfortunately, this coding implies an ordering on the outcomes, putting drug overdose in between stroke and epileptic seizure, and insisting that the difference between stroke and drug overdose is the same as the difference between drug overdose and epileptic seizure. In practice there is no particular reason that this needs to be the case. For instance, one could choose an

132 4. Classification

equally reasonable coding,

Y =

⎧ ⎪⎨

⎪⎩

1 if epileptic seizure;

2 if stroke;

3 if drug overdose.

which would imply a totally different relationship among the three condi- tions. Each of these codings would produce fundamentally different linear models that would ultimately lead to different sets of predictions on test observations. If the response variable’s values did take on a natural ordering, such as

mild, moderate, and severe, and we felt the gap between mild and moderate was similar to the gap between moderate and severe, then a 1, 2, 3 coding would be reasonable. Unfortunately, in general there is no natural way to convert a qualitative response variable with more than two levels into a quantitative response that is ready for linear regression. For a binary (two level) qualitative response, the situation is better. For

binary instance, perhaps there are only two possibilities for the patient’s med- ical condition: stroke and drug overdose. We could then potentially use the dummy variable approach from Section 3.3.1 to code the response as follows:

Y =

{ 0 if stroke;

1 if drug overdose.

We could then fit a linear regression to this binary response, and predict drug overdose if Ŷ > 0.5 and stroke otherwise. In the binary case it is not hard to show that even if we flip the above coding, linear regression will produce the same final predictions. For a binary response with a 0/1 coding as above, regression by least

squares is not completely unreasonable: it can be shown that the Xβ̂ ob- tained using linear regression is in fact an estimate of Pr(drug overdose|X) in this special case. However, if we use linear regression, some of our es- timates might be outside the [0, 1] interval (see Figure 4.2), making them hard to interpret as probabilities! Nevertheless, the predictions provide an ordering and can be interpreted as crude probability estimates. Curiously, it turns out that the classifications that we get if we use linear regression to predict a binary response will be the same as for the linear discriminant analysis (LDA) procedure we discuss in Section 4.4. To summarize, there are at least two reasons not to perform classifica-

tion using a regression method: (a) a regression method cannot accommo- date a qualitative response with more than two classes; (b) a regression method will not provide meaningful estimates of Pr(Y |X), even with just two classes. Thus, it is preferable to use a classification method that is truly suited for qualitative response values. In the next section, we present logistic regression, which is well-suited for the case of a binary qualita-

4.3 Logistic Regression 133

0 500 1000 1500 2000 2500

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

Balance

P ro

b a b

ili ty

o f

D e

fa u

| | || | ||| ||| || | ||| || || | || | || || ||| | | | ||| || || || || ||| || | ||| || ||| ||| || ||| |||| | ||| ||| | || | || | || || | ||| | || ||| || || | || | ||| || | ||| || ||| | ||| || || ||| | ||

|| ||| || || | |||| | || || | ||| || || || | || | | | |

||| | ||| |||| | || || ||| || ||| || |

| || |

| |

|| || ||| | || || ||| | || || || || |||| | ||

| ||| | || || || || ||| || |

|| | ||| | | ||| ||| || | || | | || || ||| || || | || || |||||||| || || | || | |||| | || || || || | || ||| || |||

|||

| | |

| | |||||

|| || || || || || ||| | || || |||| || | | ||| ||| | | | || || || || | ||

| | || || || ||| || | ||| ||| | ||| |||| | || |

| || ||| | | | ||| ||| | || | | |||| | || | ||||| || | ||| || || || |

| || || | ||||| | | || || | |||| | ||| | ||| | | ||| | | ||| || ||| | || ||||

| || |

| || || | || | | | || || | || || ||| |||| |||

| |||

|| | | || ||| | || ||| || || | ||| || | || || || |||| | ||| | | ||| || ||| | ||| |||| |

||| || | | ||

| || || |||| ||| ||| | | || | || || ||| ||| ||| | ||| || | || ||| || | | ||| || || ||| ||

| || || || | || || || || || || |||| ||

| |||| | | | ||| || |||| || |

||| | ||| ||| || || | || || || || |||| || | | ||| | |||| | |

||| | ||||| || || | || || || | ||| || || |

|||| |||| | | || || || || | || || |||| | | ||| |

|| || | | |||| || || || | |||| || || || || ||| || || | || | | || || | || || ||

|| | ||| | ||| |

|||||| | | ||| || | || || | | ||| | | | | || ||| | | |||| | ||

|| | || |

| ||| || | ||| | || || ||

| || | || | | || | || || | | |

| |||

||| | || || || | | | || || ||

|| || | | ||| | |||| || ||| || || || || || ||| ||| | ||| | | |||| || ||| | | ||

|| | | || ||| || | ||| || ||| || || || | || || || || ||

|| || |

|| | || ||| |||| | |||

| || | | ||| || | ||| ||| | || | | | ||| || || || | ||| || | ||| || || ||

|| | ||

|| | | || ||| || ||| ||| | || ||| |||| | || | | || |||| |

|| || | || ||| || | || || | || | ||| | || ||| || || || || | || || | || || || || | || | |||| | ||| | || || | || || || || ||| || | | || | ||| ||| | ||| | | |||

|||| || | | || | ||| ||| || | ||| || | | | |||| |

||| | || ||| | || |||| || | || || | | || || ||| ||| | ||| || || || || | |

|| || | ||| ||||| | | | ||| | | ||| || || || || || | |

|||| ||| |

| ||| |

||| ||| | ||||| | || || || || | | || || || || || ||| ||| | || | ||

||| | ||| || | ||| | |||| || | | || |||| | || || || || || | |

||| | | || |||| | || || | |

|| | || | || |||| | ||

| || || ||| || | | ||| || || ||

|| || ||| ||| || || || ||| || || ||| || | | | ||| || || ||||| || | ||| | || | |

| || || |

| || | | || || | ||||| || | || || || || || | || || || | ||

|| ||| || || || ||| | || |||| | | || || || | |

|| | || || ||| | ||| || ||| | | ||| ||| || || ||| || ||| || || ||||

|||

||| | || || || |

| || || ||| ||| || | |||

||| || || ||| || |

|| || || | ||| || | || | || |

| || ||| | | ||| || ||| | || | || | || | | || || || | || || | |||

|||| | || | | || | || | ||| | ||| | || | || || || || ||| || |||| || | ||| || || || || | ||

||| || | |

| | ||| | || || ||||| ||| ||| |

|| | ||| ||| ||| | ||

|| | || || ||| || | |

| | || ||||| ||| | || || ||| || || ||

|| | |

||| || | || | || | | ||||| | || ||| || |||

|| | | |||| | | || | || | |||| | | | | | ||| | |||

|||| || ||| || ||

||| || |

|| | || || || | || | |||||| |

|| |

||| | | || ||| |

| | ||| ||||| ||| | | || || || |

| || | ||| || || || || | | ||| || | | | ||| | || || || |||| || |

| || | ||| | | || ||| || | | ||| | | ||| | | ||| ||| ||| | |

| || | ||| || ||| | || |

| ||| | || ||| |||

||| | | || || || |||| ||| | ||| ||| ||| || || ||| | ||| |||| | || ||| | || || | || ||| || || || || || || | || || || |||| || || || ||| |

|| || || ||| ||

|| | ||| | || ||| | || | | | ||| | || || || | ||| |||| || | || || | | |||| | || ||| || | ||| ||| || | || | | | ||| ||

| ||| | || | | | |

|||| || || | ||| | || |||||| || || || ||| ||| | || ||| | |||| ||| | || || |

| || | || || || || || ||| | || || || || | ||| || | || ||| || || |||| ||

|| | ||| || || || || | | ||| ||| ||| || ||| |||| ||||| |

|||| | | || |||| ||| ||| || || ||| || || || || ||||| ||| | || || | || | ||| ||| |

| ||| || || || || || || |||| || |||| || || |

||| || || || || | |||

|| ||| | ||| || ||| || ||||| ||| | | || |||| || || | ||| || | || ||| | | ||||| | ||| ||| || | | | | || | | || || || | || |||| | ||

| || || || || || || || | | || || | | || | ||| ||| | | ||| ||

|| ||| || ||| || | || || ||| |||| | ||| || || ||| | | | ||| || | ||| |||| || || |||| || | || | ||| || | || | ||| || || || || ||| || |||| ||| || ||||| | || || |

|| || | ||| || ||

|| | | |||| | ||| ||| ||| || || || ||| || || || | || || || ||| | | ||||

|| | |||| || || ||

| || | |

|| || | |||| || | ||| ||| | || || || | || | | ||

|| || |

|| | ||| || |||| ||| || |

|| | || ||

|| | | || | || ||| ||| || | || |

| || ||| | |||| |||| || | ||||| | | |

|| ||| ||||

||| | || || ||| || | ||| | || || | || || || | ||| || |

|| || || | || ||

| || ||| ||

| || | | ||

|| || | || |

|| |||| | ||||| | | || | | |||||| || ||| ||| || || |||| | | || |||| || ||

| |

||| ||

| || | |

|| || | || | | |||| || | | || | | || || || || || | || ||| | | |||| | |

| || || || || || || ||| | | || || ||| || | |||| | || ||| | |||| |||||

| | |

| | ||| | |||| | || || || || || || | || | || | | | || || | || | |||| | || | || || | ||| | || || ||| || || |||

| || ||| | || | || || || ||| | || |||| || | || | ||

| | ||| ||| || | |||

|| ||| || || || | | || | ||| | || ||| | ||

|| || | || | | || ||| | |||| || || | || ||||| || ||| || || || ||| ||| ||| | ||| || ||| ||| || || | ||| |

|| |

| ||||| | || ||| || || || || | ||| || || | | | ||| | || ||| ||| ||| |||| ||| ||| || || | || | |

| || | ||| ||| || || || ||| || ||| | || || | | || | || || || | || | ||| | || || || ||| ||| || || | || | ||

| || ||| || || ||| || | ||| || | || |

| || | ||| ||| || ||| | ||| || || | | ||| | ||

| | || | | |

|| | ||| || || | | ||| | | ||| | | || || |||| | || ||

|| | | || | | | ||

| ||| | | || | || ||| || || | ||| | ||| || |||| | ||| ||| || | || || ||| ||| | ||| ||| ||||| |||| ||| | | |||| ||

| |||| | || | |

||| ||| || || ||

|| |

||| ||| | |||| || ||| | || ||| | | || ||| ||

| ||| || |||| || | || | || || | | |||| || | |||

||||| || | |||| ||

| ||| || |

|| || || ||||| | | ||| | ||| | || | ||| || | || | || | | || |||

| || |||| ||| | | |

| || |||

|| || | || | ||||| | | ||| || || | | ||| || || | | | | || ||| | |||| | | || | |||| || | ||| | || |||| | | || ||| ||

|| || | || | | || | ||| ||| || || | | || || | || || ||| || || || | | | || | |||| | | || || ||| |

|| || | || || || || | ||| | || || || | ||| | || ||| ||

|| || || ||

| || ||| || || | |

| || | | || | ||| | || || ||| || |||| || || | ||

|| || || || | | |||

|| || | ||| || ||| || | | || |||| | | || ||| |

| | |

| || | ||| || | | ||| || | || ||| || | | || | | ||| | || | | ||| || ||| | ||| || ||

|| || ||| ||| || || ||| || |

|| |

| | || | || || | ||| || | ||| | | | || | || | || | | |||| | ||| | ||| ||| ||| || | ||| || || || | || |||| | | |||| ||| | | | |||| |

| || || |||| | || |

|| || || | |||

| |

| ||| ||| | || | | | |||

| ||| | ||| || || || || || | ||| || | || ||| ||| | | || |||| | |||||

| || | ||| ||| || || | | ||||| ||| | || || || ||| | ||| || | | || | || || |||| || | ||| | ||| | || || || | || ||| ||| ||| || | ||| | || ||| || || ||| || || | |||| || ||| || ||| | ||| ||| | || | || || || ||| | || ||| | || | || ||| | | ||| | |

| | ||||| | ||

|| |

|||| || |||||

|| | | || |||| | || || ||

|| ||| || || || ||| |||| || ||| | |||| | | |

| | | |||| || ||| | ||| | | || || |

|| | || | || | |

| | |

||| | || | ||| || || || ||| |||| || || ||| || || |||

| | | || | || ||| || | ||| ||| || || | ||| || | || || || ||| || ||| | || | || | || || ||| | || ||||| | |

|| | | | |||| || ||| || || || || | | || |

||| | || || ||| || | || |||

|| || ||| | | ||| ||| | || || || || || || | | | ||| | || | | || || | | || ||| || ||| | || || | |||| || || ||| || |

|| | || | | || ||| |||| | || | | || ||| | || || || |||| || ||| ||| || || || || || | ||| ||| |||| | |

| ||| | ||| ||| || || || |||| || | ||||| | || | ||||| | ||

| | || || ||| || | |||| | ||| | || ||| ||| | | || |

|| ||| ||

| || | ||| || || || |||| || ||

||| | || |||| || | || ||| || || | || | || | || || ||| |||| ||| |

||| ||| || | || || || | || || | | ||| || || | ||| ||| | ||| ||| |||| | || | || || ||| | || || ||| |

|| | ||| ||| | || || || || | || | | ||| || | ||| | || || || || | | || | | || | | |||| || ||| | | || || || ||| || || | |

| | ||| || || || |

| || || |||

| | || |||| ||| | || | ||| || | || | || ||| | || || || || |||||| || | || || | ||| ||| | | || | | || || ||| | ||| || ||||| | || ||| | ||| |||| | || | || ||| ||

|||||| | ||| || | || || || ||| || || || || | | || | ||

|| || || || ||| | ||| ||| ||| || || || || | || || ||

||| | || || || | ||| | || || | || ||| | || || ||

| | || | ||| || || | | ||| | || | |||| |||| || || | || | | ||| | || | | |||| || | |||| | || || | || || | ||| || || | ||| || | || ||| |||

| || || |||| || || |||| | ||| |||| | |||| |||| || |

|| | || | | || | | |

| ||||| | |||||| | || | | || || || || | || ||| || | || || || |

| ||| | || | ||| ||| |||| |

|| || || || || | | || | ||| | | | ||

|| || | || | |||| | ||

| || |||| || | ||||| || || | ||| | | ||| ||| | || |||| ||| |

| |||| || | ||

|| || | |||| | || ||||| | || | ||| | | || || | | | || ||| ||| || | | || | ||

|| ||| |

| | || || |||| |

|| | ||| | || || | ||| || || ||| | || || | ||| || || | || | || |||| || ||| ||| ||| || | ||| | || || | |||

| |

||| || || || | || |||| | |||| || ||| || || | ||| ||

| || || ||| || || | | ||| | || | |||| ||

|| ||| || | || | | | || | ||| | |

| || || || | || || || | ||| || ||| || ||

| || | |||| |||

| ||| || || | || |

| || ||| || | || || |||| || | ||| | || | || | ||| | || | || ||||| |

|| | || || | | | ||| | || ||||

| | | | |||| ||| |||| |||| | | | ||| | || | || || | || | |||| | || | | ||| || |

||||

| ||| | ||| ||

| || || ||| ||| || |||| ||| |

||| || | ||

||| | | || ||| | || |||| || | || ||| || || | ||| | ||| || |||

|| ||| || ||| | | || || | || | ||| | || | | || | ||| |||| || || ||||| | ||| | || || || || | || | ||||

|| ||| || ||| | | || || | ||| |||| | |

|| | | ||| || ||

| || || || |

| | | || | || ||| ||| || | || || || |||

| || || ||||| | | || ||

|| | || || | ||| || | || | ||| |

|| | || | |||

|| | |||| || ||| | ||| | | ||| || | || || || | | || |||| ||

| || | || || | || ||| | ||| ||| ||| | ||| | || | ||| ||||| | | | ||

| | | || | ||| || || ||

| || |||| | ||| | |||| || | | ||| || | ||

|| | | ||||| | || ||| || || || ||| |||| |||

| | | ||| |||| | || |||| | || | | || || | | || || || | ||||| ||| |||

|| ||| ||| ||

| | || | || ||||| ||| || || || || | ||| || | | | | ||

| ||| | || || | | || ||| || |||| ||| || | | ||| | ||| | || || || | || | || | || || | || |

| || || |||| ||||| | || | || | || |||| | || | || || |||| | | ||| ||

| || || || ||| || || | ||||| || || | ||| || |||| | || ||| ||| || || | |||| || | ||| ||| || | ||| | | || | || || | | || | | || | || |

||| | | ||| ||| || | || | || ||| | |

| ||| || || | | || ||| | | | || | || ||| || | || || | || || || || || || || ||| | || || || | || ||||

| ||| ||| |

| ||| | || | | ||||| | ||| || || || ||| || | ||| | || || ||| || | || | |

| |

||| ||||| | |||| || || ||| || | || | || ||| | ||

| | ||| | ||| | |||| || || | |||| | | ||

||| | | || | | ||| || | ||| || ||| || || ||| ||| ||| | || | | | || || ||| || | ||| |||| || || ||| | ||| |||| ||||| || ||| || || | | ||| || || || |||| ||| ||

||| ||| ||| | || | ||| | || | | || ||| | || || ||| | || | ||| || | | || ||| | ||| | || || || |||| | | ||

||||| || | | || || || ||||| | ||| || ||| ||| | || || || | | || || ||| || | |

|| | || |||| || ||

|| || | | || | | ||

|| || | || || ||

| || || || || | || ||| | || | | ||| | || || |||| |||| || | ||

| || || | || || ||| | | || ||| || || || | | | ||||

|| ||| ||| || || || | || || || | | || | || | ||| | || ||| || | || ||| || | || || || || ||| ||| | | ||| | || | || || || || | |||| || || | || || ||| | || | |||| | || | || | | | || || | || || ||| | | ||| |||| | ||| || | || ||| | || | ||| || ||| || || || ||| || | || || || | || |||| || | || || || ||| || || ||| | | | || | | | || |||

|| || || ||||| || | |||| |

| | | || | | | |||||| ||| || | | ||| | ||| | ||| |||| | | | | || |||| | ||

| ||

| | || || ||| || || ||| || | || || | ||| || || || || || || | || | ||| ||

|| | | || || | || ||| ||| || |

|| |||| | || | || || | | || | |||| || ||| || | | || | || | || || ||

| | || | |||| | || || | || |||| |||| | || | |||

|||| | ||| || | | |

| | || |||

| ||| |||| | | || | | ||| || | | || | | || | || | || |

||| || ||| | |

||| || ||| ||| ||| ||

|| | || || |||| ||| | || || | ||| || | ||

| |

|| ||||| || | || | ||

| || ||| || |

|| | ||

| |||| || | || | ||| | ||| ||| || || || || || |||| | ||| | ||| |||

| || || | | || ||| || | || ||| || ||||| | |||

|| |||| | ||| | |||| |||| || || || | ||| | ||| ||| || | || | | || ||| || ||| | | | || ||| | | | || || | | || || || || || | ||| | ||| ||| | || | ||| || | || | || || | ||| ||| || || || | ||||| | ||

|| |||| || ||| ||| | |||

|| || | | ||| | | |

| | || | || || ||| || || || | || | || || || | | | ||| |||| || | || ||| | || || | | |

|| || || || || ||||| | | || || ||| | | ||| | ||| ||| | | |

| | ||| || || ||| || || || || || || | ||||| || | || || | || || || | | ||| || || | ||| || | | ||| ||| ||| |

||| || ||| || |

||| || | || ||| || ||| ||| | || | ||| ||| || ||| ||| || | | | || || | || || | | | ||| | | || || || | || | | ||

|| | || || || || ||| | || |||| || || || | | ||| | || | ||| | || || | ||

| |||

| || || || | |||| || ||| | || ||||

||| | || ||

| |||

| | || | ||||| ||| || || || | | | ||| | | || |||| || | ||||| ||| ||| | |||| | | || ||| || || || |||| || || | || ||| || |||| |||| | | | ||

|| ||| ||| || | || | || |||| |||| | || ||| ||| ||||||| || ||| | || | || ||| | | | || || |||| || || || | || | |

| || || || | || ||||

| || ||| | | || ||| || | ||| | | | |

||| | | |||| || || || || | || | ||| | || | ||| |

| || | || || ||| | || | | || | |||| || || ||| | || ||| ||| |

| || ||| || | ||| | || | || | ||| ||| || || |||| || | || ||| | || | ||| || || |||| ||| || |

||||

| ||||| ||| |

| | || ||| | |

|| | || ||||| | ||| || | |

| || || ||

||| || | | |||||| |

|||| | || ||

|| | |

|||| | |

| | || || | ||| || | ||

| | ||| || ||| || ||| || || | | || ||| | || ||| | || |||| ||||| || | ||||| | | || || ||| | ||| | || || |

|| || ||| || || ||| | || || || | || |||| || || | | |||| | | | ||

|| | | || ||| | | | | || ||| || || ||| ||| || | || || ||| ||| ||| ||| || || || | | | || | | || ||| | || || ||| ||| || || | | ||| ||| || |||| | | || | || | || || ||| | | || || | ||| || ||| | || |

| ||| || || |

|| || ||| ||| | || || |

|| || | ||| ||| || | |||| | || || ||| || ||||| || || | || ||

|| |||| | || || || || | || | ||| || | |

| |||| ||| |

| ||| || ||| | ||| || | | |

| |||||| |

| | || |||| || ||| || || || | || |||||

| | ||| ||| | | || | | ||| | || | | ||| |

| |||| ||| || || | ||| | | | ||

0 500 1000 1500 2000 2500

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

Balance

P ro

b a b

ili ty

o f

D e

fa u

|| ||| || || | |||| | || || | ||| || || || | || | | | |

||| | ||| |||| | || || ||| || ||| || |

| || |

| |

|| || ||| | || || ||| | || || || || |||| | ||

| ||| | || || || || ||| || |

|| | ||| | | ||| ||| || | || | | || || ||| || || | || || |||||||| || || | || | |||| | || || || || | || ||| || |||

|||

| | |

| | |||||

|| || || || || || ||| | || || |||| || | | ||| ||| | | | || || || || | ||

| | || || || ||| || | ||| ||| | ||| |||| | || |

| || ||| | | | ||| ||| | || | | |||| | || | ||||| || | ||| || || || |

| || || | ||||| | | || || | |||| | ||| | ||| | | ||| | | ||| || ||| | || ||||

| || |

| || || | || | | | || || | || || ||| |||| |||

| |||

|| | | || ||| | || ||| || || | ||| || | || || || |||| | ||| | | ||| || ||| | ||| |||| |

||| || | | ||

| || || |||| ||| ||| | | || | || || ||| ||| ||| | ||| || | || ||| || | | ||| || || ||| ||

| || || || | || || || || || || |||| ||

| |||| | | | ||| || |||| || |

||| | ||| ||| || || | || || || || |||| || | | ||| | |||| | |

||| | ||||| || || | || || || | ||| || || |

|||| |||| | | || || || || | || || |||| | | ||| |

|| || | | |||| || || || | |||| || || || || ||| || || | || | | || || | || || ||

|| | ||| | ||| |

|||||| | | ||| || | || || | | ||| | | | | || ||| | | |||| | ||

|| | || |

| ||| || | ||| | || || ||

| || | || | | || | || || | | |

| |||

||| | || || || | | | || || ||

|| || | | ||| | |||| || ||| || || || || || ||| ||| | ||| | | |||| || ||| | | ||

|| | | || ||| || | ||| || ||| || || || | || || || || ||

|| || |

|| | || ||| |||| | |||

| || | | ||| || | ||| ||| | || | | | ||| || || || | ||| || | ||| || || ||

|| | ||

|| | | || ||| || ||| ||| | || ||| |||| | || | | || |||| |

|| || | || ||| || | || || | || | ||| | || ||| || || || || | || || | || || || || | || | |||| | ||| | || || | || || || || ||| || | | || | ||| ||| | ||| | | |||

|||| || | | || | ||| ||| || | ||| || | | | |||| |

||| | || ||| | || |||| || | || || | | || || ||| ||| | ||| || || || || | |

|| || | ||| ||||| | | | ||| | | ||| || || || || || | |

|||| ||| |

| ||| |

||| ||| | ||||| | || || || || | | || || || || || ||| ||| | || | ||

||| | ||| || | ||| | |||| || | | || |||| | || || || || || | |

||| | | || |||| | || || | |

|| | || | || |||| | ||

| || || ||| || | | ||| || || ||

|| || ||| ||| || || || ||| || || ||| || | | | ||| || || ||||| || | ||| | || | |

| || || |

| || | | || || | ||||| || | || || || || || | || || || | ||

|| ||| || || || ||| | || |||| | | || || || | |

|| | || || ||| | ||| || ||| | | ||| ||| || || ||| || ||| || || ||||

|||

||| | || || || |

| || || ||| ||| || | |||

||| || || ||| || |

|| || || | ||| || | || | || |

| || ||| | | ||| || ||| | || | || | || | | || || || | || || | |||

|||| | || | | || | || | ||| | ||| | || | || || || || ||| || |||| || | ||| || || || || | ||

||| || | |

| | ||| | || || ||||| ||| ||| |

|| | ||| ||| ||| | ||

|| | || || ||| || | |

| | || ||||| ||| | || || ||| || || ||

|| | |

||| || | || | || | | ||||| | || ||| || |||

|| | | |||| | | || | || | |||| | | | | | ||| | |||

|||| || ||| || ||

||| || |

|| | || || || | || | |||||| |

|| |

||| | | || ||| |

| | ||| ||||| ||| | | || || || |

| || | ||| || || || || | | ||| || | | | ||| | || || || |||| || |

| || | ||| | | || ||| || | | ||| | | ||| | | ||| ||| ||| | |

| || | ||| || ||| | || |

| ||| | || ||| |||

||| | | || || || |||| ||| | ||| ||| ||| || || ||| | ||| |||| | || ||| | || || | || ||| || || || || || || | || || || |||| || || || ||| |

|| || || ||| ||

|| | ||| | || ||| | || | | | ||| | || || || | ||| |||| || | || || | | |||| | || ||| || | ||| ||| || | || | | | ||| ||

| ||| | || | | | |

|||| || || | ||| | || |||||| || || || ||| ||| | || ||| | |||| ||| | || || |

| || | || || || || || ||| | || || || || | ||| || | || ||| || || |||| ||

|| | ||| || || || || | | ||| ||| ||| || ||| |||| ||||| |

|||| | | || |||| ||| ||| || || ||| || || || || ||||| ||| | || || | || | ||| ||| |

| ||| || || || || || || |||| || |||| || || |

||| || || || || | |||

|| ||| | ||| || ||| || ||||| ||| | | || |||| || || | ||| || | || ||| | | ||||| | ||| ||| || | | | | || | | || || || | || |||| | ||

| || || || || || || || | | || || | | || | ||| ||| | | ||| ||

|| ||| || ||| || | || || ||| |||| | ||| || || ||| | | | ||| || | ||| |||| || || |||| || | || | ||| || | || | ||| || || || || ||| || |||| ||| || ||||| | || || |

|| || | ||| || ||

|| | | |||| | ||| ||| ||| || || || ||| || || || | || || || ||| | | ||||

|| | |||| || || ||

| || | |

|| || | |||| || | ||| ||| | || || || | || | | ||

|| || |

|| | ||| || |||| ||| || |

|| | || ||

|| | | || | || ||| ||| || | || |

| || ||| | |||| |||| || | ||||| | | |

|| ||| ||||

||| | || || ||| || | ||| | || || | || || || | ||| || |

|| || || | || ||

| || ||| ||

| || | | ||

|| || | || |

|| |||| | ||||| | | || | | |||||| || ||| ||| || || |||| | | || |||| || ||

| |

||| ||

| || | |

|| || | || | | |||| || | | || | | || || || || || | || ||| | | |||| | |

| || || || || || || ||| | | || || ||| || | |||| | || ||| | |||| |||||

| | |

| | ||| | |||| | || || || || || || | || | || | | | || || | || | |||| | || | || || | ||| | || || ||| || || |||

| || ||| | || | || || || ||| | || |||| || | || | ||

| | ||| ||| || | |||

|| ||| || || || | | || | ||| | || ||| | ||

|| || | || | | || ||| | |||| || || | || ||||| || ||| || || || ||| ||| ||| | ||| || ||| ||| || || | ||| |

|| |

| ||||| | || ||| || || || || | ||| || || | | | ||| | || ||| ||| ||| |||| ||| ||| || || | || | |

| || | ||| ||| || || || ||| || ||| | || || | | || | || || || | || | ||| | || || || ||| ||| || || | || | ||

| || ||| || || ||| || | ||| || | || |

| || | ||| ||| || ||| | ||| || || | | ||| | ||

| | || | | |

|| | ||| || || | | ||| | | ||| | | || || |||| | || ||

|| | | || | | | ||

| ||| | | || | || ||| || || | ||| | ||| || |||| | ||| ||| || | || || ||| ||| | ||| ||| ||||| |||| ||| | | |||| ||

| |||| | || | |

||| ||| || || ||

|| |

||| ||| | |||| || ||| | || ||| | | || ||| ||

| ||| || |||| || | || | || || | | |||| || | |||

||||| || | |||| ||

| ||| || |

|| || || ||||| | | ||| | ||| | || | ||| || | || | || | | || |||

| || |||| ||| | | |

| || |||

|| || | || | ||||| | | ||| || || | | ||| || || | | | | || ||| | |||| | | || | |||| || | ||| | || |||| | | || ||| ||

|| || | || | | || | ||| ||| || || | | || || | || || ||| || || || | | | || | |||| | | || || ||| |

|| || | || || || || | ||| | || || || | ||| | || ||| ||

|| || || ||

| || ||| || || | |

| || | | || | ||| | || || ||| || |||| || || | ||

|| || || || | | |||

|| || | ||| || ||| || | | || |||| | | || ||| |

| | |

| || | ||| || | | ||| || | || ||| || | | || | | ||| | || | | ||| || ||| | ||| || ||

|| || ||| ||| || || ||| || |

|| |

| | || | || || | ||| || | ||| | | | || | || | || | | |||| | ||| | ||| ||| ||| || | ||| || || || | || |||| | | |||| ||| | | | |||| |

| || || |||| | || |

|| || || | |||

| |

| ||| ||| | || | | | |||

| ||| | ||| || || || || || | ||| || | || ||| ||| | | || |||| | |||||

| | ||||| | ||

|| |

|||| || |||||

|| | | || |||| | || || ||

|| ||| || || || ||| |||| || ||| | |||| | | |

| | | |||| || ||| | ||| | | || || |

|| | || | || | |

| | |

||| | || | ||| || || || ||| |||| || || ||| || || |||

| | | || | || ||| || | ||| ||| || || | ||| || | || || || ||| || ||| | || | || | || || ||| | || ||||| | |

|| | | | |||| || ||| || || || || | | || |

||| | || || ||| || | || |||

|| || ||| | | ||| ||| | || || || || || || | | | ||| | || | | || || | | || ||| || ||| | || || | |||| || || ||| || |

|| | || | | || ||| |||| | || | | || ||| | || || || |||| || ||| ||| || || || || || | ||| ||| |||| | |

| ||| | ||| ||| || || || |||| || | ||||| | || | ||||| | ||

| | || || ||| || | |||| | ||| | || ||| ||| | | || |

|| ||| ||

| || | ||| || || || |||| || ||

||| | || |||| || | || ||| || || | || | || | || || ||| |||| ||| |

||| ||| || | || || || | || || | | ||| || || | ||| ||| | ||| ||| |||| | || | || || ||| | || || ||| |

|| | ||| ||| | || || || || | || | | ||| || | ||| | || || || || | | || | | || | | |||| || ||| | | || || || ||| || || | |

| | ||| || || || |

| || || |||

| | || |||| ||| | || | ||| || | || | || ||| | || || || || |||||| || | || || | ||| ||| | | || | | || || ||| | ||| || ||||| | || ||| | ||| |||| | || | || ||| ||

|||||| | ||| || | || || || ||| || || || || | | || | ||

|| || || || ||| | ||| ||| ||| || || || || | || || ||

||| | || || || | ||| | || || | || ||| | || || ||

| | || | ||| || || | | ||| | || | |||| |||| || || | || | | ||| | || | | |||| || | |||| | || || | || || | ||| || || | ||| || | || ||| |||

| || || |||| || || |||| | ||| |||| | |||| |||| || |

|| | || | | || | | |

| ||||| | |||||| | || | | || || || || | || ||| || | || || || |

| ||| | || | ||| ||| |||| |

|| || || || || | | || | ||| | | | ||

|| || | || | |||| | ||

| || |||| || | ||||| || || | ||| | | ||| ||| | || |||| ||| |

| |||| || | ||

|| || | |||| | || ||||| | || | ||| | | || || | | | || ||| ||| || | | || | ||

|| ||| |

| | || || |||| |

|| | ||| | || || | ||| || || ||| | || || | ||| || || | || | || |||| || ||| ||| ||| || | ||| | || || | |||

| |

||| || || || | || |||| | |||| || ||| || || | ||| ||

| || || ||| || || | | ||| | || | |||| ||

|| ||| || | || | | | || | ||| | |

| || || || | || || || | ||| || ||| || ||

| || | |||| |||

| ||| || || | || |

| || ||| || | || || |||| || | ||| | || | || | ||| | || | || ||||| |

|| | || || | | | ||| | || ||||

| | | | |||| ||| |||| |||| | | | ||| | || | || || | || | |||| | || | | ||| || |

||||

| ||| | ||| ||

| || || ||| ||| || |||| ||| |

||| || | ||

||| | | || ||| | || |||| || | || ||| || || | ||| | ||| || |||

|| ||| || ||| | | || || | || | ||| | || | | || | ||| |||| || || ||||| | ||| | || || || || | || | ||||

|| ||| || ||| | | || || | ||| |||| | |

|| | | ||| || ||

| || || || |

| | | || | || ||| ||| || | || || || |||

| || || ||||| | | || ||

|| | || || | ||| || | || | ||| |

|| | || | |||

|| | |||| || ||| | ||| | | ||| || | || || || | | || |||| ||

| || | || || | || ||| | ||| ||| ||| | ||| | || | ||| ||||| | | | ||

| | | || | ||| || || ||

| || |||| | ||| | |||| || | | ||| || | ||

|| | | ||||| | || ||| || || || ||| |||| |||

| | | ||| |||| | || |||| | || | | || || | | || || || | ||||| ||| |||

|| ||| ||| ||

| | || | || ||||| ||| || || || || | ||| || | | | | ||

| ||| | || || | | || ||| || |||| ||| || | | ||| | ||| | || || || | || | || | || || | || |

| || || |||| ||||| | || | || | || |||| | || | || || |||| | | ||| ||

| || || || ||| || || | ||||| || || | ||| || |||| | || ||| ||| || || | |||| || | ||| ||| || | ||| | | || | || || | | || | | || | || |

||| | | ||| ||| || | || | || ||| | |

| ||| || || | | || ||| | | | || | || ||| || | || || | || || || || || || || ||| | || || || | || ||||

| ||| ||| |

| ||| | || | | ||||| | ||| || || || ||| || | ||| | || || ||| || | || | |

| |

||| ||||| | |||| || || ||| || | || | || ||| | ||

| | ||| | ||| | |||| || || | |||| | | ||

||| | | || | | ||| || | ||| || ||| || || ||| ||| ||| | || | | | || || ||| || | ||| |||| || || ||| | ||| |||| ||||| || ||| || || | | ||| || || || |||| ||| ||

||| ||| ||| | || | ||| | || | | || ||| | || || ||| | || | ||| || | | || ||| | ||| | || || || |||| | | ||

||||| || | | || || || ||||| | ||| || ||| ||| | || || || | | || || ||| || | |

|| | || |||| || ||

|| || | | || | | ||

|| || | || || ||

| || || || || | || ||| | || | | ||| | || || |||| |||| || | ||

| || || | || || ||| | | || ||| || || || | | | ||||

|| ||| ||| || || || | || || || | | || | || | ||| | || ||| || | || ||| || | || || || || ||| ||| | | ||| | || | || || || || | |||| || || | || || ||| | || | |||| | || | || | | | || || | || || ||| | | ||| |||| | ||| || | || ||| | || | ||| || ||| || || || ||| || | || || || | || |||| || | || || || ||| || || ||| | | | || | | | || |||

|| || || ||||| || | |||| |

| | | || | | | |||||| ||| || | | ||| | ||| | ||| |||| | | | | || |||| | ||

| ||

| | || || ||| || || ||| || | || || | ||| || || || || || || | || | ||| ||

|| | | || || | || ||| ||| || |

|| |||| | || | || || | | || | |||| || ||| || | | || | || | || || ||

| | || | |||| | || || | || |||| |||| | || | |||

|||| | ||| || | | |

| | || |||

| ||| |||| | | || | | ||| || | | || | | || | || | || |

||| || ||| | |

||| || ||| ||| ||| ||

|| | || || |||| ||| | || || | ||| || | ||

| |

|| ||||| || | || | ||

| || ||| || |

|| | ||

| |||| || | || | ||| | ||| ||| || || || || || |||| | ||| | ||| |||

| || || | | || ||| || | || ||| || ||||| | |||

|| |||| | ||| | |||| |||| || || || | ||| | ||| ||| || | || | | || ||| || ||| | | | || ||| | | | || || | | || || || || || | ||| | ||| ||| | || | ||| || | || | || || | ||| ||| || || || | ||||| | ||

|| |||| || ||| ||| | |||

|| || | | ||| | | |

| | || | || || ||| || || || | || | || || || | | | ||| |||| || | || ||| | || || | | |

|| || || || || ||||| | | || || ||| | | ||| | ||| ||| | | |

| | ||| || || ||| || || || || || || | ||||| || | || || | || || || | | ||| || || | ||| || | | ||| ||| ||| |

||| || ||| || |

||| || | || ||| || ||| ||| | || | ||| ||| || ||| ||| || | | | || || | || || | | | ||| | | || || || | || | | ||

|| | || || || || ||| | || |||| || || || | | ||| | || | ||| | || || | ||

| |||

| || || || | |||| || ||| | || ||||

||| | || ||

| |||

| | || | ||||| ||| || || || | | | ||| | | || |||| || | ||||| ||| ||| | |||| | | || ||| || || || |||| || || | || ||| || |||| |||| | | | ||

|| ||| ||| || | || | || |||| |||| | || ||| ||| ||||||| || ||| | || | || ||| | | | || || |||| || || || | || | |

| || || || | || ||||

| || ||| | | || ||| || | ||| | | | |

||| | | |||| || || || || | || | ||| | || | ||| |

| || | || || ||| | || | | || | |||| || || ||| | || ||| ||| |

| || ||| || | ||| | || | || | ||| ||| || || |||| || | || ||| | || | ||| || || |||| ||| || |

||||

| ||||| ||| |

| | || ||| | |

|| | || ||||| | ||| || | |

| || || ||

||| || | | |||||| |

|||| | || ||

|| | |

|||| | |

| | || || | ||| || | ||

| | ||| || ||| || ||| || || | | || ||| | || ||| | || |||| ||||| || | ||||| | | || || ||| | ||| | || || |

|| || ||| || || ||| | || || || | || |||| || || | | |||| | | | ||

|| | | || ||| | | | | || ||| || || ||| ||| || | || || ||| ||| ||| ||| || || || | | | || | | || ||| | || || ||| ||| || || | | ||| ||| || |||| | | || | || | || || ||| | | || || | ||| || ||| | || |

| ||| || || |

|| || ||| ||| | || || |

|| || | ||| ||| || | |||| | || || ||| || ||||| || || | || ||

|| |||| | || || || || | || | ||| || | |

| |||| ||| |

| ||| || ||| | ||| || | | |

| |||||| |

| | || |||| || ||| || || || | || |||||

| | ||| ||| | | || | | ||| | || | | ||| |

| |||| ||| || || | ||| | | | ||

FIGURE 4.2. Classification using the Default data. Left: Estimated probabil- ity of default using linear regression. Some estimated probabilities are negative! The orange ticks indicate the 0/1 values coded for default(No or Yes). Right: Predicted probabilities of default using logistic regression. All probabilities lie between 0 and 1.

tive response; in later sections we will cover classification methods that are appropriate when the qualitative response has two or more classes.

4.3 Logistic Regression

Consider again the Default data set, where the response default falls into one of two categories, Yes or No. Rather than modeling this response Y directly, logistic regression models the probability that Y belongs to a par- ticular category. For the Default data, logistic regression models the probability of default.

For example, the probability of default given balance can be written as

Pr(default = Yes|balance).

The values of Pr(default = Yes|balance), which we abbreviate p(balance), will range between 0 and 1. Then for any given value of balance, a prediction can be made for default. For example, one might predict default = Yes for any individual for whom p(balance) > 0.5. Alternatively, if a company wishes to be conservative in predicting individuals who are at risk for de- fault, then they may choose to use a lower threshold, such as p(balance) > 0.1.

4.3.1 The Logistic Model

How should we model the relationship between p(X) = Pr(Y = 1|X) and X? (For convenience we are using the generic 0/1 coding for the response.)

134 4. Classification

In Section 4.2 we considered using a linear regression model to represent these probabilities:

p(X) = β0 + β1X. (4.1)

If we use this approach to predict default=Yes using balance, then we obtain the model shown in the left-hand panel of Figure 4.2. Here we see the problem with this approach: for balances close to zero we predict a negative probability of default; if we were to predict for very large balances, we would get values bigger than 1. These predictions are not sensible, since of course the true probability of default, regardless of credit card balance, must fall between 0 and 1. This problem is not unique to the credit default data. Any time a straight line is fit to a binary response that is coded as 0 or 1, in principle we can always predict p(X) < 0 for some values of X and p(X) > 1 for others (unless the range of X is limited). To avoid this problem, we must model p(X) using a function that gives

outputs between 0 and 1 for all values of X. Many functions meet this description. In logistic regression, we use the logistic function,

logistic function

p(X) = eβ0+β1X

1 + eβ0+β1X . (4.2)

To fit the model (4.2), we use a method called maximum likelihood, which maximum likelihoodwe discuss in the next section. The right-hand panel of Figure 4.2 illustrates

the fit of the logistic regression model to the Default data. Notice that for low balances we now predict the probability of default as close to, but never below, zero. Likewise, for high balances we predict a default probability close to, but never above, one. The logistic function will always produce an S-shaped curve of this form, and so regardless of the value of X, we will obtain a sensible prediction. We also see that the logistic model is better able to capture the range of probabilities than is the linear regression model in the left-hand plot. The average fitted probability in both cases is 0.0333 (averaged over the training data), which is the same as the overall proportion of defaulters in the data set. After a bit of manipulation of (4.2), we find that

p(X)

1−p(X) = eβ0+β1X. (4.3)

The quantity p(X)/[1−p(X)] is called the odds, and can take on any value odds

between 0 and ∞. Values of the odds close to 0 and ∞ indicate very low and very high probabilities of default, respectively. For example, on average 1 in 5 people with an odds of 1/4 will default, since p(X) = 0.2 implies an odds of 0.2

1−0.2 = 1/4. Likewise, on average nine out of every ten people with

an odds of 9 will default, since p(X) = 0.9 implies an odds of 0.9 1−0.9 = 9.

Odds are traditionally used instead of probabilities in horse-racing, since they relate more naturally to the correct betting strategy.

4.3 Logistic Regression 135

By taking the logarithm of both sides of (4.3), we arrive at

log

( p(X)

1−p(X)

) = β0 + β1X. (4.4)

The left-hand side is called the log odds or logit. We see that the logistic log odds

logit regression model (4.2) has a logit that is linear in X. Recall from Chapter 3 that in a linear regression model, β1 gives the

average change in Y associated with a one-unit increase in X. By contrast, in a logistic regression model, increasing X by one unit changes the log odds by β1 (4.4). Equivalently, it multiplies the odds by e

β1 (4.3). However, because the relationship between p(X) and X in (4.2) is not a straight line, β1 does not correspond to the change in p(X) associated with a one-unit increase in X. The amount that p(X) changes due to a one-unit change in X depends on the current value of X. But regardless of the value of X, if β1 is positive then increasing X will be associated with increasing p(X), and if β1 is negative then increasing X will be associated with decreasing p(X). The fact that there is not a straight-line relationship between p(X) and X, and the fact that the rate of change in p(X) per unit change in X depends on the current value of X, can also be seen by inspection of the right-hand panel of Figure 4.2.

4.3.2 Estimating the Regression Coefficients

The coefficients β0 and β1 in (4.2) are unknown, and must be estimated based on the available training data. In Chapter 3, we used the least squares approach to estimate the unknown linear regression coefficients. Although we could use (non-linear) least squares to fit the model (4.4), the more general method of maximum likelihood is preferred, since it has better sta- tistical properties. The basic intuition behind using maximum likelihood to fit a logistic regression model is as follows: we seek estimates for β0 and β1 such that the predicted probability p̂(xi) of default for each individual, using (4.2), corresponds as closely as possible to the individual’s observed default status. In other words, we try to find β̂0 and β̂1 such that plugging these estimates into the model for p(X), given in (4.2), yields a number close to one for all individuals who defaulted, and a number close to zero for all individuals who did not. This intuition can be formalized using a mathematical equation called a likelihood function:

likelihood function

ℓ(β0, β1) = ∏

i:yi=1

p(xi) ∏

i′:yi′ =0

(1−p(xi′)). (4.5)

The estimates β̂0 and β̂1 are chosen to maximize this likelihood function. Maximum likelihood is a very general approach that is used to fit many

of the non-linear models that we examine throughout this book. In the linear regression setting, the least squares approach is in fact a special case

136 4. Classification

Coefficient Std. error z-statistic p-value Intercept −10.6513 0.3612 −29.5 <0.0001 balance 0.0055 0.0002 24.9 <0.0001

TABLE 4.1. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using balance. A one-unit increase in balance is associated with an increase in the log odds of default by 0.0055 units.

of maximum likelihood. The mathematical details of maximum likelihood are beyond the scope of this book. However, in general, logistic regression and other models can be easily fit using statistical software such as R, and so we do not need to concern ourselves with the details of the maximum likelihood fitting procedure. Table 4.1 shows the coefficient estimates and related information that

result from fitting a logistic regression model on the Default data in order to predict the probability of default=Yes using balance. We see that β̂1 = 0.0055; this indicates that an increase in balance is associated with an increase in the probability of default. To be precise, a one-unit increase in balance is associated with an increase in the log odds of default by 0.0055 units. Many aspects of the logistic regression output shown in Table 4.1 are

similar to the linear regression output of Chapter 3. For example, we can measure the accuracy of the coefficient estimates by computing their stan- dard errors. The z-statistic in Table 4.1 plays the same role as the t-statistic in the linear regression output, for example in Table 3.1 on page 68. For instance, the z-statistic associated with β1 is equal to β̂1/SE(β̂1), and so a large (absolute) value of the z-statistic indicates evidence against the null

hypothesis H0 : β1 = 0. This null hypothesis implies that p(X) = eβ0

1+eβ0 : in

other words, that the probability of default does not depend on balance. Since the p-value associated with balance in Table 4.1 is tiny, we can reject H0. In other words, we conclude that there is indeed an association between balance and probability of default. The estimated intercept in Table 4.1 is typically not of interest; its main purpose is to adjust the average fitted probabilities to the proportion of ones in the data (in this case, the overall default rate).

4.3.3 Making Predictions

Once the coefficients have been estimated, we can compute the probability of default for any given credit card balance. For example, using the coeffi- cient estimates given in Table 4.1, we predict that the default probability for an individual with a balance of $1, 000 is

p̂(X) = eβ̂0+β̂1X

1 + eβ̂0+β̂1X =

e−10.6513+0.0055×1,000

1 + e−10.6513+0.0055×1,000 = 0.00576,

4.3 Logistic Regression 137

Coefficient Std. error z-statistic p-value Intercept −3.5041 0.0707 −49.55 <0.0001 student[Yes] 0.4049 0.1150 3.52 0.0004

TABLE 4.2. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using student status. Student status is encoded as a dummy variable, with a value of 1 for a student and a value of 0 for a non-student, and represented by the variable student[Yes] in the table.

which is below 1 %. In contrast, the predicted probability of default for an individual with a balance of $2, 000 is much higher, and equals 0.586 or 58.6 %. One can use qualitative predictors with the logistic regression model us-

ing the dummy variable approach from Section 3.3.1. As an example, the Default data set contains the qualitative variable student. To fit a model that uses student status as a predictor variable, we simply create a dummy variable that takes on a value of 1 for students and 0 for non-students. The logistic regression model that results from predicting probability of default from student status can be seen in Table 4.2. The coefficient associated with the dummy variable is positive, and the associated p-value is statis- tically significant. This indicates that students tend to have higher default probabilities than non-students:

P̂r(default=Yes|student=Yes) = e−3.5041+0.4049×1

1 + e−3.5041+0.4049×1 = 0.0431,

P̂r(default=Yes|student=No) = e−3.5041+0.4049×0

1 + e−3.5041+0.4049×0 = 0.0292.

4.3.4 Multiple Logistic Regression

We now consider the problem of predicting a binary response using multiple predictors. By analogy with the extension from simple to multiple linear regression in Chapter 3, we can generalize (4.4) as follows:

log

( p(X)

1−p(X)

) = β0 + β1X1 + · · · + βpXp, (4.6)

where X = (X1, . . . , Xp) are p predictors. Equation 4.6 can be rewritten as

p(X) = eβ0+β1X1+···+βpXp

1 + eβ0+β1X1+···+βpXp . (4.7)

Just as in Section 4.3.2, we use the maximum likelihood method to estimate β0, β1, . . . , βp. Table 4.3 shows the coefficient estimates for a logistic regression model

that uses balance, income (in thousands of dollars), and student status to predict probability of default. There is a surprising result here. The p-

138 4. Classification

Coefficient Std. error z-statistic p-value Intercept −10.8690 0.4923 −22.08 <0.0001 balance 0.0057 0.0002 24.74 <0.0001 income 0.0030 0.0082 0.37 0.7115 student[Yes] −0.6468 0.2362 −2.74 0.0062

TABLE 4.3. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using balance, income, and student status. Student status is encoded as a dummy variable student[Yes], with a value of 1 for a student and a value of 0 for a non-student. In fitting this model, income was measured in thousands of dollars.

values associated with balance and the dummy variable for student status are very small, indicating that each of these variables is associated with the probability of default. However, the coefficient for the dummy variable is negative, indicating that students are less likely to default than non- students. In contrast, the coefficient for the dummy variable is positive in Table 4.2. How is it possible for student status to be associated with an increase in probability of default in Table 4.2 and a decrease in probability of default in Table 4.3? The left-hand panel of Figure 4.3 provides a graph- ical illustration of this apparent paradox. The orange and blue solid lines show the average default rates for students and non-students, respectively, as a function of credit card balance. The negative coefficient for student in the multiple logistic regression indicates that for a fixed value of balance and income, a student is less likely to default than a non-student. Indeed, we observe from the left-hand panel of Figure 4.3 that the student default rate is at or below that of the non-student default rate for every value of balance. But the horizontal broken lines near the base of the plot, which show the default rates for students and non-students averaged over all val- ues of balance and income, suggest the opposite effect: the overall student default rate is higher than the non-student default rate. Consequently, there is a positive coefficient for student in the single variable logistic regression output shown in Table 4.2. The right-hand panel of Figure 4.3 provides an explanation for this dis-

crepancy. The variables student and balance are correlated. Students tend to hold higher levels of debt, which is in turn associated with higher prob- ability of default. In other words, students are more likely to have large credit card balances, which, as we know from the left-hand panel of Fig- ure 4.3, tend to be associated with high default rates. Thus, even though an individual student with a given credit card balance will tend to have a lower probability of default than a non-student with the same credit card balance, the fact that students on the whole tend to have higher credit card balances means that overall, students tend to default at a higher rate than non-students. This is an important distinction for a credit card company that is trying to determine to whom they should offer credit. A student is

4.3 Logistic Regression 139

500 1000 1500 2000

0 .0

0 .2

0 .4

0 .6

0 .8

Credit Card Balance

D e

fa u

lt R

a te

No Yes

0 5 0 0

1 0 0 0

1 5 0 0

2 0 0 0

2 5 0 0

Student Status

C re

d it C

a rd

B a

la n

c e

FIGURE 4.3. Confounding in the Default data. Left: Default rates are shown for students (orange) and non-students (blue). The solid lines display default rate as a function of balance, while the horizontal broken lines display the overall default rates. Right: Boxplots of balance for students (orange) and non-students (blue) are shown.

riskier than a non-student if no information about the student’s credit card balance is available. However, that student is less risky than a non-student with the same credit card balance! This simple example illustrates the dangers and subtleties associated

with performing regressions involving only a single predictor when other predictors may also be relevant. As in the linear regression setting, the results obtained using one predictor may be quite different from those ob- tained using multiple predictors, especially when there is correlation among the predictors. In general, the phenomenon seen in Figure 4.3 is known as confounding.

confounding By substituting estimates for the regression coefficients from Table 4.3

into (4.7), we can make predictions. For example, a student with a credit card balance of $1, 500 and an income of $40, 000 has an estimated proba- bility of default of

p̂(X) = e−10.869+0.00574×1,500+0.003×40−0.6468×1

1 + e−10.869+0.00574×1,500+0.003×40−0.6468×1 = 0.058. (4.8)

A non-student with the same balance and income has an estimated prob- ability of default of

p̂(X) = e−10.869+0.00574×1,500+0.003×40−0.6468×0

1 + e−10.869+0.00574×1,500+0.003×40−0.6468×0 = 0.105. (4.9)

(Here we multiply the income coefficient estimate from Table 4.3 by 40, rather than by 40,000, because in that table the model was fit with income measured in units of $1, 000.)

140 4. Classification

4.3.5 Multinomial Logistic Regression

We sometimes wish to classify a response variable that has more than two classes. For example, in Section 4.2 we had three categories of medical con- dition in the emergency room: stroke, drug overdose, epileptic seizure. However, the logistic regression approach that we have seen in this section only allows for K = 2 classes for the response variable. It turns out that it is possible to extend the two-class logistic regression

approach to the setting of K > 2 classes. This extension is sometimes known as multinomial logistic regression. To do this, we first select a single

multinomial logistic regression

class to serve as the baseline; without loss of generality, we select the Kth class for this role. Then we replace the model (4.7) with the model

Pr(Y = k|X = x) = eβk0+βk1x1+···+βkpxp

1 + ∑K−1

l=1 e βl0+βl1x1+···+βlpxp

(4.10)

for k = 1, . . . , K−1, and

Pr(Y = K|X = x) = 1

1 + ∑K−1

l=1 e βl0+βl1x1+···+βlpxp

. (4.11)

It is not hard to show that for k = 1, . . . , K−1,

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) = βk0 + βk1x1 + · · · + βkpxp. (4.12)

Notice that (4.12) is quite similar to (4.6). Equation 4.12 indicates that once again, the log odds between any pair of classes is linear in the features. It turns out that in (4.10)–(4.12), the decision to treat the Kth class as

the baseline is unimportant. For example, when classifying emergency room visits into stroke, drug overdose, and epileptic seizure, suppose that we fit two multinomial logistic regression models: one treating stroke as the baseline, another treating drug overdose as the baseline. The coefficient estimates will differ between the two fitted models due to the differing choice of baseline, but the fitted values (predictions), the log odds between any pair of classes, and the other key model outputs will remain the same. Nonetheless, interpretation of the coefficients in a multinomial logistic

regression model must be done with care, since it is tied to the choice of baseline. For example, if we set epileptic seizure to be the baseline, then we can interpret βstroke0 as the log odds of stroke versus epileptic seizure, given that x1 = . . . = xp = 0. Furthermore, a one-unit increase in Xj is associated with a βstrokej increase in the log odds of stroke over epileptic seizure. Stated another way, if Xj increases by one unit, then

Pr(Y = stroke|X = x) Pr(Y = epileptic seizure|X = x)

increases by eβstrokej .

4.4 Generative Models for Classification 141

We now briefly present an alternative coding for multinomial logistic regression, known as the softmax coding. The softmax coding is equivalent

softmax to the coding just described in the sense that the fitted values, log odds between any pair of classes, and other key model outputs will remain the same, regardless of coding. But the softmax coding is used extensively in some areas of the machine learning literature (and will appear again in Chapter 10), so it is worth being aware of it. In the softmax coding, rather than selecting a baseline class, we treat all K classes symmetrically, and assume that for k = 1, . . . , K,

Pr(Y = k|X = x) = eβk0+βk1x1+···+βkpxp

∑K l=1 e

βl0+βl1x1+···+βlpxp . (4.13)

Thus, rather than estimating coefficients for K − 1 classes, we actually estimate coefficients for all K classes. It is not hard to see that as a result of (4.13), the log odds ratio between the kth and k′th classes equals

log

( Pr(Y = k|X = x) Pr(Y = k′|X = x)

) = (βk0 − βk′0) + (βk1 − βk′1)x1 + · · · + (βkp − βk′p)xp.

(4.14)

4.4 Generative Models for Classification

Logistic regression involves directly modeling Pr(Y = k|X = x) using the logistic function, given by (4.7) for the case of two response classes. In statistical jargon, we model the conditional distribution of the response Y , given the predictor(s) X. We now consider an alternative and less direct approach to estimating these probabilities. In this new approach, we model the distribution of the predictors X separately in each of the response classes (i.e. for each value of Y ). We then use Bayes’ theorem to flip these around into estimates for Pr(Y = k|X = x). When the distribution of X within each class is assumed to be normal, it turns out that the model is very similar in form to logistic regression. Why do we need another method, when we have logistic regression?

There are several reasons:

• When there is substantial separation between the two classes, the parameter estimates for the logistic regression model are surprisingly unstable. The methods that we consider in this section do not suffer from this problem.

• If the distribution of the predictors X is approximately normal in each of the classes and the sample size is small, then the approaches in this section may be more accurate than logistic regression.

• The methods in this section can be naturally extended to the case of more than two response classes. (In the case of more than two

142 4. Classification

response classes, we can also use multinomial logistic regression from Section 4.3.5.)

Suppose that we wish to classify an observation into one of K classes, where K ≥ 2. In other words, the qualitative response variable Y can take on K possible distinct and unordered values. Let πk represent the overall or prior probability that a randomly chosen observation comes from the

prior kth class. Let fk(X) ≡ Pr(X|Y = k)1 denote the density function of X

density functionfor an observation that comes from the kth class. In other words, fk(x) is

relatively large if there is a high probability that an observation in the kth class has X ≈ x, and fk(x) is small if it is very unlikely that an observation in the kth class has X ≈ x. Then Bayes’ theorem states that

Bayes’ theorem

Pr(Y = k|X = x) = πkfk(x)

∑K l=1 πlfl(x)

. (4.15)

In accordance with our earlier notation, we will use the abbreviation pk(x) = Pr(Y = k|X = x); this is the posterior probability that an observation

posterior X = x belongs to the kth class. That is, it is the probability that the observation belongs to the kth class, given the predictor value for that observation. Equation 4.15 suggests that instead of directly computing the posterior

probability pk(x) as in Section 4.3.1, we can simply plug in estimates of πk and fk(x) into (4.15). In general, estimating πk is easy if we have a random sample from the population: we simply compute the fraction of the training observations that belong to the kth class. However, estimating the density function fk(x) is much more challenging. As we will see, to estimate fk(x), we will typically have to make some simplifying assumptions. We know from Chapter 2 that the Bayes classifier, which classifies an

observation x to the class for which pk(x) is largest, has the lowest possible error rate out of all classifiers. (Of course, this is only true if all of the terms in (4.15) are correctly specified.) Therefore, if we can find a way to estimate fk(x), then we can plug it into (4.15) in order to approximate the Bayes classifier. In the following sections, we discuss three classifiers that use different

estimates of fk(x) in (4.15) to approximate the Bayes classifier: linear dis- criminant analysis, quadratic discriminant analysis, and naive Bayes.

4.4.1 Linear Discriminant Analysis for p = 1

For now, assume that p = 1—that is, we have only one predictor. We would like to obtain an estimate for fk(x) that we can plug into (4.15) in order to estimate pk(x). We will then classify an observation to the class for which

1Technically, this definition is only correct if X is a qualitative random variable. If X is quantitative, then fk(x)dx corresponds to the probability of X falling in a small region dx around x.

4.4 Generative Models for Classification 143

pk(x) is greatest. To estimate fk(x), we will first make some assumptions about its form. In particular, we assume that fk(x) is normal or Gaussian. In the one-

normal Gaussian

dimensional setting, the normal density takes the form

fk(x) = 1

√ 2πσk

exp

( −

2σ2k (x−µk)2

) , (4.16)

where µk and σ 2 k are the mean and variance parameters for the kth class.

For now, let us further assume that σ21 = · · · = σ2K: that is, there is a shared variance term across all K classes, which for simplicity we can denote by σ2. Plugging (4.16) into (4.15), we find that

pk(x) = πk

1√ 2πσ

exp ( − 1

2σ2 (x−µk)2

)

∑K l=1 πl

1√ 2πσ

exp ( − 1

2σ2 (x−µl)2

). (4.17)

(Note that in (4.17), πk denotes the prior probability that an observation belongs to the kth class, not to be confused with π ≈ 3.14159, the math- ematical constant.) The Bayes classifier2 involves assigning an observation X = x to the class for which (4.17) is largest. Taking the log of (4.17) and rearranging the terms, it is not hard to show3 that this is equivalent to assigning the observation to the class for which

δk(x) = x · µk σ2 −

µ2k 2σ2

+ log(πk) (4.18)

is largest. For instance, if K = 2 and π1 = π2, then the Bayes classifier assigns an observation to class 1 if 2x (µ1 −µ2) > µ21 − µ22, and to class 2 otherwise. The Bayes decision boundary is the point for which δ1(x) = δ2(x); one can show that this amounts to

x = µ21 −µ22

2(µ1 −µ2) =

µ1 + µ2 2

. (4.19)

An example is shown in the left-hand panel of Figure 4.4. The two normal density functions that are displayed, f1(x) and f2(x), represent two distinct classes. The mean and variance parameters for the two density functions are µ1 = −1.25, µ2 = 1.25, and σ21 = σ22 = 1. The two densities overlap, and so given that X = x, there is some uncertainty about the class to which the observation belongs. If we assume that an observation is equally likely to come from either class—that is, π1 = π2 = 0.5—then by inspection of (4.19), we see that the Bayes classifier assigns the observation to class 1 if x < 0 and class 2 otherwise. Note that in this case, we can compute the Bayes classifier because we know that X is drawn from a Gaussian distribution within each class, and we know all of the parameters involved. In a real-life situation, we are not able to calculate the Bayes classifier.

2Recall that the Bayes classifier assigns an observation to the class for which pk(x) is largest. This is different from Bayes’ theorem in (4.13), which allows us to manipulate conditional distributions.

3See Exercise 2 at the end of this chapter.

144 4. Classification

−4 −2 0 2 4 −3 −2 −1 0 1 2 3 4

0 1

2 3

4 5

FIGURE 4.4. Left: Two one-dimensional normal density functions are shown. The dashed vertical line represents the Bayes decision boundary. Right: 20 obser- vations were drawn from each of the two classes, and are shown as histograms. The Bayes decision boundary is again shown as a dashed vertical line. The solid vertical line represents the LDA decision boundary estimated from the training data.

In practice, even if we are quite certain of our assumption that X is drawn from a Gaussian distribution within each class, to apply the Bayes classifier we still have to estimate the parameters µ1, . . . , µK, π1, . . . , πK, and σ2. The linear discriminant analysis (LDA) method approximates the

linear discriminant analysis

Bayes classifier by plugging estimates for πk, µk, and σ 2 into (4.18). In

particular, the following estimates are used:

µ̂k = 1

∑

i:yi=k

σ̂2 = 1

n−K

K∑

k=1

∑

i:yi=k

(xi − µ̂k)2 (4.20)

where n is the total number of training observations, and nk is the number of training observations in the kth class. The estimate for µk is simply the average of all the training observations from the kth class, while σ̂2 can be seen as a weighted average of the sample variances for each of the K classes. Sometimes we have knowledge of the class membership probabili- ties π1, . . . , πK, which can be used directly. In the absence of any additional information, LDA estimates πk using the proportion of the training obser- vations that belong to the kth class. In other words,

π̂k = nk/n. (4.21)

The LDA classifier plugs the estimates given in (4.20) and (4.21) into (4.18), and assigns an observation X = x to the class for which

δ̂k(x) = x · µ̂k σ̂2 −

µ̂2k 2σ̂2

+ log(π̂k) (4.22)

4.4 Generative Models for Classification 145

is largest. The word linear in the classifier’s name stems from the fact that the discriminant functions δ̂k(x) in (4.22) are linear functions of x (as

discriminant functionopposed to a more complex function of x).

The right-hand panel of Figure 4.4 displays a histogram of a random sample of 20 observations from each class. To implement LDA, we began by estimating πk, µk, and σ

2 using (4.20) and (4.21). We then computed the decision boundary, shown as a black solid line, that results from assigning an observation to the class for which (4.22) is largest. All points to the left of this line will be assigned to the green class, while points to the right of this line are assigned to the purple class. In this case, since n1 = n2 = 20, we have π̂1 = π̂2. As a result, the decision boundary corresponds to the midpoint between the sample means for the two classes, (µ̂1 + µ̂2)/2. The figure indicates that the LDA decision boundary is slightly to the left of the optimal Bayes decision boundary, which instead equals (µ1 + µ2)/2 = 0. How well does the LDA classifier perform on this data? Since this is simulated data, we can generate a large number of test observations in order to compute the Bayes error rate and the LDA test error rate. These are 10.6 % and 11.1 %, respectively. In other words, the LDA classifier’s error rate is only 0.5 % above the smallest possible error rate! This indicates that LDA is performing pretty well on this data set. To reiterate, the LDA classifier results from assuming that the obser-

vations within each class come from a normal distribution with a class- specific mean and a common variance σ2, and plugging estimates for these parameters into the Bayes classifier. In Section 4.4.3, we will consider a less stringent set of assumptions, by allowing the observations in the kth class to have a class-specific variance, σ2k.

4.4.2 Linear Discriminant Analysis for p >1

We now extend the LDA classifier to the case of multiple predictors. To do this, we will assume that X = (X1, X2, . . . , Xp) is drawn from a multi- variate Gaussian (or multivariate normal) distribution, with a class-specific

multivariate Gaussianmean vector and a common covariance matrix. We begin with a brief review

of this distribution. The multivariate Gaussian distribution assumes that each individual pre-

dictor follows a one-dimensional normal distribution, as in (4.16), with some correlation between each pair of predictors. Two examples of multivariate Gaussian distributions with p = 2 are shown in Figure 4.5. The height of the surface at any particular point represents the probability that both X1 and X2 fall in a small region around that point. In either panel, if the sur- face is cut along the X1 axis or along the X2 axis, the resulting cross-section will have the shape of a one-dimensional normal distribution. The left-hand panel of Figure 4.5 illustrates an example in which Var(X1) = Var(X2) and Cor(X1, X2) = 0; this surface has a characteristic bell shape. However, the bell shape will be distorted if the predictors are correlated or have unequal variances, as is illustrated in the right-hand panel of Figure 4.5. In this situation, the base of the bell will have an elliptical, rather than circular,

146 4. Classification

x1x1

x 2x 2

FIGURE 4.5. Two multivariate Gaussian density functions are shown, with p = 2. Left: The two predictors are uncorrelated. Right: The two variables have a correlation of 0.7.

shape. To indicate that a p-dimensional random variable X has a multi- variate Gaussian distribution, we write X ∼ N(µ, Σ). Here E(X) = µ is the mean of X (a vector with p components), and Cov(X) = Σ is the p×p covariance matrix of X. Formally, the multivariate Gaussian density is defined as

f(x) = 1

(2π)p/2|Σ|1/2 exp

( − 1

2 (x−µ)T Σ−1(x−µ)

) . (4.23)

In the case of p > 1 predictors, the LDA classifier assumes that the observations in the kth class are drawn from a multivariate Gaussian dis- tribution N(µk, Σ), where µk is a class-specific mean vector, and Σ is a covariance matrix that is common to all K classes. Plugging the density function for the kth class, fk(X = x), into (4.15) and performing a little bit of algebra reveals that the Bayes classifier assigns an observation X = x to the class for which

δk(x) = x T Σ−1µk −

2 µTk Σ

−1µk + log πk (4.24)

is largest. This is the vector/matrix version of (4.18). An example is shown in the left-hand panel of Figure 4.6. Three equally-

sized Gaussian classes are shown with class-specific mean vectors and a common covariance matrix. The three ellipses represent regions that con- tain 95 % of the probability for each of the three classes. The dashed lines are the Bayes decision boundaries. In other words, they represent the set of values x for which δk(x) = δℓ(x); i.e.

xT Σ−1µk − 1

2 µTk Σ

−1µk = x T Σ−1µl −

2 µTl Σ

−1µl (4.25)

for k ̸= l. (The log πk term from (4.24) has disappeared because each of the three classes has the same number of training observations; i.e. πk is

4.4 Generative Models for Classification 147

−4 −2 0 2 4

− 4

− 2

0 2

−4 −2 0 2 4

− 4

− 2

0 2

X1X1 X

X 2

FIGURE 4.6. An example with three classes. The observations from each class are drawn from a multivariate Gaussian distribution with p = 2, with a class-spe- cific mean vector and a common covariance matrix. Left: Ellipses that contain 95 % of the probability for each of the three classes are shown. The dashed lines are the Bayes decision boundaries. Right: 20 observations were generated from each class, and the corresponding LDA decision boundaries are indicated using solid black lines. The Bayes decision boundaries are once again shown as dashed lines.

the same for each class.) Note that there are three lines representing the Bayes decision boundaries because there are three pairs of classes among the three classes. That is, one Bayes decision boundary separates class 1 from class 2, one separates class 1 from class 3, and one separates class 2 from class 3. These three Bayes decision boundaries divide the predictor space into three regions. The Bayes classifier will classify an observation according to the region in which it is located. Once again, we need to estimate the unknown parameters µ1, . . . , µK,

π1, . . . , πK, and Σ; the formulas are similar to those used in the one- dimensional case, given in (4.20). To assign a new observation X = x, LDA plugs these estimates into (4.24) to obtain quantities δ̂k(x), and clas- sifies to the class for which δ̂k(x) is largest. Note that in (4.24) δk(x) is a linear function of x; that is, the LDA decision rule depends on x only through a linear combination of its elements. As previously discussed, this is the reason for the word linear in LDA. In the right-hand panel of Figure 4.6, 20 observations drawn from each of

the three classes are displayed, and the resulting LDA decision boundaries are shown as solid black lines. Overall, the LDA decision boundaries are pretty close to the Bayes decision boundaries, shown again as dashed lines. The test error rates for the Bayes and LDA classifiers are 0.0746 and 0.0770, respectively. This indicates that LDA is performing well on this data. We can perform LDA on the Default data in order to predict whether

or not an individual will default on the basis of credit card balance and

148 4. Classification

True default status No Yes Total

Predicted No 9644 252 9896 default status Yes 23 81 104

Total 9667 333 10000

TABLE 4.4. A confusion matrix compares the LDA predictions to the true de- fault statuses for the 10,000 training observations in the Default data set. Ele- ments on the diagonal of the matrix represent individuals whose default statuses were correctly predicted, while off-diagonal elements represent individuals that were misclassified. LDA made incorrect predictions for 23 individuals who did not default and for 252 individuals who did default.

student status.4 The LDA model fit to the 10,000 training samples results in a training error rate of 2.75 %. This sounds like a low error rate, but two caveats must be noted.

• First of all, training error rates will usually be lower than test error rates, which are the real quantity of interest. In other words, we might expect this classifier to perform worse if we use it to predict whether or not a new set of individuals will default. The reason is that we specifically adjust the parameters of our model to do well on the training data. The higher the ratio of parameters p to number of samples n, the more we expect this overfitting to play a role. For

overfitting these data we don’t expect this to be a problem, since p = 2 and n = 10, 000.

• Second, since only 3.33 % of the individuals in the training sample defaulted, a simple but useless classifier that always predicts that an individual will not default, regardless of his or her credit card balance and student status, will result in an error rate of 3.33 %. In other words, the trivial null classifier will achieve an error rate that

null is only a bit higher than the LDA training set error rate.

In practice, a binary classifier such as this one can make two types of errors: it can incorrectly assign an individual who defaults to the no default category, or it can incorrectly assign an individual who does not default to the default category. It is often of interest to determine which of these two types of errors are being made. A confusion matrix, shown for the Default

confusion matrixdata in Table 4.4, is a convenient way to display this information. The

table reveals that LDA predicted that a total of 104 people would default. Of these people, 81 actually defaulted and 23 did not. Hence only 23 out of 9,667 of the individuals who did not default were incorrectly labeled.

4The careful reader will notice that student status is qualitative — thus, the normality assumption made by LDA is clearly violated in this example! However, LDA is often remarkably robust to model violations, as this example shows. Naive Bayes, discussed in Section 4.4.4, provides an alternative to LDA that does not assume normally distributed predictors.

4.4 Generative Models for Classification 149

This looks like a pretty low error rate! However, of the 333 individuals who defaulted, 252 (or 75.7 %) were missed by LDA. So while the overall error rate is low, the error rate among individuals who defaulted is very high. From the perspective of a credit card company that is trying to identify high-risk individuals, an error rate of 252/333 = 75.7 % among individuals who default may well be unacceptable. Class-specific performance is also important in medicine and biology,

where the terms sensitivity and specificity characterize the performance of sensitivity

specificity a classifier or screening test. In this case the sensitivity is the percentage of true defaulters that are identified; it equals 24.3 %. The specificity is the percentage of non-defaulters that are correctly identified; it equals (1− 23/9667) = 99.8 %. Why does LDA do such a poor job of classifying the customers who de-

fault? In other words, why does it have such low sensitivity? As we have seen, LDA is trying to approximate the Bayes classifier, which has the low- est total error rate out of all classifiers. That is, the Bayes classifier will yield the smallest possible total number of misclassified observations, re- gardless of the class from which the errors stem. Some misclassifications will result from incorrectly assigning a customer who does not default to the default class, and others will result from incorrectly assigning a customer who defaults to the non-default class. In contrast, a credit card company might particularly wish to avoid incorrectly classifying an individual who will default, whereas incorrectly classifying an individual who will not de- fault, though still to be avoided, is less problematic. We will now see that it is possible to modify LDA in order to develop a classifier that better meets the credit card company’s needs. The Bayes classifier works by assigning an observation to the class for

which the posterior probability pk(X) is greatest. In the two-class case, this amounts to assigning an observation to the default class if

Pr(default = Yes|X = x) > 0.5. (4.26) Thus, the Bayes classifier, and by extension LDA, uses a threshold of 50 % for the posterior probability of default in order to assign an observation to the default class. However, if we are concerned about incorrectly pre- dicting the default status for individuals who default, then we can consider lowering this threshold. For instance, we might label any customer with a posterior probability of default above 20 % to the default class. In other words, instead of assigning an observation to the default class if (4.26) holds, we could instead assign an observation to this class if

Pr(default = Yes|X = x) > 0.2. (4.27) The error rates that result from taking this approach are shown in Table 4.5. Now LDA predicts that 430 individuals will default. Of the 333 individuals who default, LDA correctly predicts all but 138, or 41.4 %. This is a vast improvement over the error rate of 75.7 % that resulted from using the threshold of 50 %. However, this improvement comes at a cost: now 235

150 4. Classification

True default status No Yes Total

Predicted No 9432 138 9570 default status Yes 235 195 430

Total 9667 333 10000

TABLE 4.5. A confusion matrix compares the LDA predictions to the true de- fault statuses for the 10,000 training observations in the Default data set, using a modified threshold value that predicts default for any individuals whose posterior default probability exceeds 20 %.

0.0 0.1 0.2 0.3 0.4 0.5

0 .0

0 .2

0 .4

0 .6

Threshold

E rr

o r

R a te

FIGURE 4.7. For the Default data set, error rates are shown as a function of the threshold value for the posterior probability that is used to perform the assign- ment. The black solid line displays the overall error rate. The blue dashed line represents the fraction of defaulting customers that are incorrectly classified, and the orange dotted line indicates the fraction of errors among the non-defaulting customers.

individuals who do not default are incorrectly classified. As a result, the overall error rate has increased slightly to 3.73 %. But a credit card company may consider this slight increase in the total error rate to be a small price to pay for more accurate identification of individuals who do indeed default. Figure 4.7 illustrates the trade-off that results from modifying the thresh-

old value for the posterior probability of default. Various error rates are shown as a function of the threshold value. Using a threshold of 0.5, as in (4.26), minimizes the overall error rate, shown as a black solid line. This is to be expected, since the Bayes classifier uses a threshold of 0.5 and is known to have the lowest overall error rate. But when a threshold of 0.5 is used, the error rate among the individuals who default is quite high (blue dashed line). As the threshold is reduced, the error rate among individuals who default decreases steadily, but the error rate among the individuals who do not default increases. How can we decide which threshold value is best? Such a decision must be based on domain knowledge, such as detailed information about the costs associated with default. The ROC curve is a popular graphic for simultaneously displaying the

ROC curve two types of errors for all possible thresholds. The name “ROC” is his- toric, and comes from communications theory. It is an acronym for receiver

4.4 Generative Models for Classification 151

ROC Curve

False positive rate

T ru

e p

o s it iv

e r

a te

0.0 0.2 0.4 0.6 0.8 1.0

0 .0

0 .2

0 .4

0 .6

0 .8

1 .0

FIGURE 4.8. A ROC curve for the LDA classifier on the Default data. It traces out two types of error as we vary the threshold value for the posterior probability of default. The actual thresholds are not shown. The true positive rate is the sensitivity: the fraction of defaulters that are correctly identified, using a given threshold value. The false positive rate is 1-specificity: the fraction of non-defaulters that we classify incorrectly as defaulters, using that same threshold value. The ideal ROC curve hugs the top left corner, indicating a high true positive rate and a low false positive rate. The dotted line represents the “no information” classifier; this is what we would expect if student status and credit card balance are not associated with probability of default.

operating characteristics. Figure 4.8 displays the ROC curve for the LDA classifier on the training data. The overall performance of a classifier, sum- marized over all possible thresholds, is given by the area under the (ROC) curve (AUC). An ideal ROC curve will hug the top left corner, so the larger

area under the (ROC) curve

the AUC the better the classifier. For this data the AUC is 0.95, which is close to the maximum of one so would be considered very good. We expect a classifier that performs no better than chance to have an AUC of 0.5 (when evaluated on an independent test set not used in model training). ROC curves are useful for comparing different classifiers, since they take into account all possible thresholds. It turns out that the ROC curve for the logistic regression model of Section 4.3.4 fit to these data is virtually indis- tinguishable from this one for the LDA model, so we do not display it here. As we have seen above, varying the classifier threshold changes its true

positive and false positive rate. These are also called the sensitivity and one sensitivity

minus the specificity of our classifier. Since there is an almost bewildering specificity

array of terms used in this context, we now give a summary. Table 4.6 shows the possible results when applying a classifier (or diagnostic test)

152 4. Classification

True class − or Null + or Non-null Total

Predicted − or Null True Neg. (TN) False Neg. (FN) N∗ class + or Non-null False Pos. (FP) True Pos. (TP) P∗

Total N P

TABLE 4.6. Possible results when applying a classifier or diagnostic test to a population.

Name Definition Synonyms False Pos. rate FP/N Type I error, 1−Specificity True Pos. rate TP/P 1−Type II error, power, sensitivity, recall Pos. Pred. value TP/P∗ Precision, 1−false discovery proportion Neg. Pred. value TN/N∗

TABLE 4.7. Important measures for classification and diagnostic testing, derived from quantities in Table 4.6.

to a population. To make the connection with the epidemiology literature, we think of “+” as the “disease” that we are trying to detect, and “−” as the “non-disease” state. To make the connection to the classical hypothesis testing literature, we think of “−” as the null hypothesis and “+” as the alternative (non-null) hypothesis. In the context of the Default data, “+” indicates an individual who defaults, and “−” indicates one who does not. Table 4.7 lists many of the popular performance measures that are used in

this context. The denominators for the false positive and true positive rates are the actual population counts in each class. In contrast, the denominators for the positive predictive value and the negative predictive value are the total predicted counts for each class.

4.4.3 Quadratic Discriminant Analysis

As we have discussed, LDA assumes that the observations within each class are drawn from a multivariate Gaussian distribution with a class- specific mean vector and a covariance matrix that is common to all K classes. Quadratic discriminant analysis (QDA) provides an alternative

quadratic discriminant analysis

approach. Like LDA, the QDA classifier results from assuming that the observations from each class are drawn from a Gaussian distribution, and plugging estimates for the parameters into Bayes’ theorem in order to per- form prediction. However, unlike LDA, QDA assumes that each class has its own covariance matrix. That is, it assumes that an observation from the kth class is of the form X ∼ N(µk, Σk), where Σk is a covariance matrix for the kth class. Under this assumption, the Bayes classifier assigns an

4.4 Generative Models for Classification 153

observation X = x to the class for which

δk(x) = − 1

2 (x−µk)T Σ−1k (x−µk)−

2 log |Σk| + log πk

= − 1

2 xT Σ−1k x + x

T Σ−1k µk − 1

2 µTk Σ

−1 k µk −

2 log |Σk| + log πk

(4.28)

is largest. So the QDA classifier involves plugging estimates for Σk, µk, and πk into (4.28), and then assigning an observation X = x to the class for which this quantity is largest. Unlike in (4.24), the quantity x appears as a quadratic function in (4.28). This is where QDA gets its name. Why does it matter whether or not we assume that the K classes share a

common covariance matrix? In other words, why would one prefer LDA to QDA, or vice-versa? The answer lies in the bias-variance trade-off. When there are p predictors, then estimating a covariance matrix requires esti- mating p(p+1)/2 parameters. QDA estimates a separate covariance matrix for each class, for a total of Kp(p+1)/2 parameters. With 50 predictors this is some multiple of 1,275, which is a lot of parameters. By instead assum- ing that the K classes share a common covariance matrix, the LDA model becomes linear in x, which means there are Kp linear coefficients to esti- mate. Consequently, LDA is a much less flexible classifier than QDA, and so has substantially lower variance. This can potentially lead to improved prediction performance. But there is a trade-off: if LDA’s assumption that the K classes share a common covariance matrix is badly off, then LDA can suffer from high bias. Roughly speaking, LDA tends to be a better bet than QDA if there are relatively few training observations and so reducing variance is crucial. In contrast, QDA is recommended if the training set is very large, so that the variance of the classifier is not a major concern, or if the assumption of a common covariance matrix for the K classes is clearly untenable. Figure 4.9 illustrates the performances of LDA and QDA in two scenarios.

In the left-hand panel, the two Gaussian classes have a common correla- tion of 0.7 between X1 and X2. As a result, the Bayes decision boundary is linear and is accurately approximated by the LDA decision boundary. The QDA decision boundary is inferior, because it suffers from higher vari- ance without a corresponding decrease in bias. In contrast, the right-hand panel displays a situation in which the orange class has a correlation of 0.7 between the variables and the blue class has a correlation of −0.7. Now the Bayes decision boundary is quadratic, and so QDA more accurately approximates this boundary than does LDA.

4.4.4 Naive Bayes

In previous sections, we used Bayes’ theorem (4.15) to develop the LDA and QDA classifiers. Here, we use Bayes’ theorem to motivate the popular naive Bayes classifier.

naive Bayes

154 4. Classification

−4 −2 0 2 4

− 4

− 3

− 2

− 1

0 1

−4 −2 0 2 4

− 4

− 3

− 2

− 1

0 1

X1X1 X

X 2

FIGURE 4.9. Left: The Bayes (purple dashed), LDA (black dotted), and QDA (green solid) decision boundaries for a two-class problem with Σ1 = Σ2. The shading indicates the QDA decision rule. Since the Bayes decision boundary is linear, it is more accurately approximated by LDA than by QDA. Right: Details are as given in the left-hand panel, except that Σ1 ̸= Σ2. Since the Bayes decision boundary is non-linear, it is more accurately approximated by QDA than by LDA.

Recall that Bayes’ theorem (4.15) provides an expression for the pos- terior probability pk(x) = Pr(Y = k|X = x) in terms of π1, . . . , πK and f1(x), . . . , fK(x). To use (4.15) in practice, we need estimates for π1, . . . , πK and f1(x), . . . , fK(x). As we saw in previous sections, estimating the prior probabilities π1, . . . , πK is typically straightforward: for instance, we can estimate π̂k as the proportion of training observations belonging to the kth class, for k = 1, . . . , K. However, estimating f1(x), . . . , fK(x) is more subtle. Recall that fk(x)

is the p-dimensional density function for an observation in the kth class, for k = 1, . . . , K. In general, estimating a p-dimensional density function is challenging. In LDA, we make a very strong assumption that greatly sim- plifies the task: we assume that fk is the density function for a multivariate normal random variable with class-specific mean µk, and shared covariance matrix Σ. By contrast, in QDA, we assume that fk is the density function for a multivariate normal random variable with class-specific mean µk, and class-specific covariance matrix Σk. By making these very strong assump- tions, we are able to replace the very challenging problem of estimating K p-dimensional density functions with the much simpler problem of estimat- ing K p-dimensional mean vectors and one (in the case of LDA) or K (in the case of QDA) (p×p)-dimensional covariance matrices. The naive Bayes classifier takes a different tack for estimating f1(x), . . . ,

fK(x). Instead of assuming that these functions belong to a particular family of distributions (e.g. multivariate normal), we instead make a single assumption:

4.4 Generative Models for Classification 155

Within the kth class, the p predictors are independent.

Stated mathematically, this assumption means that for k = 1, . . . , K,

fk(x) = fk1(x1)×fk2(x2)× · · ·× fkp(xp), (4.29)

where fkj is the density function of the jth predictor among observations in the kth class. Why is this assumption so powerful? Essentially, estimating a p-dimen-

sional density function is challenging because we must consider not only the marginal distribution of each predictor — that is, the distribution of

marginal distributioneach predictor on its own — but also the joint distribution of the predictors

joint distribution

— that is, the association between the different predictors. In the case of a multivariate normal distribution, the association between the different predictors is summarized by the off-diagonal elements of the covariance matrix. However, in general, this association can be very hard to charac- terize, and exceedingly challenging to estimate. But by assuming that the p covariates are independent within each class, we completely eliminate the need to worry about the association between the p predictors, because we have simply assumed that there is no association between the predictors! Do we really believe the naive Bayes assumption that the p covariates

are independent within each class? In most settings, we do not. But even though this modeling assumption is made for convenience, it often leads to pretty decent results, especially in settings where n is not large enough relative to p for us to effectively estimate the joint distribution of the predic- tors within each class. In fact, since estimating a joint distribution requires such a huge amount of data, naive Bayes is a good choice in a wide range of settings. Essentially, the naive Bayes assumption introduces some bias, but reduces variance, leading to a classifier that works quite well in practice as a result of the bias-variance trade-off. Once we have made the naive Bayes assumption, we can plug (4.29) into

(4.15) to obtain an expression for the posterior probability,

Pr(Y = k|X = x) = πk ×fk1(x1)×fk2(x2)× · · ·× fkp(xp)

∑K l=1 πl ×fl1(x1)×fl2(x2)× · · ·× flp(xp)

(4.30)

for k = 1, . . . , K. To estimate the one-dimensional density function fkj using training data

x1j, . . . , xnj, we have a few options.

• If Xj is quantitative, then we can assume that Xj|Y = k ∼ N(µjk, σ2jk). In other words, we assume that within each class, the jth predictor is drawn from a (univariate) normal distribution. While this may sound a bit like QDA, there is one key difference, in that here we are assum- ing that the predictors are independent; this amounts to QDA with an additional assumption that the class-specific covariance matrix is diagonal.

156 4. Classification

True default status No Yes Total

Predicted No 9615 241 9856 default status Yes 52 92 144

Total 9667 333 10000

TABLE 4.8. Comparison of the naive Bayes predictions to the true default status for the 10, 000 training observations in the Default data set, when we predict default for any observation for which P(Y = default|X = x) > 0.5.

• If Xj is quantitative, then another option is to use a non-parametric estimate for fkj. A very simple way to do this is by making a his- togram for the observations of the jth predictor within each class. Then we can estimate fkj(xj) as the fraction of the training obser- vations in the kth class that belong to the same histogram bin as xj. Alternatively, we can use a kernel density estimator, which is essen-

kernel density estimator

tially a smoothed version of a histogram.

• If Xj is qualitative, then we can simply count the proportion of train- ing observations for the jth predictor corresponding to each class. For instance, suppose that Xj ∈ {1, 2, 3}, and we have 100 observations in the kth class. Suppose that the jth predictor takes on values of 1, 2, and 3 in 32, 55, and 13 of those observations, respectively. Then we can estimate fkj as

f̂kj(xj) =

⎧ ⎪⎨

⎪⎩

0.32 if xj = 1

0.55 if xj = 2

0.13 if xj = 3.

We now consider the naive Bayes classifier in a toy example with p = 3 predictors and K = 2 classes. The first two predictors are quantitative, and the third predictor is qualitative with three levels. Suppose further that π̂1 = π̂2 = 0.5. The estimated density functions f̂kj for k = 1, 2 and j = 1, 2, 3 are displayed in Figure 4.10. Now suppose that we wish to classify a new observation, x∗ = (0.4, 1.5, 1)T . It turns out that in this example, f̂11(0.4) = 0.368, f̂12(1.5) = 0.484, f̂13(1) = 0.226, and f̂21(0.4) = 0.030, f̂22(1.5) = 0.130, f̂23(1) = 0.616. Plugging these estimates into (4.30) results in posterior probability estimates of Pr(Y = 1|X = x∗) = 0.944 and Pr(Y = 2|X = x∗) = 0.056. Table 4.8 provides the confusion matrix resulting from applying the naive

Bayes classifier to the Default data set, where we predict a default if the posterior probability of a default — that is, P(Y = default|X = x) — ex- ceeds 0.5. Comparing this to the results for LDA in Table 4.4, our findings are mixed. While LDA has a slightly lower overall error rate, naive Bayes correctly predicts a higher fraction of the true defaulters. In this implemen- tation of naive Bayes, we have assumed that each quantitative predictor is

4.4 Generative Models for Classification 157

Density estimates for class k=1

f̂11 f̂12 f̂13

−4 −2 0 2 4

F re

q u e n cy

−2 0 2 4 1 2 3

Density estimates for class k=2

f̂21 f̂22 f̂23

−4 −2 0 2 4

F re

q u e n cy

−2 0 2 4 1 2 3

FIGURE 4.10. In the toy example in Section 4.4.4, we generate data with p = 3 predictors and K = 2 classes. The first two predictors are quantitative, and the third predictor is qualitative with three levels. In each class, the estimated density for each of the three predictors is displayed. If the prior probabilities for the two classes are equal, then the observation x∗ = (0.4, 1.5, 1)T has a 94.4% posterior probability of belonging to the first class.

True default status No Yes Total

Predicted No 9320 128 9448 default status Yes 347 205 552

Total 9667 333 10000

TABLE 4.9. Comparison of the naive Bayes predictions to the true default status for the 10, 000 training observations in the Default data set, when we predict default for any observation for which P(Y = default|X = x) > 0.2.

drawn from a Gaussian distribution (and, of course, that within each class, each predictor is independent). Just as with LDA, we can easily adjust the probability threshold for

predicting a default. For example, Table 4.9 provides the confusion matrix resulting from predicting a default if P(Y = default|X = x) > 0.2. Again,

158 4. Classification

the results are mixed relative to LDA with the same threshold (Table 4.5). Naive Bayes has a higher error rate, but correctly predicts almost two-thirds of the true defaults. In this example, it should not be too surprising that naive Bayes does

not convincingly outperform LDA: this data set has n = 10,000 and p = 4, and so the reduction in variance resulting from the naive Bayes assumption is not necessarily worthwhile. We expect to see a greater pay-off to using naive Bayes relative to LDA or QDA in instances where p is larger or n is smaller, so that reducing the variance is very important.

4.5 A Comparison of Classification Methods

4.5.1 An Analytical Comparison

We now perform an analytical (or mathematical) comparison of LDA, QDA, naive Bayes, and logistic regression. We consider these approaches in a setting with K classes, so that we assign an observation to the class that maximizes Pr(Y = k|X = x). Equivalently, we can set K as the baseline class and assign an observation to the class that maximizes

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) (4.31)

for k = 1, . . . , K. Examining the specific form of (4.31) for each method provides a clear understanding of their similarities and differences. First, for LDA, we can make use of Bayes’ Theorem (4.15) as well as

the assumption that the predictors within each class are drawn from a multivariate normal density (4.23) with class-specific mean and shared co- variance matrix in order to show that

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) = log

( πkfk(x)

πKfK(x)

)

= log

( πk exp

( −1

2 (x−µk)T Σ−1(x−µk)

)

πK exp ( −1

2 (x−µK)T Σ−1(x−µK)

) )

= log

( πk πK

) −

2 (x−µk)T Σ−1(x−µk)

+ 1

2 (x−µK)T Σ−1(x−µK)

= log

( πk πK

) −

2 (µk + µK)

T Σ−1(µk −µK)

+ xT Σ−1(µk −µK)

= ak + p∑

j=1

bkjxj, (4.32)

4.5 A Comparison of Classification Methods 159

where ak = log (

πk πK

) − 1

2 (µk + µK)

T Σ−1(µk − µK) and bkj is the jth component of Σ−1(µk −µK). Hence LDA, like logistic regression, assumes that the log odds of the posterior probabilities is linear in x. Using similar calculations, in the QDA setting (4.31) becomes

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) = ak +

p∑

j=1

bkjxj + p∑

j=1

p∑

l=1

ckjlxjxl, (4.33)

where ak, bkj, and ckjl are functions of πk, πK, µk, µK, Σk and ΣK. Again, as the name suggests, QDA assumes that the log odds of the posterior probabilities is quadratic in x. Finally, we examine (4.31) in the naive Bayes setting. Recall that in

this setting, fk(x) is modeled as a product of p one-dimensional functions fkj(xj) for j = 1, . . . , p. Hence,

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) = log

( πkfk(x)

πKfK(x)

)

= log

( πk ∏p

j=1 fkj(xj)

πK ∏p

j=1 fKj(xj)

)

= log

( πk πK

) +

p∑

j=1

log

( fkj(xj)

fKj(xj)

)

= ak + p∑

j=1

gkj(xj), (4.34)

where ak = log (

πk πK

) and gkj(xj) = log

( fkj(xj) fKj(xj)

) . Hence, the right-hand

side of (4.34) takes the form of a generalized additive model, a topic that is discussed further in Chapter 7. Inspection of (4.32), (4.33), and (4.34) yields the following observations

about LDA, QDA, and naive Bayes:

• LDA is a special case of QDA with ckjl = 0 for all j = 1, . . . , p, l = 1, . . . , p, and k = 1, . . . , K. (Of course, this is not surprising, since LDA is simply a restricted version of QDA with Σ1 = · · · = ΣK = Σ.)

• Any classifier with a linear decision boundary is a special case of naive Bayes with gkj(xj) = bkjxj. In particular, this means that LDA is a special case of naive Bayes! This is not at all obvious from the descriptions of LDA and naive Bayes earlier in the chapter, since each method makes very different assumptions: LDA assumes that the features are normally distributed with a common within-class covariance matrix, and naive Bayes instead assumes independence of the features.

160 4. Classification

• If we model fkj(xj) in the naive Bayes classifier using a one-dimensional Gaussian distribution N(µkj, σ

2 j ), then we end up with gkj(xj) =

bkjxj where bkj = (µkj − µKj)/σ2j . In this case, naive Bayes is actu- ally a special case of LDA with Σ restricted to be a diagonal matrix with jth diagonal element equal to σ2j .

• Neither QDA nor naive Bayes is a special case of the other. Naive Bayes can produce a more flexible fit, since any choice can be made for gkj(xj). However, it is restricted to a purely additive fit, in the sense that in (4.34), a function of xj is added to a function of xl, for j ̸= l; however, these terms are never multiplied. By contrast, QDA includes multiplicative terms of the form ckjlxjxl. Therefore, QDA has the potential to be more accurate in settings where interactions among the predictors are important in discriminating between classes.

None of these methods uniformly dominates the others: in any setting, the choice of method will depend on the true distribution of the predictors in each of the K classes, as well as other considerations, such as the values of n and p. The latter ties into the bias-variance trade-off. How does logistic regression tie into this story? Recall from (4.12) that

multinomial logistic regression takes the form

log

( Pr(Y = k|X = x) Pr(Y = K|X = x)

) = βk0 +

p∑

j=1

βkjxj.

This is identical to the linear form of LDA (4.32): in both cases,

log (

Pr(Y =k|X=x) Pr(Y =K|X=x)

) is a linear function of the predictors. In LDA, the co-

efficients in this linear function are functions of estimates for πk, πK, µk, µK, and Σ obtained by assuming that X1, . . . , Xp follow a normal distri- bution within each class. By contrast, in logistic regression, the coefficients are chosen to maximize the likelihood function (4.5). Thus, we expect LDA to outperform logistic regression when the normality assumption (approxi- mately) holds, and we expect logistic regression to perform better when it does not. We close with a brief discussion of K-nearest neighbors (KNN), intro-

duced in Chapter 2. Recall that KNN takes a completely different approach from the classifiers seen in this chapter. In order to make a prediction for an observation X = x, the training observations that are closest to x are identified. Then X is assigned to the class to which the plurality of these observations belong. Hence KNN is a completely non-parametric approach: no assumptions are made about the shape of the decision boundary. We make the following observations about KNN:

• Because KNN is completely non-parametric, we can expect this ap- proach to dominate LDA and logistic regression when the decision

4.5 A Comparison of Classification Methods 161

boundary is highly non-linear, provided that n is very large and p is small.

• In order to provide accurate classification, KNN requires a lot of ob- servations relative to the number of predictors—that is, n much larger than p. This has to do with the fact that KNN is non-parametric, and thus tends to reduce the bias while incurring a lot of variance.

• In settings where the decision boundary is non-linear but n is only modest, or p is not very small, then QDA may be preferred to KNN. This is because QDA can provide a non-linear decision boundary while taking advantage of a parametric form, which means that it requires a smaller sample size for accurate classification, relative to KNN.

• Unlike logistic regression, KNN does not tell us which predictors are important: we don’t get a table of coefficients as in Table 4.3.

4.5.2 An Empirical Comparison

We now compare the empirical (practical) performance of logistic regres- sion, LDA, QDA, naive Bayes, and KNN. We generated data from six dif- ferent scenarios, each of which involves a binary (two-class) classification problem. In three of the scenarios, the Bayes decision boundary is linear, and in the remaining scenarios it is non-linear. For each scenario, we pro- duced 100 random training data sets. On each of these training sets, we fit each method to the data and computed the resulting test error rate on a large test set. Results for the linear scenarios are shown in Figure 4.11, and the results for the non-linear scenarios are in Figure 4.12. The KNN method requires selection of K, the number of neighbors (not to be con- fused with the number of classes in earlier sections of this chapter). We performed KNN with two values of K: K = 1, and a value of K that was chosen automatically using an approach called cross-validation, which we discuss further in Chapter 5. We applied naive Bayes assuming univariate Gaussian densities for the features within each class (and, of course — since this is the key characteristic of naive Bayes — assuming independence of the features). In each of the six scenarios, there were p = 2 quantitative predictors.

The scenarios were as follows:

Scenario 1: There were 20 training observations in each of two classes. The observations within each class were uncorrelated random normal variables with a different mean in each class. The left-hand panel of Figure 4.11 shows that LDA performed well in this setting, as one would expect since this is the model assumed by LDA. Logistic regression also performed quite well, since it assumes a linear decision boundary. KNN performed poorly because

162 4. Classification

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 1

0 .2

5 0

.3 0

0 .3

5 0

.4 0

0 .4

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 2

0 .1

5 0

.2 0

0 .2

5 0

.3 0

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 3

0 .2

0 0

.2 5

0 .3

0 0

.3 5

0 .4

0 0

.4 5

0 .5

FIGURE 4.11. Boxplots of the test error rates for each of the linear scenarios described in the main text.

it paid a price in terms of variance that was not offset by a reduction in bias. QDA also performed worse than LDA, since it fit a more flexible classifier than necessary. The performance of naive Bayes was slightly better than QDA, because the naive Bayes assumption of independent predictors is correct.

Scenario 2: Details are as in Scenario 1, except that within each class, the two predictors had a correlation of −0.5. The center panel of Figure 4.11 indicates that the performance of most methods is similar to the previ- ous scenario. The notable exception is naive Bayes, which performs very poorly here, since the naive Bayes assumption of independent predictors is violated.

Scenario 3: As in the previous scenario, there is substantial negative cor- relation between the predictors within each class. However, this time we generated X1 and X2 from the t-distribution, with 50 observations per class. t-

distributionThe t-distribution has a similar shape to the normal distribution, but it has a tendency to yield more extreme points—that is, more points that are far from the mean. In this setting, the decision boundary was still linear, and so fit into the logistic regression framework. The set-up violated the assumptions of LDA, since the observations were not drawn from a normal distribution. The right-hand panel of Figure 4.11 shows that logistic regres- sion outperformed LDA, though both methods were superior to the other approaches. In particular, the QDA results deteriorated considerably as a consequence of non-normality. Naive Bayes performed very poorly because the independence assumption is violated.

Scenario 4: The data were generated from a normal distribution, with a correlation of 0.5 between the predictors in the first class, and correlation of −0.5 between the predictors in the second class. This setup corresponded to the QDA assumption, and resulted in quadratic decision boundaries. The left-hand panel of Figure 4.12 shows that QDA outperformed all of the other approaches. The naive Bayes assumption of independent predictors is violated, so naive Bayes performs poorly.

4.5 A Comparison of Classification Methods 163

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 4

0 .3

0 0

.3 5

0 .4

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 5

0 .1

8 0

.2 0

0 .2

2 0

.2 4

0 .2

6 0

.2 8

0 .3

0 0

.3 2

K N

N −

K N

N −

C V

L D

L o

g is

tic

N B

ay e

Q D

SCENARIO 6

0 .1

5 0

.2 0

0 .2

5 0

.3 0

0 .3

5 0

.4 0

0 .4

FIGURE 4.12. Boxplots of the test error rates for each of the non-linear sce- narios described in the main text.

Scenario 5: The data were generated from a normal distribution with un- correlated predictors. Then the responses were sampled from the logistic function applied to a complicated non-linear function of the predictors. The center panel of Figure 4.12 shows that both QDA and naive Bayes gave slightly better results than the linear methods, while the much more flexi- ble KNN-CV method gave the best results. But KNN with K = 1 gave the worst results out of all methods. This highlights the fact that even when the data exhibits a complex non-linear relationship, a non-parametric method such as KNN can still give poor results if the level of smoothness is not chosen correctly.

Scenario 6: The observations were generated from a normal distribution with a different diagonal covariance matrix for each class. However, the sample size was very small: just n = 6 in each class. Naive Bayes performed very well, because its assumptions are met. LDA and logistic regression performed poorly because the true decision boundary is non-linear, due to the unequal covariance matrices. QDA performed a bit worse than naive Bayes, because given the very small sample size, the former incurred too much variance in estimating the correlation between the predictors within each class. KNN’s performance also suffered due to the very small sample size.

These six examples illustrate that no one method will dominate the oth- ers in every situation. When the true decision boundaries are linear, then the LDA and logistic regression approaches will tend to perform well. When the boundaries are moderately non-linear, QDA or naive Bayes may give better results. Finally, for much more complicated decision boundaries, a non-parametric approach such as KNN can be superior. But the level of smoothness for a non-parametric approach must be chosen carefully. In the next chapter we examine a number of approaches for choosing the correct level of smoothness and, in general, for selecting the best overall method. Finally, recall from Chapter 3 that in the regression setting we can accom-

modate a non-linear relationship between the predictors and the response

164 4. Classification

by performing regression using transformations of the predictors. A similar approach could be taken in the classification setting. For instance, we could create a more flexible version of logistic regression by including X2, X3, and even X4 as predictors. This may or may not improve logistic regres- sion’s performance, depending on whether the increase in variance due to the added flexibility is offset by a sufficiently large reduction in bias. We could do the same for LDA. If we added all possible quadratic terms and cross-products to LDA, the form of the model would be the same as the QDA model, although the parameter estimates would be different. This device allows us to move somewhere between an LDA and a QDA model.

4.6 Generalized Linear Models

In Chapter 3, we assumed that the response Y is quantitative, and ex- plored the use of least squares linear regression to predict Y . Thus far in this chapter, we have instead assumed that Y is qualitative. However, we may sometimes be faced with situations in which Y is neither qualitative nor quantitative, and so neither linear regression from Chapter 3 nor the classification approaches covered in this chapter is applicable. As a concrete example, we consider the Bikeshare data set. The response

is bikers, the number of hourly users of a bike sharing program in Wash- ington, DC. This response value is neither qualitative nor quantitative: instead, it takes on non-negative integer values, or counts. We will consider

counts predicting bikers using the covariates mnth (month of the year), hr (hour of the day, from 0 to 23), workingday (an indicator variable that equals 1 if it is neither a weekend nor a holiday), temp (the normalized temperature, in Celsius), and weathersit (a qualitative variable that takes on one of four possible values: clear; misty or cloudy; light rain or light snow; or heavy rain or heavy snow.) In the analyses that follow, we will treat mnth, hr, and weathersit as

qualitative variables.

4.6.1 Linear Regression on the Bikeshare Data

To begin, we consider predicting bikers using linear regression. The results are shown in Table 4.10. We see, for example, that a progression of weather from clear to cloudy

results in, on average, 12.89 fewer bikers per hour; however, if the weather progresses further to rain or snow, then this further results in 53.60 fewer bikers per hour. Figure 4.13 displays the coefficients associated with mnth and the coefficients associated with hr. We see that bike usage is highest in the spring and fall, and lowest during the winter months. Furthermore, bike usage is greatest around rush hour (9 AM and 6 PM), and lowest overnight. Thus, at first glance, fitting a linear regression model to the Bikeshare data set seems to provide reasonable and intuitive results.

4.6 Generalized Linear Models 165

Coefficient Std. error z-statistic p-value Intercept 73.60 5.13 14.34 0.00 workingday 1.27 1.78 0.71 0.48 temp 157.21 10.26 15.32 0.00 weathersit[cloudy/misty] -12.89 1.96 -6.56 0.00 weathersit[light rain/snow] -66.49 2.97 -22.43 0.00 weathersit[heavy rain/snow] -109.75 76.67 -1.43 0.15

TABLE 4.10. Results for a least squares linear model fit to predict bikers in the Bikeshare data. The predictors mnth and hr are omitted from this table due to space constraints, and can be seen in Figure 4.13. For the qualitative variable weathersit, the baseline level corresponds to clear skies.

●

− 4

0 −

2 0

0 2

Month

C o

e ff

ic ie

n t

J F M A M J J A S O N D

● ●

● ● ●

●

● ● ● ●

●

5 10 15 20

− 1

0 0

0 5

0 1

0 0

2 0

Hour

C o

e ff

ic ie

n t

FIGURE 4.13. A least squares linear regression model was fit to predict bikers in the Bikeshare data set. Left: The coefficients associated with the month of the year. Bike usage is highest in the spring and fall, and lowest in the winter. Right: The coefficients associated with the hour of the day. Bike usage is highest during peak commute times, and lowest overnight.

But upon more careful inspection, some issues become apparent. For example, 9.6% of the fitted values in the Bikeshare data set are negative: that is, the linear regression model predicts a negative number of users during 9.6% of the hours in the data set. This calls into question our ability to perform meaningful predictions on the data, and it also raises concerns about the accuracy of the coefficient estimates, confidence intervals, and other outputs of the regression model. Furthermore, it is reasonable to suspect that when the expected value

of bikers is small, the variance of bikers should be small as well. For instance, at 2 AM during a heavy December snow storm, we expect that extremely few people will use a bike, and moreover that there should be little variance associated with the number of users during those conditions. This is borne out in the data: between 1 AM and 4 AM, in December, January, and February, when it is raining, there are 5.05 users, on average,

166 4. Classification

5 10 15 20

0 1

0 0

2 0

0 3

0 0

4 0

0 5

0 0

6 0

Hour

N u

m b

e r

o f

B ik

e rs

5 10 15 20

0 1

2 3

4 5

Hour

L o

g (N

u m

b e

r o

f B

ik e

rs )

FIGURE 4.14. Left: On the Bikeshare dataset, the number of bikers is dis- played on the y-axis, and the hour of the day is displayed on the x-axis. Jitter was applied for ease of visualization. For the most part, as the mean number of bikers increases, so does the variance in the number of bikers. A smoothing spline fit is shown in green. Right: The log of the number of bikers is now displayed on the y-axis.

with a standard deviation of 3.73. By contrast, between 7 AM and 10 AM, in April, May, and June, when skies are clear, there are 243.59 users, on average, with a standard deviation of 131.7. The mean-variance relationship is displayed in the left-hand panel of Figure 4.14. This is a major violation of the assumptions of a linear model, which state that Y =

∑p j=1 Xjβj +ϵ,

where ϵ is a mean-zero error term with variance σ2 that is constant, and not a function of the covariates. Therefore, the heteroscedasticity of the data calls into question the suitability of a linear regression model. Finally, the response bikers is integer-valued. But under a linear model,

Y = β0 + ∑p

j=1 Xjβj + ϵ, where ϵ is a continuous-valued error term. This means that in a linear model, the response Y is necessarily continuous- valued (quantitative). Thus, the integer nature of the response bikers sug- gests that a linear regression model is not entirely satisfactory for this data set. Some of the problems that arise when fitting a linear regression model

to the Bikeshare data can be overcome by transforming the response; for instance, we can fit the model

log(Y ) = p∑

j=1

Xjβj + ϵ.

Transforming the response avoids the possibility of negative predictions, and it overcomes much of the heteroscedasticity in the untransformed data, as is shown in the right-hand panel of Figure 4.14. However, it is not quite a satisfactory solution, since predictions and inference are made in terms of the log of the response, rather than the response. This leads to challenges in interpretation, e.g. “a one-unit increase in Xj is associated with an increase in the mean of the log of Y by an amount βj”. Furthermore, a

4.6 Generalized Linear Models 167

log transformation of the response cannot be applied in settings where the response can take on a value of 0. Thus, while fitting a linear model to a transformation of the response may be an adequate approach for some count-valued data sets, it often leaves something to be desired. We will see in the next section that a Poisson regression model provides a much more natural and elegant approach for this task.

4.6.2 Poisson Regression on the Bikeshare Data

To overcome the inadequacies of linear regression for analyzing the Bikeshare data set, we will make use of an alternative approach, called Poisson regression. Before we can talk about Poisson regression, we must first in-

Poisson regressiontroduce the Poisson distribution.

Poisson distribution

Suppose that a random variable Y takes on nonnegative integer values, i.e. Y ∈ {0, 1, 2, . . .}. If Y follows the Poisson distribution, then

Pr(Y = k) = e−λλk

k! for k = 0, 1, 2, . . . . (4.35)

Here, λ > 0 is the expected value of Y , i.e. E(Y ). It turns out that λ also equals the variance of Y , i.e. λ = E(Y ) = Var(Y ). This means that if Y follows the Poisson distribution, then the larger the mean of Y , the larger its variance. (In (4.35), the notation k!, pronounced “k factorial”, is defined as k! = k × (k −1)× (k −2)× . . .×3×2×1.) The Poisson distribution is typically used to model counts; this is a

natural choice for a number of reasons, including the fact that counts, like the Poisson distribution, take on nonnegative integer values. To see how we might use the Poisson distribution in practice, let Y denote the number of users of the bike sharing program during a particular hour of the day, under a particular set of weather conditions, and during a particular month of the year. We might model Y as a Poisson distribution with mean E(Y ) = λ = 5. This means that the probability of no users during this particular hour is Pr(Y = 0) = e

−550

0! = e−5 = 0.0067 (where 0! = 1 by convention). The

probability that there is exactly one user is Pr(Y = 1) = e −551

1! = 5e−5 =

0.034, the probability of two users is Pr(Y = 2) = e −552

2! = 0.084, and so

on. Of course, in reality, we expect the mean number of users of the bike

sharing program, λ = E(Y ), to vary as a function of the hour of the day, the month of the year, the weather conditions, and so forth. So rather than modeling the number of bikers, Y , as a Poisson distribution with a fixed mean value like λ = 5, we would like to allow the mean to vary as a function of the covariates. In particular, we consider the following model for the mean λ = E(Y ), which we now write as λ(X1, . . . , Xp) to emphasize that it is a function of the covariates X1, . . . , Xp:

log(λ(X1, . . . , Xp)) = β0 + β1X1 + · · · + βpXp (4.36)

168 4. Classification

Coefficient Std. error z-statistic p-value Intercept 4.12 0.01 683.96 0.00 workingday 0.01 0.00 7.5 0.00 temp 0.79 0.01 68.43 0.00 weathersit[cloudy/misty] -0.08 0.00 -34.53 0.00 weathersit[light rain/snow] -0.58 0.00 -141.91 0.00 weathersit[heavy rain/snow] -0.93 0.17 -5.55 0.00

TABLE 4.11. Results for a Poisson regression model fit to predict bikers in the Bikeshare data. The predictors mnth and hr are omitted from this table due to space constraints, and can be seen in Figure 4.15. For the qualitative variable weathersit, the baseline corresponds to clear skies.

or equivalently

λ(X1, . . . , Xp) = e β0+β1X1+···+βpXp. (4.37)

Here, β0, β1, . . . , βp are parameters to be estimated. Together, (4.35) and (4.36) define the Poisson regression model. Notice that in (4.36), we take the log of λ(X1, . . . , Xp) to be linear in X1, . . . , Xp, rather than hav- ing λ(X1, . . . , Xp) itself be linear in X1, . . . , Xp, in order to ensure that λ(X1, . . . , Xp) takes on nonnegative values for all values of the covariates. To estimate the coefficients β0, β1, . . . , βp, we use the same maximum

likelihood approach that we adopted for logistic regression in Section 4.3.2. Specifically, given n independent observations from the Poisson regression model, the likelihood takes the form

ℓ(β0, β1, . . . , βp) = n∏

i=1

e−λ(xi)λ(xi) yi

yi! , (4.38)

where λ(xi) = e β0+β1xi1+···+βpxip, due to (4.37). We estimate the coef-

ficients that maximize the likelihood ℓ(β0, β1, . . . , βp), i.e. that make the observed data as likely as possible. We now fit a Poisson regression model to the Bikeshare data set. The

results are shown in Table 4.11 and Figure 4.15. Qualitatively, the results are similar to those from linear regression in Section 4.6.1. We again see that bike usage is highest in the spring and fall and during rush hour, and lowest during the winter and in the early morning hours. Moreover, bike usage increases as the temperature increases, and decreases as the weather worsens. Interestingly, the coefficient associated with workingday is statistically significant under the Poisson regression model, but not under the linear regression model. Some important distinctions between the Poisson regression model and

the linear regression model are as follows:

• Interpretation: To interpret the coefficients in the Poisson regression model, we must pay close attention to (4.37), which states that an increase in Xj by one unit is associated with a change in E(Y ) = λ by a factor of exp(βj). For example, a change in weather from clear

4.6 Generalized Linear Models 169

●

● ●

●

● ●

●

− 0

.6 −

0 .4

− 0

.2 0

.0 0

Month

C o

e ff

ic ie

n t

J F M A M J J A S O N D

●

● ●

● ● ● ●

●

● ●

●

5 10 15 20

− 2

− 1

0 1

Hour

C o

e ff

ic ie

n t

FIGURE 4.15. A Poisson regression model was fit to predict bikers in the Bikeshare data set. Left: The coefficients associated with the month of the year. Bike usage is highest in the spring and fall, and lowest in the winter. Right: The coefficients associated with the hour of the day. Bike usage is highest during peak commute times, and lowest overnight.

to cloudy skies is associated with a change in mean bike usage by a factor of exp(−0.08) = 0.923, i.e. on average, only 92.3% as many people will use bikes when it is cloudy relative to when it is clear. If the weather worsens further and it begins to rain, then the mean bike usage will further change by a factor of exp(−0.5) = 0.607, i.e. on average only 60.7% as many people will use bikes when it is rainy relative to when it is cloudy.

• Mean-variance relationship: As mentioned earlier, under the Poisson model, λ = E(Y ) = Var(Y ). Thus, by modeling bike usage with a Poisson regression, we implicitly assume that mean bike usage in a given hour equals the variance of bike usage during that hour. By contrast, under a linear regression model, the variance of bike usage always takes on a constant value. Recall from Figure 4.14 that in the Bikeshare data, when biking conditions are favorable, both the mean and the variance in bike usage are much higher than when conditions are unfavorable. Thus, the Poisson regression model is able to handle the mean-variance relationship seen in the Bikeshare data in a way that the linear regression model is not.5

overdispersion

• nonnegative fitted values: There are no negative predictions using the Poisson regression model. This is because the Poisson model itself only allows for nonnegative values; see (4.35). By contrast, when we

5In fact, the variance in the Bikeshare data appears to be much higher than the mean, a situation referred to as overdispersion. This causes the Z-values to be inflated in Table 4.11. A more careful analysis should account for this overdispersion to obtain more accurate Z-values, and there are a variety of methods for doing this. But they are beyond the scope of this book.

170 4. Classification

fit a linear regression model to the Bikeshare data set, almost 10% of the predictions were negative.

4.6.3 Generalized Linear Models in Greater Generality

We have now discussed three types of regression models: linear, logistic and Poisson. These approaches share some common characteristics:

1. Each approach uses predictors X1, . . . , Xp to predict a response Y . We assume that, conditional on X1, . . . , Xp, Y belongs to a certain family of distributions. For linear regression, we typically assume that Y follows a Gaussian or normal distribution. For logistic regression, we assume that Y follows a Bernoulli distribution. Finally, for Poisson regression, we assume that Y follows a Poisson distribution.

2. Each approach models the mean of Y as a function of the predictors. In linear regression, the mean of Y takes the form

E(Y |X1, . . . , Xp) = β0 + β1X1 + · · · + βpXp, (4.39) i.e. it is a linear function of the predictors. For logistic regression, the mean instead takes the form

E(Y |X1, . . . , Xp) = Pr(Y = 1|X1, . . . , Xp)

= eβ0+β1X1+···+βpXp

1 + eβ0+β1X1+···+βpXp , (4.40)

while for Poisson regression it takes the form

E(Y |X1, . . . , Xp) = λ(X1, . . . , Xp) = eβ0+β1X1+···+βpXp. (4.41)

Equations (4.39)–(4.41) can be expressed using a link function, η, which link function

applies a transformation to E(Y |X1, . . . , Xp) so that the transformed mean is a linear function of the predictors. That is,

η(E(Y |X1, . . . , Xp)) = β0 + β1X1 + · · · + βpXp. (4.42) The link functions for linear, logistic and Poisson regression are η(µ) = µ, η(µ) = log(µ/(1−µ)), and η(µ) = log(µ), respectively. The Gaussian, Bernoulli and Poisson distributions are all members of a

wider class of distributions, known as the exponential family. Other well- exponential familyknown members of this family are the exponential distribution, the Gamma

exponential distribution

distribution, and the negative binomial distribution. In general, we can per-

Gamma distribution negative binomial distribution

form a regression by modeling the response Y as coming from a particular member of the exponential family, and then transforming the mean of the response so that the transformed mean is a linear function of the predictors via (4.42). Any regression approach that follows this very general recipe is known as a generalized linear model (GLM). Thus, linear regression, logistic

generalized linear model

regression, and Poisson regression are three examples of GLMs. Other ex- amples not covered here include Gamma regression and negative binomial

Gamma regression

regression.

negative binomial regression

4.7 Lab: Classification Methods 171

4.7 Lab: Classification Methods

4.7.1 The Stock Market Data

We will begin by examining some numerical and graphical summaries of the Smarket data, which is part of the ISLR2 library. This data set consists of percentage returns for the S&P 500 stock index over 1, 250 days, from the beginning of 2001 until the end of 2005. For each date, we have recorded the percentage returns for each of the five previous trading days, Lag1 through Lag5. We have also recorded Volume (the number of shares traded on the previous day, in billions), Today (the percentage return on the date in question) and Direction (whether the market was Up or Down on this date). Our goal is to predict Direction (a qualitative response) using the other features.

> library(ISLR2)

> names(Smarket)

[1] "Year" "Lag1" "Lag2" "Lag3" "Lag4"

[6] "Lag5" "Volume" "Today" "Direction"

> dim(Smarket)

[1] 1250 9

> summary(Smarket)

Year Lag1 Lag2

Min. :2001 Min. : -4.92200 Min. : -4.92200

1st Qu .:2002 1st Qu .: -0.63950 1st Qu .: -0.63950

Median :2003 Median : 0.03900 Median : 0.03900

Mean :2003 Mean : 0.00383 Mean : 0.00392

3rd Qu .:2004 3rd Qu.: 0.59675 3rd Qu.: 0.59675

Max. :2005 Max. : 5.73300 Max. : 5.73300

Lag3 Lag4 Lag5

Min. : -4.92200 Min. : -4.92200 Min. : -4.92200

1st Qu .: -0.64000 1st Qu .: -0.64000 1st Qu .: -0.64000

Median : 0.03850 Median : 0.03850 Median : 0.03850

Mean : 0.00172 Mean : 0.00164 Mean : 0.00561

3rd Qu.: 0.59675 3rd Qu.: 0.59675 3rd Qu.: 0.59700

Max. : 5.73300 Max. : 5.73300 Max. : 5.73300

Volume Today Direction

Min. :0.356 Min. : -4.92200 Down :602

1st Qu .:1.257 1st Qu .: -0.63950 Up :648

Median :1.423 Median : 0.03850

Mean :1.478 Mean : 0.00314

3rd Qu .:1.642 3rd Qu.: 0.59675

Max. :3.152 Max. : 5.73300

> pairs(Smarket)

The cor() function produces a matrix that contains all of the pairwise correlations among the predictors in a data set. The first command below gives an error message because the Direction variable is qualitative.

> cor(Smarket)

Error in cor(Smarket) : ‘x’ must be numeric

> cor(Smarket[, -9])

172 4. Classification

Year Lag1 Lag2 Lag3 Lag4 Lag5

Year 1.0000 0.02970 0.03060 0.03319 0.03569 0.02979

Lag1 0.0297 1.00000 -0.02629 -0.01080 -0.00299 -0.00567

Lag2 0.0306 -0.02629 1.00000 -0.02590 -0.01085 -0.00356

Lag3 0.0332 -0.01080 -0.02590 1.00000 -0.02405 -0.01881

Lag4 0.0357 -0.00299 -0.01085 -0.02405 1.00000 -0.02708

Lag5 0.0298 -0.00567 -0.00356 -0.01881 -0.02708 1.00000

Volume 0.5390 0.04091 -0.04338 -0.04182 -0.04841 -0.02200

Today 0.0301 -0.02616 -0.01025 -0.00245 -0.00690 -0.03486

Volume Today

Year 0.5390 0.03010

Lag1 0.0409 -0.02616

Lag2 -0.0434 -0.01025

Lag3 -0.0418 -0.00245

Lag4 -0.0484 -0.00690

Lag5 -0.0220 -0.03486

Volume 1.0000 0.01459

Today 0.0146 1.00000

As one would expect, the correlations between the lag variables and to- day’s returns are close to zero. In other words, there appears to be little correlation between today’s returns and previous days’ returns. The only substantial correlation is between Year and Volume. By plotting the data, which is ordered chronologically, we see that Volume is increasing over time. In other words, the average number of shares traded daily increased from 2001 to 2005.

> attach(Smarket)

> plot(Volume)

4.7.2 Logistic Regression

Next, we will fit a logistic regression model in order to predict Direction using Lag1 through Lag5 and Volume. The glm() function can be used to fit

glm() many types of generalized linear models, including logistic regression. The

generalized linear model

syntax of the glm() function is similar to that of lm(), except that we must pass in the argument family = binomial in order to tell R to run a logistic regression rather than some other type of generalized linear model.

> glm.fits <- glm(

Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , data = Smarket , family = binomial

)

> summary(glm.fits)

Call:

glm(formula = Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , family = binomial , data = Smarket)

Deviance Residuals:

Min 1Q Median 3Q Max

4.7 Lab: Classification Methods 173

-1.45 -1.20 1.07 1.15 1.33

Coefficients :

Estimate Std. Error z value Pr(>|z|)

(Intercept) -0.12600 0.24074 -0.52 0.60

Lag1 -0.07307 0.05017 -1.46 0.15

Lag2 -0.04230 0.05009 -0.84 0.40

Lag3 0.01109 0.04994 0.22 0.82

Lag4 0.00936 0.04997 0.19 0.85

Lag5 0.01031 0.04951 0.21 0.83

Volume 0.13544 0.15836 0.86 0.39

( Dispersion parameter for binomial family taken to be 1)

Null deviance: 1731.2 on 1249 degrees of freedom

Residual deviance: 1727.6 on 1243 degrees of freedom

AIC: 1742

Number of Fisher Scoring iterations : 3

The smallest p-value here is associated with Lag1. The negative coefficient for this predictor suggests that if the market had a positive return yesterday, then it is less likely to go up today. However, at a value of 0.15, the p-value is still relatively large, and so there is no clear evidence of a real association between Lag1 and Direction. We use the coef() function in order to access just the coefficients for this

fitted model. We can also use the summary() function to access particular aspects of the fitted model, such as the p-values for the coefficients.

> coef(glm.fits)

(Intercept) Lag1 Lag2 Lag3 Lag4

-0.12600 -0.07307 -0.04230 0.01109 0.00936

Lag5 Volume

0.01031 0.13544

> summary(glm.fits)$coef

Estimate Std. Error z value Pr(>|z|)

(Intercept) -0.12600 0.2407 -0.523 0.601

Lag1 -0.07307 0.0502 -1.457 0.145

Lag2 -0.04230 0.0501 -0.845 0.398

Lag3 0.01109 0.0499 0.222 0.824

Lag4 0.00936 0.0500 0.187 0.851

Lag5 0.01031 0.0495 0.208 0.835

Volume 0.13544 0.1584 0.855 0.392

> summary(glm.fits)$coef[, 4]

(Intercept) Lag1 Lag2 Lag3 Lag4

0.601 0.145 0.398 0.824 0.851

Lag5 Volume

0.835 0.392

The predict() function can be used to predict the probability that the market will go up, given values of the predictors. The type = "response" option tells R to output probabilities of the form P(Y = 1|X), as opposed to other information such as the logit. If no data set is supplied to the

174 4. Classification

predict() function, then the probabilities are computed for the training data that was used to fit the logistic regression model. Here we have printed only the first ten probabilities. We know that these values correspond to the probability of the market going up, rather than down, because the contrasts() function indicates that R has created a dummy variable with a 1 for Up.

> glm.probs <- predict(glm.fits , type = "response ")

> glm.probs [1:10]

1 2 3 4 5 6 7 8 9 10

0.507 0.481 0.481 0.515 0.511 0.507 0.493 0.509 0.518 0.489

> contrasts (Direction)

Down 0

Up 1

In order to make a prediction as to whether the market will go up or down on a particular day, we must convert these predicted probabilities into class labels, Up or Down. The following two commands create a vector of class predictions based on whether the predicted probability of a market increase is greater than or less than 0.5.

> glm.pred <- rep("Down", 1250)

> glm.pred[glm.probs > .5] = "Up"

The first command creates a vector of 1,250 Down elements. The second line transforms to Up all of the elements for which the predicted probability of a market increase exceeds 0.5. Given these predictions, the table() function

table() can be used to produce a confusion matrix in order to determine how many observations were correctly or incorrectly classified.

> table(glm.pred , Direction)

Direction

glm.pred Down Up

Down 145 141

Up 457 507

> (507 + 145) / 1250

[1] 0.5216

> mean(glm.pred == Direction)

[1] 0.5216

The diagonal elements of the confusion matrix indicate correct predictions, while the off-diagonals represent incorrect predictions. Hence our model correctly predicted that the market would go up on 507 days and that it would go down on 145 days, for a total of 507 + 145 = 652 correct predictions. The mean() function can be used to compute the fraction of days for which the prediction was correct. In this case, logistic regression correctly predicted the movement of the market 52.2 % of the time. At first glance, it appears that the logistic regression model is working

a little better than random guessing. However, this result is misleading because we trained and tested the model on the same set of 1, 250 ob- servations. In other words, 100% − 52.2% = 47.8%, is the training error

4.7 Lab: Classification Methods 175

rate. As we have seen previously, the training error rate is often overly optimistic—it tends to underestimate the test error rate. In order to better assess the accuracy of the logistic regression model in this setting, we can fit the model using part of the data, and then examine how well it predicts the held out data. This will yield a more realistic error rate, in the sense that in practice we will be interested in our model’s performance not on the data that we used to fit the model, but rather on days in the future for which the market’s movements are unknown. To implement this strategy, we will first create a vector corresponding

to the observations from 2001 through 2004. We will then use this vector to create a held out data set of observations from 2005.

> train <- (Year < 2005)

> Smarket .2005 <- Smarket [!train , ]

> dim(Smarket .2005)

[1] 252 9

> Direction .2005 <- Direction [! train]

The object train is a vector of 1,250 elements, corresponding to the ob- servations in our data set. The elements of the vector that correspond to observations that occurred before 2005 are set to TRUE, whereas those that correspond to observations in 2005 are set to FALSE. The object train is a Boolean vector, since its elements are TRUE and FALSE. Boolean vectors can

boolean be used to obtain a subset of the rows or columns of a matrix. For instance, the command Smarket[train, ] would pick out a submatrix of the stock market data set, corresponding only to the dates before 2005, since those are the ones for which the elements of train are TRUE. The ! symbol can be used to reverse all of the elements of a Boolean vector. That is, !train is a vector similar to train, except that the elements that are TRUE in train get swapped to FALSE in !train, and the elements that are FALSE in train get swapped to TRUE in !train. Therefore, Smarket[!train, ] yields a sub- matrix of the stock market data containing only the observations for which train is FALSE—that is, the observations with dates in 2005. The output above indicates that there are 252 such observations. We now fit a logistic regression model using only the subset of the obser-

vations that correspond to dates before 2005, using the subset argument. We then obtain predicted probabilities of the stock market going up for each of the days in our test set—that is, for the days in 2005.

> glm.fits <- glm(

Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , data = Smarket , family = binomial , subset = train

)

> glm.probs <- predict(glm.fits , Smarket .2005 ,

type = "response")

Notice that we have trained and tested our model on two completely sep- arate data sets: training was performed using only the dates before 2005,

176 4. Classification

and testing was performed using only the dates in 2005. Finally, we com- pute the predictions for 2005 and compare them to the actual movements of the market over that time period.

> glm.pred <- rep("Down", 252)

> glm.pred[glm.probs > .5] <- "Up"

> table(glm.pred , Direction .2005)

Direction .2005

glm.pred Down Up

Down 77 97

Up 34 44

> mean(glm.pred == Direction .2005)

[1] 0.48

> mean(glm.pred != Direction .2005)

[1] 0.52

The != notation means not equal to, and so the last command computes the test set error rate. The results are rather disappointing: the test error rate is 52 %, which is worse than random guessing! Of course this result is not all that surprising, given that one would not generally expect to be able to use previous days’ returns to predict future market performance. (After all, if it were possible to do so, then the authors of this book would be out striking it rich rather than writing a statistics textbook.) We recall that the logistic regression model had very underwhelming p-

values associated with all of the predictors, and that the smallest p-value, though not very small, corresponded to Lag1. Perhaps by removing the variables that appear not to be helpful in predicting Direction, we can obtain a more effective model. After all, using predictors that have no relationship with the response tends to cause a deterioration in the test error rate (since such predictors cause an increase in variance without a corresponding decrease in bias), and so removing such predictors may in turn yield an improvement. Below we have refit the logistic regression using just Lag1 and Lag2, which seemed to have the highest predictive power in the original logistic regression model.

> glm.fits <- glm(Direction ∼ Lag1 + Lag2 , data = Smarket , family = binomial , subset = train)

> glm.probs <- predict(glm.fits , Smarket .2005 ,

type = "response")

> glm.pred <- rep("Down", 252)

> glm.pred[glm.probs > .5] <- "Up"

> table(glm.pred , Direction .2005)

Direction .2005

glm.pred Down Up

Down 35 35

Up 76 106

> mean(glm.pred == Direction .2005)

[1] 0.56

> 106 / (106 + 76)

[1] 0.582

4.7 Lab: Classification Methods 177

Now the results appear to be a little better: 56% of the daily movements have been correctly predicted. It is worth noting that in this case, a much simpler strategy of predicting that the market will increase every day will also be correct 56% of the time! Hence, in terms of overall error rate, the logistic regression method is no better than the naive approach. However, the confusion matrix shows that on days when logistic regression predicts an increase in the market, it has a 58% accuracy rate. This suggests a possible trading strategy of buying on days when the model predicts an in- creasing market, and avoiding trades on days when a decrease is predicted. Of course one would need to investigate more carefully whether this small improvement was real or just due to random chance. Suppose that we want to predict the returns associated with particular

values of Lag1 and Lag2. In particular, we want to predict Direction on a day when Lag1 and Lag2 equal 1.2 and 1.1, respectively, and on a day when they equal 1.5 and −0.8. We do this using the predict() function.

> predict(glm.fits ,

newdata =

data.frame(Lag1 = c(1.2 , 1.5) , Lag2 = c(1.1 , -0.8)),

type = "response"

)

1 2

0.4791 0.4961

4.7.3 Linear Discriminant Analysis

Now we will perform LDA on the Smarket data. In R, we fit an LDA model using the lda() function, which is part of the MASS library. Notice that the

lda() syntax for the lda() function is identical to that of lm(), and to that of glm() except for the absence of the family option. We fit the model using only the observations before 2005.

> library(MASS)

> lda.fit <- lda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)

> lda.fit

Call:

lda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)

Prior probabilities of groups:

Down Up

0.492 0.508

Group means:

Lag1 Lag2

Down 0.0428 0.0339

Up -0.0395 -0.0313

178 4. Classification

Coefficients of linear discriminants :

LD1

Lag1 -0.642

Lag2 -0.514

> plot(lda.fit)

The LDA output indicates that π̂1 = 0.492 and π̂2 = 0.508; in other words, 49.2 % of the training observations correspond to days during which the market went down. It also provides the group means; these are the average of each predictor within each class, and are used by LDA as estimates of µk. These suggest that there is a tendency for the previous 2 days’ returns to be negative on days when the market increases, and a tendency for the previous days’ returns to be positive on days when the market declines. The coefficients of linear discriminants output provides the linear combination of Lag1 and Lag2 that are used to form the LDA decision rule. In other words, these are the multipliers of the elements of X = x in (4.24). If −0.642×Lag1−0.514×Lag2 is large, then the LDA classifier will predict a market increase, and if it is small, then the LDA classifier will predict a market decline. The plot() function produces plots of the linear discriminants, obtained

by computing −0.642×Lag1−0.514×Lag2 for each of the training obser- vations. The Up and Down observations are displayed separately. The predict() function returns a list with three elements. The first ele-

ment, class, contains LDA’s predictions about the movement of the market. The second element, posterior, is a matrix whose kth column contains the posterior probability that the corresponding observation belongs to the kth class, computed from (4.15). Finally, x contains the linear discriminants, described earlier.

> lda.pred <- predict(lda.fit , Smarket .2005)

> names(lda.pred)

[1] "class" "posterior" "x"

As we observed in Section 4.5, the LDA and logistic regression predictions are almost identical.

> lda.class <- lda.pred$class

> table(lda.class , Direction .2005)

Direction .2005

lda.pred Down Up

Down 35 35

Up 76 106

> mean(lda.class == Direction .2005)

[1] 0.56

Applying a 50 % threshold to the posterior probabilities allows us to recre- ate the predictions contained in lda.pred$class.

> sum(lda.pred$posterior[, 1] >= .5)

[1] 70

4.7 Lab: Classification Methods 179

> sum(lda.pred$posterior[, 1] < .5)

[1] 182

Notice that the posterior probability output by the model corresponds to the probability that the market will decrease:

> lda.pred$posterior [1:20 , 1]

> lda.class [1:20]

If we wanted to use a posterior probability threshold other than 50 % in order to make predictions, then we could easily do so. For instance, suppose that we wish to predict a market decrease only if we are very certain that the market will indeed decrease on that day—say, if the posterior probability is at least 90 %.

> sum(lda.pred$posterior[, 1] > .9)

[1] 0

No days in 2005 meet that threshold! In fact, the greatest posterior prob- ability of decrease in all of 2005 was 52.02 %.

4.7.4 Quadratic Discriminant Analysis

We will now fit a QDA model to the Smarket data. QDA is implemented in R using the qda() function, which is also part of the MASS library. The

qda() syntax is identical to that of lda().

> qda.fit <- qda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)

> qda.fit

Call:

qda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)

Prior probabilities of groups:

Down Up

0.492 0.508

Group means:

Lag1 Lag2

Down 0.0428 0.0339

Up -0.0395 -0.0313

The output contains the group means. But it does not contain the coef- ficients of the linear discriminants, because the QDA classifier involves a quadratic, rather than a linear, function of the predictors. The predict() function works in exactly the same fashion as for LDA.

> qda.class <- predict(qda.fit , Smarket .2005)$class

> table(qda.class , Direction .2005)

Direction .2005

qda.class Down Up

Down 30 20

180 4. Classification

Up 81 121

> mean(qda.class == Direction .2005)

[1] 0.599

Interestingly, the QDA predictions are accurate almost 60 % of the time, even though the 2005 data was not used to fit the model. This level of accu- racy is quite impressive for stock market data, which is known to be quite hard to model accurately. This suggests that the quadratic form assumed by QDA may capture the true relationship more accurately than the linear forms assumed by LDA and logistic regression. However, we recommend evaluating this method’s performance on a larger test set before betting that this approach will consistently beat the market!

4.7.5 Naive Bayes

Next, we fit a naive Bayes model to the Smarket data. Naive Bayes is im- plemented in R using the naiveBayes() function, which is part of the e1071

naiveBayes() library. The syntax is identical to that of lda() and qda(). By default, this implementation of the naive Bayes classifier models each quantitative fea- ture using a Gaussian distribution. However, a kernel density method can also be used to estimate the distributions.

> library(e1071)

> nb.fit <- naiveBayes (Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)

> nb.fit

Naive Bayes Classifier for Discrete Predictors

Call:

naiveBayes.default(x = X, y = Y, laplace = laplace)

A-priori probabilities :

Down Up

0.492 0.508

Conditional probabilities :

Lag1

Y [,1] [,2]

Down 0.0428 1.23

Up -0.0395 1.23

Lag2

Y [,1] [,2]

Down 0.0339 1.24

Up -0.0313 1.22

The output contains the estimated mean and standard deviation for each variable in each class. For example, the mean for Lag1 is 0.0428 for Direction=Down, and the standard deviation is 1.23. We can easily verify this:

4.7 Lab: Classification Methods 181

> mean(Lag1[train ][ Direction[train] == "Down"])

[1] 0.0428

> sd(Lag1[train ][ Direction[train] == "Down"])

[1] 1.23

The predict() function is straightforward.

> nb.class <- predict(nb.fit , Smarket .2005)

> table(nb.class , Direction .2005)

Direction .2005

nb.class Down Up

Down 28 20

Up 83 121

> mean(nb.class == Direction .2005)

[1] 0.591

Naive Bayes performs very well on this data, with accurate predictions over 59% of the time. This is slightly worse than QDA, but much better than LDA. The predict() function can also generate estimates of the probability

that each observation belongs to a particular class.

> nb.preds <- predict(nb.fit , Smarket .2005 , type = "raw")

> nb.preds [1:5 , ]

Down Up

[1,] 0.487 0.513

[2,] 0.476 0.524

[3,] 0.465 0.535

[4,] 0.475 0.525

[5,] 0.490 0.510

4.7.6 K-Nearest Neighbors

We will now perform KNN using the knn() function, which is part of the knn()

class library. This function works rather differently from the other model- fitting functions that we have encountered thus far. Rather than a two-step approach in which we first fit the model and then we use the model to make predictions, knn() forms predictions using a single command. The function requires four inputs.

1. A matrix containing the predictors associated with the training data, labeled train.X below.

2. A matrix containing the predictors associated with the data for which we wish to make predictions, labeled test.X below.

3. A vector containing the class labels for the training observations, labeled train.Direction below.

4. A value for K, the number of nearest neighbors to be used by the classifier.

182 4. Classification

We use the cbind() function, short for column bind, to bind the Lag1 and cbind()

Lag2 variables together into two matrices, one for the training set and the other for the test set.

> library(class)

> train.X <- cbind(Lag1 , Lag2)[train , ]

> test.X <- cbind(Lag1 , Lag2)[!train , ]

> train.Direction <- Direction[train]

Now the knn() function can be used to predict the market’s movement for the dates in 2005. We set a random seed before we apply knn() because if several observations are tied as nearest neighbors, then R will randomly break the tie. Therefore, a seed must be set in order to ensure reproducibil- ity of results.

> set.seed (1)

> knn.pred <- knn(train.X, test.X, train.Direction , k = 1)

> table(knn.pred , Direction .2005)

Direction .2005

knn.pred Down Up

Down 43 58

Up 68 83

> (83 + 43) / 252

[1] 0.5

The results using K = 1 are not very good, since only 50 % of the observa- tions are correctly predicted. Of course, it may be that K = 1 results in an overly flexible fit to the data. Below, we repeat the analysis using K = 3.

> knn.pred <- knn(train.X, test.X, train.Direction , k = 3)

> table(knn.pred , Direction .2005)

Direction .2005

knn.pred Down Up

Down 48 54

Up 63 87

> mean(knn.pred == Direction .2005)

[1] 0.536

The results have improved slightly. But increasing K further turns out to provide no further improvements. It appears that for this data, QDA provides the best results of the methods that we have examined so far. KNN does not perform well on the Smarket data but it does often provide

impressive results. As an example we will apply the KNN approach to the Caravan data set, which is part of the ISLR2 library. This data set includes 85 predictors that measure demographic characteristics for 5,822 individuals. The response variable is Purchase, which indicates whether or not a given individual purchases a caravan insurance policy. In this data set, only 6 % of people purchased caravan insurance.

> dim(Caravan)

[1] 5822 86

4.7 Lab: Classification Methods 183

> attach(Caravan)

> summary(Purchase)

No Yes

5474 348

> 348 / 5822

[1] 0.0598

Because the KNN classifier predicts the class of a given test observation by identifying the observations that are nearest to it, the scale of the variables matters. Variables that are on a large scale will have a much larger effect on the distance between the observations, and hence on the KNN classifier, than variables that are on a small scale. For instance, imagine a data set that contains two variables, salary and age (measured in dollars and years, respectively). As far as KNN is concerned, a difference of $1,000 in salary is enormous compared to a difference of 50 years in age. Consequently, salary will drive the KNN classification results, and age will have almost no effect. This is contrary to our intuition that a salary difference of $1,000 is quite small compared to an age difference of 50 years. Furthermore, the importance of scale to the KNN classifier leads to another issue: if we measured salary in Japanese yen, or if we measured age in minutes, then we’d get quite different classification results from what we get if these two variables are measured in dollars and years. A good way to handle this problem is to standardize the data so that all

standardize variables are given a mean of zero and a standard deviation of one. Then all variables will be on a comparable scale. The scale() function does just

scale() this. In standardizing the data, we exclude column 86, because that is the qualitative Purchase variable.

> standardized .X <- scale(Caravan[, -86])

> var(Caravan[, 1])

[1] 165

> var(Caravan[, 2])

[1] 0.165

> var( standardized .X[, 1])

[1] 1

> var( standardized .X[, 2])

[1] 1

Now every column of standardized.X has a standard deviation of one and a mean of zero. We now split the observations into a test set, containing the first 1,000

observations, and a training set, containing the remaining observations. We fit a KNN model on the training data using K = 1, and evaluate its performance on the test data.

> test <- 1:1000

> train.X <- standardized .X[-test , ]

> test.X <- standardized .X[test , ]

> train.Y <- Purchase[-test]

184 4. Classification

> test.Y <- Purchase[test]

> set.seed (1)

> knn.pred <- knn(train.X, test.X, train.Y, k = 1)

> mean(test.Y != knn.pred)

[1] 0.118

> mean(test.Y != "No")

[1] 0.059

The vector test is numeric, with values from 1 through 1, 000. Typing standardized.X[test, ] yields the submatrix of the data containing the observations whose indices range from 1 to 1, 000, whereas typing standardized.X[-test, ] yields the submatrix containing the observations whose indices do not range from 1 to 1, 000. The KNN error rate on the 1,000 test observations is just under 12 %. At first glance, this may ap- pear to be fairly good. However, since only 6 % of customers purchased insurance, we could get the error rate down to 6 % by always predicting No regardless of the values of the predictors! Suppose that there is some non-trivial cost to trying to sell insurance

to a given individual. For instance, perhaps a salesperson must visit each potential customer. If the company tries to sell insurance to a random selection of customers, then the success rate will be only 6 %, which may be far too low given the costs involved. Instead, the company would like to try to sell insurance only to customers who are likely to buy it. So the overall error rate is not of interest. Instead, the fraction of individuals that are correctly predicted to buy insurance is of interest. It turns out that KNN with K = 1 does far better than random guessing

among the customers that are predicted to buy insurance. Among 77 such customers, 9, or 11.7 %, actually do purchase insurance. This is double the rate that one would obtain from random guessing.

> table(knn.pred , test.Y)

test.Y

knn.pred No Yes

No 873 50

Yes 68 9

> 9 / (68 + 9)

[1] 0.117

Using K = 3, the success rate increases to 19 %, and with K = 5 the rate is 26.7 %. This is over four times the rate that results from random guessing. It appears that KNN is finding some real patterns in a difficult data set!

> knn.pred <- knn(train.X, test.X, train.Y, k = 3)

> table(knn.pred , test.Y)

test.Y

knn.pred No Yes

No 920 54

Yes 21 5

> 5 / 26

[1] 0.192

> knn.pred <- knn(train.X, test.X, train.Y, k = 5)

4.7 Lab: Classification Methods 185

> table(knn.pred , test.Y)

test.Y

knn.pred No Yes

No 930 55

Yes 11 4

> 4 / 15

[1] 0.267

However, while this strategy is cost-effective, it is worth noting that only 15 customers are predicted to purchase insurance using KNN with K = 5. In practice, the insurance company may wish to expend resources on convincing more than just 15 potential customers to buy insurance. As a comparison, we can also fit a logistic regression model to the data.

If we use 0.5 as the predicted probability cut-off for the classifier, then we have a problem: only seven of the test observations are predicted to purchase insurance. Even worse, we are wrong about all of these! However, we are not required to use a cut-off of 0.5. If we instead predict a purchase any time the predicted probability of purchase exceeds 0.25, we get much better results: we predict that 33 people will purchase insurance, and we are correct for about 33 % of these people. This is over five times better than random guessing!

> glm.fits <- glm(Purchase ∼ ., data = Caravan , family = binomial , subset = -test)

Warning message:

glm.fits: fitted probabilities numerically 0 or 1 occurred

> glm.probs <- predict(glm.fits , Caravan[test , ],

type = "response ")

> glm.pred <- rep("No", 1000)

> glm.pred[glm.probs > .5] <- "Yes"

> table(glm.pred , test.Y)

test.Y

glm.pred No Yes

No 934 59

Yes 7 0

> glm.pred <- rep("No", 1000)

> glm.pred[glm.probs > .25] <- "Yes"

> table(glm.pred , test.Y)

test.Y

glm.pred No Yes

No 919 48

Yes 22 11

> 11 / (22 + 11)

[1] 0.333

4.7.7 Poisson Regression

Finally, we fit a Poisson regression model to the Bikeshare data set, which measures the number of bike rentals (bikers) per hour in Washington, DC. The data can be found in the ISLR2 library.

186 4. Classification

> attach(Bikeshare)

> dim(Bikeshare)

[1] 8645 15

> names(Bikeshare)

[1] "season" "mnth" "day" "hr"

[5] "holiday" "weekday" "workingday" "weathersit"

[9] "temp" "atemp" "hum" "windspeed"

[13] "casual" "registered " "bikers"

We begin by fitting a least squares linear regression model to the data.

> mod.lm <- lm(

bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare

)

> summary(mod.lm)

Call:

lm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare)

Residuals:

Min 1Q Median 3Q Max

-299.00 -45.70 -6.23 41.08 425.29

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) -68.632 5.307 -12.932 < 2e -16 ***

mnthFeb 6.845 4.287 1.597 0.110398

mnthMarch 16.551 4.301 3.848 0.000120 ***

mnthApril 41.425 4.972 8.331 < 2e -16 ***

mnthMay 72.557 5.641 12.862 < 2e -16 ***

Due to space constraints, we truncate the output of summary(mod.lm). In mod.lm, the first level of hr (0) and mnth (Jan) are treated as the baseline values, and so no coefficient estimates are provided for them: implicitly, their coefficient estimates are zero, and all other levels are measured relative to these baselines. For example, the Feb coefficient of 6.845 signifies that, holding all other variables constant, there are on average about 7 more riders in February than in January. Similarly there are about 16.5 more riders in March than in January. The results seen in Section 4.6.1 used a slightly different coding of the

variables hr and mnth, as follows:

> contrasts (Bikeshare$hr) = contr.sum (24)

> contrasts (Bikeshare$mnth) = contr.sum (12)

> mod.lm2 <- lm(

bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare

)

> summary(mod.lm2)

Call:

lm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare)

4.7 Lab: Classification Methods 187

Residuals:

Min 1Q Median 3Q Max

-299.00 -45.70 -6.23 41.08 425.29

Coefficients :

Estimate Std. Error t value Pr(>|t|)

(Intercept) 73.597 5.132 14.340 < 2e -16 ***

mnth1 -46.087 4.086 -11.281 < 2e-16 ***

mnth2 -39.242 3.539 -11.088 < 2e-16 ***

mnth3 -29.536 3.155 -9.361 < 2e -16 ***

mnth4 -4.662 2.741 -1.701 0.08895 .

What is the difference between the two codings? In mod.lm2, a coefficient estimate is reported for all but the last level of hr and mnth. Importantly, in mod.lm2, the coefficient estimate for the last level of mnth is not zero: instead, it equals the negative of the sum of the coefficient estimates for all of the other levels. Similarly, in mod.lm2, the coefficient estimate for the last level of hr is the negative of the sum of the coefficient estimates for all of the other levels. This means that the coefficients of hr and mnth in mod.lm2 will always sum to zero, and can be interpreted as the difference from the mean level. For example, the coefficient for January of −46.087 indicates that, holding all other variables constant, there are typically 46 fewer riders in January relative to the yearly average. It is important to realize that the choice of coding really does not matter,

provided that we interpret the model output correctly in light of the coding used. For example, we see that the predictions from the linear model are the same regardless of coding:

> sum (( predict(mod.lm) - predict(mod.lm2))^2)

[1] 1.426e -18

The sum of squared differences is zero. We can also see this using the all.equal() function:

all.equal()

> all.equal(predict(mod.lm), predict(mod.lm2))

To reproduce the left-hand side of Figure 4.13, we must first obtain the coefficient estimates associated with mnth. The coefficients for January through November can be obtained directly from the mod.lm2 object. The coefficient for December must be explicitly computed as the negative sum of all the other months.

> coef.months <- c(coef(mod.lm2)[2:12] ,

-sum(coef(mod.lm2)[2:12]))

To make the plot, we manually label the x-axis with the names of the months.

> plot(coef.months , xlab = "Month", ylab = " Coefficient ",

xaxt = "n", col = "blue", pch = 19, type = "o")

188 4. Classification

> axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A",

"M", "J", "J", "A", "S", "O", "N", "D"))

Reproducing the right-hand side of Figure 4.13 follows a similar process.

> coef.hours <- c(coef(mod.lm2)[13:35] ,

-sum(coef(mod.lm2)[13:35]))

> plot(coef.hours , xlab = "Hour", ylab = " Coefficient ",

col = "blue", pch = 19, type = "o")

Now, we consider instead fitting a Poisson regression model to the Bikeshare data. Very little changes, except that we now use the function glm() with the argument family = poisson to specify that we wish to fit a Poisson regression model:

> mod.pois <- glm(

bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare , family = poisson

)

> summary(mod.pois)

Call:

glm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , family = poisson , data = Bikeshare)

Deviance Residuals:

Min 1Q Median 3Q Max

-20.7574 -3.3441 -0.6549 2.6999 21.9628

Coefficients :

Estimate Std. Error z value Pr(>|z|)

(Intercept) 4.118245 0.006021 683.964 < 2e -16 ***

mnth1 -0.670170 0.005907 -113.445 < 2e-16 ***

mnth2 -0.444124 0.004860 -91.379 < 2e -16 ***

mnth3 -0.293733 0.004144 -70.886 < 2e -16 ***

mnth4 0.021523 0.003125 6.888 5.66e-12 ***

We can plot the coefficients associated with mnth and hr, in order to repro- duce Figure 4.15:

> coef.mnth <- c(coef(mod.pois)[2:12] ,

-sum(coef(mod.pois)[2:12]))

> plot(coef.mnth , xlab = "Month", ylab = " Coefficient ",

xaxt = "n", col = "blue", pch = 19, type = "o")

> axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A", "M",

"J", "J", "A", "S", "O", "N", "D"))

> coef.hours <- c(coef(mod.pois)[13:35] ,

-sum(coef(mod.pois)[13:35]))

> plot(coef.hours , xlab = "Hour", ylab = " Coefficient ",

col = "blue", pch = 19, type = "o")

We can once again use the predict() function to obtain the fitted values (predictions) from this Poisson regression model. However, we must use the argument type = "response" to specify that we want R to output exp(β̂0 + β̂1X1 +. . .+β̂pXp) rather than β̂0 +β̂1X1 +. . .+β̂pXp, which it will output by default.

4.8 Exercises 189

> plot(predict(mod.lm2), predict(mod.pois , type = "response"))

> abline (0, 1, col = 2, lwd = 3)

The predictions from the Poisson regression model are correlated with those from the linear model; however, the former are non-negative. As a result the Poisson regression predictions tend to be larger than those from the linear model for either very low or very high levels of ridership. In this section, we used the glm() function with the argument family =

poisson in order to perform Poisson regression. Earlier in this lab we used the glm() function with family = binomial to perform logistic regression. Other choices for the family argument can be used to fit other types of GLMs. For instance, family = Gamma fits a gamma regression model.

4.8 Exercises

Conceptual

1. Using a little bit of algebra, prove that (4.2) is equivalent to (4.3). In other words, the logistic function representation and logit represen- tation for the logistic regression model are equivalent.

2. It was stated in the text that classifying an observation to the class for which (4.17) is largest is equivalent to classifying an observation to the class for which (4.18) is largest. Prove that this is the case. In other words, under the assumption that the observations in the kth class are drawn from a N(µk, σ

2) distribution, the Bayes classifier assigns an observation to the class for which the discriminant function is maximized.

3. This problem relates to the QDA model, in which the observations within each class are drawn from a normal distribution with a class- specific mean vector and a class specific covariance matrix. We con- sider the simple case where p = 1; i.e. there is only one feature.

Suppose that we have K classes, and that if an observation belongs to the kth class then X comes from a one-dimensional normal dis- tribution, X ∼ N(µk, σ2k). Recall that the density function for the one-dimensional normal distribution is given in (4.16). Prove that in this case, the Bayes classifier is not linear. Argue that it is in fact quadratic.

Hint: For this problem, you should follow the arguments laid out in Section 4.4.1, but without making the assumption that σ21 = . . . = σ

2 K.

4. When the number of features p is large, there tends to be a deteri- oration in the performance of KNN and other local approaches that

190 4. Classification

perform prediction using only observations that are near the test ob- servation for which a prediction must be made. This phenomenon is known as the curse of dimensionality, and it ties into the fact that

curse of di- mensionalitynon-parametric approaches often perform poorly when p is large. We

will now investigate this curse.

(a) Suppose that we have a set of observations, each with measure- ments on p = 1 feature, X. We assume that X is uniformly (evenly) distributed on [0, 1]. Associated with each observation is a response value. Suppose that we wish to predict a test obser- vation’s response using only observations that are within 10 % of the range of X closest to that test observation. For instance, in order to predict the response for a test observation with X = 0.6, we will use observations in the range [0.55, 0.65]. On average, what fraction of the available observations will we use to make the prediction?

(b) Now suppose that we have a set of observations, each with measurements on p = 2 features, X1 and X2. We assume that (X1, X2) are uniformly distributed on [0, 1] × [0, 1]. We wish to predict a test observation’s response using only observations that are within 10 % of the range of X1 and within 10 % of the range of X2 closest to that test observation. For instance, in order to predict the response for a test observation with X1 = 0.6 and X2 = 0.35, we will use observations in the range [0.55, 0.65] for X1 and in the range [0.3, 0.4] for X2. On average, what fraction of the available observations will we use to make the prediction?

(c) Now suppose that we have a set of observations on p = 100 fea- tures. Again the observations are uniformly distributed on each feature, and again each feature ranges in value from 0 to 1. We wish to predict a test observation’s response using observations within the 10 % of each feature’s range that is closest to that test observation. What fraction of the available observations will we use to make the prediction?

(d) Using your answers to parts (a)–(c), argue that a drawback of KNN when p is large is that there are very few training obser- vations “near” any given test observation.

(e) Now suppose that we wish to make a prediction for a test obser- vation by creating a p-dimensional hypercube centered around the test observation that contains, on average, 10 % of the train- ing observations. For p = 1, 2, and 100, what is the length of each side of the hypercube? Comment on your answer.

4.8 Exercises 191

Note: A hypercube is a generalization of a cube to an arbitrary number of dimensions. When p = 1, a hypercube is simply a line segment, when p = 2 it is a square, and when p = 100 it is a 100-dimensional cube.

5. We now examine the differences between LDA and QDA.

(a) If the Bayes decision boundary is linear, do we expect LDA or QDA to perform better on the training set? On the test set?

(b) If the Bayes decision boundary is non-linear, do we expect LDA or QDA to perform better on the training set? On the test set?

(c) In general, as the sample size n increases, do we expect the test prediction accuracy of QDA relative to LDA to improve, decline, or be unchanged? Why?

(d) True or False: Even if the Bayes decision boundary for a given problem is linear, we will probably achieve a superior test er- ror rate using QDA rather than LDA because QDA is flexible enough to model a linear decision boundary. Justify your answer.

6. Suppose we collect data for a group of students in a statistics class with variables X1 = hours studied, X2 = undergrad GPA, and Y = receive an A. We fit a logistic regression and produce estimated coefficient, β̂0 = −6, β̂1 = 0.05, β̂2 = 1.

(a) Estimate the probability that a student who studies for 40 h and has an undergrad GPA of 3.5 gets an A in the class.

(b) How many hours would the student in part (a) need to study to have a 50 % chance of getting an A in the class?

7. Suppose that we wish to predict whether a given stock will issue a dividend this year (“Yes” or “No”) based on X, last year’s percent profit. We examine a large number of companies and discover that the mean value of X for companies that issued a dividend was X̄ = 10, while the mean for those that didn’t was X̄ = 0. In addition, the variance of X for these two sets of companies was σ̂2 = 36. Finally, 80 % of companies issued dividends. Assuming that X follows a nor- mal distribution, predict the probability that a company will issue a dividend this year given that its percentage profit was X = 4 last year.

Hint: Recall that the density function for a normal random variable is f(x) = 1√

2πσ2 e−(x−µ)

2/2σ2. You will need to use Bayes’ theorem.

8. Suppose that we take a data set, divide it into equally-sized training and test sets, and then try out two different classification procedures.

192 4. Classification

First we use logistic regression and get an error rate of 20 % on the training data and 30 % on the test data. Next we use 1-nearest neigh- bors (i.e. K = 1) and get an average error rate (averaged over both test and training data sets) of 18 %. Based on these results, which method should we prefer to use for classification of new observations? Why?

9. This problem has to do with odds.

(a) On average, what fraction of people with an odds of 0.37 of defaulting on their credit card payment will in fact default?

(b) Suppose that an individual has a 16 % chance of defaulting on her credit card payment. What are the odds that she will de- fault?

10. Equation 4.32 derived an expression for log (

Pr(Y =k|X=x) Pr(Y =K|X=x)

) in the

setting where p > 1, so that the mean for the kth class, µk, is a p- dimensional vector, and the shared covariance Σ is a p × p matrix. However, in the setting with p = 1, (4.32) takes a simpler form, since the means µ1, . . . , µK and the variance σ

2 are scalars. In this simpler setting, repeat the calculation in (4.32), and provide expressions for ak and bkj in terms of πk, πK, µk, µK, and σ

11. Work out the detailed forms of ak, bkj, and bkjl in (4.33). Your answer should involve πk, πK, µk, µK, Σk, and ΣK.

12. Suppose that you wish to classify an observation X ∈ R into apples and oranges. You fit a logistic regression model and find that

P̂r(Y = orange|X = x) = exp(β̂0 + β̂1x)

1 + exp(β̂0 + β̂1x) .

Your friend fits a logistic regression model to the same data using the softmax formulation in (4.13), and finds that

P̂r(Y = orange|X = x) = exp(α̂orange0 + α̂orange1x)

exp(α̂orange0 + α̂orange1x) + exp(α̂apple0 + α̂apple1x) .

(a) What is the log odds of orange versus apple in your model?

(b) What is the log odds of orange versus apple in your friend’s model?

(c) Suppose that in your model, β̂0 = 2 and β̂1 = −1. What are the coefficient estimates in your friend’s model? Be as specific as possible.

4.8 Exercises 193

(d) Now suppose that you and your friend fit the same two models on a different data set. This time, your friend gets the coefficient estimates α̂orange0 = 1.2, α̂orange1 = −2, α̂apple0 = 3, α̂apple1 = 0.6. What are the coefficient estimates in your model?

(e) Finally, suppose you apply both models from (d) to a data set with 2,000 test observations. What fraction of the time do you expect the predicted class labels from your model to agree with those from your friend’s model? Explain your answer.

Applied

13. This question should be answered using the Weekly data set, which is part of the ISLR2 package. This data is similar in nature to the Smarket data from this chapter’s lab, except that it contains 1, 089 weekly returns for 21 years, from the beginning of 1990 to the end of 2010.

(a) Produce some numerical and graphical summaries of the Weekly data. Do there appear to be any patterns?

(b) Use the full data set to perform a logistic regression with Direction as the response and the five lag variables plus Volume as predictors. Use the summary function to print the results. Do any of the predictors appear to be statistically significant? If so, which ones?

(c) Compute the confusion matrix and overall fraction of correct predictions. Explain what the confusion matrix is telling you about the types of mistakes made by logistic regression.