Discussion 15 - 205

Chapter-4-LinearRegression.pdf

Home >Business & Finance homework help >Operations Management homework help >Discussion 15 - 205

Linear Regression Chapter 4

Introduction (Slide 1 of 2) • Managerial decisions are often based on the relationship between two or

more variables: • Example: After considering the relationship between advertising

expenditures and sales, a marketing manager might attempt to predict sales for a given level of advertising expenditures.

• Sometimes a manager will rely on intuition to judge how two variables are related.

• If data can be obtained, a statistical procedure called regression analysis can be used to develop an equation showing how the variables are related.

Introduction (Slide 2 of 2) • Dependent variable or response: Variable being predicted.

• Independent variables or predictor variables: Variables being used to predict the value of the dependent variable.

• Simple linear regression: A regression analysis for which any one unit change in the independent variable, x, is assumed to result in the same change in the dependent variable, y.

• Multiple linear regression: A regression analysis involving two or more independent variables.

Simple Linear Regression Model Regression Model

Estimated Regression Equation

Simple Linear Regression Model (Slide 1 of 5) Regression Model: • The equation that describes how y is related to x and an error term.

• Parameters: The characteristics of the population, 0 1 and . 

• Random variable: Error term, .

• The error term accounts for the variability in y that cannot be explained by the linear relationship between x and y.

Simple Linear Regression Model (Slide 2 of 5) Estimated Regression Equation:

• The parameter values are usually not known and must be estimated using sample data.

• Sample statistics 0 1(denoted and )b b are computed as estimates of the population parameters

0 1 and .  • Substituting the values of the sample statistics 0 1 0 1 and for and inb b  

the regression equation and dropping the error term, we obtain the estimated regression for simple linear regression.

Simple Linear Regression Model (Slide 3 of 5) • In the estimated simple linear regression equation:

0 1ŷ b b x= +

• ŷ = Estimate for the mean value of y corresponding to a given value of x.

• 0b = Estimated y-intercept.

• 1b = Estimated slope.

• The graph of the estimated simple linear regression equation is called the estimated regression line.

• In general, ŷ is the point estimator of ( ) ,E y x the mean value of y for a given value of x.

Simple Linear Regression Model (Slide 4 of 5) Figure 7.1: The Estimation Process in Simple Linear Regression

Simple Linear Regression Model (Slide 5 of 5) Figure 7.2: Possible Regression Lines in Simple Linear Regression

Least Squares Method Least Squares Estimates of the Regression Parameters

Using Excel’s Chart Tools to Compute the Estimated Regression Equation

Least Squares Method (Slide 1 of 15) • Least squares method: A procedure for using sample data to find

the estimated regression equation.

• Determine the values of 0 1 and .b b

• Interpretation of 0 1 and :b b

• The slope 1b is the estimated change in the mean of the dependent

variable y that is associated with a one unit increase in the independent variable x.

• The y-intercept 0b is the estimated value of the dependent variable y when the independent variable x is equal to 0.

Least Squares Method (Slide 2 of 15) Table 7.1: Miles Traveled and Travel Time for 10 Butler Trucking Company Driving Assignments

Driving Assignment i x = Miles Traveled

y = Travel Time

(hours)

1 100 9.3

2 50 4.8

3 50 8.9

4 100 6.5

5 50 4.2

6 80 6.2

7 75 7.4

8 65 6.0

9 90 7.6

10 90 6.1

Least Squares Method (Slide 3 of 15) Figure 7.3: Scatter Chart of Miles Traveled and Travel Time for Sample of 10 Butler Trucking Company Driving Assignments

Least Squares Method (Slide 4 of 15)

Least Squares Method (Slide 5 of 15)

• thi residual: The error made using the regression model to estimate the mean value of the dependent variable for the thi observation.

• Denoted as ˆ .i i ie y y= −

• Hence,

( ) 2 2

1 1

ˆmin min n n

i i i i i

y y e = =

− = 

• We are finding the regression that minimizes the sum of squared errors.

Least Squares Method (Slide 6 of 15) Least Squares Estimates of the Regression Parameters:

• For the Butler Trucking Company data in Table 7.1: • Estimated slope of 1 0.0678.b =

• y-intercept of 0 1.2739.b =

• The estimated simple linear regression model:

1 ˆ 1.2739 0.0678y x= +

Least Squares Method (Slide 7 of 15) • Interpretation of 1:b If the length of a driving assignment were 1 unit

(1 mile) longer, the mean travel time for that driving assignment would be 0.0678 units (0.0678 hours, or approximately 4 minutes) longer.

• Interpretation of 0:b If the driving distance for a driving assignment was 0 units (0 miles), the mean travel time would be 1.2739 units (1.2739 hours, or approximately 76 minutes).

Least Squares Method (Slide 8 of 15) • Experimental region: The range of values of the independent

variables in the data used to estimate the model. • The regression model is valid only over this region.

• Extrapolation: Prediction of the value of the dependent variable outside the experimental region. • It is risky.

Least Squares Method (Slide 9 of 15) • Butler Trucking Company example: Use the estimated model and the

known values for miles traveled for a driving assignment (x) to estimate mean travel time in hours.

• For example, the first driving assignment in Table 7.1 has a value for miles traveled of 100.x =

• The mean travel time in hours for this driving assignment is estimated to be:

( )1ˆ 1.2739 0.0678 100 8.0539y = + =

• The resulting residual of the estimate is:

1 1 1 ˆ 9.3 8.0539 1.2461e y y= − = − =

Least Squares Method (Slide 10 of 15) Table 7.2: Predicted Travel Time and Residuals for 10 Butler Trucking Company Driving Assignments

Least Squares Method (Slide 11 of 15) Figure 7.4: Scatter Chart of Miles Traveled and Travel Time for Butler Trucking Company Driving Assignments with Regression Line Superimposed

Least Squares Method (Slide 12 of 15) Figure 7.5: A Geometric Interpretation of the Least Squares Method

Least Squares Method (Slide 13 of 15) Using Excel’s Chart Tools to Compute the Estimated Regression Equation:

• After constructing a scatter chart with Excel’s chart tools:

1. Right-click on any data point and select Add Trendline.

2. In the Format Trendline task pane, in the Trendline Options area:

• Select Linear.

• Select Display Equation on chart.

Least Squares Method (Slide 14 of 15) Figure 7.6: Scatter Chart and Estimated Regression Line for Butler Trucking Company

Least Squares Method (Slide 15 of 15) Slope Equation

( )( )

( )

1 1

i i i

i i

x x y y

x x

− −

−



y-Intercept Equation

0 1b y b x= −

value of the independent variable for the th observation.

value of the dependent variable for the th observation.

mean value for the independent variable.

mean value for the dependent var

x i

y i

= iable.

total number of observations.n =

Assessing the Fit of the Simple Linear Regression Model The Sums of Squares

The Coefficient of Determination

Using Excel’s Chart Tools to Compute the Coefficient of Determination

Assessing the Fit of the Simple Linear Regression Model (Slide 1 of 10) The Sums of Squares:

• Sum of squares due to error: The value of SSE is a measure of the error in using the estimated regression equation to predict the values of the dependent variable in the sample.

From Table 7.2, 2

SSE 8.0288 n

i i

e =

= =

Assessing the Fit of the Simple Linear Regression Model (Slide 2 of 10) Figure 7.7: The Sample Mean y as a Predictor of Travel Time for Butler Trucking Company

Assessing the Fit of the Simple Linear Regression Model (Slide 3 of 10) • Table 7.3 shows the sum of squared deviations obtained by using

the sample mean 6.7y = to predict the value of travel time in hours for each driving assignment in the sample.

• Butler Trucking Example: For the thi driving assignment in the sample, the difference iy y− provides a measure of the error involved in using y to predict travel time for the thi driving assignment.

Assessing the Fit of the Simple Linear Regression Model (Slide 4 of 10) • The corresponding sum of squares is called the total sum of

squares (SST).

Assessing the Fit of the Simple Linear Regression Model (Slide 5 of 10) Table 7.3: Calculations for the Sum of Squares Total for the Butler Trucking Simple Linear Regression

Assessing the Fit of the Simple Linear Regression Model (Slide 6 of 10) Figure 7.8: Deviations About the Estimated Regression Line and the Line y y= for the Third Butler Trucking Company Driving Assignment

Assessing the Fit of the Simple Linear Regression Model (Slide 7 of 10)

• Measures how much the ŷ values on the estimated regression line deviate from .y

• Relation between SST, SSR, and SSE: SST = SSR + SSE

where • SST = total sum of squares • SSR = sum of squares due to regression • SSE = sum of squares due to error.

Assessing the Fit of the Simple Linear Regression Model (Slide 8 of 10) The Coefficient of Determination: • The ratio SSR/SST used to evaluate the goodness of fit for the estimated

regression equation; this ratio is called the coefficient of determination and is denoted by 2.r

• Take values between zero and one.

• Interpreted as the percentage of the total sum of squares that can be explained by using the estimated regression equation.

Assessing the Fit of the Simple Linear Regression Model (Slide 9 of 10) Using Excel’s Chart Tools to Compute the Coefficient of Determination:

• To compute the coefficient of determination using the scatter chart in Figure 7.3:

1. Right-click on any data point in the scatter chart and select Add Trendline.

2. When the Format Trendline task pane appears: Select Display R- squared value on chart in the Trendline Options area.

Assessing the Fit of the Simple Linear Regression Model (Slide 10 of 10) Figure 7.9: Scatter Chart and Estimated Regression Line with Coefficient of Determination 2r for Butler Trucking Company

The Multiple Regression Model Regression Model

Estimated Multiple Regression Equation

Least Squares Method and Multiple Regression

Butler Trucking Company and Multiple Regression

Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation

The Multiple Regression Model (Slide 1 of 11) Regression Model:

• y = dependent variable.

• 1 2, , . . ., qx x x = independent variables.

• 0 1 2, , , . . . , q    = parameters.

•  = error term (accounts for the variability in y that cannot be explained by the linear effect of the q independent variables).

The Multiple Regression Model (Slide 2 of 11) Regression Model (cont.): • Interpretation of slope coefficient :j Represents the change in the mean

value of the dependent variable y that corresponds to a one unit increase in the independent variable ,jx holding the values of all other independent variables in the model constant.

• The multiple regression equation that describes how the mean value of y is related to 1 2, , . . ., :qx x x

( )2 2 0 1 1 2 2, , . . . , q q qE y x x x x x x   = + + + +

The Multiple Regression Model (Slide 3 of 11) Estimated Multiple Regression Equation:

The Multiple Regression Model (Slide 4 of 11) Least Squares Method and Multiple Regression: • The least squares method is used to develop the estimated multiple

regression equation:

• Finding ( ) 2 2

0 1 2 1 1 ˆ, , , , that satisfy min min .

n n

q i i ii i b b b b y y e

= = − = 

• Uses sample data to provide the values of 0 1 2, , , , qb b b b that minimize the sum of squared residuals.

The Multiple Regression Model (Slide 5 of 11) Figure 7:10: The Estimation Process for Multiple Regression

The Multiple Regression Model (Slide 6 of 11) Butler Trucking Company and Multiple Regression:

• The estimated simple linear regression equation, ˆ 1.2739 0.0678 .i iy x= +

• The linear effect of the number of miles traveled explains 66.41%

( )2 0.6641r = of the variability in travel time in the sample data. • This implies, 33.59% of the variability in sample travel times remains

unexplained

• The managers might want to consider adding one or more independent variables, such as number of deliveries, to the model to explain some of the remaining variability in the dependent variable.

The Multiple Regression Model (Slide 7 of 11) Butler Trucking Company and Multiple Regression (cont.):

• Estimated multiple linear regression with two independent variables:

0 1 1 2 2ŷ b b x b x= + +

• ŷ = Estimated mean travel time.

• 1x = Distance traveled.

• 2x = Number of deliveries.

• The SST, SSR, SSE and 2R are computed using the formulas discussed earlier.

The Multiple Regression Model (Slide 8 of 11) Using Excel’s Regression Tool to Develop the Estimated Multiple Regression Equation:

Figure 7.11: Data Analysis Tools Box

The Multiple Regression Model (Slide 9 of 11) Figure 7.12: Regression Dialog Box

The Multiple Regression Model (Slide 10 of 11) Figure 7.13: Excel Regression Output for the Butler Trucking Company with Miles and Deliveries as Independent Variables

The Multiple Regression Model (Slide 11 of 11) Figure 7.14: Graph of the Regression Equation for Multiple Regression Analysis with Two Independent Variables

Inference and Regression Conditions Necessary for Valid Inference in the Least Squares Regression Model

Testing Individual Regression Parameters

Addressing Nonsignificant Independent Variables

Multicollinearity

Inference and Regression (Slide 1 of 15) • Statistical inference: Process of making estimates and drawing

conclusions about one or more characteristics of a population (the value of one or more parameters) through the analysis of sample data drawn from the population.

• In regression, inference is commonly used to estimate and draw conclusions about:

• The regression parameters 0 1 2, , , , .q   

• The mean value and/or the predicted value of the dependent variable y for specific values of the independent variables * * *1 2, , , .qx x x

• Consider both hypothesis testing and interval estimation.

Inference and Regression (Slide 2 of 15) Conditions Necessary for Valid Inference in the Least Squares Regression Model:

• For any given combination of values of the independent variables 1 2, , , ,qx x x the population of potential error terms  is normally

distributed with a mean of 0 and a constant variance.

• The values of  are statistically independent.

Inference and Regression (Slide 3 of 15) Figure 7.15: Illustration of the Conditions for Valid Inference in Regression

Inference and Regression (Slide 4 of 15) Figure 7.16: Example of a Random Error Pattern in a Scatter Chart of Residuals and Predicted Values of the Dependent Variable

Inference and Regression (Slide 5 of 15) Figure 7.17: Examples of Diagnostic Scatter Charts of Residuals from Four Regressions

Inference and Regression (Slide 6 of 15) Figure 7.18: Excel Residual Plots for the Butler Trucking Company Multiple Regression

Inference and Regression (Slide 7 of 15) Figure 7.19: Table of the First Several Predicted Values ŷ and Residuals e Generated by the Excel Regression Tool

Inference and Regression (Slide 8 of 15) Figure 7.20: Scatter Chart of Predicted Values ŷ and Residuals e

Inference and Regression (Slide 9 of 15) Testing Individual Regression Parameters:

• To determine whether statistically significant relationships exist between the dependent variable y and each of the independent variables

1 2, , , qx x x individually.

• If a 0,j = there is no linear relationship between the dependent variable y and the independent variable .jx

• If a 0,j  there is a linear relationship between y and .jx

Inference and Regression (Slide 10 of 15) Testing Individual Regression Parameters (cont.):

• Use a t test to test the hypothesis that a regression parameter j is zero.

• The test statistic for this t test, . j

b t

s =

jb s = Estimated standard deviation of .jb

• As the magnitude of t increases (as t deviates from zero in either direction), we are more likely to reject the hypothesis that the regression parameter j is zero.

Inference and Regression (Slide 11 of 15) Testing Individual Regression Parameters (cont.):

• Confidence interval can be used to test whether each of the regression parameters

0 1 2, , , , q    is equal to zero.

• Confidence interval: An estimate of a population parameter that provides an interval believed to contain the value of the parameter at some level of confidence.

• Confidence level: Indicates how frequently interval estimates based on samples of the same size taken from the same population using identical sampling techniques will contain the true value of the parameter we are estimating.

Inference and Regression (Slide 12 of 15) Addressing Nonsignificant Independent Variables:

• If practical experience dictates that the nonsignificant independent variable has a relationship with the dependent variable, the independent variable should be left in the model.

• If the model sufficiently explains the dependent variable without the nonsignificant independent variable, then consider rerunning the regression without the nonsignificant independent variable.

• The appropriate treatment of the inclusion or exclusion of the y-intercept when 0b is not statistically significant may require special consideration.

Inference and Regression (Slide 13 of 15) Multicollinearity:

• Multicollinearity refers to the correlation among the independent variables in multiple regression analysis.

• In t tests for the significance of individual parameters, the difficulty caused by multicollinearity is that it is possible to conclude that a parameter associated with one of the multicollinear independent variables is not significantly different from zero when the independent variable actually has a strong relationship with the dependent variable.

• This problem is avoided when there is little correlation among the independent variables.

Inference and Regression (Slide 14 of 15) Figure 7.21: Excel Regression Output for the Butler Trucking Company with Miles and Gasoline Consumption as Independent Variables

Inference and Regression (Slide 15 of 15) Figure 7.22: Scatter Chart of Miles and Gasoline Consumed for Butler Trucking Company

Big Data and Regression (Slide 1 of 2) Multicollinearity (cont.):

• Testing for an overall regression relationship:

• Use an F test based on the F probability distribution.

• If the F test leads us to reject the hypothesis that the values of 1 2, , , qb b b are all zero:

• Conclude that there is an overall regression relationship.

• Otherwise, conclude that there is no overall regression relationship.

Big Data and Regression (Slide 2 of 2) Multicollinearity (cont.):

• Testing for an overall regression relationship (cont.): • The test statistic generated by the sample data for this test is:

( )

SSR

SSE 1

q F

n q =

− −

• SSR = Sum of squares due to regression. • SSE = Sum of squares due to error. • q = the number of independent variables in the regression model. • n = the number of observations in the sample.

• Larger values of F provide stronger evidence of an overall regression relationship.

Categorical Independent Variables Butler Trucking Company and Rush Hour

Interpreting the Parameters

More Complex Categorical Variables

Categorical Independent Variables (Slide 1 of 10) Butler Trucking Company and Rush Hour:

• Dependent variable, y: Travel time.

• Independent variables: miles traveled ( )1x and number of deliveries ( )2 .x

• Categorical variable: rush hour ( )3x = 0 if an assignment did not include travel on the congested segment of highway during afternoon rush hour.

• Categorical variable: rush hour ( )3x = 1 if an assignment included travel on the congested segment of highway during afternoon rush hour.

Categorical Independent Variables (Slide 2 of 10) Figure 7.23: Histograms of the Residuals for Driving Assignments That Included Travel on a Congested Segment of a Highway During the Afternoon Rush Hour and Residuals for Driving Assignments That Did Not

Categorical Independent Variables (Slide 3 of 10) Figure 7.24: Excel Data and Output for Butler Trucking with Miles Traveled 1( ),x Number of Deliveries

2( ),x and the Highway Rush Hour Dummy Variable 3( ),x as the Independent Variables

Categorical Independent Variables (Slide 4 of 10) Interpreting the Parameters:

• The model estimates that travel time increases by: • 0.0672 hours for every increase of 1 mile traveled, holding constant the

number of deliveries and whether the driving assignment route requires the driver to travel on the congested segment of a highway during the afternoon rush hour period.

• 0.6735 hours for every delivery, holding constant the number of miles traveled and whether the driving assignment route requires the driver to travel on the congested segment of a highway during the afternoon rush hour period.

Categorical Independent Variables (Slide 5 of 10) Interpreting the Parameters (cont.):

• The model estimates that travel time increases by (cont.):

• 0.9980 hours if the driving assignment route requires the driver to travel on the congested segment of a highway during the afternoon rush hour period, holding constant the number of miles traveled and the number of deliveries.

• 2 0.8838R = indicates that the regression model explains approximately 88.4% of the variability in travel time for the driving assignments in the sample.

Categorical Independent Variables (Slide 6 of 10) Interpreting the Parameters (cont.):

• Compare the regression model for the case when 3 0x = and when 3 1.x =

• When 3 0:x =

• When 3 1:x =

Categorical Independent Variables (Slide 7 of 10) More Complex Categorical Variables:

• If a categorical variable has k levels, 1k − dummy variables are required, with each dummy variable corresponding to one of the levels of the categorical variable and coded as 0 or 1.

• Example: • Suppose a manufacturer of vending machines organized the sales territories

for a particular state into three regions: A, B, and C.

• The managers want to use regression analysis to help predict the number of vending machines sold per week.

• Suppose the managers believe sales region is one of the important factors in predicting the number of units sold.

Categorical Independent Variables (Slide 8 of 10) More Complex Categorical Variables (cont.):

• Example (cont.):

• Sales region: categorical variable with three levels (A, B, and C).

• Number of dummy variables = 3 2 1.− =

• Each variable can be coded 0 or 1 as:

1 1 if sales Region B; 0 if otherwise.x =

2 1 if sales Region C; 0 if otherwise.x =

• The values of 1 2 and are:x x

Categorical Independent Variables (Slide 9 of 10) More Complex Categorical Variables (cont.):

• Example (cont.): • The regression equation relating the mean number of units sold to the

dummy variables is written as:

0 1 1 2 2ŷ b b x b x= + +

• Observations corresponding to Region A correspond to 1 20, 0,x x= = so the estimated mean number of units sold in Region A is:

( ) ( )0 1 2 0ˆ 0 0y b b b b= + + =

Categorical Independent Variables (Slide 10 of 10) More Complex Categorical Variables (cont.):

• Example (cont.): • Observations corresponding to Region B are coded 1 21, 0,x x= =

so the estimated mean number of units sold in Region B is:

( ) ( )0 1 2 0 1ˆ 1 0y b b b b b= + + = +

• Observations corresponding to Region C are coded 1 20, 1,x x= = so the estimated mean number of units sold in Region C is:

( ) ( )0 1 2 0 2ˆ 0 1y b b b b b= + + = +

Modeling Nonlinear Relationships Quadratic Regression Models

Piecewise Linear Regression Models

Interaction Between Independent Variables

Modeling Nonlinear Relationships (Slide 1 of 16) Figure 7.25: Scatter Chart for the Reynolds Example

Modeling Nonlinear Relationships (Slide 2 of 16) Figure 7.26: Excel Regression Output for the Reynolds Example

Modeling Nonlinear Relationships (Slide 3 of 16) Figure 7.27: Scatter Chart of the Residuals and Predicted Values of the Dependent Variable for the Reynolds Simple Linear Regression

Modeling Nonlinear Relationships (Slide 4 of 16) Quadratic Regression Models:

• In the Reynolds example, to account for the curvilinear relationship between months employed and scales sold, we could include the square of the number of months the salesperson has been employed as a second independent variable.

• An estimated regression equation given by:

• Equation (7.18) corresponds to a quadratic regression model.

Modeling Nonlinear Relationships (Slide 5 of 16) Figure 7.28: Relationships That Can Be Fit with a Quadratic Regression Model

Modeling Nonlinear Relationships (Slide 6 of 16) Figure 7.29: Excel Data for the Reynolds Quadratic Regression Model

Modeling Nonlinear Relationships (Slide 7 of 16) Figure 7.30: Excel Output for the Reynolds Quadratic Regression Model

Modeling Nonlinear Relationships (Slide 8 of 16) Figure 7.31: Scatter Chart of the Residuals and Predicted Values of the Dependent Variable for the Reynolds Quadratic Regression Model

Modeling Nonlinear Relationships (Slide 9 of 16) Piecewise Linear Regression Models:

• For the Reynolds data, as an alternative to a quadratic regression model:

• Recognize that below some value of Months Employed, the relationship between Months Employed and Sales appears to be positive and linear.

• Whereas the relationship between Months Employed and Sales appears to be negative and linear for the remaining observations.

• Piecewise linear regression model: This model will allow us to fit these relationships as two linear regressions that are joined at the value of Months at which the relationship between Months Employed and Sales changes.

Modeling Nonlinear Relationships (Slide 10 of 16) Piecewise Linear Regression Models (cont.):

• Knot: The value of the independent variable at which the relationship between dependent variable and independent variable changes; also called breakpoint.

• For the Reynolds data, knot is the value of the independent variable Months Employed at which the relationship between Months Employed and Sales changes.

Modeling Nonlinear Relationships (Slide 11 of 16) Figure 7.32: Possible Position of Knot ( )

k x

Modeling Nonlinear Relationships (Slide 12 of 16) Piecewise Linear Regression Models (cont.):

• Define a dummy variable:

( )

1 Months.

value of the knot (90 months for the Reynolds example).

the knot dummy variable.

• Then fit the following estimated regression equation:

Modeling Nonlinear Relationships (Slide 13 of 16) Figure 7.33: Data and Excel Output for the Reynolds Piecewise Linear Regression Model

Modeling Nonlinear Relationships (Slide 14 of 16) Interaction Between Independent Variables: • Interaction: This occurs when the relationship between the dependent

variable and one independent variable is different at various values of a second independent variable.

• The estimated multiple linear regression equation is given as:

Modeling Nonlinear Relationships (Slide 15 of 16) Figure 7.34: Mean Unit Sales (1,000s) as a Function of Selling Price and Advertising Expenditures

Modeling Nonlinear Relationships (Slide 16 of 16) Figure 7.35: Excel Output for the Tyler Personal Care Linear Regression Model with Interaction

Model Fitting Variable Selection Procedures

Overfitting

Model Fitting (Slide 1 of 11) Variable Selection Procedures: • Special procedures are sometimes employed to select the independent

variables to include in the regression model.

• Iterative procedures: At each step of the procedure, a single independent variable is added or removed and the new model is evaluated. Iterative procedures include:

• Backward elimination.

• Forward selection.

• Stepwise selection.

• Best subsets procedure: Evaluates regression models involving different subsets of the independent variables.

Model Fitting (Slide 2 of 11) Variable Selection Procedures (cont.):

• Backward elimination procedure: • Begins with the regression model that includes all of the independent

variables under consideration.

• At each step, backward elimination considers the removal of an independent variable according to some criterion.

• Stops when all independent variables in the model are significant at a specified level of significance.

Model Fitting (Slide 3 of 11) Variable Selection Procedures (cont.):

• Forward selection procedure: • Begins with none of the independent variables under consideration

included in the regression model.

• At each step, forward selection considers the addition of an independent variable according to some criterion.

• Stops when there are no independent variables not currently in the model that meet the criterion for being added to the regression model.

Model Fitting (Slide 4 of 11) Variable Selection Procedures (cont.):

• Stepwise selection procedure: • Begins with none of the independent variables under consideration

included in the regression model.

• The analyst establishes both a criterion for allowing independent variables to enter the model and a criterion for allowing independent variables to remain in the model.

• To initiate the procedure, the most significant independent variable is added to the empty model if its level of significance satisfies the entering threshold.

Model Fitting (Slide 5 of 11) Variable Selection Procedures (cont.):

• Stepwise selection procedure (cont.): • Each subsequent step involves two intermediate steps:

• First, the remaining independent variables not in the current model are evaluated, and the most significant one is added to the model.

• Then the independent variables in the current model are evaluated, and the least significant one is removed.

• Stops when no independent variables not currently in the model have a level of significance for remaining in the regression model.

Model Fitting (Slide 6 of 11) Variable Selection Procedures (cont.):

• Best subsets procedure: • Simple linear regressions for each of the independent variables under

consideration are generated, and then the multiple regressions with all combinations of two independent variables under consideration are generated, and so on.

• Once a regression model has been generated for every possible subset of the independent variables under consideration, the entire collection of regression models can be compared and evaluated.

Model Fitting (Slide 7 of 11) Overfitting: • Overfitting generally results from creating an overly complex model to

explain idiosyncrasies in the sample data.

• In regression analysis, this often results from the use of complex functional forms or independent variables that do not have meaningful relationships with the dependent variable.

• If a model is overfit to the sample data, it will perform better on the sample data used to fit the model than it will on other data from the population.

• Thus, an overfit model can be misleading about its predictive capability and its interpretation.

Model Fitting (Slide 8 of 11) Overfitting (cont.):

• How does one avoid overfitting a model? • Use only independent variables that you expect to have real and meaningful

relationships with the dependent variable.

• Use complex models, such as quadratic models and piecewise linear regression models, only when you have a reasonable expectation that such complexity provides a more accurate depiction of what you are modeling.

• Do not let software dictate your model; use iterative modeling procedures, such as the stepwise and best-subsets procedures, only for guidance and not to generate your final model.

Model Fitting (Slide 9 of 11) Overfitting (cont.):

• How does one avoid overfitting a model? (cont.): • If you have access to a sufficient quantity of data, assess your model on data

other than the sample data that were used to generate the model (this is referred to as cross-validation).

• Three possible ways to execute cross-validation:

• Holdout method.

• k-fold cross-validation.

• Leave-one-out cross-validation.

Model Fitting (Slide 10 of 11) Overfitting (cont.):

• Holdout method: The sample data are randomly divided into mutually exclusive and collectively exhaustive training and validation sets.

• Training set: The data set used to build the candidate models that appear to make practical sense.

• Validation set: The set of data used to compare model performances and ultimately select a model for predicting values of the dependent variable.

Model Fitting (Slide 11 of 11) Overfitting (cont.):

• k-fold cross-validation: The sample data are randomly divided into k equal-sized, mutually exclusive, and collectively exhaustive subsets called folds, and k iterations are executed.

• Leave-one-out cross-validation: For a sample of n observations, an iteration consists of estimating the model on 1n− observations and evaluating the model on the single observation that was omitted from the training data.

Big Data and Regression Inference and Very Large Samples

Model Selection

Big Data and Regression (Slide 1 of 6) Inference and Very Large Samples: • Virtually all relationships between independent variables and the

dependent variable will be statistically significant if the sample is sufficiently large.

• That is, if the sample size is very large, there will be little difference in the

jb values generated by different random samples.

Big Data and Regression (Slide 2 of 6) Figure 7.36: Excel Regression Output for Credit Card Company Example

Big Data and Regression (Slide 3 of 6) Table 7.4: Regression Parameter Estimates and the Corresponding p values for 10 Multiple Regression Models, Each Estimated on 50 Observations from the LargeCredit Data

Big Data and Regression (Slide 4 of 6) Figure 7.37: Excel Regression Output for Credit Card Company Example after Adding Number of Hours per Week Spent Watching Television

Big Data and Regression (Slide 5 of 6) Model Selection: • When dealing with large samples, it is often more difficult to discern the

most appropriate model.

• If developing a regression model for explanatory purposes, the practical significance of the estimated regression coefficients should be considered when interpreting the model and considering which variables to keep in the model.

• If developing a regression model to make future predictions, the selection of the independent variables to include in the regression model should be based on the predictive accuracy on observations that have not been used to train the model.

Big Data and Regression (Slide 6 of 6) Figure 7.38: Predictive Accuracy on LargeCredit Validation Set

Prediction with Regression

Prediction with Regression (Slide 1 of 2) • In addition to the point estimate, there are two types of interval

estimates associated with the regression equation: • A confidence interval is an interval estimate of the mean y value given

values of the independent variables.

• A prediction interval is an interval estimate of an individual y value given values of the independent variables.

Prediction with Regression (Slide 2 of 2) Table 7.5: Predicted Values and 95% Confidence Intervals and Prediction Intervals for 10 New Butler Trucking Routes