BUS 308 DQ 1 & 2 *****ORIGINAL WORK ONLY*****
loka9502
bus308_chapter_10.pdf
10
Multiple Regression: Using More Than One Predictor
Learning Objectives
After reading this chapter, you should be able to:
• Determine how to correlate more than two variables.
• Explain how to control redundant information with multiple predictors.
• Use multiple predictors to determine the value of a criterion.
• Complete a multiple regression problem using Excel.
iStockphoto/Thinkstock
tan81004_10_c10_247-272.indd 247 2/22/13 3:43 PM
CHAPTER 10Section 10.1 Multiple Correlation
Chapter Outline
10.1 Multiple Correlation Calculating the Multiple Correlation Another Multiple R Problem
10.2 Semi-Partial Correlation
10.3 Multiple Regression Picturing How Multiple Regression Looks The Regression Equation Completing a Multiple Regression Problem Another Multiple Regression Problem
10.4 Determining the Error in a Regression Solution Solidifying the Multiple Regression Procedure Over-Fitting the Data and Shrinkage
10.5 Multiple Regression With Excel
Chapter Summary
Introduction
As valuable as the Chapter 9 processes are for understanding how a business outcome can be predicted by a correlated predictor, simple regression, or regression with one predictor, does not accommodate most business realities. Retail sales are related to more than just the sales associates’ years of experience. If the problem were really that sim- ple, hiring decisions would be very straightforward: Simply hire the most experienced applicant. The reality of course, is that business outcomes are a function of many factors. In terms of regression, often criterion variables can only be understood adequately by multiple predic- tor variables. This chapter explores regression pro- cedures that can include more than one predictor. Using more than one predictor in a regression prob- lem constitutes what is called multiple regression.
10.1 Multiple Correlation
Suppose that a manager at an electronics store believes that sales associates’ produc-tivity is related to their level of job satisfaction, but it’s also related to the experience they gain through their tenure with the organization. Since those with the highest level of job satisfaction also tend to be those who remain with the organization the longest, the two predictors are themselves correlated. Can two correlated predictors make a better estimation of sales productivity than either variable by itself? Since the two variables are correlated, they are both going to provide some of the same information. How can the redundancy be controlled so that the prediction is optimal, but not artificially inflated?
Key Terms: Multiple regres- sion is regression with more than one predictor.
tan81004_10_c10_247-272.indd 248 2/22/13 3:43 PM
CHAPTER 10Section 10.1 Multiple Correlation
If sales associates’ productivity (the criterion variable) is predicted by their tenure (pre- dictor x1) as well as their job satisfaction (predictor x2), the resulting regression problem is different from what was done in Chapter 9 in two ways. First, recall that in Chapter 8, the Pearson correlation gauged the relationship between two continuous variables, and that correlation was the foundation for bivariate regression that came up in Chapter 9. By way of introduction, we noted that the square of the Pearson r indicated how much of any change in the criterion variable could be explained by changes in the predictor variable. The r2 value provided an indicator of how well that single x variable was likely to predict the value of y. A second predictor means that there is an additional relationship to be con- sidered, the relationship between that second predictor and the criterion variable. At issue will be how much additional information about y is provided by that second x variable.
That second predictor variable introduces another difference between simple and multi- ple regression. It is that multiple predictor variables, besides each being correlated with the criterion variable, y, are often correlated with each other. The correlation between x1 and x2 was noted above when it was observed that those with the longest tenure with the electronics company are also likely to have the highest levels of job satisfaction. This cor- relation creates a problem. If the predictors are correlated with each other, it means that they share some of the same information. As long as they are not perfectly correlated with each other (r < 11.0/21.0) they each probably contain information about the criterion variable that is unique the particular predictor. That unique information is what makes both variables valuable as predictors of the criterion. How can what is unique be sepa- rated from what is common to the other predictor? To use them both without distinguish- ing between what is common and what is unique is to run the risk of “double-counting” the common information and getting an inaccurate prediction of the criterion as a result.
The initial problem is one of determining how well both of the predictor variables cor- relate with the criterion variable. The strength of that more complicated relationship will be determined with a new statistic. It is called multiple correlation, or sometimes just multiple R. This statistic is actually an extension of the Pearson Correlation that will
determine the degree to which any number of pre- dictors—any number of x variables—are correlated with a single criterion variable, y. To keep the exam- ples in this chapter simple, the number of predictors will be limited to two. However, the logic and related analysis can be extended to include any number of predictors.
Multiple R can be adapted to determine the strength of the relationship between any number of predictor variables, x1, x2, . . . xn, with a single criterion variable, y.
The problem of determining the degree to which multiple x variables overlap each other is gauged by another statistic called semi-partial correlation. That statistic will gauge how well a second x variable is correlated with y, beyond the level at which the first x is correlated with the criterion.
Key Terms: Multiple corre- lation determines the correla- tion between a combination of x variables and a y variable.
tan81004_10_c10_247-272.indd 249 2/22/13 3:43 PM
CHAPTER 10Section 10.1 Multiple Correlation
Formula 10.1 Ryx1x2 5 Å r2 yx1 1 r2yx2 2 2ryx1ryx2rx1x2
1 2 r2 x1x2
Where
Ryx1x2 is the multiple correlation of the criterion, y, with the combination of the two predictors, designated x1 and x2. ryx1 is the Pearson Correlation of the criterion, y, and the first predictor, x1. ryx2 is the Pearson Correlation of the criterion, y, and the second predictor, x2. rx1x2 is the Pearson Correlation of the two predictors, x1 and x2.
Table 10.1 contains data for the example of predicting sales productivity (in thousands of dollars) using associates’ tenure in years and job satisfaction, which was measured using a survey instrument.
ryx1, the correlation between the criterion and the second predictor, which is ryx2, and the correlation between the two predictors, rx1x2.
When there are multiple x variables correlated with y, and the x variables are correlated with each other, semi-partial correlation will determine the degree to which second, third, and so on x variables are corre- lated with y beyond what the other x variables have already accounted for.
The semi-partial correlation provides a conceptual introduction to regression coefficients that, when there are more than one, must each be adjusted according to the correlation between pre- dictors so that the contribution that each x variable makes to the prediction of y is unique.
Calculating the Multiple Correlation
Formula 10.1 is the formula for multiple R. There are three components to calculating the solution. They are the correlation between the criterion and the first predictor, which is
Key Terms: Semi-partial correlation determines the x / y relationship, controlling for what x may have in common with other x variables. A similar control occurs with partial regression coefficients.
tan81004_10_c10_247-272.indd 250 2/22/13 3:44 PM
CHAPTER 10Section 10.1 Multiple Correlation
Table 10.1: The relationship among tenure, job satisfaction, and sales productivity
Tenure (years) Job Satisfaction Sales ($k)
1.0 5 1.4
.5 2 .2
3.0 11 3.5
2.5 7 1.8
4.0 13 3.5
8.5 14 4.0
7.0 11 3.8
3.5 8 2.4
2.75 5 3.6
4.0 10 4.2
The first step is to determine the three Pearson Correlations involved. Using the Cor- relation procedure in Excel’s Data Analysis menu results in the following bivariate, Pearson Correlations:
• The correlation of tenure with job satisfaction is rx1x2 5 .820. • The correlation of sales productivity with tenure is ryx1 5 .742. • The correlation of sales productivity with job satisfaction is ryx2 5 .796.
The three Pearson Correlations are all that are needed to complete the multiple correlation of sales productivity with a combination of experience and job satisfaction. Substituting the values for the notation yields:
Ryx1x2 5 Å .7422 1 .7962 2 21.7422 1.7962 1.8202
1 2 .8202 5 .811
Note that the multiple correlation of sales with the combination of experience and job satisfaction is greater than the correlation of either bivariate correlation of either predictor variable with the criterion variable. This is to be expected. The fact that the x variables are correlated with the criterion variable indicates that they both have something in common with that variable. Since the x variables are not perfectly correlated with each other, it means that they each measure something different. The fact that Ryx1x2 is greater than either ryx1 or ryx2 indicates that however much overlap there might be between the two x variables, they each also measure something unique. If that unique component is also something that each x variable has in common with y, the multiple correlation value, Ryx1x2 can become a good deal more robust than either of the bivariate correlations.
In Chapter 9 the Pearson Correlation was squared (r2) to produce the coefficient of deter- mination and calculate how much of the variance in one variable can be explained by changes in the other. The same thing can be done with Ryx1x2. The square of the multiple correlation (R2) indicates how much of the variance in y can be explained by changes in the x variables, something illustrated in the Venn diagram that is Figure 10.1 (Diekhoff,
tan81004_10_c10_247-272.indd 251 2/22/13 3:44 PM
CHAPTER 10Section 10.1 Multiple Correlation
1992). In the example above, tenure and job satisfaction account for about 66% (.8112) of the variance in sales productivity. The other side of that result, of course, is that about 34% of the variance in sales cannot be explained by tenure and job satisfaction. The 34% is a reminder that predictions of sales figures from employees’ tenure and job satisfaction are never going to be completely accurate because other variables that also have something to do with sales results have not been included in the analysis. Perhaps how aggressive the salespeople are is a factor in their sales success. If it is, by omitting that variable from the prediction, whatever information it can contribute is likewise omitted, and the accuracy of the prediction must diminish correspondingly. Regression solutions based on imper- fect correlations cannot provide consistent, perfectly accurate predictions. Over several predictions and with the inclusion of additional relevant predictors, however, they will be consistently better than chance predictions.
Figure 10.1: Three related variables
Adapted from: Diekhoff, G. (1992). Statistics for the social and behavior sciences: Univariate, bivariate, and multivariate. Dubuque, IA: William C. Brown Publishers
Another Multiple R Problem
Suppose a shift supervisor is interested in how a combination of a worker’s manual dex- terity and job satisfaction might be related to the number of components the worker com- pletes during a shift. Treating the number of components completed as the y variable, and manual dexterity as x1 and job satisfaction as x2, the shift supervisor collects the relevant data from a number of workers and determines the following. First the bivariate Pearson correlations:
ryx1 5 .625
ryx2 5 .413
rx1x2 5 .373
y
Area 1 Area 3
Area 2
x 2x1
tan81004_10_c10_247-272.indd 252 2/22/13 3:44 PM
CHAPTER 10Section 10.2 Semi-Partial Correlation
With Formula 10.1,
Ryx1x2 5 Å r2 yx1 1 r2 yx2 2 2ryx1ryx2rx1x
1 2 r2x1x2
and the appropriate correlation values substituted,
5 Å .6252 1 .4132 2 21.6252 1.4132 1.3732
1 2 3.732
5 ".428 5.654
The correlation of a combination of manual dexterity and job satisfaction with the number of components the workers complete during a shift is Ryx1x2 5 .654. If the multiple R value is squared (.6542 5 .428), it indicates that about 43% of the variability in the number of components completed during a shift is explained by a combination of workers’ manual dex- terity and their job satisfaction.
10.2 Semi-Partial Correlation
Multiple regression, we noted earlier, introduces two related problems not present when there is only one predictor. The problem of how to determine the relationship between a combination of more than one predictor and a single criterion variable is solved with the multivariate extension of the Pearson Correlation, the multiple R, procedure.
The second problem with multiple predictors is that multiple predictors, in addition to being correlated with the criterion variable, are also usually correlated with each other. As a consequence, there must be a way to determine how much of the correlation between each predictor and a criterion variable is unique and so will contribute to the quality of the prediction, and how much is shared between predictors and so is redundant informa- tion. This problem is resolved with the other procedure mentioned earlier, semi-partial correlation.
Figure 10.1 illustrates the correlation that two variables have with a third. Applying it to our example, it is the correlation of manual dexterity and job satisfaction, two “x” vari- ables, to the number of components completed in a shift, the y variable, which is repre- sented by the upper circle. The areas designated “1” and “2” indicate the variance shared between manual dexterity, x1, and the number of components completed, y. In fact if the proportion of the total area of either variable that is covered by “1” plus “2” were cal- culated, it would be the ryx12 amount, the proportion of the variance in either of manual dexterity and number of components completed, that is explained by changes in the other. Similarly, the areas represented by “2” and “3” indicate variance shared between job sat- isfaction, x2 , and the number of components completed, y, and represents the proportion of the variance in either variable that can be explained by changes in the other.
Review Question A: What statistic will indicate the correla- tion between office productivity and a combination of employees’ experience and their ages?
tan81004_10_c10_247-272.indd 253 2/22/13 3:44 PM
CHAPTER 10Section 10.2 Semi-Partial Correlation
The way Formula 10.2 is written (ry(x2x1)) indicates that this test will determine the rela- tionship between a second x variable (x2) and y, controlling for whatever portion of x2 has already been accounted for in the correlation between y and the first variable, x1. As such, the procedure will indicate how much information about y the x2 variable provides, in addition to what x1 has already introduced. It solves the “double counting” problem noted earlier. Substituting the above Pearson Correlation values into the semi-partial cor- relation formula, the result is:
ry1x2x12 5 .413 2 1.6252 1.3732
"1 2 .3732 5 .194
The ry(x2x1) 5 .194 value is the correlation of job satisfaction with the number of components completed during a shift, with whatever job satisfaction also has in common with man- ual dexterity “partialed out.” One way to think of the result is that if the value is squared, it indicates how much additional information about y can be determined by adding the second x variable. So although job satisfaction is correlated with the number of components completed at ryx2 5 .413, the additional information contributed by job satisfaction over and above the manual dexterity/components completed correlation is a good deal more modest. More than half the correlation value indicating what job satisfaction reveals about the number of
components completed is also present in the correlation between manual dexterity and the number of components completed.
The square of the semi-partial correlation is interpreted the same way as the square of a Pearson correlation. The r2y(x2x1) value will indicate the proportion of variance in com- ponents completed that is explained by job satisfaction, beyond what manual dexterity
Review Question B: If multiple worker characteristics are each correlated with how long the worker will stay on the job, what can be said about multiple R com- pared to the bivariate correlations?
Formula 10.2 ry1x2x12 5 ryx2 2 1ryx1 2 1rx1x2 2
"1 2 r2 x1x2
The problem is that part of what x1 (manual dexterity) has in common with y, the “2” area designated in Figure 10.1, is also shared between x2 (job satisfaction) and y. The fact that both predictors occupy this portion illustrates that this information about y is provided by both the x variables. The most accurate correlation of y with the predictors requires that this information be included in the analysis just a single time since to include it a second time suggests that it is new information. To put it simply, the task is to use the informa- tion that a second predictor provides about a criterion variable, but only the portion of the information that is not already represented in the relationship between the first predictor and the criterion variable.
The semi-partial correlation procedure will determine the strength of the correlation between a second predictor and a criterion variable, controlling or holding out of the result, whatever information about y is already included in the relationship with the first predictor. The formula for semi-partial correlation is the following:
tan81004_10_c10_247-272.indd 254 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
explains. The square of .194 5.038, about 3.8% of the variance in number of components completed is explained by changes in the job satisfaction variable, beyond what manual dexterity has already explained. That result indicates a good deal of overlap between the two x variables.
10.3 Multiple Regression
Although it is not used directly in multiple regression, the semi-partial correlation pro-cedure is an instructive way to introduce an important, related concept. Whenever there is more than one predictor variable it is quite likely that the predictors are themselves correlated and the regression procedure has to be adjusted so that the unique contribution of each predictor can be assessed. So, part of the multiple regression procedure involves eliminating the redundant information in second, third, and so on predictor variables in a manner that is analogous to what occurs in a semi-partial correlation.
In simple, or bivariate, regression, the regression solution included a regression coefficient that indicated the slope of the regression line. In multiple regression, as the discussion that follows will indicate in greater detail, a separate regression coefficient is calculated for each predictor variable. Because each is adjusted for what previous regression coefficients have already explained about how to position the line, the coefficients are actually the partial regression coefficients referred to earlier. Any information about how y changes when the particular x increases by 1.0 (which is how the regression coefficient is defined) has to be adjusted for the impact on y of the other regression coefficients in the problem.
Picturing How Multiple Regression Looks
The regression equation for bivariate problems was based on two subcomponents, val- ues for the intercept and the slope, both of which had to be calculated before completing the solution. In Chapter 9, the intercept and slope values were represented by a regres- sion line in a graph. Once the line is positioned, a value of x on the horizontal axis of the graph can be used to predict the corresponding value of y by moving from the x value vertically to the regression line, and then horizontally left to the vertical axis, which has the values of y.
That image is not difficult to represent visually since it involves just two dimensions. For multiple regression the concept is similar, but the image is more difficult to represent with line drawings since it involves more than two dimensions. In a regression problem with two predictors, which is the simplest form of multiple regression, each predictor contrib- utes its own regression line. With two x axes for the predictors, as well as the y axis, repre- senting the solution visually requires a three-dimensional image.
Think of standing in a room on the second floor of a building facing one of the corners in the room where two walls meet.
• The vertical line where the two walls intersect is like the y axis in the graph. • Where the floor meets the corner of the room represents y 5 0. • Negative values of y could be represented in the corner below the floor of the
room.
tan81004_10_c10_247-272.indd 255 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
• The floor running along the bottom of the wall to your left is like the axis for the x1 variable.
• The floor running along the bottom of the wall to your right is like the axis for the x2 variable.
• Each of the two x variables produces a separate slope that can either rise or fall depending upon whether the variable’s correlation with y is positive or negative. As with simple regression, the slopes indicate the independent impact that each x variable has upon y.
• If the two slopes are joined along their lengths, they create a regression “plane” rather than the regression line from the simple regression solution.
• In this image, the values for x1 and x2 are plotted horizontally along their respective axes to the point that indicates the particular value for x1 or x2, and then perpendicular to those axes toward the middle of the room to where the two values intersect each other.
• Proceeding from the point of intersection vertically to the regression plane, and then horizontally back to the y axis (the corner of the room) indicates the value of y that corresponds to the values for x1 and x2.
With the limitations inherent in two-dimensional line drawings, these several compo- nents are represented in Figure 10.2. The figure is how a regression solution might be represented for a situation where the correlations between each x value and y are positive, and the value of the y intercept is likewise positive.
Figure 10.2: Picturing a multiple regression solution
The Regression Equation
The general form of the multiple regression equation is the following:
x 1 x2
y
Regression plane
The x 1 value
The x 2 value
The vertical point from the intersection of the x
1 and x
2 values. This is the point
of contact with the regression “plane.”
The value of y, given x 1 and x
2
Formula 10.3 y’ 5 a 1 b1x1 1 b2x2 1 . . . 1 bkxk
tan81004_10_c10_247-272.indd 256 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
Formula 10.4 a 5 My 2 b1Mx1 2 b2Mx2
Where
y’ 5 the value of the variable to be predicted. a is the value of the regression intercept or constant, indicating the value of y when the values of all predictors are 0. b1 . . . bk are the values of the regression coefficients associated with predictors 1 2 k. x1 . . . xk are the values of the predictors.
Remember that multiple regression coefficients are actually partial regression coefficients. Each coefficient includes only the information about the criterion variable that does not also emerge in the other coefficients. For example:
• The b1 coefficient explains how much y will change for every increase of 1.0 in the first predictor variable, x1, when the effects of the other predictors (x2 . . . xk) are held constant.
• The second regression coefficient, b2, explains how much y will change when- ever x2 increases by 1.0, when the effects of the other predictors (x1 and x3 . . . xk) are held constant.
With two predictors, the multiple regression equation has its simplest form:
y’ 5 a 1 b1x1 1 b2x2
As with simple regression, developing a solution requires that values be determined for the intercept (the a value), and then for each regression coefficient. There are two in this case, b1 and b2.
The formula for the intercept, or the regression constant value is:
The formulas for two regression coefficients are:
Formula 10.5 b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b
Formula 10.6 b2 5 ryx2 2 ryx1 rx1x2
1 2 r2x1x2 a
sy sx2
b
tan81004_10_c10_247-272.indd 257 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
Note how the b1 and b2 equations would change if the x1 and x2 variables, manual dexterity and motive strength in the earlier example, were uncorrelated. If rx1x2 5 0, then
• b1 would reduce to ryx1a sy sx1
b , and
• b2 would become ryx2a sy sx2
b
This is a reminder that the balance of formulas 10.5 and 10.6 are the components that are required to control the redundancy between the predictor variables. If the predic-
tor variables are uncorrelated, there is no need to introduce control in one regression coefficient for the effect of the other predictor, and the two equations reduce to the same form they had in simple regression, which is what we saw in Chapter 9. However, the reality is that when multiple variables correlate with a criterion variable, they also tend to correlate with each other. Uncorrelated predictor variables are unlikely to occur in a business setting.
Completing a Multiple Regression Problem
Earlier we completed a multiple correlation problem for the relationship between sales productivity and a combination of employee tenure and employee job satisfaction. To take the next step, suppose a regional manager wishes to know what can be expected for someone who transfers into the region with 5 years of job experience (the tenure variable) and a job satisfaction rating of 9. The first step is to gather the relevant descriptive statis- tics. The means and standard deviations can be calculated from the data provided in Table 10.1, which results in the following statistics:
Mean Standard Deviation
Experience 3.675 2.461
Job Satisfaction 8.600 3.864
Sales 2.840 1.328
The correlation matrix below summarizes the correlations between each pair of variables that were calculated earlier in the chapter. Correlation matrices are commonly used as an efficient way to visually present correlations among multiple pairs of variables. The cells highlighted in gray are sometimes left blank because they either indicate the variable’s correlation with itself (r 5 1.0 along the diagonal), or because the coefficient is already represented in one of the other cells (e.g., the correlation between tenure and job satisfac- tion is the same as the correlation between job satisfaction and tenure).
Tenure Job Satisfaction Sales Tenure 1 .820 .742
Job Satisfaction .820 1 .796
Sales .742 .796 1
Review Question C: Why must the regres- sion coefficients in multiple regression be partial regression coefficients?
tan81004_10_c10_247-272.indd 258 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
The regression equation and the formulas for the constant value and the regression coef- ficients are as follows:
y’ 5 a 1 b1x1 1 b2x2
a 5 My 2 b1Mx1 2 b2Mx2
b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b
b2 5 ryx2 2 ryx1 rx1x2
1 2 r2x1x2 a
sy sx2
b
As with the simple regression problems, the regression coefficients must be solved before tackling the regression constant value,
b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b
Treating tenure as x1, job satisfaction as x2, and sales as the y variable, inserting the rel- evant descriptive statistics and completing the calculations provides the following for the first regression coefficient:
b1 5 .742 2 1.7962 1.8202
1 2 .8202 a1.328
2.461 b
5 .089 .328
1.5402 5 .147
For the second regression coefficient,
b2 5 .796 2 1.7422 1.8202
1 2 .8202 a1.328
3.864 b
5 .188 .328
1.3442 5 .197
With the regression coefficients determined, the regression constant can be calculated:
a 5 My 2 b1Mx1 2 b2Mx2
5 2.84 2 .147(3.675) 2 .197(8.60) 5 .606
tan81004_10_c10_247-272.indd 259 2/22/13 3:44 PM
CHAPTER 10Section 10.3 Multiple Regression
The values for x1 and x2 were 5 years of tenure and a job satisfaction rating of 9, respectively. When they are inserted into the regression equation along with the values calculated for the regression constant and the two regression coefficients, the result is as follows:
y’ 5 a 1 b1x1 1 b2x2
5 .606 1 (.147)(5) 1 (.197)(9)
5 3.114
For someone with a job tenure of 5 years who has a job satisfaction score of 9, the best prediction for sales productivity will be about $3,114.
Another Multiple Regression Problem
In an information technology (IT) company, bids for new contracts are based upon how much overhead cost is expected and the number of billable hours that are involved in the work. Data for the most recent 24 projects result in the descriptive statistics that follow.
In the correlations below, project overheads are designated x1, billable hours are desig- nated x2, and the bid amounts are the criterion variable, y:
Correlations:
x1 x2 y
x1 1.000 .919 833.000
x2 .919 1.000 .736
y .833 .736 1.000
Descriptive Statistics:
Mean Standard Deviation
Overheads ($000) 1438.000 198.441
Billable hours 52.750 12.502
Bid amounts ($000) 465.500 108.893
A team is preparing a bid for a new project. The project is expected to have $375,000 in overhead costs and 38 billable hours. How much should the bid be?
tan81004_10_c10_247-272.indd 260 2/22/13 3:44 PM
CHAPTER 10Section 10.4 Determining the Error in a Regression Solution
Formula 10.7 SEmest 5 sy "11 2 R2 yx1x22
y’ 5 a 1 b1x1 1 b2x2
a 5 My 2 b1Mx1 2 b2Mx2
b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b
5 .833 2 1.7362 1.9192
1 2 .9192 a108.893
98.441 b 5 1.115
b2 5 ryx2 2 ryx1 rx1x2
1 2 r2 x1x2 a
sy sx2
b
5 .736 2 1.8332 1.9192
1 2 9192 a108.893
12.502 b 5 21.655
a 5 465.5 2 (1.115)(438) 2 (21.655)(52.75) 5 64.431
y’ 5 a 1 b1x1 1 b2x2
5 64.431 1 1.115(375) 1 21.655(38) 5 419.666
The data suggest that the bid should be for $419,666. Notice in this problem that both the overhead costs and the number of billable hours for this project are well below the means for those two variables. In the case of billable hours, the value is more than a standard deviation below what is typical. However, the predicted value is only about half a stan- dard deviation below the mean. This is an example of the “regression to the mean” phe- nomenon that was first noted in Chapter 9. Extreme values of the predictor (x) variables tend to result in less extreme predicted values of the criterion, y.
10.4 Determining the Error in a Regression Solution As it was for simple regression, the statistic that quantifies the amount of error in a regression solution is another standard error value. In this application, it is the
standard error of the multiple estimate. The only difference between the standard error of the multi- ple estimate and the standard error of the estimate in Chapter 9 is that here, to accommodate multiple predictors, the correlation measure employed is the multiple correlation, Ryx1x2 rather than a bivariate correlation. That makes the formula for the standard error of the multiple estimate as follows:
Key Terms: The standard error of the multiple estimate is a measure of error variance in multiple regression.
tan81004_10_c10_247-272.indd 261 2/22/13 3:44 PM
CHAPTER 10Section 10.4 Determining the Error in a Regression Solution
Where
sy 5 the standard deviation of the criterion variable R2yx1x2 5 the square of the multiple correlation of the criterion variable with the two predictors
Examining the formula is a reminder that if there were no correlation between the predic- tors and the criterion variable (if Ryx1x2 5 0), the standard deviation of the criterion vari- able would indicate the standard error of the multiple estimate. If there were a perfect cor- relation between the predictors and the criterion variable (if Ryx1x2 5 1), then there would be no error (SEmest 5 0). Using the data from Another Multiple Regression Problem just above determining the standard error of the multiple estimate begins with a calculation for the multiple correlation statistic.
Ryx1x2 5 Å r2yx1 1 r
2 yx2 2 2ryx1ryx2 rx1x2
1 2 r2x1x2
Substituting in the Pearson correlation values from the last example results in the follow- ing calculation of the multiple correlation value.
Ryx1x2 5 .8332 1 .7362 2 21.8332 1.7362 1.9192
1 2 .9192 5 .836
In the course of calculating Ryx1x2 note that if what is probably the last step is omitted, taking the square root, the resulting value is R2, which is the value that the SEmest formula requires. In this instance, that value is R2yx1x2 5 .699. Recall that this value represents the proportion of the variance in the crite- rion (bid amount) accounted for by the predictors (overheads and billable hours). In this case, it is about 70%.
Relying now on Formula 10.7,
SEmest 5 sy"11 2 R2yx1x2 2
5 108.893"11 2 .6992 5 59.742
The SEmest seems not to be very informative by itself. In Chapter 11 we will include this value in a process that will give it more meaning.
Review Question D: The largest value the standard error of the multiple estimate can have is the standard deviation of the crite- rion variable. When does that occur?
tan81004_10_c10_247-272.indd 262 2/22/13 3:44 PM
CHAPTER 10Section 10.4 Determining the Error in a Regression Solution
Solidifying the Multiple Regression Procedure
One more example may provide a healthy repetition of the procedures involved in com- pleting a multiple regression problem. A safety engineer is examining the relationship between the number of hours shift workers ordinarily work in a week, the number of hours they typically sleep at night, and the number of days lost to injury or illness during the previous three-month period. The data for these variables are in Table 10.2. The safety engineer is going to use number of hours worked per week and the number of hours slept per night to predict the number of days lost to injury/illness. Specifically, the assessor will need to know:
a. Whether the bivariate correlations are statistically significant. b. How much of the variance in days lost to injury can be explained by a combina-
tion of the number of hours worked and the amount of sleep received. c. What the correlation is between amount of sleep and days lost to injury when
hours worked per week is controlled. d. How much time is lost to injury or sickness for someone who works a 40-hour
shift each week and sleeps 8 hours per night. e. What the standard error of the multiple estimate is for the solution.
The data and descriptive statistics for 10 randomly selected workers are in Table 10.2.
Table 10.2: Days lost to injury/illness when working 40 hours a week and sleeping 8 hours a night
Worker Hours Worked Hours Slept Days Missed
1 35 9 .5
2 40 8 0
3 40 7 1
4 40 8 0
5 50 5 4
6 45 6 3
7 38 8 .5
8 40 8 0
9 46 7 1
10 50 6 3
Mean Standard Deviation
Days missed (y) 1.30 1.476
Hours worked (x1) 42.40 5.082
Hours slept (x2) 7.20 1.229
tan81004_10_c10_247-272.indd 263 2/22/13 3:44 PM
CHAPTER 10Section 10.4 Determining the Error in a Regression Solution
The correlations are: Days missed (y) Hours worked (x1) Hours slept (x2)
Days missed (y) 1.0 .834 2.925
Hours worked (x1) 1.0 2.921
Hours slept (x2) 1.0
a. Checking the Table 8.1 critical value for 8 degrees of freedom indicates that all these correlation coefficients exceed r.05(8) 5 .632 and are statistically significant. Specifically, the number of hours worked are positively correlated and the num- ber of hours slept are negatively correlated with days missed. In other words, workers who worked longer hours and slept fewer hours missed more days due to injury/illness.
b. Item “b” above asks for the square of the multiple correlation of days lost with a combination of hours worked and hours slept. That correlation is as follows:
Ryx1x2 5 Å r2yx1 1 r
2 yx2 2 2ryx1ryx2rx1x2 1 2 r2x1x2
5 Å .8342 1 2.9252 2 21.8342 12.9252 12.9212
1 2 12.9212 2
5 .926
The square of the multiple correlation will indicate how much of days lost can be explained by a combination of hours worked and hours slept. In this case, R2yx1x2 5 .858. Thus, about 86% of the variance in days lost to injury/illness is accounted for by the number of hours worked and the number of hours slept.
c. Item “c” asks for the semi-partial correlation between the amount of sleep, the x2 variable, and the number of days lost, y, controlling for hours worked in the number of hours slept:
ry1x2x12 5 ryx2 2 1ryx1 2 1rx1x2 2
"1 2 r2 x1x2 5
2.925 2 1.8342 12.9212 "1 2 12.9212 2
5 2.403
The bivariate correlation between the amount of sleep the worker typically gets and the number of days off work is a very robust ryx2 5 2.925. How- ever, when what the number of hours of sleep variable has in common with the number of hours worked per week is controlled, the remaining correlation with number of days lost, while still negative, is a much more modest ry(x2x1) 5 2.403. Note that the hours worked and hours slept are highly negatively correlated, which makes intuitive sense. The more hours a worker works, the fewer the hours left for sleep. The problem that the semi-partial correlation reveals here is that a great deal of the information that the hours slept variable contributes has already been accounted for by the number of hours worked variable.
d. Item “d” asks how much time is lost to injury or sickness for someone who works a 40-hour shift each week and sleeps 8 hours per night. It requires a regression solution:
tan81004_10_c10_247-272.indd 264 2/22/13 3:44 PM
CHAPTER 10Section 10.5 Multiple Regression With Excel
y’ 5 a 1 b1x1 1 b2x2
b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b 5
.834 2 12.9252 12.9212 1 2 12.9212 2 a
1.476 5.082
b 5 2.035
b2 5 ryx2 2 ryx1 rx1x2
1 2 r2x1x2 a
sy sx2
b 5 2.925 2 1.8342 12.9212
1 2 12.9212 2 a 1.476 1.229
b 5 21.242
a 5 My 2 b1Mx1 2 b2Mx2 5 1.30 2 (2.035)(42.40) 2 (21.242)(7.20) 5 11.726
For the prediction of time lost for someone who works a 40 hour shift and sleeps 8 hours per night:
y’ 5 11.726 1 (2.035)(40) 1 (21.242)(8) 5 .39
The typical person who works a 40-hour shift and sleeps 8 hours per night will likely lose about 4/10 of a day to injury or sickness in a three- month period.
e. The standard error of the multiple estimate can be calculated as follows:
SEmest 5 sy "1 2 R2 yx1x2 5 1.476 "1 2 .9262 5 .557
Over-Fitting the Data and Shrinkage
Most statistical results tend to be relatively unstable when the sample size is small. This is a problem that isn’t unique to regression, but it can be a particular risk with a multiple regression solution based on 10 people. The sample size was kept small to make the cal- culations easy to manage, but it would be unrealistic to expect this solution to hold up with data from a new sample. The smaller the sample, the greater risk of the over-fitting problem discussed in Chapter 9. Recall that over-fitting means that a solution is tailored so closely to a particular sample that it does not predict well with new data. In other words, the solution does not generalize very well. Small samples tend to have descriptive characteristics that are not representative of the characteristics of the general population. As long as the correlations involved are statistically significant, the solutions generated will tend to be more accurate than random guesses, but they will lack the accuracy of solu- tions based on larger samples.
10.5 Multiple Regression With Excel
Excel makes the multiple regression procedure very straightforward. In fact the com-mands are almost identical to the commands for simple regression (Chapter 9). The days lost as a function of number of hours worked and the amount of sleep will serve as an illustration so that the result can be compared with the longhand calculations. Figure 10.3 provides a screenshot of the solution. Note that:
tan81004_10_c10_247-272.indd 265 2/22/13 3:44 PM
CHAPTER 10Section 10.5 Multiple Regression With Excel
• The values for the criterion variable, y, are listed in column A, with cell A1 containing the label for the variable, “days missed.”
• The values for x1 and x2 are listed with their labels in columns B and C, respectively.
• Once the data are entered, select Data Analysis from the Data menu and scroll down to regression.
• Highlight regression and click OK. • The cells containing the criterion variable are cells A1 to A11 if the label is
included, so in the box for input Y range indicate A1:A11. • The cells containing the two predictor variables and their labels are cells B1 to
B11 and C1 to C11. In the box for input X range indicate B1:C11. • Click the box for labels so that Excel treats A1, B1, and C1 as labels for the
variables rather than as the location for some of the measures. • To have the output where it does not overwrite the original data, indicate
something other than A1 to C11. In the screenshot, the output range was A15. • Click OK.
Figure 10.3: Completing a multiple regression problem on Excel
The table titled Summary Output provides a list of some of the same statistics calculated in the longhand procedure. The value for Ryx1x2 titled Multiple R is the same as the value determined earlier. Because the sample size is so small, Excel automatically calculates an Adjusted R Square value that would be safer to use if this solution were to be applied to new data. It is that adjusted value that is used to calculate the standard error of the multiple estimate statistic, titled Standard Error, which because of that adjustment has a slightly different value than the one calculated longhand.
tan81004_10_c10_247-272.indd 266 2/22/13 3:44 PM
CHAPTER 10Chapter Summary
As was the case with simple regression (Chapter 9), Excel automatically produces a test of the entire regression model. The ANOVA test is an analysis of whether there is a linear relationship between the criterion variable and a combination of the predictors. The sig- nificant F indicates that there is. However, recall that when there is just a single predictor as there was in the last chapter, the ANOVA result will always be the same as the test of the individual predictor in the final table. When there are multiple predictors, this changes.
The final table is where the constant value and the regression coefficients are presented. With minor differences, they have the same values as the longhand calculations. The table includes significance tests for the constant and the two regression coefficients. Although the ANOVA indicated that the entire model is statistically significant, this final table pro- vides an opportunity to test individual predictors. Note from the “P-value” column that the number of hours slept is a statistically significant predictor of the number of days that the worker will miss due to injury or illness (t 5 22.818, p 5 .026), but the hours worked is not (p 5 .755). Be reminded that these are both partial regression coefficients. Each of them quantifies what the particular predictor variable contributes to determining the value of y, controlling for the other predictor. The standard error and confidence interval statistics (last four columns) in this table were not addressed in the longhand calculations. A related topic, the confidence intervals of the prediction, will come up in Chapter 11.
Since the partial regression coefficient for hours worked is not a significant predictor of the number of days the worker will miss, would it be a good idea to delete it from the analysis? In multiple regression, the whole is sometimes greater than the sum of its parts. As long as there is a statistically significant correlation between the predictor and the criterion, the resulting partial regression coefficient, which may not be statistically significant by itself, can still make a contribution to the prediction of y. This is suggested by the fact that the Ryx1x2 correlation is larger than either of the two bivariate correlations of the predictors with the criterion. While what the hours worked variable contributes may be minor, there is something there, and the better course is to leave it as part of the analysis. For example, while there might not be a direct relationship between hours worked and days missed, it is possible that there is an indirect relationship so that hours worked predicts hours slept, which in turn predicts days missed. More advanced mul- tiple regression procedures allow for testing such types of “mediated” relationships, but the point is that not everything hinges on whether a particular partial regression coef- ficient is statistically significant.
Chapter Summary
Regression is a natural extension of correlation. The idea that variables correlate because they share common information is a powerful concept, and in the case of two vari- ables was the basis for the simple regression discussion in Chapter 9. Here that concept was extended to more than two variables. Information from two (or more) predictors can be combined to make a more accurate prediction of a third variable (Objective 3) than one predictor can provide. The statistic that provides an overall view of the relationship among multiple predictors and a criterion is multiple correlation (Ryx1x2), in the case of two predictor variables. Specifically, multiple R indicates the degree to which a combina- tion of x variables correlates with a y variable. In doing so, it provides some evidence of how well the x variables might predict the y variable (Objective 1).
tan81004_10_c10_247-272.indd 267 2/22/13 3:44 PM
CHAPTER 10Answers to Review Questions
Multiple R does not indicate how much of the information among multiple x variables might be redundant, however. This is an important issue when multiple predictors are correlated with each other, as they frequently are. The semi-partial correlation (ry(x2x1)) determines the correlation of each predictor with the criterion, while controlling the degree to which that predictor also correlates with other predictors. In doing so, semi- partial correlation provides a way to determine the degree to which multiple x variables contain the same information, a repetition that must be eliminated if predictions based on multiple regression are to be accurate (Objectives 2 and 3). Although we did not pursue relationships more complex than those involving two predictors, the semi-partial correla- tion can be extended to control for redundant information in any number of predictors.
Adding predictors will reduce unexplained (error) variance, as long as the predictors are significantly correlated with the criterion variable, and so long as each predictor contrib- utes something unique (Objective 3). Unexplained variance is never entirely eliminated, however. Even with the most relevant predictors, there is likely to be an error component. As with bivariate regression, error is calculated with the standard error of the estimate, renamed the standard error of the multiple estimate when there is more than one predictor variable.
Also similar to bivariate regression, a regression solution, particularly a solution based on a small sample, can be difficult to generalize. Solutions that fit the data from which they were calculated substantially better than they predict for new data are said to over-fitted to the sample. The number of predictors often compounds the problem of over-fitting, something that underscores the value of large samples in regression solutions.
There is great value in the longhand calculations undertaken in this chapter. Working through each component can be very helpful to illustrating how each component of the multiple regression solution is completed, and why each step is necessary. While the logic involved is less clear, the entire multiple regression task is handled much more efficiently with Excel (Objective 4). At least in the PC version (sorry, Mac users) Excel makes short work of calculating multiple R, the regression coefficients, as well as the constant value. The output even includes significance tests for the individual coefficients.
Answers to Review Questions
A. A multiple correlation will determine the relationship between a criterion and a combination of any number of predictors.
B. If multiple variables are each correlated with a criterion, combining the mul- tiple variables in a multiple correlation (R) statistic will result in a multiple correlation value larger than any of the bivariate correlations.
C. Partial regression coefficients are necessary because multiple predictors typi- cally contain some of the same information, a redundancy that must be con- trolled if the prediction is to be accurate.
D. The standard error of the multiple estimate will equal the value of the stan- dard deviation of the criterion variable only if R 5 0.
tan81004_10_c10_247-272.indd 268 2/22/13 3:44 PM
CHAPTER 10Chapter Formulas
Chapter Formulas
Formula 10.1 Ryx1x2 5 Å r2yx1 1 r
2 yx2 2 2ryx1 ryx2 rx1x2 1 2 r2 x1x2
This is the formula for multiple correlation. It is used when the question is of the relation- ship between two or more x variables and a y variable.
Formula 10.2 ry1x2x12 5
ryx2 2 1ryx1 2 1rx1x2 2 "1 2 r2 x1x2
This is the formula for semi-partial correlation. It indicates the correlation between y and x2 when the influence of x1 is ruled out of x2.
Formula 10.3 y’ 5 a 1 b1x1 1 b2x2
This is the formula for multiple regression when there are two predictors (x1 and x2).
Formula 10.4 a 5 My 2 b1Mx1 2 b2Mx2
This is the formula for the intercept, or the constant value in multiple regression.
Formula 10.5 b1 5 ryx1 2 ryx2 rx1x2
1 2 r2x1x2 a
sy sx1
b
This is the formula for the first regression coefficient in a multiple regression problem that involves two predictors.
Formula 10.6 b2 5 ryx2 2 ryx1 rx1x2
1 2 r2x1x2 a
sy sx2
b
This is the formula for the second regression coefficient in a multiple regression problem with two predictors.
Formula 10.7 SEmest 5 sy"11 2 R2 yx1x2 2
The standard error of multiple estimate is the estimate of rediction error in a multiple regression problem.
tan81004_10_c10_247-272.indd 269 2/22/13 3:44 PM
CHAPTER 10Management Application Exercises
Management Application Exercises
Unless otherwise stated, use p 5 .05 in all your answers.
A national chain of tax preparation offices selects tax accountants based on a combination of their analytical and interpersonal skills. You have been hired to assess the effectiveness of this selection strategy in terms of the extent to which these skills can predict customer loyalty. Customer loyalty is measured as the number of customers who request the same accountant by name two years in a row.
Analytical Interpersonal Loyalty
Analytical
M 5 34.701
s 5 5.935
.505 .788
Interpersonal
M 5 11.661
s 5 1.734
.649
Loyalty
M 5 66.338
s 5 8.772
1. Using the data from the table above, determine the multiple correlation (R) of cus- tomer loyalty (y) with a combination of analytical (x1) and interpersonal skills (x2).
2. What is the correlation between analytical skills and customer loyalty if interper- sonal skills are controlled in analytical skills?
3. What is the correlation between interpersonal skills and customer loyalty if ana- lytical skills are controlled in interpersonal skills?
4. If a new tax accountant has an analytical score of 39 and an interpersonal score of 10, what customer loyalty score is predicted?
5. With reference to item 4, how much does customer loyalty change when analytical ability increases by 1.0?
6. What will be the SEmest for the solution to item 4?
7. From the data above, what customer loyalty score is the best prediction for a tax accountant with an analytical score of 27 and an interpersonal score of 8?
8. If length of service and age are both significantly correlated with job satisfaction, why must any correlation between service and age be a factor in a regression solution?
tan81004_10_c10_247-272.indd 270 2/22/13 3:44 PM
CHAPTER 10Key Terms
9. With reference to item 1, what does the R2 value indicate?
10. What customer loyalty score is the best prediction from an analytical score of 45 and an interpersonal score of 15?
Why is the predicted value less extreme than either of the x values?
Key Terms
• Multiple regression is regression employing more than one predictor to estimate the value of a criterion.
• Multiple correlation, or multiple R, indicates the strength of the relationship between a criterion and multiple predictors.
• Semi-partial correlation gauges the relationship between any two variables, con- trolling the influence of one or more additional variables on one of the two variables.
• Because multiple predictors in most business problems are correlated with each other as well as with the variable to be predicted, partial regression coefficients must be calculated to control information present in a particular predictor already included in the model by other predictors.
• The standard error of the multiple estimate is the measure of error variance in multiple regression problems. Its magnitude is directly related to how well predic- tors correlate with the criterion and to how much variability there is in the criterion.
tan81004_10_c10_247-272.indd 271 2/22/13 3:44 PM
tan81004_10_c10_247-272.indd 272 2/22/13 3:44 PM
