BUS 308 DQ 1 & 2 *****ORIGINAL WORK ONLY*****
loka9502
bus308_chapter_09.pdf
9
Simple Regression: Predicting One Variable From Another
Learning Objectives
After reading this chapter, you should be able to:
• Develop the connection between correlation and prediction.
• Explain what is needed to fit a regression line.
• Predict a value based on the measure of a related variable.
iStockphoto/Thinkstock
tan81004_09_c09_219-246.indd 219 2/22/13 3:43 PM
CHAPTER 9Introduction
Chapter Outline
9.1 The Language of Regression Which Is the Predictor? Picturing Regression About Linearity Coping With Less-Than-Perfect Correlations Understanding the Least Squares Criterion
9.2 Ordinary Least Squares Regression With One Predictor The Regression Equation Calculating a Regression Solution Practicing the Regression Solution Interpreting the Results Error The Standard Error of the Estimate Another Regression Problem
9.3 Regression with Excel Shrinkage and Over-Fitting the Sample Regression to the Mean
9.4 The Requirements for Ordinary Least Squares Regression
Chapter Summary
Introduction
Predicting the future is a central requirement in business decision making. Managers use existing data to predict the future values of other variables of interest. For example, marketing data are used to predict future sales. Financial analysts and investors also use predictions. Financial ratios, for instance, are used to predict future stock prices. Although these predictions may not be perfectly accurate, they are usually more accurate than hap- hazard, uninformed decision making. In statistics, if two variables are connected, the use of one variable to predict another yields better-than-chance estimates. Managers who use statistics to predict the variables they are interested in can make more educated guesses about the future, which can inform better decisions.
Recall from Chapter 8 that when two variables are correlated, they co-vary, which means that they move together. Correlated variables share some variance, which means that they contain some of the same underlying characteristics and thus reflect some of the same information. For example, if organizational commitment and intentions to quit are cor- related (of course negatively), it is because to some degree they both measure the same underlying characteristic of desire to identify with the organization. The more highly two variables are correlated, the greater the quantity of whatever is measured that the two variables have in common.
tan81004_09_c09_219-246.indd 220 2/22/13 3:43 PM
CHAPTER 9Introduction
Also recall that the coefficient of determination (r2) indicates the proportion of one vari- able that can be accounted for by the other. If organizational commitment and intentions to quit are correlated r 5 2.6, then r2 5 .36. The coefficient of determination tells us that 36% of whatever turnover intentions measures can be explained by variations in orga- nizational commitment. Another way to say this is that the coefficient of determination indicates how much information two correlated variables have in common.
If correlated variables share information, it seems logical that if we had information about the value of one of those variables, we could make a better-than-chance prediction of what the value of the other would be. For example:
• Because age and height are correlated for teenagers, knowing how old some- one is should allow for a reasonably accurate estimate of how tall that person is. Or perhaps turning that analysis around, information about how tall a teen is should allow a reasonable prediction of age.
• If education and income are correlated, and someone’s years of schooling are known, it seems feasible that someone should be able to make an educated guess about the person’s level of income.
• If a mutual fund invests heavily in the banking sector, and banks’ profits are related to the prime rate, the value of funds in the mutual fund should be predictable from the prime rate.
Regression allows a precise mathematical prediction of the value of one variable from the value of another with which it is correlated. Karl Gauss, the man who defined the charac- teristics of the normal (Gaussian) distribution, began developing the mathematics behind regression in the early part of the 19th century. Many others have also made contributions to this kind of quantitative analysis. Their work allows experts in a variety of fields to use regression procedures in their decision making.
• Meteorologists use changes in barometric pressure to predict the kind of weather that is coming. Because drops in barometric pressure are predictors of violent storms in the Great Plains states and in the southeastern part of the United States, meteorologists watch particularly for dramatic drops in air pressure.
• Oddsmakers use data on a team’s past performance, any injuries to key players, and the quality of the opponent to predict the outcomes in games.
• Psychologists can use genetic and social factors, including the history of alcohol abuse in a family, to predict an individual’s predisposition to abuse drugs.
• Closer to our subject, economists gather data on unemployment rates, whole- sale inventories, and consumer spending to predict the rate at which the economy will grow. This approach is effective because each of those variables correlates with economic expansion.
Each of the above four scenarios is possible because of correlations between variables. Correlations pave the way for prediction. In each of the bulleted examples, someone could mathematically calculate the value of one variable based on the value of another. It is even possible to determine the probability that a specified result will emerge. Prediction is par- ticularly important for managers because it allows for proactive planning. Rather than waiting for some important event to emerge and then reacting to it, the probability and
tan81004_09_c09_219-246.indd 221 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
intensity of the event can be anticipated with some precision, and then managers can plan and initiate appropriate courses of action. Prediction is a basis for sound decision making.
9.1 The Language of Regression
There are many types of regression procedures. The concepts involved and much of the language is consistent in most instances. One element that is common to all regression procedures is that when variables have information in common, the measure of one can be used to predict the measure of the other. In the some of the more common statistical pack- ages, the variable to be predicted is called the “dependent variable.” The variable used to make the prediction is called the “independent variable.”
Those terms were part of the discussion of the z, t, and F tests under the hypothesis of difference, and we did not find them particularly objectionable there. However, regres- sion is based on correlation, and there is a particular danger in correlation discussions of obscuring the distinctions between correlation and causation, as we noted in Chapter 8. To help avoid this “slippery slope,” we will adjust the language in the regression discussion. Rather than dependent and independent variable, as common as those terms may have become, the references here will be to the criterion variable and predictor variable. The criterion variable is the one whose value is predicted. The predictor variable, of course, is the one employed to make the prediction. This language will help minimize the risk of drifting toward assumptions about causal relationships. The point is not that the predictor does not cause the criterion variable. That may be exactly the case. For example, it would be difficult to argue that rising interest rates do not directly impact consumer spending— that interest rates do not cause spending to decrease. The point is that the correlation, which is the founda- tion for regression, is not usually sufficient by itself to establish cause.
Although the terms “criterion” and “predictor” are used here for descriptive purposes, some shorthand indicators will be needed as well. The symbols used in regression will be the same that were used for correlation problems in the last chapter, “x” and “y.” The difference is that in Chapter 8, which variable was x and which was y was an arbitrary assignment. Here the designation will be more particular:
• x for the predictor variable, and • y for the criterion variable.
Which Is the Predictor?
The confusion that can occur when equating correlation with causation is increased by recognizing that either variable in a significant correlation can be used to predict the other. If there is a correlation between interest rates and the level of consumer spending, it means that each variable is equally related to the other.
Key Terms: The correlation between variables makes it possible to predict the value of a criterion variable from the value of a predictor variable.
tan81004_09_c09_219-246.indd 222 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
• Someone could predict consumer spending from the interest rates banks charge. • But turn it around, and the problem becomes one of predicting the interest
rates that banks charge from the level of consumer spending.
Either variable in a statistically significant correlation can predict the other. From the point of view of the mathematics involved, which variable is used to predict which does not matter. However, sometimes practical considerations dictate which variable must be the predictor and which must be the criterion. Even when the correlation is known, some- times new data on one of the variables is more difficult to gather than it is for the other. In such cases, the more accessible variable may become the predictor and then the less avail- able variable may be predicted rather than measured directly.
Of the two variables in the bulleted example, consumer spending is probably more dif- ficult to measure than the interest rates that banks charge. Interest rates can usually be determined from the financial section of the newspaper, or from an online news source. Consumer spending is more involved, including information from several different sources, so in such circumstances it probably makes more sense to use the bank rates, which are easier to access, as the predictor for the consumer spending variable.
Sometimes there are considerations besides the ease with which a variable is measured. Perhaps social skills among servers at a restaurant chain are correlated with average daily revenue. Although either variable is equally correlated with the other, revenue probably is the more important variable from the manager’s point of view. Mathematically, as long as the two variables are correlated, revenue can predict social skills among the staff members just as accurately as social skills predict revenue, but social skills are probably viewed as a means to an end—the end being revenue. Thus, it makes sense to use social skills as the predictor and revenue as the criterion.
Finally, for management decision-making purposes, it is often more useful to designate the variable that can be manipulated as the predictor. For example, knowing that servers’ social skills can predict revenues, the management of the restaurant chain can create a training program to enhance servers’ social skills. If the program is effective and servers’ social skills improve, it can be predicted that revenues will also rise. On the other hand, it is neither feasible nor particularly logical to attempt to improve servers’ social skills by increasing revenues.
Picturing Regression
Visually, correlation and regression are very similar, because they are both based on the hypothesis of association. The tool that was used to picture correlation in Chapter 8 was the scatter plot, which is an easy way to graphically represent regression proce- dures as well. Recall that each point in a scatter plot represents a pair of measures. The distance from the vertical axis is the measure of the x variable, and the distance from the horizontal axis is the measure of the y variable. When they are used for regression, the assignment of the x and y variables is more specific. In regression:
• y indicates the criterion variable and is plotted on the vertical axis. • x indicates the predictor variable and is plotted on the horizontal axis.
tan81004_09_c09_219-246.indd 223 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
Remember that when there is a linear relationship between two variables (one of the requirements for the Pearson Correlation), the “pattern” formed by the points is generally from lower left to upper right for positive correlations and from upper left to lower right for negative correlations. In either of those diagonal patterns, highly correlated variables are indicated by relatively little scatter along the length of the pattern.
Since correlation and regression are both based on the concept of association between variables, and since correlation came up in the last chapter, it is a good platform from which to introduce regression. Suppose that the research and development (R&D) depart- ment housed in a company that manufactures household cleaning products has received budgets that have grown consistently over a 20-year period. Happily for those in the R&D department, sales have also grown over the same period. Put more directly, the two vari- ables are correlated. A positive correlation suggests that as R&D budgets increase, so do sales, although there tends to be an interval between the time when the product is com- pleted and the time required to prepare it for retail sales. Because the correlation between R&D budgets and sales of household products is positive, it seems safe to say that the larger the R&D budget is, the greater sales revenue will be once the products are available to consumers.
At this point, those in the R&D department might take the conceptual “next step” and argue that R&D budgets will predict sales volume two years hence. That hypothesis makes the R&D budget the predictor variable, x, and sales revenue the criterion variable, y. With R&D money rounded to the nearest $100,000, and sales revenue in $millions, the data are as follows:
R&D Budget (x) Sales Revenue After Two Years (y)
1. 1 1.5
2. 2 1.8
3. 3 2.2
4. 3 2.0
5. 5 2.0
6. 5 2.1
7. 7 2.4
8. 7 2.2
9. 8 2.4
10. 10 2.7
11. 10 2.6
12. 11 2.9
13. 13 3.0
14. 15 3.0
15. 16 3.1
(continued)
tan81004_09_c09_219-246.indd 224 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
(continued)
R&D Budget (x) Sales Revenue After Two Years (y)
16. 16 2.7
17. 16 3.3
18. 17 3.0
19. 18 3.4
20. 20 4.0
If the data are placed in columns in an Excel spreadsheet, just as they are in the preceding table (the numbering down the left is automatic in Excel and not considered one of the data columns), the commands for creating the scatter plot are then simply Insert and then Scatter. With some additions to label the axes and title the graph, the result is Figure 9.1.
Figure 9.1: A scatter plot for the relationship between R&D budget and sales revenue
To complete the scatter plot by hand rather than having Excel produce it, draw the verti- cal and horizontal axes of the graph at right angles to each other. Number the x-axis from 0 at the point of the y-axis to 20 in equal increments to the left (numbering by 2s might be helpful); number the y-axis from 0 at the point of the x-axis to 4.0 in equal increments. For each pair of data, indicate the confluence of the sales revenue (vertical axis) with the R&D budget two years earlier (horizontal axis). Once all pairs of data are plotted, four things will be evident:
R&D Budget in Hundreds of Thousands
S a le
s in
M ill
io n s
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0 0 5 2510 15 20
tan81004_09_c09_219-246.indd 225 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
1. There is a correlation between the R&D budget and the sales revenue two years later. If this were not the case, the dots would have no particular pattern and would scatter randomly throughout the graph.
2. The fact that the pattern is from lower left to upper right indicates that the relationship between these two variables is positive; as one increases, so does the other. If the relationship had been negative, the data pattern would have been from the upper left to the lower right. This might occur, for example, if someone plotted the relationship between the number of vacation and sick days taken by the R&D staff and their productivity as a group. In that case, as the value of one variable increases, the other decreases.
3. The correlation is substantial, indicated by the fact that there is not much scatter in the scatter plot.
4. The relationship appears to be linear. There is no suggestion in the graph that the relationship changes from positive to negative, or even to neutral at some point.
About Linearity
The last point about the relationship between x and y being linear is particularly impor- tant. Recall that one of the requirements for the Pearson Correlation in Chapter 8 is that the relationship must be linear. The type of regression discussed in this chapter is based
on the Pearson Correlation and so likewise depends upon a linear relationship between x and y. There are other regres- sion approaches when the relationship between variables is not linear, but in the present example, linearity is essential.
Because of the linear relationship between variables, any sta- tistically significant correlation allows a straight line to be scribed through the data points so that it is as close as possible
to as many of the data points as possible. The result might look like the line through the data points in the graph in Figure 9.2 (a).
Review Question A: What is the visual evidence in a scat- ter plot for a weak correlation?
tan81004_09_c09_219-246.indd 226 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
Figure 9.2: The regression line
That line through the data points is called a regression line. If it is positioned so that it is as close as possible to as many of the 20 data points as it can be and still be a straight line, it can be used to determine any value of y from a specified value of x (or for that matter, any value of x from a specified value of y). For example, someone using the graph can select any value of x along the horizontal axis, go vertically from that x value up to the line drawn through the dots, and then move left horizontally from the line to where the y-axis is. The value of the y variable at the point at which the y-axis is encountered will be the
predicted value of y for the specified x value. This is shown in Figure 9.2 (b).
Although the R&D budget was not $1.25 million for any of the 20 years in the data set, the position- ing of the regression line allows someone to predict what sales revenue would likely be 2 years later if that were the R&D budget. Perhaps the CEO asks the
R&D director, “If your budget for next year is $1.25 million, how much sales revenue can you predict 2 years from now?” If the regression line is positioned correctly, the researcher can find 1.25 on the x-axis, travel vertically up to the regression line, and then move left to
R&D Budget in Hundreds of Thousands
0 5 2510 15 20
5
4
3
2
1
0
S a le
s in
M ill
io n s
(a) A Regression Line Through a Scatterplot
The Relationship Between R&D Budget and Sales
R&D Budget in Hundreds of Thousands
0 5 2510 15 20
5
4
3
2
1
0
S a le
s in
M ill
io n s
(b) Using the Regression Line to Determine y from x
The Relationship Between R&D Budget and Sales
Key Terms: The regression line is the visual representation of the regression equation. The line’s position in a graph is fixed by the intercept and slope values.
tan81004_09_c09_219-246.indd 227 2/22/13 3:43 PM
CHAPTER 9Section 9.1 The Language of Regression
the y-axis to determine that $1.25 million in R&D predicts sales revenue of about $3 mil- lion. This type of prediction is of critical importance in budget allocation decisions and in a good many other business situations.
Coping With Less-Than-Perfect Correlations
Although the R&D director in the above example could predict that a budget of $1.25 million would produce sales of “about” $3 million, a number of factors might affect the accuracy of this prediction. The director cannot be precise about future sales because the correlation between the two variables is less than perfect. Future sales of $3 million might be the best possible prediction given the data, but there is likely to be some margin of error. Maybe for a budget of $1.25 million, sales of $2.9 million would turn out to be a bet- ter prediction, or perhaps $3.1 million.
A glance at the scatter plot indicates that indeed there is some scatter. For example, there were three years when the budget was the same $1.6 million, but the sales revenues were different. This indicates that factors besides the size of the R&D budget affect sales. Left unmeasured in this analysis are changes in general economic conditions, changes in the quality of competing products, the budget devoted to advertising, and a host of other variables. Such variables aside, as long as the correlation between the predictor and crite- rion variables is statistically significant, the predicted value of y from the value of x will be more accurate than a number of random guesses.
Because correlations are never perfect, prediction error is inevitable, and a certain amount of it is tolerable. No one sues the United States Commerce Department if the prediction for job growth is off by 1⁄10 of a percent for the year. People do not ordinarily petition the television station to fire the weather- man when the forecast high for the day is wrong by a couple of degrees. It is helpful, however, to have some measure of how extensive the error is likely to be. Later in this chapter, procedures will be investigated that estimate the amount of error, a value that when coupled with the predicted value of y provides a useful gauge of prediction accuracy.
Understanding the Least Squares Criterion
Although the scatter plot provides a helpful way to introduce the concept of regression, relying on the scattered points in a graph to predict one variable from the other is nei- ther practical nor as accurate as analysts need to be. Rather than a visual guess about the best fit of the regression line, its positioning is actually determined mathematically with a regression equation. For the particular kind of regression in this chapter, the equation must satisfy what is called the least squares criterion, which means that the regression line must be positioned so that the sum of all possible prediction errors has its lowest pos- sible value.
Look at Figure 9.2 again. If the correlation between R&D budgets and later sales were per- fect, the data points would form a straight line. For each value of x there would be just one corresponding value of y. With a perfect correlation, those three years in which the R&D
Review Question B: If for a particular data set all the prediction errors were calculated for an infinite number of regression proce- dures, what would be the value of their sum?
tan81004_09_c09_219-246.indd 228 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
budget was $1.6 million would all have the same associated sales revenues values. To “fit” a straight line to data that do not form a straight line, there must be some compromise. The line must be as close as possible to all the data points and remain a straight line.
Because an element of error in the regression solution is unavoidable, at least some of the predictions are going to be inaccurate, and the degree of accuracy may vary from predic- tion to prediction. If the actual value of the criterion variable were known, error would be the difference between the predicted value of that variable, symbolized by y’ (y “prime”), and the actual value, symbolized by y. If the difference between y and y’ is not zero, there is some prediction error. Those y 2 y’ differences are termed residual scores.
If someone were to rely on Figure 9.2 to make a series of predictions and then go through company records to determine what the actual sale values were for those 20 years, any dif- ference between what sales were predicted to be and what they actually were would be reflected in the residual scores. If all of those residual scores were added up, what would be their total? This is how that question looks in symbols:
(y 2 y’) 5 ?
When several predictions are made, some of the residual scores turn out to be positive (the actual value of y will be larger than the predicted value, y’) and some are negative (the actual value of y is smaller than its predicted value). Because the positive and negative residual scores offset each other when there are several, summing the residual scores from several predictions would equal zero. The positive and negative residual scores would cancel each other out.
That zero sum is unhelpful to someone wishing to know how much error there is in a regression solution. To provide an indicator of the prediction error, it would be more telling to square and then sum the residual scores. When the regression line is located in a position of best fit, the sum of those squared residual scores has its lowest possible value. Such a solution minimizes prediction error and satisfies the least squares criterion. For that reason, this particular form of regression is called ordinary least square regression.
9.2 Ordinary Least Squares Regression With One Predictor
Theoretically, there can be any number of predictor variables in a regression analysis. The multiple predictors are designated, x1, x2, x3 . . ., and so on. Multiple regression, which is regression with multiple predictors, is the subject of Chapter 10. The focus here is on regression with just one predictor. It is sometimes called simple (bivariate) regression since only two variables—a single predictor and a single criterion variable—are involved.
The regression equation provides the math that is needed to position the regression line. The process requires answers to two questions:
Key Terms: Ordinary least squares regression procedures minimize residual scores, the differences between the pre- dicted and actual values. This requirement is called the least squares criterion.
tan81004_09_c09_219-246.indd 229 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
Where
y’ 5 the predicted value of the criterion variable a 5 the intercept b 5 the slope of the regression line x 5 the value of the predictor variable e 5 prediction error
The equation indicates that the predicted value of y is based on the value of the intercept, plus the product of the slope times the predictor, plus some error. Calculating the amount of error in the regression problem is actually done separately from estimating the value of y. The “e” here in the equation is mostly a reminder that absent perfect correlations between x and y, there will always be some error in the prediction. Having noted the per- sistence of error, the e will be removed from the equation from this point forward, making the formula:
y’ 5 a 1 bx
1. At what point does the regression line cross the y-axis in the graph? 2. How much change is there in the criterion variable (y), when the predictor vari-
able (x) increases by 1.0?
The answer to the first question will produce a value for the regression intercept. The name indicates that this value is the point on the y-axis of the graph where the axis is intercepted by the regression line. Since the y-axis occurs at the point where x 5 0, the intercept is “the value of y when x 5 0.” Figure 9.2 makes it look like the regression line crosses the y-axis at about y 5 1.2. In fact the intercept value is calculated rather than visually estimated, but as a conceptual exercise, it can be helpful to look at the graph and make a guess about the value.
The answer to the second question is a value representing the slope of the line. With refer- ence once more to Figure 9.2, how much does the criterion variable (y) appear to change when the value of the predictor variable (x) increases by 1.0? How much does y change vertically when x is increased by 1.0? Perhaps a reasonable estimate is that y increases by about .2 every time x goes up by 1.0. This slope value gauges how radically the regression line inclines or declines.
The Regression Equation
The bivariate regression equation has this form:
Formula 9.1 y’ 5 a 1 bx 1 e
Key Terms: Simple, or bivari- ate regression involves deter- mining the value of the criterion when the single predictor is 0 (the intercept) and calculating the change in the criterion of increasing the predictor by 1.0 (the slope).
tan81004_09_c09_219-246.indd 230 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
Where
b 5 the slope of the regression line, or the regression coefficient rxy 5 the correlation coefficient for x and y sy 5 the standard deviation of the criterion variable, y sx 5 the standard deviation of the predictor variable, x
Calculating a Regression Solution
It was estimated earlier, based on the graph, that if the R&D budget was about $1.25 mil- lion, the sales two years later would be about $3 million. How close was that visual esti- mate to the calculated regression solution?
The steps involved in completing the regression solution are as follows:
1. Calculate the means and standard deviations for x and y. 2. Calculate the correlation of x and y. 3. Calculate the slope of the line, or the regression coefficient, b. 4. Calculate the regression intercept, a. 5. Solve the regression equation for the value of y’.
Where
a 5 the intercept b 5 is the new part, the slope of the regression line My 5 the mean of the criterion variable, y Mx 5 the mean of the predictor variable, x
The intercept value is found by taking:
1. the mean of the criterion variable 2. minus the slope value, once that is determined, times the mean of the predictor
variable.
The slope of the line has this formula:
The intercept value, a, and the slope, b, each have their own equations, but most of the terms involved are statistics that are already familiar. First the intercept:
Formula 9.2 a 5 My 2 bMx
Formula 9.3 b 5 rxy (sy/sx)
tan81004_09_c09_219-246.indd 231 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
1. For the two variables, verify that:
R&D Budget (x) Sales Revenue (y)
Means 10.150 2.615
Standard Deviations 5.941 .613
2. Use the Correlation procedure in Excel Data Analysis (Data → Data Analysis → Correlation). Indicate the cells containing the input data, indicate where the output is to go, and click OK. Verify that:
rxy 5 .946
Checking this value against the critical value from the table, rxy.05(18) 5 .444 indi- cates that the correlation of R&D budget with sales 2 years later is statistically significant. The correlation indicates that regression procedures will result in a better-than-chance prediction of the value of y for a given value of x.
3. The slope of the line is as follows:
b 5 rxy (sy/sx)
5 .946(.613/5.941)
5 .098
This value indicates that y increases by .098 units for every 1.0 unit increase in x. Earlier the estimate was that the slope of the line in the graph in Figure 9.2 might be about .2, an overestimate, as it turns out.
4. The regression line intercept is:
a 5 My 2 bMx
5 2.615 2 (.098)(10.15)
5 1.620
This value indicates that when x 5 0, y 5 1.620. The estimate based on the visual “best fit” was that the intercept would be at about y 5 1.2. That esti- mate, too, was off the mark.
5. What are sales likely to be if the R&D budget is about $1.25 million? The estimate based on a visual placement of the regression line in Figure 9.2 was of sales of about $3 million. Check this “guesstimate” by solving for y’ and comparing the answer to that estimation. Substituting x 5 $1.25 mil- lion in the regression equation (using 12.5, since the values were reported in $100,000 above):
tan81004_09_c09_219-246.indd 232 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
y’ 5 a 1 bx
5 1.620 1 (.098)(12.5)
5 2.845
Based on the data available for 20 years, the best sales prediction following a year in which the research and development budget was $1.25 million is $2.845 million, which, as it turns out, is not too far from the earlier prediction of $3 million.
The bivariate regression procedure reduces to this: Data from two significantly correlated variables can be used to predict what is unknown about one variable from what is known of the other. Making the prediction requires some descriptive sta- tistics, including the means and standard deviations of the two variables, and the coefficient of their correlation, in order to calculate three relatively straightforward equations for (1) the slope, (2) the intercept, and (3) the predicted value of y.
Practicing the Regression Solution
Note that once the calculations of the slope (b) and the intercept (a) are completed, the value of y can be predicted for any value of x. For example, what sales can be predicted 2 years later for a year in which the R&D budget was $3.0 million?
y’ 5 a 1 bx
5 1.620 1 (.098)(30)
5 4.560
Based on the data available for the 20 years, an R&D budget of $3 million will result in sales of $4.56 million 2 years later.
What would happen if the company were sold, and someone interested in short-term profits eliminated the R&D budget for a year? What would be the subsequent impact on sales? Would sales likewise fall to zero?
y’ 5 a 1 bx
5 1.620 1 (.098)(0)
5 1.620
Even with no money devoted to research and development, the solution indicates that a $0 sales year is unlikely. But this is an answer that was already available. Note that the intercept is defined as the value of y when x 5 0. In this case, x 5 0 is an R&D budget of 0. Just examining the intercept value of the intercept would have answered the question.
Review Question C: What statistic indi- cates the value of y when x 5 0?
tan81004_09_c09_219-246.indd 233 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
Interpreting the Results
Because b 5 rxy (sy/sx), the value of the regression slope is actually a particular proportion of the ratio of sy to sx. The specific proportion is determined by the strength of the corre- lation. It is largely academic because the potential for a correlation of 1.0 between any 2 variables in a business problem is so remote, but if the correlation between the 2 variables is perfect (rxy 5 1.0), the slope becomes 1 times that ratio of sy to sx. As the correlation diminishes, the value of the slope is a decreasing proportion of that ratio. If the correlation is .50, for example, the slope’s value will be half the ratio of sy to sx.
The slope of the regression line is not always a positive value. If the correlation between the predictor and criterion variables is negative, the slope will be negative, which means that for every 1.0 increase in x, the value of y declines; the regression line will move down- ward in the scatter plot from left to right. For example, the number of conflicts between a chief executive officer and the corporation’s governing board is used to predict the length of the CEO’s tenure. As the number of conflicts increase, the length of the executive’s ten- ure probably decreases. The slope in such a case would be a negative value.
A negative value for the slope is not unusual.
• Perhaps the number of hours that business students spend per week playing video games is used to predict their grade averages during their freshman year. Since the correlation is likely negative, the regression slope probably declines from left to right.
• The number of days of sick leave will likely be negatively correlated with work performance, yielding a negative slope and a downward-sloping regres- sion line.
Error
The sales that were associated with a budget of $1.25 million were predicted to be $2.845 million. It is the best prediction that the available 20 pairs of data allow. Twenty years of data are a great deal, but the results still represent only a sample, and there is always a possibility that because of sampling error, the data may be unlike the population of all relationships between R&D budgets and subsequent sales.
However, even if the data set were exhaustive and included data for single case of the rela- tionship between R&D and sales for every single producer of household clearing prod- ucts, the answer still would not necessarily be precisely accurate for one individual case. The regression equations allow the best prediction given the data, but some prediction error is inevitable as long as the correlation between predictor and criterion is , 1.0.
This reality does not mean that the regression process is flawed. It just means that it is imperfect. It is similar to the point made about calculating the various standard error sta- tistics for t-tests. The fact that there is error in the test statistic does not imply that mistakes were made. It indicates that there is variability in the data that is unaccounted for. It is the same with regression procedures. Meeting the least squares criterion minimizes error, but error cannot be entirely eliminated.
tan81004_09_c09_219-246.indd 234 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
Where
SEest 5 the standard error of estimate sy 5 the standard deviation of the criterion (y) variable
r2xy 5 the square of the correlation coefficient
For the problem where R&D budgets predicted sales, the standard error of the estimate is as follows:
For any data set, where a large number of predictions is made, some of the predictions will be too high, some will be too low, and a few might be correct. The fact that all these predic- tion errors would sum to 0, ( y 2 y’ 5 0) is small consolation to someone making a single prediction and for whom that outcome is very important. Regression procedures include a way to estimate the amount of error so that we know how much to trust the prediction.
The Standard Error of the Estimate
Recall that the standard error of the mean and the standard error of the difference were calculated for t-tests. Both those statistics are measures of error variance. For regression, there is a similar measure of error variance called the standard error of the estimate (SEest). Theoretically, the standard error of the estimate is this:
• If a very large number of regression solutions is calculated from a data set, and • for each solution, we are able to determine the residual score (the difference
between the actual and predicted values of the criterion variable, y 2 y’), • the standard error of the estimate is the standard deviation of all those resid-
ual scores.
The only way that residual scores could be deter- mined, of course, would be to have access to the actual values of y to begin with. For someone with that information, what would be the point of using regression? The definition above is for conceptual purposes. The “standard deviation of residual scores” explains what the standard error of the esti-
mate represents, but it is not helpful as a guide to how to actually determine the amount of error in a prediction.
Recall from Chapter 4 that, in theory, the standard error of the mean is the standard deviation of all the sample means in a population of sample means. In practice, that value was estimated by dividing the sample standard deviation by the square root of the number in the sample. The standard error of the estimate is determined similarly. The formula is:
Key Terms: The standard error of the estimate is the standard deviation of all pos- sible error scores in a regression problem.
Review Question D: What impact will a stronger correlation have on the standard error of the estimate?
Formula 9.4 SEest 5 Sy "11 2 r2 xy 2
tan81004_09_c09_219-246.indd 235 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
1. With the correlation between R&D budgets and subsequent sales rxy 5 .946. 2. The standard deviation of the y variable, (sales) is sy 5 .613.
The standard error of the estimate is:
SEest 5 sy "11 2 r2 xy 2
5 .613 "11 2 .9462 2
5 .199
A large SEest value indicates substantial error in the prediction. Note that several factors affect the magnitude of the standard error of the estimate.
1. The sy value is the standard deviation of the variable to be predicted. Highly variable data for the criterion measures result in large standard deviation val- ues and, as a result, large SEest values.
2. The "1 2 rxy component is such that the more highly x and y are correlated, the smaller this resulting value will be and, as a consequence, the smaller the SEest value.
The smallest SEest can be is zero, and its largest is the value of sy. Note the following:
• If the correlation between predictor and criterion is perfect (rxy 5 1.0), the
latter part of the term, "11 2 r2 xy 2 becomes the square root of 1 2 1 which is "0 , which of course is 0; sy 3 0 5 0.
• At the other extreme, if the correlation between predictor and criterion has its lowest possible value (0), the latter part of the term becomes the square root of 1 2 0, which is 1; sy 3 1 is sy.
Another Regression Problem
A second regression problem will provide more practice with calculating regression solu- tions: The manager of a catering company is using the number of people in the party to predict the cost of the drinks that are required for the event. The following are the data for 12 recently catered events:
tan81004_09_c09_219-246.indd 236 2/22/13 3:43 PM
CHAPTER 9Section 9.2 Ordinary Least Squares Regression With One Predictor
Number of People Dollar Cost of Drinks
1. 10 23.3
2. 12 27
3. 12 30.5
4. 14 34
5. 14 45
6. 16 55
7. 16 57.5
8. 16 62
9. 16 68
10. 18 70
11. 18 85
12. 18 90
The task is to predict what drinks will cost for a catered party for 14.
Number of People (x) Cost of Drinks (y)
Means 15.000 53.942
Standard Deviations 2.629 22.342
If the correlation is completed longhand, recall that each “raw” score is turned into a z score, so that there are z score equivalents for the x variable (number in the party) and for the y variable (drinks costs). The pairs of z scores are multiplied together, summed, and then divided by the number of pairs minus 1:
Formula 8.1 rxy 5 S1zx 3 zy 2
n 2 1
rxy 5 .944
tan81004_09_c09_219-246.indd 237 2/22/13 3:43 PM
CHAPTER 9Section 9.3 Regression With Excel
Or if Excel calculates the correlation, once the data are in parallel columns, the commands are Data → Data Analysis → Correlation. Indicate the cells containing the input data, enter where the output is to go, and click OK.
The slope (regression coefficient):
b 5 rxy (sy/sx)
5 .944(22.342/2.629)
5 8.022
The intercept (regression constant):
a 5 My 2 bMx
5 53.942 2 (8.022)(15.0)
5 266.388
Now to solve the equation for 14 people:
y’ 5 a 1 bx
5 266.388 1 (8.022)(14)
5 45.92
If an event is catered for 14 people, the best prediction for the cost of drinks for the event is $45.92. The standard error of the estimate for this solution is as follows:
SEest 5 sy "11 2 r2xy 2
5 22.342 "11 2 .9442 2
5 7.372
9.3 Regression With Excel
The Data Analysis option in Excel has a specific regression procedure. To complete the problem above, the procedure is as follows: • With the number in the party data in column A, and the costs of drinks data
in column B, as they probably already are if Excel was used for the correla- tion, insert a blank line at the top of the data set for the labels and enter Num- ber and Drink Costs in cells A1 and B1 respectively.
• Select the Data tab at the top of the page and then Data Analysis at the far right just below the page tabs.
tan81004_09_c09_219-246.indd 238 2/22/13 3:43 PM
CHAPTER 9Section 9.3 Regression With Excel
• In the Analysis Tools window, scroll down to Regression and click OK. • Click the Input Y Range window and drag the cursor from B2 to B13. • Click the Input X Range window and drag the cursor from A2 to A13. • Click the Output Range window and enter something like D1 so that the
output data do not overwrite the results. Click OK.
The result is the screenshot that is Figure 9.3.
Figure 9.3: Predicting the cost of drinks at a catered event based on the number of people
Some of the columns have been expanded to make it easier to read the output.
• The data under Regression Statistics include the same correlation value calcu- lated in the longhand solution. Although Excel calls the statistic Multiple R, in bivariate regression problems it is actually the Pearson Correlation, rxy.
• The R Square value is the square of the Pearson Correlation, the coefficient of determination that indicates the amount of variance in y explained by x.
• The Adjusted R Square is a value reduced from the calculated R Square value. It is diminished according to the size of the sample so that the resulting statistic can be more safely generalized to other kinds of analyses.
• The Standard Error value is the standard error of the estimate, but it is calcu- lated using the adjusted R square value rather than the original r2xy.
• The number of Observations is n 5 12.
tan81004_09_c09_219-246.indd 239 2/22/13 3:43 PM
CHAPTER 9Section 9.3 Regression With Excel
• The table labeled ANOVA has a different purpose than the one generated in previous chapters, when ANOVA was used to test mean differences among dependent or independent groups with one or more categorical predic- tors and one continuous criterion. In regression output, this table tests the assumption that there is no significant relationship between the predictor and criterion variables, x and y. However, there are some similarities. For exam- ple, the SS and MS for regression are similar to SSbet and MSbet in ANOVA in that it contains the variance accounted for by the predictor. One of the differ- ences is that in regression the predictor is a continuous variable. Moreover, an F-value is calculated based on the ratio of the variance accounted for by the predictor in relation to residual error variance (MSres). Note that the ANOVA result (F) is statistically significant. That 4.12E206 means that F in the ANOVA table could have occurred by chance, with p 5 .00000412 (the E206 means that the value is read with the decimal moved 6 places to the left). The x, y relationship is probably not random, which is good news for someone wish- ing to predict the value of y from x.
• The last table in Figure 9.3 provides the regression solution. The intercept and slope values are similar to those produced by the longhand calculations, with allowance for round-off variations. The standard error values for a and b were not calculated longhand, nor were the significance tests for those individual values. The significance tests are redundant since in this type of regression if rxy is significant, it means that x is a significant predictor of y, something already established by the ANOVA test. Notice that if you square the t-value of the predictor, 9.0042 5 81.072, which is the same as the F-value (81.073, due to rounding).
• The last four columns in the bottom table provide the confidence intervals for the intercept and slope. Confidence intervals will be covered in Chapter 11.
Shrinkage and Over-Fitting the Sample
When samples are small and correlations are rela- tively weak, the magnitude of the associated pre- diction errors increases. Even with high correlations and low data variability, there are risks involved in regression solutions. When a particular solution is tailored too closely to the related sample, the prob- lem is over-fitting the sample. It is a problem
because to maximize its value, a regression solution should predict well for new data sets as well as for the original data set. Over-fitting is always a concern in regression, but it is most problematic when samples are relatively small. Another term to describe this problem is shrinkage, which refers to the degree to which the accuracy of a solution is diminished when it is used with new data.
Review Question E: What does “shrink- age” mean in regres- sion, and how can it be avoided?
Key Terms: Over-fitting refers to a solution that gener- alizes poorly to new data sets. As the accuracy of predictions diminishes, there is reference to shrinkage.
tan81004_09_c09_219-246.indd 240 2/22/13 3:43 PM
CHAPTER 9Section 9.4 The Requirements for Ordinary Least Squares Regression
Regression to the Mean
When Sir Francis Galton, a 19th century British researcher, conducted his study of fami- lies, he noted that very tall parents tend to have children who are less tall, and very short parents tend to have children who are not quite so short. He had stumbled into what is called regression to the mean, a principle that indicates that extreme values of x always predict slightly less extreme values of y. Why is this so? The answer lies in the nature of
normal distributions, where the closer one comes to the mean, the greater the number of individuals. If an extreme value is employed as a predictor, prob- ability indicates that most of the data in any popula- tion, and therefore the most probable outcome, will be a value less distant from the mean.
In the problem where the caterer was predicting the cost of drinks, the data were as follows:
Number of People (x) Cost of Drinks (y)
Means 15.000 53.942
Standard Deviations 2.629 22.342
If a value that is two standard deviations below the mean, Mx, is the predictor, what will be the predicted cost of drinks? Regression to the mean indicates that y’ will be less than two standard deviations below the mean for costs, a value that would be $9.258. If x has a value of the mean minus two standard deviations, it would be 15.0 2 (2 3 2.629) 5 9.742 (never mind that there couldn’t be an event catered for 9.742 people). Using that value as x, the value of y’ is:
y’ 5 a 1 bx
5 266.388 1 (8.022)(9.742)
5 11.762
The predicted cost of drinks for an event catered for 9.742 people is $11.762. How- ever, the mean of y minus two standard deviations produces a different value. It is 53.942 2 (2 3 22.342) 5 9.258, a value more extreme, more distant from the mean than y’. The difference is that in the regression equation, the solution has “regressed to the mean.”
9.4 The Requirements for Ordinary Least Squares Regression
There are many different regression procedures. Each is adapted to different circum-stances having to do largely with the nature of the data. The requirements of bivariate ordinary least squares regression are:
• Data that are interval or ratio scale variables • Data that are normally distributed in their populations • A statistically significant correlation between the predictor and criterion
variables
Key Terms: Regression to the mean is a principle indicating that extreme values in a distribu- tion predict less extreme values.
tan81004_09_c09_219-246.indd 241 2/22/13 3:43 PM
CHAPTER 9Answers to Review Questions
• A linear relationship between the variables • Similar amounts of variability in the ranges of the data
Chapter Summary
The correlation coefficient is an elegant statistic. Whenever separate measures have some quality in common, correlation coefficients quantify that commonality. Regres- sion procedures capitalize on this by using what is shared to predict the level of one mea- sure from the other. Because prediction is a part of all science and of virtually every social domain as well, regression has remarkably wide application. When variables are related, but one is more difficult to measure than the other, the more accessible variable can be used to predict the more elusive variable (Objective 1).
There are many types of regression. Bivariate regression has one predictor variable and one predicted variable, the criterion variable. The math involved is called “least squares regression,” or “ordinary least squares regression,” and reveals where to position a regres- sion line so that the sum of the squared errors from a series of predictions has the lowest possible value. The line is a visual representation of the relationship between the vari- ables. It allows the prediction of y from x, and since there is no assumption about which is the cause, a prediction also of x from y, when that is helpful.
The regression line is a best case fit given the available data. The location of the regres- sion line is based on where it intercepts the y-axis (the intercept) and the line’s inclining or declining orientation (the slope) (Objective 2). The intercept and slope values make it possible to predict y from a value of x (Objective 3). The regression line and the related predictions represent compromises based on imperfect correlations between the predictor and criterion variable. The imperfections are manifested in a measure of error in the pre- dicted value of y. The standard error of the estimate indicates the magnitude of the error.
When regression solutions are tailored too closely to a data set, particularly a small data set, the solution is over-fitted to the sample. Over-fitting means that there will be enough error in the values of a and b and the positioning of the regression line that the solution will not predict equally well for other data sets. Shrinkage refers to this reduction in the value of the regression solution as it is applied to new data.
The discussion in this chapter has been confined to what is called simple, or bivariate, regression. This type of regression involves one predictor and one criterion variable. When there is more than one predictor, the analysis is multiple regression, which is the focus of Chapter 10.
Answers to Review Questions
A. The visual evidence is a great deal of “scatter” in the points in the graph. The weaker the correlation, the less well defined the pattern of dots.
B. The prediction errors would sum to zero with the overpredictions and the underpredictions cancelling each other out.
tan81004_09_c09_219-246.indd 242 2/22/13 3:43 PM
CHAPTER 9Management Application Exercises
C. The statistic that indicates the value of y when x 5 0 describes the value of the intercept, a, in the regression equation.
D. A stronger correlation will shrink the standard error of the estimate. E. Shrinkage refers to the receding value of a regression solution when it is
employed with additional data sets. The best check against shrinkage is large, representative data sets.
Chapter Formulas
Formula 9.1 y’ 5 a 1 bx This is the equation for ordinary least squares regression.
Formula 9.2 a 5 My 2 bMx This formula establishes the regression constant, or inter- cept value. It indicates the value y when x 5 0.
Formula 9.3 b 5 rxy (sy/sx) This formula indicates the slope of the regression line or the regression coefficient. It indicates the impact on y of increasing x by 1.0.
Formula 9.4. SEest 5 Sy "11 2 r2xy 2 This statistic is a measure of prediction error.
Management Application Exercises
Unless otherwise stated, use p 5 .05 in all your answers.
A large company administers aptitude tests to job applicants. The first table below, also sometimes referred to as a “correlation matrix,” summarizes the correlations between 6 aptitude test scores among a group of 52 job applicants who have completed 6 tests. Each row-column intersection provides the correlation between the aptitudes noted in the row and column labels. The second table summarizes the means and standard deviations of the 52 applicants’ scores on each test. Use the information provided in these tables to answer questions 1 through 6 below.
Correlations
problem solving
analytical compre- hension
reasoning computa- tion
verbal
problem solving
analytical .726
comprehension .833 .767
reasoning .598 .857 .686
computation .919 .734 .736 .534
verbal .714 .894 .740 .852 .675
tan81004_09_c09_219-246.indd 243 2/22/13 3:43 PM
CHAPTER 9Management Application Exercises
Descriptive Statistics
Test Mean Standard Deviation
problem solving 43.000 8.441
analytical 46.500 9.317
comprehension 46.500 8.893
reasoning 48.000 6.144
computation 52.750 7.502
verbal 54.850 5.250
1. A new applicant completes all tests except for comprehension. What other apti- tude will provide the most accurate prediction of this applicant’s comprehension ability? Explain.
2. What comprehension score is predicted for someone who has a reasoning score of 60?
3. The public relations department needs to hire someone with solid verbal skills. However, the applicants for this position were only given the analytical test. What verbal score is predicted for an applicant who has an analytical score of 57.5?
4. The office manager is hiring new administrative assistants who will have wide- ranging data management responsibilities. Applicants for this position have taken the problem-solving test, but the manager is also interested in reasoning abilities.
a. What reasoning score can be predicted for an applicant whose problem-solv- ing score is 49?
b. How much will reasoning increase for every 1.0 increase in the problem solv- ing?
c. What value will reasoning have if problem solving is 0? d. In terms of regression solutions, why is the value of reasoning relevant
when problem solving is 0? e. What is the value of the standard error of the estimate?
5. Referring to the chart at the beginning of the Exercises, what variable will provide the most accurate prediction of a potential employee’s comprehension ability? Explain.
6. The Personnel Department for a large company wishes to hire a public relations specialist. The need is for someone with solid verbal skills. Although the depart- ment has data on applicants’ analytical ability, it lacks information about vocabu- lary ability. From the chart at the beginning of the exercise, what vocabulary score is predicted for an applicant who has an analytic score of 575?
7. Use the following data to answer items a and b. The data indicate the number of “sick days” appliance installers take during a three-month period and the number of complaints filed by customers during the same interval.
tan81004_09_c09_219-246.indd 244 2/22/13 3:43 PM
CHAPTER 9Management Application Exercises
Sick Days Complaints
2 3
5 6
4 5
1 3
3 4
5 7
4 4
6 9
a. Is the correlation between number of sick days and number of customer complaints statistically significant?
b. What is the best prediction for the number of complaints that will be regis- tered for an installer who takes five sick days during the period?
8. Because machine tooling involves working equipment, and any errors that the workers make are potentially expensive for the company, the company personnel director gathers data on employees’ manual dexterity and their number of manu- facturing errors. The data are as follows:
Dexterity Errors
23 6
25 4
18 7
15 6
25 3
20 4
19 4
28 2
a. How many errors can be predicted for someone with a manual dexterity score of 22?
b. Why might shrinkage be a problem if the solution based on these data is used to predict the number of errors a different group of machine tooling employees might make?
c. What is the value of the standard error of the estimate?
tan81004_09_c09_219-246.indd 245 2/22/13 3:43 PM
CHAPTER 9Key Terms
Key Terms
• The correlation between variables makes it possible to predict the value of a criterion variable from the value of a predictor variable.
• The regression line is the visual representation of the regression equation. The line’s position in a graph is fixed by the intercept and slope values.
• Ordinary least squares regression procedures minimize residual scores, the differences between the predicted and actual values. This requirement is called the least squares criterion.
• Simple (bivariate) regression involves determining the value of a criterion vari- able based on the value of a predictor variable to which the criterion variable is correlated. The prediction is based on a value called the intercept, which is the value of the criterion when the predictor equals 0, and the value of the slope, which indicates the change in the criterion variable associated with increasing the predictor by 1.0. The slope is also known as the regression coefficient.
• The standard error of the estimate is the standard deviation of all possible error scores in a regression problem.
• Over-fitting the sample refers to a regression solution that does not generalize well to data sets other than the one for which it was calculated, either because the sample upon which the solution is based is small or because the correlation between predictor and criterion variables is relatively weak. When the accuracy of predic- tions diminishes, the phenomenon is called shrinkage.
• Regression to the mean indicates that an extreme value of x will always predict a slightly less extreme value of y’.
tan81004_09_c09_219-246.indd 246 2/22/13 3:43 PM
