ECO342: Principles of Econometrics (BSK046DS)

Multiple regression is employed rather than bivariate regression because the goal of regression is always to explain the variance in a population and provide predictive measures.

Recall that Multiple Regression as we are considering it is subject to the same requirements as bivariate regression. We are considering here still the Classical Linear Regression Model. Just as with simple regression, multiple regression can only do a good job of explaining the relationships of independent variables and dependent variables when those relationships are linear.

We can only use our ordinary least squares linear multiple regression to consider variables measured at the interval or ratio level. This is the way we have been measuring variables in this class so far. If the dependent variable is not measured this way (for instance using “dummy” or ordinal values, such as is/is not a smoker) at this level, then other, more specialized regression techniques must be used.

Independent variables should be included only if they are based on the economist’s theory about which factors influence the dependent variable. It is important that there is some theoretical or hypothetical link between dependent and independent variables rather than just helter-skelter addition of variables to run up the R-Squared.

Variables should be added to reduce specification bias in the model. When important variables are omitted, the remaining variables have not only direct explanatory power, explaining their presumed effect upon the dependent variable, but an indirect effect, explaining the effect upon the dependent variable of those independent variables which have been omitted. This is specification bias. (Gujarati & Porter, 2010)

Variables should be added to to improve the Adjusted R-squared explanation of variance. Generally, one adds additional variables to the regression equation to improve the model’s ability to explain observed variance in results and so also its predictive capability. The Adjusted R-squared is the appropriate metric for considering whether explanatory power of the model is improved because this adjusts both for sample size n and degrees of freedom k. (Recall that degrees of freedom are reduced as independent variables are added to the regression model.)

Variables that do not contribute very much to explaining the R-Squared should be dropped. This is essentially a paraphrase of the Occam’s Razor preference for simplicity and elegance in modeling.

Variables should be dropped if the sample size isn’t “large enough.” A good rule of thum is to have at least 30 observations for a bivariate regression and 10 more for each additional independent variable. (Saint-Germain, 2002). The recommendation for medical research published in one journal is similar: “As a rule of thumb multiple regression analysis should not be performed if the total number of variables is greater than the number of subjects ÷ 10.” (Journal of Tropical Pediatry, u.k.). A large number of variables can translate into a large sample size rather quickly!

This is a big deal. You have heard the expression, “Lies, damn lies and statistics.” This term was popularized by Mark Twain and, probably not surprisingly, was directed at politicians of his day. (Wikipedia, 2011). It may often be that the lying at hand is not unintentional but accidental or, at the least, unprofessional. One of the easiest traps to fall in with multiple regression is that of multicollinearity, which may have the result that one is not analyzing what one thinks one is analyzing, and so in arriving at inaccurate correlations or lack of correlations between variables.

Some writers suggest that the magic number of independent variables is something less than five. Although this clearly depends upon the particular phenomenon modeled and the sample size available, this is probably not a bad rule of thumb. The reason for preferring a smaller number of variables is the need to avoid multicollinearity. Multcollinearity is encountered if the independent variables are highly correlated with one another. It is suggested that “If a correlation coefficient matrix with all the independent variables indicates correlations of .75 or higher, then there may be a problem with multicollinearity.” (Saint-Germain, 2002).

If variables are multicollinear, they are not independent at all but, in fact, measuring the same thing or aspects of the same thing. When one variable is added to the regression equation, it tends to explain most of the variance, leaving little for the second variable.

A simple example of collinear variables could be using three variables to predict a person’s weight: (1) height and (2) waist size and (3) pants length. Clearly, there is some overlap in the variables of pants length and height. The situation could be simpler till: we could be trying to predict height and include in amongst our variables (1) weight in ounces and (2) weight in pounds. The two predictors are clearly completely redundant and appear trivial, but similar but more subtle situations may occur in real analyses. “Trying to decide which one of the two measures is a better predictor of height would be rather silly; however, this is exactly what you would try to do if you were to perform a multiple regression analysis with height as the dependent (Y) variable and the two measures of weight as the independent (X) variables. When there are very many variables involved, it is often not immediately apparent that this problem exists, and it may only manifest itself after several variables have already been entered into the regression equation.” (Statsoft, Inc., 2011)

Signs of multicollinearity include (Saint-Germain, 2002):

1) none of the t-ratios of the coefficients are statistically significant, but he F-test for the equation as a whole is significant;

2) adding an additional independent variable to the equation radically changes either the size or the sign (plus or minus) of the coefficients associated with the other independent variables

If multicollinearity is discovered, one can either leave both in while noting that multicollinearity is present or drop one of the two variables that are highly correlated. The choice will depend upon the circumstance but general preference for Occam’s Razor and simplicity suggest that dropping on of the collinear variables will be in most instances preferred.

- - - - -

Gujarati, D. & Porter, D. (2010). Chapter Four: Multiple Regression Estimation and Hypothesis Testing. Essentials of Econometrics.

Saint-Germain, M. (2002). Problems with Multiple Regresssion. PPA 696 Research Methods. Retrieved May 2, 2011 from http://www.csulb.edu/~msaintg/ppa696/696regmx.htm

(Links to an external site.)

Links to an external site.

StatSoft, Inc. (2011). Multiple Regression. Electronic Statistics Textbook. Retrieved May 2, 2011 from http://www.statsoft.com/textbook/multiple-regression/

(Links to an external site.)

Links to an external site.

Unknown. (u.k.). Multiple Regression Analysis. Journal of Tropical Pediatrics. Retrieved May 2, 2011 from www.oxfordjournals.org/our_journals/tropej/online/ma_chap3.ppt

(Links to an external site.)

Links to an external site.

Wikipedia. (2011). Lies, Damned Lies, and Statistics. Wikipedia. Retreived May 2, 2011 from http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics

(Links to an external site.)

Links to an external site.