FOR BUSINESS INTELLIGENCE
ozaiqii
regression.ppt6. Correlation and Regression
*
The mean, or average value, is the most commonly used measure of central tendency. The mean, ,is given by
Where,
Xi = Observed values of the variable X
n = Number of observations (sample size)
The mode is the value that occurs most frequently. It represents the highest peak of the distribution. The mode is a good measure of location when the variable is inherently categorical or has otherwise been grouped into categories.
Statistics Associated with Frequency Distribution Measures of Location
X
=
X
i
/
n
S
i
=
1
n
X
*
The median of a sample is the middle value when the data are arranged in ascending or descending order.
Statistics Associated with Frequency Distribution Measures of Location
*
Skewness. The tendency of the deviations from the mean to be larger in one direction than in the other. It can be thought of as the tendency for one tail of the distribution to be heavier than the other.
Kurtosis is a measure of the relative peakedness or flatness of the curve defined by the frequency distribution. The kurtosis of a normal distribution is zero. If the kurtosis is positive, then the distribution is more peaked than a normal distribution. A negative value means that the distribution is flatter than a normal distribution.
Statistics Associated with Frequency Distribution Measures of Shape
*
Find the mean, median, mode, and range for the following list of values
- 13, 18, 13, 14, 13, 16, 14, 21, 13
- The mean is the usual average: (13 + 18 + 13 + 14 + 13 + 16 + 14 + 21 + 13) ÷ 9 = 15
- The median is the middle value, so I'll have to rewrite the list in order:
13, 13, 13, 13, 14, 14, 16, 18, 21
- There are nine numbers in the list, so the middle one will be the (9 + 1) ÷ 2 = 10 ÷ 2 = 5th number (the median is the mean of the middle two values if there are an even number of numbers): 13, 13, 13, 13, 14, 14, 16, 18, 21. So the median is 14.
- The mode is the number that is repeated more often than any other: 13 is the mode.
- The largest value in the list is 21, and the smallest is 13, so the range is 21 – 13 = 8.
The range measures the spread of the data. It is simply the difference between the largest and smallest values in the sample.
Range = Xlargest – Xsmallest.
The variance is the mean squared deviation from the mean. The variance can never be negative. The variance is a measure of how far a set of numbers is spread out from the mean. http://www.mathsisfun.com/data/standard-deviation.html
Deviation the difference between the value of an observation and the mean of the population. It is a value minus its mean: x - meanx. Standard deviation is based on the square of the difference. In SPSS, select Analyze, Correlate, Bivariate; click Options; check Cross-product deviations and covariances.
The standard deviation is the square root of the variance.
Statistics Associated with Frequency Distribution Measures of Variability
s
x
=
(
X
i
-
X
)
2
n
-
1
S
i
=
1
n
*
Statistics Associated with Frequency Distribution Measures of Variability
- Covariance is a measure of how much the deviations of two variables match. The equation is: cov(x,y) = SUM[(x - meanx)(y - meany)]. In SPSS, select Analyze, Correlate, Bivariate; click Options; check Cross-product deviations and covariances.
- Correlation is a bivariate measure of association (strength) of the relationship between two variables. It varies from 0 (random relationship) to 1 (perfect linear relationship) or -1 (perfect negative linear relationship). It is usually reported in terms of its square (r2), interpreted as percent of variance explained. For instance, if r2 is .25, then the independent variable is said to explain 25% of the variance in the dependent variable. In SPSS, select Analyze, Correlate, Bivariate; check Pearson.
Correlation
- Pearson's r , the most common type sometimes, is also called product-moment correlation.
- Pearson's r is a measure of association which varies from -1 to +1, with 0 indicating no relationship (random pairing of values) and 1 indicating perfect relationship. In SPSS, select Analyze, Correlate, Bivariate; check Pearson (the default).
Multiple Regression
- The multiple regression equation takes the form y = b1x1 + b2x2 + ... + bnxn + c.
- The b's are regression coefficients, representing the amount the dependent variable y changes when the corresponding independent changes 1 unit. The c is the constant, where the regression line intercepts the y axis, representing the amount the dependent y will be when all the independent variables are 0.
- The standardized version of the b coefficients are the beta weights, and the ratio of the beta coefficients is the ratio of the relative predictive power of the independent variables.
- Associated with multiple regression is R2, multiple correlation, which is the percent of variance in the dependent variable explained collectively by all of the independent variables.
How big a sample size do I need to do multiple regression ?
- According to Tabachnick and Fidell (2001: 117), a rule of thumb for testing b coefficients is to have N >= 104 + m, where m = number of independent variables.
- Another popular rule of thumb is that there must be at least 20 times as many cases as independent variables.
*
Statistics Associated with
Regression Analysis
Regression coefficient. The estimated parameter b is usually referred to as the non-standardized regression coefficient.
Standardized regression coefficient. Also termed the beta coefficient or beta weight is used to denote the standardized regression coefficient.
Byx = Bxy = rxy
Sum of squared errors. The distances of all the points from the regression line are squared and added together to arrive at the sum of squared errors, which is a measure of total error, .
e
j
S
2
*
Conducting Regression Analysis
Plot the Scatter Diagram
A scatter diagram, or scattergram, is a plot of the values of two variables for all the cases or observations.
The most commonly used technique for fitting a straight line to a scattergram is the least-squares procedure.
In fitting the line, the least-squares procedure
minimizes the sum of squared errors, .
e
j
S
2
*
Determine the Strength and Significance of Association: Significance of r with t test
- t statistic. A t statistic with n - 2 degrees of freedom (in simple regression) can be used to test the null hypothesis that no linear relationship exists between X and Y, or H0: r = 0.
- One tests the hypothesis that the correlation is zero (p = 0) using this formula: t = [r*SQRT(n-2)]/[SQRT(1-r2)]
- If the computed t value is as high or higher than the table t value, then the researcher concludes the correlation is significant (that is, significantly different from 0). In practice, most computer programs compute the significance of correlation for the researcher without need for manual methods.
T test: H0: b=0, H1 b is not equal to zero
F test: H0 R-sqr =0, H1 R-sqr is not equal to zero
Determine the Strength and Significance of Association: F test
Another, equivalent test for examining the significance of the linear relationship between X and Y (significance of b) is the test for the significance of the coefficient of determination. The hypotheses in this case are:
H0: R2pop = 0
H1: R2pop > 0
F test. The F test is used to test the null hypothesis that the coefficient of multiple determination in the population, R2pop, is zero. This is equivalent to testing the null hypothesis, which is the same as testing the significance of the regression model as a whole. The test statistic has an F distribution with k and (n - k - 1) degrees of freedom (in multiple regression), where k = number of terms in the equation not counting the constant. F = [R2/k]/[(1 - R2 )/(n - k - 1)].
*
Significance level: p value
- In statistics, a result is called statistically significant if it is unlikely to have occurred by chance.
- The decision is often made using the p-value (see sig. in the table): if the p-value is less than the significance level, then the null hypothesis is rejected. The smaller the p-value, the more significant the result is said to be.
- Thus, we can say, “the null hypothesis is rejected”.
Variables
- Dependent variable. The dependent variable is the predicted variable in the regression equation.
- Independent variables are the predictor variables in the regression equation.
- Dummy variables are a way of adding the values of a nominal or ordinal variable to a regression equation. The standard approach to modeling categorical variables is to include the categorical variables in the regression equation by converting each level of each categorical variable into a variable of its own, usually coded 0 or 1. For instance, the categorical variable "region" may be converted into dummy variables such as "East," "West," "North," or "South." Typically "1" means the attribute of interest is present (ex., South = 1 means the case is from the region South). We have to leave one of the levels out of the regression model to avoid perfect multicollinearity (singularity; redundancy), which will prevent a solution (for example, we may leave out "North" to avoid singularity).
Regression with Dummy Variables
Product Usage Original Dummy Variable Code
Category Variable
Code D1 D2 D3
Nonusers............... 1 1 0 0
Light Users........... 2 0 1 0
Medium Users....... 3 0 0 1
Heavy Users.......... 4 0 0 0
i = a + b1D1 + b2D2 + b3D3
In this case, "heavy users" has been selected as a reference category and has not been directly included in the regression equation.
Y
*
Conducting Multiple Regression Analysis
Strength of Association
R
2
S
S
r
e
g
S
S
y
=
R2, also called multiple correlation or the coefficient of multiple determination, is the percent of the variance in the dependent explained uniquely or jointly by the independents.
*
Conducting Multiple Regression Analysis
Strength of Association
R
2
R
2
k
(
1
-
)
n
-
k
-
1
-
Adjusted R-Square is an adjustment for the fact that when one has a large number of independents.
When used for the case of a few independents, R2 and adjusted R2 will be close. When there are many independents, adjusted R2 may be noticeably lower. Always use adjusted R2 when comparing models with different numbers of independents.
R2 is adjusted for the number of independent variables and the sample
size by using the following formula:
Adjusted R2 =
*
Assumptions
Normality
The error term is normally distributed.
the distribution of variables is normal. regression assumes that the variables have normal distributions.
Linearity
The means of all these normal distributions of Y, given X, lie on a straight line with slope b.
- Absence of high multicollinearity
- No outliers
*
Normality
- A histogram of standardized residuals should show a roughly normal curve.
- Skewness and kurtosis can also be used to check normality of the variables.
- P-P plot: Another alternative for the same purpose is the normal probability plot, with the observed cumulative probabilities of occurrence of the standardized residuals on the Y axis and of expected normal probabilities of occurrence on the X axis, such that a 45-degree line will appear when observed conforms to normally expected.
Normality
The error term is normally distributed.
The variance of the error term is constant.
Homoscedasticity (also spelled homoskedasticity): Lack of homoscedasticity may mean (1) there is an interaction effect between a measured independent variable and an unmeasured independent variable not in the model; or (2) that some independent variables are skewed while others are not.
The error terms are uncorrelated. In other words, the observations have been drawn independently.
The Durbin-Watson statistic is a test to see if the assumption of independent observations is met, which is the same as testing to see if autocorrelation is present. As a rule of thumb, a Durbin-Watson statistic in the range of 1.5 to 2.5 means the researcher may reject the notion that data are autocorrelated (serially dependent) and instead may assume independence of observations.
Homoscedasticity:
error terms are constant
- Nonconstant error variance (heteroscedastivity) can indicate the need to respecify the model to include omitted independent variables. Nonconstant error variance can be observed by requesting simple residual plots, as in the illustrations below, where "Training" as independent is used to predict "Score" as dependent:
- Plot of the dependent on the X-axis against standardized predicted values on the Y axis. For the homoscedasticity assumption to be met, observations should be spread about the regression line similarly for the entire X axis. In the illustration below, which is heteroscedastic, the spread is much narrower for low values than for high values of the X variable, Score.
This plot shows heteroscedasticity
- The variance of the error term should be constant for all values of the independent variables. Heteroscedasticity occurs when the variance of the error term is not constant.
The presence of heteroscedasticity can invalidate statistical tests of significance.
Residuals
- Residuals are the difference between the observed values and those predicted by the regression equation.
Residuals
- Unstandardized residuals, referenced as RESID in SPSS, refer in a regression context to the linear difference between the location of an observation (point) and the regression line (or plane or surface) in multidimensional space.
- Standardized residuals, of course, are residuals after they have been constrained to a mean of zero and a standard deviation of 1. A rule of thumb is that outliers are points whose standardized residual is greater than 3.3 (corresponding to the .001 alpha level – confidence interval or significance level). SPSS will list "Std. Residual" if "casewise diagnostics" is requested under the Statistics button.
- Studentized residuals are constrained only to have a standard deviation of 1, but are not constrained to a mean of 0.
- Studentized deleted residuals are residuals which have been constrained to have a standard deviation of 1, after the standard deviation is calculated leaving the given case out.
Multicollinearity and its problems
- Multicollinearity refers to excessive correlation of the predictor variables. When correlation is excessive (some use the rule of thumb of r > .90), standard errors of the b and beta coefficients become large, making it difficult or impossible to assess the relative importance of the predictor variables.
- Multicollinearity is less important where the research purpose is sheer prediction since the predicted values of the dependent remain stable, but multicollinearity is a severe problem when the research purpose includes causal modeling.
Multicollinearity can result in several problems, including:
The partial regression coefficients may not be estimated precisely. The standard errors are likely to be high.
The magnitudes as well as the signs of the partial regression coefficients may change from sample to sample.
It becomes difficult to assess the relative importance of the independent variables in explaining the variation in the dependent variable.
Predictor variables may be incorrectly included or removed in stepwise regression.
Test of multicollinearity
- Inspection of the correlation matrix reveals only bivariate multicollinearity, with the typical criterion being bivariate correlations > .90. To assess multivariate multicollinearity, one uses tolerance or VIF (Variance-inflation factor)
- Tolerance:As a rule of thumb, if tolerance is less than .20, a problem with multicollinearity is indicated. In SPSS, select Analyze, Regression, Linear; click Statistics; check Collinearity diagnostics to get tolerance.
- VIF is the variance inflation factor, which is simply the reciprocal of tolerance. VIF >= 4 is an arbitrary but common cut-off criterion for deciding when a given independent variable displays "too much" multicollinearity: values above 4 suggest a multicollinearity problem. Some researchers use the more lenient cutoff of 5.0 or even 10.0 to signal when multicollinearity is a problem.
Run simple regression
A simple procedure for adjusting for multicollinearity is to use not variables with low multicollinearity.
Alternatively, the set of independent variables can be transformed into a new set of predictors that are mutually independent by using techniques such as principal components analysis or factor analysis.
More specialized techniques, such as, ridge regression can also be used.
Remedies of Multicollinearity
*
Outliers
The removal of outliers from the data set under analysis can at times dramatically affect the performance of a regression model. Outliers should be removed if there is reason to believe that other variables not in the model explain why the outlier cases are unusual -- that is, outliers may well be cases which need a separate model. Alternatively, outliers may suggest that additional explanatory variables need to be brought into the model (that is, the model needs re-specification).
We can check outliers with any one of the five measures of case influence statistics (DfBeta, standardized DfBeta, DfFit, standardized DfFit, and the covariance ratio) and distance measures (Mahalanobis, Cook's D, and leverage).
we can confidently use the following three ways to detect outliers: casewise diagnostic, Mahalanobis distancem (D2), and DfBETA.
Check Influential cases (outliers) with Influence statistics
Influence statistics in SPSS are selected under the Save button dialog.
DfBeta, called standardized DfBeta in SPSS, measures the change in b coefficients (measured in standard errors) due to excluding a case from the dataset. A DfBeta coefficient is computed for every observations. If DfBeta > 0, the case increases the slope; if < 0, the case decreases the slope. The case may be considered an influential outlier if |DfBeta| > 2. In an alternative rule of thumb, a case may be an outlier if |DfBeta|> 2/SQRT(n).
Standardized DfBeta. Once DfBeta is standardized, it is easier to interpret. The threshold of SDFBETA is usually set at ±2
DfFit. DfFit measues how much the estimate (predicted value) changes as a result of a particular observation being dropped from analysis. The dfFit measure is quite similar to Cook's D.
Standardized DfFit. Once DfFit is standardized, it is easier to interpret. A rule of thumb flags as outliers those observations whose standardized DfBeta value is > twice the square root of p/N, where p is the number of parameters in the model and N is sample size.
Covariance ratio. This ratio compares the determinant of the covariance matrix with and without inclusion of a given case. The closer the covariance ratio approaches 1.0, the less influential the observation.
Check influential case with Distance Measures
Distance measures in SPSS are also selected under the Save button dialog.
Centered leverage statistic, h, also called the hat-value, is available to identify cases which influence regression coefficients more than others. The leverage statistic varies from 0 (no influence on the model) to almost 1 (completely determines the model). The maximum value is (N-1)/N, where N is sample size. A rule of thumb is that cases with leverage under .2 are not a problem, but if a case has leverage over .5, the case has undue leverage and should be examined for the possibility of measurement error or the need to model such cases separately.
Mahalabobis distance. The higher the Mahalanobis distance for a case, the more that case's values on independent variables diverge from average values. As a rule of thumb, the maximum Mahalanobis distance should not exceed the critical chi-squared value with degrees of freedom equal to number of predictors and alpha =.001, or else outliers may be a problem in the data.
Cook's distance, D, is another measure of the influence of a case. Observations with larger D values than the rest of the data are those which have unusual influence or leverage. Fox (1991: 34) suggests as a cut-off for detecting influential cases, values of D greater than 4/(N - k - 1), where N is sample size and k is the number of independents. Others suggest D > 1 as the criterion to constitute a strong indication of an outlier problem, with D > 4/n the criterion to indicate a possible problem.
Casewise Diagnostics
