Statistical Testing and Examples of Standardized Test in Research

Last week, we looked at how to make decisions about whether single sample outcomes could have come from specific populations; that is, if the sample results reasonably reflected what we would expect from a specific population with known parameters.  It should come as no surprise that we can extend this approach to the comparison of multiple samples.  For the most part, these tests are not about specific populations but rather whether the multiple samples could have come from the same or similar populations.

We continue to use the hypothesis testing procedure for these questions and tests.  One noticeable difference exists in the hypothesis statements.  In the 2-sample and multiple sample tests, neither the null nor the alternate hypothesis have a specific value in them. Recall, when testing the defect rate last week, our null was Ho: Rejection rate mean = 10%.  If we continue with rejection rate example, a null hypothesis around rejection rates could be:

Ho: Rejection rate mean1 = rejection rate mean2; where we would be testing to see if the rate were the same between two different production lines, plants, or time periods.

The other guidelines on writing null and alternate hypotheses remain the same; when developing the null and alternate hypothesis:

1. Look at the question being asked.

2. If the wording implies an equality could exist (equal to, at least, no more than, etc.), we have a null hypothesis and we write it exactly as the question asks.

3. If the wording does not suggest an equality (less than, more than, etc.), it refers to the alternate hypothesis. Write the alternate first.

4. Then, for whichever hypothesis statement you wrote, develop the other to contain all the other possible outcomes. An = null should have a =/= alternate, an => null should have a < alternate; a <= null should have a > alternate, and vice versa.

5. The order the variables are listed in each hypothesis must be the same – if we list males first in the null, we need to list males first in the alternate. This minimizes confusion in interpreting results.

Two-Sample Statistical Tests

The two sample tests follow pretty much the same logic and situations as the one-sample tests we examined last week.  The most commonly performed tests involve variances, means, proportions, and distributions.  Again, to allow us to compare outcomes between and among tests, we will use a single example for the variance, proportion, and mean test examples.  The example will look at the question of whether males and females are paid the same for the work they do.  Our sample will include the variables of salary, the job grade level, and gender for 50 employees chosen randomly for the study from a single company.  (These are obviously not all the variables we would want to consider if we were doing an actual equal pay study, but they will suffice to demonstrate the tests presented this week.)  Here is a screen shoot of the data.

Examples

As we did last week, our examination of the data set in our attempt to answer the question needs to start with examining variance equality for two reasons.  First, as we mentioned last week understanding the impact of variance on our measures is important. Second, our test for mean equality differs depending upon the variance equality between the data sets.

Variance Tests

The F-test is used to test variance equality between two populations represented by samples.  Excel has two formats for this test.  The F-TEST Two-Sample for Variances is found in the Data Analysis section of the Data tab; the outcome from this test provides us with summary statistics and the p-value for a one-tailed test (generally, not what we are looking for).  The other is the F.TEST found in the Fx and Formulas statistical function lists.  This outcome provides only the two-tail p-value.  An example of each is shown in the screen print below.

Note that the two-tail p-value from the F.TEST is twice the one-tail p-value from the F-Test output.  This will always be the case, so if the F-test format is preferred, all that needs doing is doubling the one-tail p-value to get the two-tail value to make our decision.

Either of these tests can be used for more than 2 groups by choosing the extreme variance values to use.

Mean Tests

T-Tests.  T-tests are performed for testing means from two samples.  Three versions exist, for data sets with:

· Equal variances

· Unequal variances

· Paired data (two measures on each sample member/item).

As with the F-test, Excel has two locations where these tests can be found.  The T.TEST function, found in the Fx and Formulas lists, asks for each data list, the number of tails (for either a one or two tail test outcome), and the type (paired, equal variance, or unequal variance).  The t-tests listed in the Data Analysis tab identify each test separately.  The screen shot below shows the outcomes of each test.

The paired T-Test initially confuses some students.  Paired means two measures on the same item, such as a pre-and post-test for students, the salary and midpoint value for employees in a company, etc.  Some think that the salary measure used above taken on males and females is an example of paired data – it is NOT.  Needless to say, the variables must be measuring the same thing.

Special Case – The One Sample T-Test

Microsoft’s version of Excel does not provide a one-sample T-Test function.  The expectation is that it would be performed using the one-sample t formula:

t = (mean – Ho claim)/(sample standard deviation/square root(sample size)), df = n-1.

However, if one of the two samples presented for the two-sample unequal variance t-test has a variance of 0 (meaning all values are the same), both the t formula and the df formula default to the one-sample formulas.  (The interested student is encouraged to verify this.)

This means that with the proper setup of the data, we can use Excel’s Two-Sample unequal variance T-Test to perform a one-sample t-test analysis for us.  All that is required is that one sample (the variable 1 range) contain our sample values while the second sample (the variable 2 range) contain only the Ho value that we are testing the sample against.  Typically, this second variable would be titled Ho Value.  The following screen shot shows a very small data sample, the Ho Value sample, and the resulting T-test output, for a test of the claim that the mean = 75.  (Note: the hypothesis testing steps are not show, as this example for setting up a one sample t-test).

Tests for Multiple Groups

Multiple Means

ANOVA.  If we have more than two groups of means to test, we can use the Analysis of Variance, AKA ANOVA Single Factor test.  ANOVA assumes that the variances of the populations are equal.  It then compares the variance of the entire sample to the average variance of the individual sample groups.  If the overall variance is close to the average variance, it means that the group centers (means) must be close together; if the overall variance is much larger than the individual group average, the groups must be spread out and the means are not equal.  Continuing with the salary example, we can test to see if the mean salary for each grade is equal.  The screen shot below shows an ANOVA test of the claim of all means are equal.  (Note, the grades associated with each salary are not shown.)

The Single Factor ANOVA output is a bit more complex than the other tests we have looked at so far.  The Summary table gives us descriptive statistics for each of the groups we have.  The p-value for this example is shown as 1.04E-35.  The E stands for exponent of 10, so this p-value is actually 10-35 which equals 0.0000000000000000000000000000000000104.  Reading this format of a decimal simply requires us to move the decimal point “x” (where x was 35 in this example) number of places.  If the E is followed by a negative sign, we move the decimal point to the left (giving us small values); if the E is followed by only the number, we move the decimal point to the right (giving us large values).  Almost always, a p-value with an E-x in an ANOVA or T-test output means we have a p-value smaller than 0.05.

Determining Differences.  When we reject the null hypothesis, a logical follow-up question is often, which differences are meaningful?  There are several approaches to answering this question; all involve a pair by pair comparison, and most require access statistical tables not available within Excel.

One approach that we can use in our Excel worksheet involves developing confidence intervals around the difference in group means.  (Note: Confidence intervals allow us to develop a range that contains the value we are looking for with a known level of confidence such as 95%.  We will discuss this again later.)

All of the required information for these intervals is available from the ANVOA output.  The basic approach is to

1. Find the difference between each pair of means

2. To this value, add and subtract a measure of the variation in the data (due to sample error, we know our sample means are not exactly equal to the population parameter, so we need to take this sample error into account – our real difference might be a bit larger or smaller than the samples show).

3. Examine the ranges to see if 0 is included (alternately, do the endpoints have different signs a + and -); if so the real population difference could be 0 and the means do not significantly differ.

The formula for the interval that we will build in Excel is:

(mean1 – mean2) +/- t*sqrt(MSW * (1/n1 + 1/n2)) (Lind, Marchel, & Wathen, 2008).

The value of the means for each variable is found in the Summary table, as is the count (n) for each variable.  The MSW is the MS for within that is found in the ANOVA table, and we find t with the t.inv function from Excel.

So, let’s walk thru constructing an interval for grades A and B, and then we can look at what it might look like in an Excel spreadsheet.  From our example output above, we have:

Mean A = 23.5 (rounded)

Mean B = 31.7 (rounded)

n for A = 15

n for B = 7

MSW = 8.64 (rounded)

T has a df equal to that of MSW (44 in this case), and the probability is our 0.05 for a 95% interval.  T.inv(0.05, 44) equals 2.015 (rounded).

So, for grades A and B, our mean difference = 31.7 – 23.5 = 8.2

The +/- term is t * sqrt(MSW * (1/n1 +1/n2)). Plugging in our values gives us

2.015* sqrt(8.64 * (1/15 + 1/7) = 2.71.

So, our interval is 8.2 +/- 2.71 = 5.49 to 10.91 (rounded).

Since 0 is not in this range, we can say that the mean salaries for grades A and B differ significantly.  Setting this up in Excel (using cell references as the examples on the left show) give us the following:

So, all the grade average salary differences are significantly different from each other.

Other Uses.  Excel has two other ANOVA tests that are interesting, but we will not cover them in any detail.  They are the ANOVA Two-Factor with Replication and the ANOVA Two-Factor without Replication.

The ANOVA Two-Factor without Replication (AKA Two-Way Analysis of Variance or Within Subjects F) examines two factors at the same time.  When we looked at the compa-ratios by grade, we had a single factor: compa-ratios.  If we added a second factor, for example gender1, we would have two-factors being examined by the same test.

The input table for a Two-Factor without Replication ANOVA looks like this:

In each cell would be a single measure, for example the highlighted cell at the intersection of grade A and Males, would have a single measure; perhaps the mean or median compa-ratio for this group.

The output would look like a single factor ANOVA table with the addition of a row for rows.  The single factor ANOVA already has the row for columns, called Between Groups.

So, we would be able to see if the mean for the column variable significantly differed.  We would be able to see if the mean for the row variable significantly differed. All with one test.  Doing with would have us develop two pairs of hypothesis, one for each variable we are testing.

But, this form does something else for us, if desired.  We might think that gender has an impact on the compa-ratio means per grade, and would like to see if the means differ if we “held gender constant” or “blocked the impact of gender.”  In this case, we would only have a single hypothesis test, something like Ho: Means do not differ, controlling for gender.

A criticism or weakness of this approach is that it uses fewer measures than a single factor ANOVA does, so it contains less information to make our decision on.  Still, this is an interesting tool.

The Two Factor ANOVA with Replication (AKA Factorial ANOVA or Two-Way ANOVA with Interaction) is similar to the Two-Factor ANOVA without Replication in that we can test two variables/factors at the same time.  The input table would look similar except it would have several rows under each factor (such as Gender1 above) to allow for several measures for each of the variable-pairs (such as Male and grade A, Female and grade C, etc.).

The results for each of the factors is interpreted the same way, if the p-value is less than 0.05, we have significant differences in the mean.  However; very often, what is of more interest with this version is the interaction impact (Ho: No interaction between factors).  Interactions occur when, using our compa-ratio means as an example, the graphed line showing the averages per grade for one gender is not the same as the line for the other gender.  Maybe they both do not go up in the same way, or one jumps and drops while the other smoothly increases, etc.

Interaction is one way of demonstrating a different impact of one variable on outcomes – a way of suggesting that the variables are not treated equally.  Interested students are encouraged to research these different approaches on the web for more information.

Distributions

Last week we compared a single sample distribution to a known or desired population distribution to see if the sample could have come from the population.  This week we expand this comparison to multiple groups to see if they could have come from the same or similar populations.  We use a slightly different form of the Chi Square test, called the Contingency Table analysis for this test.  One question we might have in looking at average pay differences across a company could be: are males and females distributed in a similar fashion across grade levels. If not, differing average salaries would be expected, and the impact of grades would need to be considered in answering any question about pay differences.

With the Contingency Table test, the value of the Chi Square test statistic is the same calculation as with the Goodness of Fit test: ∑ (observed – expected)2/expected over all the table cells.  Often the expected frequency count does not exist, and needs generating.  In this situation, we basically use an average for each cell; this average is simply the row sum times the column sum divided by the overall count.  The df for the Contingency Table test equals the (row count -1)*(column count -1).  An example this test is shown in the screen print below, where the distribution of employees across the six grades is examined to see if males and females are distributed in a similar pattern.  Both the manual and Excel approaches are shown.

Conclusion

This week we have examined techniques to test multiple group outcomes, including variances, means, and distributions.  In each case, the hypothesis testing procedure was used to provide the structure for the analysis and interpretation of the results.  Details on the assumptions of each of the approaches were not discussed here, and should be reviewed in the associated chapters assigned to this week’s topics.