Probably no term or concept causes the beginning statistics student more confusion than statistical significance. But it doesn’t have to be that way for you. Although it’s a powerful idea, it is also relatively straightforward and can be understood by anyone in a basic statistics class, even you!

We need an example of a study to illustrate the points we want to make. Let’s take E. Duckett and M. Richards’s “Maternal Employment and Young Adolescents’ Daily Experiences in Single Mother Families” (paper presented at the Society for Research in Child Development, Kansas City, MO, 1989—a long time ago in a galaxy far away [Kansas City, Missouri … ]). These two authors examined the attitudes of 436 fifth- through ninth-grade adolescents toward maternal employment. Even though the presentation took place some years ago, it’s a perfect example for illustrating many of the important ideas at the heart of this chapter.

Specifically, the two researchers investigated whether differences are present between the attitudes of adolescents whose mothers work and the attitudes of adolescents whose mothers do not work. They also examined some other factors, but for this example, we’ll stick with the mothers-who-work and mothers-who-don’t-work groups. One more thing. Let’s add the word significant to our discussion of differences, and we have a research hypothesis something like this:

There is a significant difference in attitude toward maternal employment between adolescents whose mothers work and adolescents whose mothers do not work.

What we mean by the word significant is that any difference between the attitudes of the two groups is due to some systematic influence and not due to chance. In this example, the researchers think that influence is whether or not mothers work. We assume that all of the other factors that might account for any differences between groups were controlled (and we can do this through some research design choices). Thus, the only thing left to account for the differences between adolescents’ attitudes is whether or not their mothers work. Right? Yes. Finished? Not quite.

If Only We Were Perfect

Because our world is not a perfect one, we must allow for some leeway in how confident we are that only those factors we identify could cause any difference between groups. In other words, you need to be able to say that although you are pretty sure the difference between the two groups of adolescents is due to something other than chance, you cannot be absolutely, 100%, positively, unequivocally, indisputably (get the picture?) sure. There’s always a chance, no matter how small, that you are wrong. The improbable is probable: So said Aristotle, and he was right. No matter how small the chance, there’s always a chance.

And, by the way, the whole notion of the normal curve’s tails never, ever really touching the x-axis (as we mentioned in the last chapter) is directly relevant to our discussion here. If the tails did touch, the probability of an event being very extreme in one or the other tail would be absolutely zero. But since they do not touch, there is always a chance, no matter how perfect we might be, that an event can occur—no matter how small and unlikely its probability might be.

Why? Many reasons. For example, you could just be wrong. Maybe during this one experiment, differences between adolescents’ attitudes were not due to whether their mothers worked or didn’t work but were due to some other factor that was inadvertently not accounted for, such as a speech given by the local Mothers Who Work Club that several students attended. How about if the people in one group were mostly adolescent males and the people in the other group were mostly adolescent females? That could be the source of a difference as well. If you are a good researcher and do your homework, you can account for such differences, but it’s always possible that you can’t. And as a good researcher, you have to take that possibility into account.

So what do you do? In most scientific endeavors that involve testing hypotheses (such as the group differences example here), there is bound to be a certain amount of error that cannot be controlled—this is the chance factor that we have been talking about in the past few chapters. The level of chance or risk you are willing to take is expressed as a significance level, a term that unnecessarily strikes fear in the hearts of even strong men and women.

Significance level (here’s the quick-and-dirty definition) is the risk associated with not being 100% confident that what you observe in an experiment is due to the treatment or what was being tested—in our example, whether or not mothers worked. If you read that significant findings occurred at the .05 level (or p < .05 in tech talk and what you regularly see in professional journals), the translation is that there is 1 chance in 20 (or 5 in 100 or .05 or 5%) that any differences found were not due to the hypothesized reason (whether mom works) but to some random reason (yep—chance). Your job is to reduce this likelihood as much as possible by removing all of the competing reasons for any differences that you observed. Because you cannot fully eliminate the likelihood (because no one can control every potential factor), you assign some level of probability and report your results with that caveat.

In sum (and in practice), the researcher defines a level of risk that he or she is willing to take. If the results fall within the region that says, “This could not have occurred by chance alone—something else is going on,” the researcher knows that the null hypothesis (which states an equality) is not the most attractive explanation for the observed outcomes. Instead, the research hypothesis (that there is an inequality or a difference) is the favored explanation.

Let’s take a look at another example, this one hypothetical.

A researcher is interested in seeing whether there is a difference between the academic achievement of children who participated in a preschool program and that of children who did not participate. The null hypothesis is that the two groups are equal to each other on some measure of achievement.

The research hypothesis is that the mean score for the group of children who participated in the program is higher than the mean score for the group of children who did not participate in the program.

As a good researcher, your job is to show (as best you can—and no one is so perfect that he or she can account for everything) that any difference that exists between the two groups is due only to the effects of the preschool experience and no other factor or combination of factors. Through a variety of techniques (that you’ll learn about in your research methods class!), you control or eliminate all the possible sources of difference, such as the influence of parents’ education, number of children in the family, and so on. Once these other potential explanatory variables are removed, the only remaining alternative explanation for differences is the effect of the preschool experience itself.

But can you be absolutely (which is pretty darn) sure? No, you cannot. Why? First, because you can never be sure that you are testing a sample that identically reflects the profile of the population. And even if the sample perfectly represents the population, there are always other influences that might affect the outcome and that you inadvertently missed when designing the experiment. There’s always the possibility of error (another word for chance).

By concluding that the differences in test scores are due to differences in treatment, you accept some risk. This degree of risk is, in effect (drumroll, please), the level of statistical significance at which you are willing to operate.

Statistical significance (here’s the formal definition) is the degree of risk you are willing to take that you will reject a null hypothesis when it is actually true. In our preceding example, the null says that there is no difference between the two sample groups (remember, the null is always a statement of equality). In your data, however, you did find a difference. That is, given the evidence you have so far, group membership seems to have an effect on achievement scores. In reality, however, maybe there is no difference in the whole population(s). If you reject the null you stated, you would be making an error. That type of error is also known as a Type I error. To use as much jargon as possible, statistical significance is the chance of making a Type I error.

So the next step is to develop a set of steps to test whether our findings indicate that error is responsible for differences or whether it is more likely that actual differences are responsible.

The World’s Most Important Table (for This Semester Only)

Here’s what it all boils down to.

A null hypothesis can be true or false. Either there really is no difference between groups (to use a common null hypothesis), or there really and truly is an inequality (such as the difference between two groups). But remember, you’ll never know this true state because the null cannot be tested directly (remember that the null applies only to the population and, for a variety of reasons we have talked about, the population cannot be directly tested).

And, as a crackerjack statistician, you can choose to either reject or accept the null hypothesis. Right? These four different conditions create the table you see in Table 9.1.

Let’s look at each cell.

More About Table 9.1

Table 9.1 has four important cells that describe the relationship between the nature of the null (whether it’s true or not) and your action (accept or reject the null hypothesis). As you can see, the null can be either true or false, and you can either reject or accept it.

The most important thing about understanding this table is the fact that the researcher never really knows the true nature of the null hypothesis and whether there really is or is not a difference between groups. Why? Because the population (which the null represents) is never directly tested. Why? Because it’s usually impractical to do so, and that’s why we have inferential statistics.

Table 9.1 ⬢ Different Types of Errors

Action You Take
		Accept the Null Hypothesis	Reject the Null Hypothesis
*True nature of the null hypothesis*	The null hypothesis is really true.	1 ☺ Bingo, you accepted a null when it is true and there is really no difference between groups.	2 ☹ Oops—you made a Type I error and rejected a null hypothesis even when there really is no difference between groups. The chance of making a Type I error is also represented by the Greek letter alpha, or α.
	The null hypothesis is really false.	3 ☹ Uh-oh—you made a Type II error and accepted a false null hypothesis. The chance of making Type II errors is represented by the Greek letter beta, or β.	4 ☺ Good job, you rejected the null hypothesis when there really are differences between the two groups. The chance of doing this is called power, or 1 − β.

· ☺ Cell 1 in Table 9.1 represents a situation in which the null hypothesis is really true (there’s no difference between groups) and the researcher made the correct decision accepting the null. No problem here. In our example, our results would show that there is no difference between the two groups of children, and we have acted correctly by not rejecting the null that there is no difference.

· ☹ Oops. Cell 2 represents a serious error. Here, we have rejected the null hypothesis (that there is no difference) when it is really true (and there is no difference between groups). Even though there is no difference between the two groups of children, we will conclude there is, and that’s an error—clearly a boo-boo, called a Type I error .

· ☹ Uh-oh, another type of error. Cell 3 represents a serious error as well. Here, we have accepted the null hypothesis (that there is no difference) when it is really false (and, indeed, there is a difference between groups). We have said that even though there is a difference between the two groups of children, we will conclude there is not—also clearly a boo-boo, this time known as a Type II error .

· ☺ Cell 4 represents a situation where the null hypothesis is really false and the researcher made the correct decision in rejecting it. No problem here. In our example, our results show that there is a difference between the two groups of children, and we have acted correctly by rejecting the null that states there is no difference.

So, if .05 is good and .01 is even “better,” why not set your Type I level of risk at .000001? For the very good reason that you would be so rigorous in being confident that your null hypotheses are really false that you would almost never reject a null, even when you should! Such a stringent Type I error rate allows for little leeway—indeed, the research hypothesis might be true but you’d never reach that conclusion with a too-rigid Type I level of error.

LIGHTBOARD LECTURE VIDEO

Type I and Type II Errors

So let's talk about hypothesis testing. When you think about it, there are, what, four possible outcomes when you test a hypothesis. Two of them are correct, but two of them are mistakes. Let's think about each of these things. Remember the way you test a hypothesis is you realize there's something that's real, and some guess you've made, and you want them to match. And in terms of reality, there are just two possibilities when it comes to some statistical relationship. One possibility is there is no relationship. That says relationship, trust me. The other possibility is there is a relationship. That still says relationship. So there are only two possibilities. Now, also, you could have one of two guesses. You could guess that there's no relationship or you could guess that there is a relationship. And we call these what? The no relationship, we call that a null hypothesis, right? And there is a relationship, we call that the researcher's hypothesis or the alternative hypothesis. But this is what the researcher thinks. This is what we have to pretend to believe in order to test things statistically. So let's look at the possible outcomes here. If in real life, there's no relationship, and you've guessed there's no relationship, well, you've made the right decision. Yay. If in real life, there is a relationship, and you think there's no relationship, you've made a mistake. That's an error. That's too bad. Now, the other two possibilities. If you think there is a relationship, but in real life, there isn't, well that's an error, too. And when I say you think there's a relationship and in real life there isn't, what I mean is you have concluded that there is a relationship after you did a statistical test, but you've made a mistake. And up here when I say that there is a relationship and you think there isn't, what I mean is after you've done your statistical analysis, you conclude that there is a relationship when there really isn't. Now, that leaves one other possibility, that in reality, there is a relationship, and you guess there is a relationship, so again, that's a happy right decision to have made. So look at these two errors here. We have names and statistics for these two types of errors, and the type of error we think is the worst is when we have concluded that we have statistically significant results, and there is a relationship, and everyone's happy, but in real life in the population, there isn't. We call that a type I error. That's bad because you might be recommending a new drug that doesn't work. You might be recommending a way of teaching reading that doesn't work. There's this other type of error. And for some reason, no one seems to care as much about this, and this is a type II error. We use Roman numerals there for the I and II. A type II error is when in fact you haven't found significance. So you'll say, yeah, I was right. There's no relationship there. But in real life, there is. This would be like saying, hey, there's a miracle drug out there that really would save lives. Oh, but this study didn't find it. So two types of errors, type I, type II this is where you found significance when you should not have. This is where you missed out and didn't find significant results when you should have.

Back to Type I Errors

Let’s focus a bit more on Cell 2, where a Type I error was made, because this is the focus of our discussion.

This Type I error, or level of significance, has certain values associated with it that define the risk you are willing to take in any test of the null hypothesis. The conventional levels set are between .01 and .05.

For example, if the level of significance is .01, it means that on any one test of the null hypothesis, there is a 1% chance you will reject the null hypothesis when the null is true and conclude that there is a group difference when there really is no group difference at all.

If the level of significance is .05, it means that on any one test of the null hypothesis, there is a 5% chance you will reject it (and conclude that there is a group difference) when the null is true and there really is no group difference at all. Notice that the level of significance is associated with an independent test of the null and is based on a “what if” way of thinking. If the null hypothesis is true in the population, what are the chances I would have found a result this big (like the difference between two groups) in my sample?

In a research report, statistical significance is usually represented as p < .05, read as “the probability of observing that outcome is less than .05” and often expressed in a report or journal article simply as “significant at the .05 level.”

With the introduction of fancy-schmancy software such as SPSS and Excel that can do statistical analysis, there’s no longer the worry about the imprecision of such statements as “p < .05” or “p < .01.” For example, p < .05 can mean anything from .000 to .049999, right? Instead, software such as SPSS and Excel gives you the exact probability, such as .013 or .158, of the risk you are willing to take that you will commit a Type I error. So, when you see in a research article the statement that “p < .05,” it means that the value of p is equal to anything from .00 to .049999999999 … (you get the picture). Likewise, when you see “p > .05” or “p = ns” (for nonsignificant), it means that the probability of rejecting a true null exceeds .05 and, in fact, can range from .0500001 to 1.00 (or close to 1.00; remember those curves never touch the ground, so we can never be sure that the null is true). So, it’s actually terrific when we know the exact probability of an outcome because we can measure more precisely the risk we are willing to take. But what to do if the p value is exactly .05? If SPSS or Excel (or any other program) generates a value of .05, extend the number of decimal places—it may really be .04999999999.

As discussed earlier, there is another kind of error you can make, which, along with the Type I error, is shown in Table 9.1. A Type II error (Cell 3 in the chart) occurs when you incorrectly accept a false null hypothesis. For example, there may really be differences between the populations represented by the sample groups, but you mistakenly conclude there are not.

When talking about the significance of a finding, you might hear the word power used. Power is a type of probability statement of how well a statistical test can detect and reject a null hypothesis when it is false. Mathematically, it’s calculated by subtracting the proportional chance of making a Type II error from 1.0. A more powerful test is always more desirable than a less powerful test, because the more powerful one lets you get to the heart of what’s false and what’s not.

Ideally, you want to minimize both Type I and Type II errors, but doing so is not always easy or under your control. You have complete control over the Type I error level or the amount of risk that you are willing to take (because you actually set the level itself). Type II errors, however, are not as directly controlled but instead are related to factors such as sample size. Type II errors are particularly sensitive to the number of subjects in a sample, and as that number increases, the probability of Type II error decreases. In other words, as the sample characteristics more closely match those of the population (achieved by increasing the sample size), the likelihood that you will accept a false null hypothesis decreases.

CORE CONCEPTS IN STATS VIDEO

Errors in Hypothesis Testing, Statistical Power, and Effect Size

Let's calculate a number of inferential statistics. First, state two statistical hypotheses regarding what is believed to exist in the population. The first is the null hypothesis. What it implies is that the difference or the change or what we call the effect in the population does not exist. The mutually exclusive alternative to the null hypothesis is the alternative hypothesis, which implies that the difference, change, or effect does exist. In the second step, you're going to make one of two decisions-- either reject the null hypothesis or not reject the null hypothesis. Because the decision is based on probability, the reality exists that the decision you make could be correct or incorrect, or you can makes two types of errors. The first error is known as Type I error. Type I error can be thought of as rejecting a true null hypothesis, which sounds pretty weird. So another way to think about it is when you're rejecting the null hypothesis when you shouldn't. This means that an effect exists when in reality it does not. The probability of making a Type I error is the same as the probability of rejecting the null hypothesis. It's called alpha, which is the probability of a statistic needed to reject the null hypothesis. The probability of making a Type I error is equal to alpha, which is traditionally set to Every time you make the decision to reject the null hypothesis, there is made the incorrect decision. You've concluded that an effect exists, when in reality, it does not. You can reduce the probability of making a Type I error by making it harder to reject the null hypothesis. For example, let's say you set alpha to something smaller, from To reject the null hypothesis, the probability of the statistic has to be less than which is more stringent than So why not reduce the probability of making Type I error? When you reduce the probability of saying in effect exists when it does not, this increases the likelihood of not saying an effect exists when it does. Reducing the probability of Type I error increases the probability of making the other type of error. A Type II error occurs when you don't reject the null hypothesis. So lowering the probability of Type II error means increasing the likelihood of rejecting the null hypothesis. The reality is the probability of committing a Type II error is difficult to calculate. Why? Think of it this way. You've concluded that an effect does not exist, because it's possible that in reality, it does not exist. Or it could be that the effect does exist, but you didn't find it. Imagine you go to Mars looking for life. You don't find it. It could be because there is no life on Mars, or it could be because there is life on Mars, and you just didn't find it. There's no way of telling. And that's why you can't exactly calculate the probability of a Type II error. However, you can reduce the probability of a Type II error. A Type II error occurs when you don't reject the null hypothesis. So lowering the probability of Type II error means increasing the likelihood of rejecting the null hypothesis. Increasing the probability of rejecting the null is the concept known as statistical power. Statistical power is the probability of detecting an effect when in fact it exists. There's a number of ways of increasing statistical power. One way is to increase your sample size. Because the bigger your sample, the more likely you are to detect effects. Another way is to increase alpha. Let's say you set alpha at This implies you're going to reject the null hypothesis if the probability of your statistic is less than That's a more lenient requirement than and you've increase the likelihood of rejecting the null. The problem with this is that the probability of Type II error increases the likelihood of Type I error. So reducing the probability of saying an effect does not exist when it does increases the likelihood of saying an effect exists when it does not. As you can see, there are two types of errors you can make in hypothesis testing, and they are directly linked to each other. They're important to know, because they affect the ability of researchers to accurately and appropriately interpret the results of their statistical analyses.

SIGNIFICANCE VERSUS MEANINGFULNESS

What an interesting situation arises for the researcher when he or she discovers that the results of an experiment indeed are statistically significant! You know technically what statistical significance means—that the null hypothesis is not a reasonable explanation for what was observed. Now, if your experimental design and other considerations were well taken care of, statistically significant results are unquestionably the first step toward making a contribution to the literature in your field. However, the value of statistical significance and its importance or meaningfulness must be kept in perspective.

For example, let’s take the case where a very large sample of illiterate adults (say, 10,000) is divided into two groups. One group receives intensive training to read using traditional teaching, and the other receives intensive training to read using computers. The average score for Group 1 (which learned in the traditional way) is 75.6 on a reading test, the dependent variable. The average score on the reading test for Group 2 (which learned using the computer) is 75.7. The amount of variance in both groups is about equal. As you can see, the difference in score is only one tenth of 1 point or 0.1 (75.6 vs. 75.7), but when a t test for the significance between independent means is applied, the results are significant at the .01 level, indicating that computers work better than traditional teaching methods. (Chapters 11 and 12 discuss t tests, the kind we would use in such a situation.)

The difference of 0.1 is indeed statistically significant, at the .01 level, but is it meaningful? Does the improvement in test scores (by such a small margin) provide sufficient rationale for the $300,000 it costs to fund the program? Or is the difference negligible enough that it can be ignored, even if it is statistically significant?

Here are some conclusions about the importance of statistical significance that we can reach, given this and the countless other possible examples:

· Statistical significance, in and of itself, is not very meaningful unless the study that is conducted has a sound conceptual base that lends some meaning to the significance of the outcome.

· Statistical significance cannot be interpreted independently of the context within which the outcomes occur. For example, if you are the superintendent in a school system, are you willing to retain children in Grade 1 if the retention program significantly raises their standardized test scores by one half point?

· Although statistical significance is important as a concept, it is not the end-all and certainly should not be the only goal of scientific research. That is the reason why we set out to test hypotheses rather than prove them. If our study is designed correctly, then even null results tell you something very important. If a particular treatment does not work, this is important information that others need to know about. If your study is designed well, then you should know why the treatment does not work, and the next person down the line can design his or her study taking into account the valuable information you have provided.

Researchers treat the reporting of statistical significance in many different ways in their written reports. Some use words such as significant (assuming that if something is significant, it is statistically so) or the entire phrase statistically significant. But some also use the phrase marginally significant, where the probability associated with a finding might be .04, or nearly significant, where the probability is something like .06. What to do? You’re the boss, if your own data are being analyzed or if you are reviewing someone else’s. Use your noodle and consider all the dimensions of the work being done. If .051, within the context of the question being asked and answered, is “good enough,” then it is. Whether outside reviewers agree is a source of great debate and a good topic for class discussion. It is only custom that .05 is the common level of significance required.

Almost every discipline has “other” terms for this significant versus meaningful distinction, but the issue is generally considered to concern the same elements. For example, health care professionals refer to the meaningful part of the equation as “clinical significance” rather than “meaningfulness.” It’s the same idea—they just use a different term given the setting in which their outcomes occur.

Ever hear of “publication bias”? It’s where a preset significance value of .05 is used as the only criterion in the serious consideration of a paper for publication. It’s not exactly .05 or bust, but in times past and even today, some editorial boards hold up significance values such as .05 or .01 as the holy grail of getting things right. If those values are not reached, then the findings cannot be significant, let alone meaningful, according to some people’s judgment. Now, there’s something to be said for consistency throughout a field, but today’s cool stats tools such as SPSS and Excel allow us to pinpoint the exact probability associated with an outcome rather than an all-or-nothing criterion such as .05, which dooms some meaningful work before it is even discussed. Many journals ask researchers to report effect size (the size of the relation among variables, such as a correlation, or a standardized difference between groups), in addition to significance levels, so a fuller picture is available. Be sophisticated—make up your own mind based on all the evidence.

AN INTRODUCTION TO INFERENTIAL STATISTICS

Whereas descriptive statistics are used to describe a sample’s characteristics, inferential statistics are used to infer something about the population based on the sample’s characteristics.

At several points throughout the first half of Statistics for People Who (Think They) Hate Statistics, we have emphasized that a hallmark of good scientific research is choosing a sample in such a way that it is representative of the population from which it was selected. The process then becomes an inferential one, in which you infer from the smaller sample to the larger population based on the results of tests (and experiments) conducted using the sample.

Before we start discussing individual inferential tests, let’s go through the logic of how the inferential method works.

How Inference Works

Here are the general steps of a research project to see how the process of inference might work. We’ll stay with adolescents’ attitudes toward mothers working as an example.

Here’s the sequence of events that might happen:

1. The researcher selects representative samples of adolescents who have mothers who work and adolescents who have mothers who do not work. These are selected in such a way that the samples represent the populations from which they are drawn. For example, they might be chosen randomly from a long list of potential participants.

2. Each adolescent is administered a test to assess his or her attitude. The mean scores for groups are computed and compared using some test.

3. A conclusion is reached as to whether the difference between the scores is the result of chance (meaning some factor other than moms working is responsible for the difference) or the result of “true” and statistically significant differences between the two groups (meaning the results are due to moms working).

4. A conclusion is reached as to the relationship between maternal employment and adolescents’ attitudes in the population from which the sample was originally drawn. In other words, an inference, based on the results of an analysis of the sample data, is made about the population of all adolescents.

How to Select What Test to Use

Step 3 brings us to ask the question, “How do I select the appropriate statistical test to determine whether a difference between groups exists?” Heaven knows, there are plenty of them, and you have to decide which one to use and when to use it.

Well, the best way to learn which test to use is to be an experienced statistician who has taken lots of courses in this area and participated in lots of research. Experience is still the greatest teacher. In fact, there’s no way you can really learn what to use and when to use it unless you’ve had the real-life, applied opportunity to actually use these tools. And as a result of taking this course, you are learning how to use these very tools.

But the basic reasons for why a particular statistical test is the right one to use can be simplified into a few characteristics about your research question. So, for our purposes and to get started, we’ve created this nice little flowchart (aka cheat sheet) of sorts that you see in Figure 9.1. You have to have some idea of what you’re doing, so selecting the correct statistical test does not put the rest of your study on autopilot, but it certainly is a good place to get started.

Don’t think for a second that Figure 9.1 takes the place of your need to learn about when these different tests are appropriate. The flowchart is here only to help you get started.

This is really important. We just wrote that selecting the appropriate statistical test is not necessarily an easy thing to do. And the best way to learn how to do it is to do it, and that means practicing and even taking more statistics courses. The simple flowchart we present here works, in general, but use it with caution. When you make a decision, check with your professor or some other person who has been through this stuff and feels more confident than you might (and who also knows more!).

Here’s How to Use the Chart

1. Assume that you’re very new to this statistics stuff (which you are) and that you have some idea of what these tests of significance are, but you’re pretty lost as far as deciding which one to use when.

2. Answer the question at the top of the flowchart.

3. Proceed down the chart by answering each of the questions until you get to the end of the chart. That’s the statistical test you should use. This is not rocket science, and with some practice (which you will get throughout this part of the book), you’ll be able to quickly and reliably select the appropriate test. Each of the remaining chapters in this part of the book will begin with a chart like the one you see in Figure 9.1 and take you through the specific steps to get to the test statistic you should use.

Figure 9.1 ⬢ A quick (but not always great) approach to determining which statistical test to use

Does the cute flowchart in Figure 9.1 contain all the statistical tests there are? Not by a long shot. There are hundreds, but the ones in Figure 9.1 are the ones used most often. And if you are going to become familiar with the research in your own field, you are bound to run into these.

AN INTRODUCTION TO TESTS OF SIGNIFICANCE

What inferential statistics does best is allow decisions to be made about populations based on information about samples. One of the most useful tools for doing this is a test of statistical significance that can be applied to different types of situations, depending on the nature of the question being asked, the level of measurement used for your variables, and the form of the null hypothesis.

For example, do you want to look at the difference between two groups, such as whether boys score significantly differently than girls do on some test? Or the relationship between two variables, such as self-esteem and depression? The two cases call for different approaches, but both will result in a test of a null hypothesis using a specific test of statistical significance.

How a Test of Significance Works: The Plan

Tests of significance are based on the fact that each type of null hypothesis has associated with it a particular type of statistic. And each of the statistics is associated with a special distribution that you use to estimate the likelihood of your outcomes having occurred by chance.

Here are the general steps to take in the application of a statistical test to any null hypothesis. These steps will serve as a model for each of the chapters in Part IV.

1. A statement of the null hypothesis. Do you remember that the null hypothesis is a statement of equality? The null hypothesis is what we assume is the “true” state of affairs given no other information on which to make a judgment.

2. Setting the level of risk (or the level of significance or chance of making a Type I error) associated with the null hypothesis. With any research hypothesis comes a certain degree of risk that you are wrong. The smaller this error is (such as .01 compared with .05), the less risk you are willing to take. No test of a hypothesis is completely risk free because you never really know the “true” relationship between variables. Remember that it is traditional to set the Type I error rate at .05 or lower.

3. Selection of the appropriate test statistic. Each null hypothesis has associated with it a particular test statistic. You can learn what test is related to what type of question in this part of the book.

4. Computation of the test statistic value. The test statistic value (also called the obtained value or observed value) is the result or product of a specific statistical calculation. For example, there are test statistics for the significance of the difference between the averages of two groups, for the significance of the difference of a correlation coefficient from zero, and for the significance of the difference between two proportions. You’ll actually compute the test statistic and come up with a numerical value.

5. Determination of the value needed for rejection of the null hypothesis using the appropriate table of critical values for the particular statistic. Each test statistic (along with group size and the risk you are willing to take) has a critical value associated with it. This is the value you would expect the test statistic to yield if the null hypothesis is indeed true. In practice, when computer software is used, these tables of critical values are replaced by the computer’s calculations.

6. Comparison of the obtained value with the critical value. This is the crucial step. Here, the value you obtained from the test statistic (the one you computed) is compared with the value (the critical value) you would expect to find by chance alone.

7. If the obtained value is more extreme than the critical value, the null hypothesis cannot be accepted. That is, the null hypothesis’s statement of equality (reflecting chance) is not the most attractive explanation for differences that were found. Here is where the real beauty of the inferential method shines through. Only if your obtained value is more extreme than what would happen by chance (meaning that the result of the test statistic is not a result of some chance fluctuation) can you say that any differences you obtained are not due to chance and that the equality stated by the null hypothesis is not the most attractive explanation for any differences you might have found. Instead, the differences are more likely due to the treatment or whatever your independent variable is.

8. If the obtained value does not exceed the critical value, the null hypothesis is the most attractive explanation. If you cannot show that the difference you obtained is due to something other than chance (such as the treatment), then the difference must be due to chance or something you have no control over. In other words, the null is the best explanation.

Here’s the Picture That’s Worth a Thousand Words

What you see in Figure 9.2 represents the eight steps we just went through. This is a visual representation of what happens when the obtained and critical values are compared. In this example, the significance level is set at .05, or 5%. It could have been set at .01, or 1%.

Figure 9.2 ⬢ Comparing obtained values with critical values and making decisions about rejecting or accepting the null hypothesis

In examining Figure 9.2, note the following:

1. The entire curve represents all the possible outcomes based on a specific null hypothesis, such as the difference between two groups or the significance of a correlation coefficient.

2. The critical value is the point beyond which the obtained outcomes are judged to be so rare that the conclusion is that the obtained outcome is not due to chance but to some other factor. In this example, we define rare as having a less than 5% chance of occurring.

3. If the outcome representing the obtained value falls to the left of the critical value (it is less extreme), the conclusion is that the null hypothesis is the most attractive explanation for any differences that are observed. In other words, the obtained value falls in the region (95% of the area under the curve) where we expect most outcomes to occur if the null hypothesis is true.

4. If the obtained value falls to the right of the critical value (it is more extreme), the conclusion is that the research hypothesis is the most attractive explanation for any differences that are observed. In other words, the obtained value falls in the region (5% of the area under the curve) where we would expect only outcomes due to something other than chance to occur.

BE EVEN MORE CONFIDENT

You now know that probabilities can be associated with outcomes—that’s been an ongoing theme for the past two chapters. Now we are going to say the same thing in a bit different way and introduce a new idea called confidence intervals.

A confidence interval (or CI) is the best estimate of the range of a population value (or population parameter) that we can come up with given the sample value (or sample statistic). Say we knew the mean spelling score for a sample of 20 third graders (of all the third graders in a school district). How much confidence could we have that the population mean will fall between two scores? A 95% confidence interval would be correct 95% of the time.

You already know that the probability of a raw score falling within ±1.96 z scores or standard deviations is 95%, right? (See page 145 in Chapter 8 if you need some review.) Or the probability of a raw score falling within ±2.56 z scores or standard deviations is 99%. If we use the positive or negative raw scores equivalent to those z scores, we have ourselves a confidence interval.

Let’s fool around with some real numbers.

Let’s say that the mean spelling score for a random sample of 100 sixth graders is 64 (out of 75 words) and the standard deviation is 5. What confidence can we have in predicting the population mean for the average spelling score for the entire population of sixth graders?

The 95% confidence interval is equal to

64±1.96(5),64±1.96(5),

or a range from 54.2 to 73.8, so at the least you can say with 95% confidence that the population mean for the average spelling score for all sixth graders falls between those two scores.

Want to be more confident? The 99% confidence interval would be computed as follows:

64±2.56(5),64±2.56(5),

or a range from 51.2 to 76.8, so you can conclude with 99% confidence that the population mean falls between those two scores.

Do keep in mind that most statisticians would probably actually use the standard deviation of the mean (called the standard error of the mean) to compute confidence intervals for this question, but we have kept the example simple to introduce the concept.

While we are using the standard deviation to compute confidence intervals as we introduce the concept, many people would choose to use the standard error of the mean or SEM (see Chapter 10). The standard error of the mean is the standard deviation of all the sample means that could, in theory, be selected from the population. Remember that both the standard deviation and the standard error of the mean are “errors” in measurement that surround a certain “true” point (which in our case would be the true mean and the true amount of variability). The use of the SEM is a bit more complex, but it is an alternative way of computing and understanding confidence intervals and makes sense because we are trying to guess the range of possible values for a mean, not an individual score.

Why does the confidence interval itself get larger as the probability of your being correct increases (from, say, 95% to 99%)? Because the larger range of the confidence interval (in this case, 19.6 [73.8, 54.2] for a 95% confidence interval vs. 25.6 [76.8, 51.2] for a 99% confidence interval) allows you to encompass a larger number of possible outcomes and you can thereby be more confident. Ha! Isn’t this stuff cool? Statisticians are proud of the fact that we can be more confident if we are willing to be less exact.

People Who Loved Statistics

David Blackwell (1919–2010) was a great statistician and mathematician and the first African American to be selected for membership in the National Academy of Sciences. He entered college at age 16, had his master’s degree by 20, and a PhD by 22. Further training and an early career as a professor were hampered by racial discrimination policies. Although brilliant, he was unable to study at Princeton in the 1940s, where he was not allowed to attend lectures and was initially rejected for a teaching position at University of California, Berkeley because of his race. Ten years later, however, he was hired at Berkeley and was a leader there for 30 years. We write about Dr. Blackwell in this chapter because most of his work centered on accuracy in estimation. He wrote one of the first Bayesian statistics books and developed a statistical technique called the Rao-Blackwell estimator, which provides what is often the best way to guess a population value using a sample value.

CORE CONCEPTS IN STATS VIDEO

Estimating the Mean of a Population

Many of the research situations discussed in this video series involve a population mean mu. The value of mu is stated in the null hypothesis. For example, mu could be equal to some value or it could be that the mu of one population is equal to the mu of another population. In these research situations, you'll either reject or not reject the null hypothesis. It isn't always possible to data value of mu, because we may not know what mu is. In situations such as this, we become more interested in estimating the value of mu. Imagine a political candidate is unsure how voters perceive her and she wants to estimate the percentage of voters who view her favorably. Taking into account the possibility of error, she wants to create a range or interval of values that contain the population mean with a desired degree of confidence. This interval of values is what's known as a confidence interval. Here's the formula for the confidence interval for the mean. This is an example of interval estimation. Interval estimation involves estimating an unknown population parameter by specifying a range or interval of values within which you have a certain degree of confidence the population parameter will fall. Interval estimation can be used in a variety of research situations. In this video, we'll calculate the confidence interval of the mean. This is an interval of values for a variable that has a stated probability of containing an unknown population mean. The formula starts with a sample mean x bar. Creating the confidence interval for the mean means moving to the left and right of the sample mean so that you create an interval or a range of values that has a stated confidence which contains the unknown population mean. You'll see we're not making a decision regarding the population mean as a certain value. Instead, we're estimating what we think the population mean might be. Just what is a confidence interval? Is it the probability of an interval or the probability of a population mean. A confidence interval for the mean is an interval that has a stated probability of containing the population mean. In other words, we could state that there is a 95% probability that the interval from this to this contains the unknown population mean. When we do interval estimation, we identify two scores, an upper and lower boundary. The interval between those two boundaries has a 95% probability of containing the population mean. This implies that our interval will either contain the population mean or it will not, and we have identified the only two scores for that sample that will contain the population mean 95% percent of the time.

Real-World Stats

It’s really interesting how different disciplines can learn from one another when they share, and it’s a shame that this does not happen more often. This is one of the reasons why interdisciplinary studies are so vital—they create an environment where new and old ideas can be used in new and old settings.

One such discussion took place in a medical journal that devotes itself to articles on anesthesia. The focus was a discussion of the relative merits of statistical versus clinical significance, and Drs. Timothy Houle and David Stump pointed out that many large clinical trials obtain a high level of statistical significance with minuscule differences between groups (just as we talked about earlier in the chapter), making the results clinically irrelevant. However, the authors pointed out that with proper marketing, billions can be made from results of dubious clinical importance. This is really a caveat emptor or buyer-beware state of affairs. Clearly, there are a few very good lessons here about whether the significance of an outcome is really meaningful or not. How to know? Look at the substance behind the results and the context within which the outcomes are found.

Want to know more? Go online or to the library and find …

Houle, T. T., & Stump, D. A. (2008). Statistical significance versus clinical significance. Seminars in Cardiothoracic and Vascular Anesthesia, 12, 5– 6.

Summary

So, now you know exactly how the concept of significance works, and all that is left is to apply it to a variety of research questions. That’s what we’ll start with in the next chapter and continue with throughout this part of the book.

Time to Practice

1. Why is significance an important construct in the study and use of inferential statistics?

2. What is statistical significance?

3. What does the (idea of the) critical value represent?

4. Given the following information and setting the level of significance at .05 for decision making, would your decision be to reject or fail to reject the null hypothesis? Provide an explanation for your conclusions.

a. The null hypothesis that there is no relationship between the type of music a person listens to and his or her propensity for crime (p < .05)

b. The null hypothesis that there is no relationship between the amount of coffee consumption and grade point average (p = .62)

c. The research hypothesis that a negative relationship exists between the number of hours worked and level of job satisfaction (p = .51)

5. What’s wrong with the following statements?

a. A Type I error of .05 means that five times out of 100 experiments, we will reject a true null hypothesis.

b. It is possible to set the Type I error rate to zero.

c. The smaller the Type I error rate, the better the results.

6. Why is it “harder” to find a significant outcome (all other things being equal) when the research hypothesis is being tested at the .01 rather than the .05 level of significance?

7. Why should we think in terms of “failing to reject” the null rather than just accepting it?

8. What is the difference between significance and meaningfulness?

9. Here’s more exploration of the significance versus meaningfulness debate:

a. Provide an example of a finding that may be both statistically significant and meaningful.

b. Now provide an example of a finding that may be statistically significant but not meaningful.

Time to Practice Video

Chapter 9: Problem 9

Chapter nine, problem nine asks you to consider the difference between statistical significance and meaningful difference. This is important because often we get statistical results that might lead us to draw a conclusion that something really important happened, but frequently, the differences that we're looking at or the relationship we're examining may not be as meaningful in the real world. We will examine one example that might be both statistically significant and meaningful and one example that is statistically significant but might not be as meaningful. Imagine two, let's call them, Snapgram pages. They each feature nine photographs, the same photographs except for one important variation. One page includes one picture of a bottle of wine and two wine glasses. The other page includes that picture but also a picture of two people drinking beer, the same two people drinking wine, and one of those people by himself with a bottle of beer. So in the first page, one of the nine images featured alcohol. In the second page, four of the nine pictures featured alcohol. The research question is, what effect do depictions of alcohol on a college student's Snapgram page have on what the viewer-- in this case, hiring officers'-- perception of that person's professionalism and the starting salary they would offer this person? Now you might imagine that more pictures of alcohol would negatively affect the offer, but how much? When we look at the paired sample T-test comparing the offer before they saw the Snapgram page and after they saw the Snapgram page, you will see that there is a mean score difference of a little over $2, This is significantly different. This is a statistically significant effect. So it went from before, they saw the resume but didn't see the Snapgram page. They offered almost $3,4 Afterwards, they offered almost $3,2 Now, is this a meaningful difference? Well, that's going to be really up to you to determine. To me, a $2, is pretty meaningful. So this gives us a sense of not only was there a consistent difference, which is what statistical significance will tell us, but also that this would have a real impact. Now let's look at a different study. The question is, do men and women who evaluate the communication style in initial interactions of college students differ? In other words, do men or women perceive the quality of college students more positively? When we look here at the T-test, you'll see that the mean scores on a variety of the traits-- approachable, intelligence, how good of a leader they are, and would you want to hire this person for the job? You also, when you look down here, will see that there is a statistically significant difference. But when you look at the mean scores, you'll see that on a scale of 1 to 1 a So the question is, even though it is statistically significant, is this difference meaningful? In other words, when men and women evaluate college students, do they evaluate them differently? When we look, we're going to see that the females consistently evaluated the students a bit more positively, but it might not be a meaningful difference. I don't know that anyone would notice a difference of on a 1 Now in case you're wondering, there is a way to quantify the meaningfulness of the difference, which we call effect size. In other chapters, you will learn more about effect size. But for this question, the takeaway message is, statistically significant does not necessarily equal meaningful and you want to think about how you answer both of those questions.

1. What does chance have to do with testing the research hypothesis for significance?

2. In Figure 9.2, there is a striped area on the right side of the illustration.

a. What does that entire striped area represent?

b. If the striped area were a larger portion underneath the curve, what would that represent?

Student Study Site

Get the tools you need to sharpen your study skills! Visit edge.sagepub.com/salkind7e to access practice quizzes, eFlashcards, original and curated videos, data sets, and more!