Research Methodologies

(Please study the online reading “Introduction to Variables” before you read this article.)

Types and characteristics of quantitative designs

A quantitative design is a type of research that investigates quantifiable relationships between variables, i.e., one objective is to find out in what ways one set of variables influences or changes another set of variables, to measure, and finally to describe the “quantity” of this influence.

As the title of this article indicates, there is not a single so-called quantitative design but there are several. Just as there are a variety of building designs to accommodate the owners’ or tenants’ specific needs, and just as there are different teaching strategies to best support individual learners, there are also different research designs to most effectively investigate specific research questions.

The most common types of quantitative designs are descriptive designs, experimental designs, and quasi-experimental designs. Researchers use any of these three quantitative design types to show relationships between variables. However, only by using a true experimental design can a researcher show that a variable causes another variable (or variables) to change. In order to show that one variable causes a change in another variable, all other possible explanations for that change need to be excluded. This can be accomplished through a process called “randomization” or “random assignment,” i.e., study participants are being “drawn” randomly from a pool of possible participants and/or assigned randomly to different conditions.

Let’s examine an example of a study using a descriptive design that shows relationships between variables and another one using an experimental design that shows ways in which one variable is the cause of a change in another variable.

Examples of quantitative designs

Study A: Descriptive research design

In study A, researchers wanted to find out if there was a relationship between depression in mothers and behavior problems in toddlers in formal day care settings. The researchers had access to a large sample (i.e., several thousands) of mothers with and without reported depression. They divided the mothers into two groups: those who had reported symptoms of depression and those who had not. Next, the researchers needed to define the term “behavior problems in toddlers.” There was a reliable rating scale available, and the researchers asked the early childhood professionals who cared for the toddlers to rate the behavior problems of those toddlers using that scale.

In this study, the independent variable, i.e., the variable that made one group of participants different from another, was “maternal depression.” The dependent variable, i.e., the variable that the researchers thought might vary depending on whether a mother was depressed or not, was “degree of behavior problem.” The researchers speculated that children whose mothers were depressed might exhibit more behavior problems in day care settings than children whose mothers were not depressed. And indeed, they found their assumption confirmed. This study is an example of a simple descriptive quantitative research design: the participants were taken as they naturally occur in real life, and the situations were not purposefully manipulated by the researcher but simply measured and then described in their pre-existing form.

Study B: Experimental research design (*)

In study B, researchers wanted to know if different methods of writing instruction promoted different degrees of reading comprehension skills in kindergarten students. They designed a true experiment: First, they decided to find out which of two writing instruction methods might yield better reading comprehension results: the interactive writing instruction method or the writing workshop method. From

their literature review they knew that other researchers had written about these two methods, and it was believed that one method was superior to the other. These researchers wanted to find out if this assumption was true.

From all available elementary schools in a specific school district, they randomly selected the kindergarten classrooms in two schools. The kindergarten students were then randomly assigned to one or the other writing instruction method. This meant that half the students were exposed to the interactive writing instruction method, and half the students to the writing workshop method. The study lasted 16 weeks, and all students’ reading comprehension level was assessed four times during the duration of the study. The researchers predicted that the two different instruction methods would result in different student reading comprehension levels, i.e., they thought one method might be better than the other in teaching reading comprehension skills to kindergarten students.

To their surprise they found out that there was no real difference in the reading comprehension levels between students of both groups. And because they had assigned their participants to the two different experimental conditions randomly, the researchers could be quite certain that nothing else but the instructional methods used (i.e., the independent variable) caused the individual students’ reading comprehension level (i.e., the dependent variable, the element that was measured).

In sum, the researchers designed a true experiment, turned their research question into a hypothesis, tested the hypothesis, and finally evaluated the results. Because they had conducted a true experiment with random assignment of the participants, they could confidently state that the two different instructional methods did not differ from each other in their effect on reading comprehension skills.

Looking more closely at the difference between descriptive and experimental designs:

Relationships vs. causation

These two research examples illustrate that there is a significant difference between the descriptive design and the experimental design: Descriptive designs can only show a relationship between variables but cannot determine if one variable is caused by the other. True experimental designs, on the other hand, can demonstrate causation. Let’s examine the two examples more closely:

Study A, using the descriptive design, shows that there is some relationship between the independent and the dependent variable: The results indicate that there is a relationship between maternal depression and toddlers’ behavior problems. However, is this one the only reasonable explanation? Could it be that there were differences in the day care settings that influenced the behavior of the toddlers? Might the toddlers’ temperament have a stronger influence on their behavior than the mothers’ depression? What about the temperaments of the child care professionals? Later in this article you will learn about ways researchers try to account for such alternate influences or explanations. Responsible researchers using a descriptive design usually mention possible other explanations in their discussion or conclusion section.

Study B, using the experimental research design, on the other hand can shows with reasonable certainty that there is not only a relationship between the variables but that the independent variable caused the specific effect on the dependent variable, i.e., the specific writing instruction methods caused the reported reading comprehension skills. The reason for this certainty is grounded in the principle of random assignment. The random sampling and/or assignment process is a proven method to assure that the effect of other variables besides the main independent variable is neutralized. Consequently, researchers have a certain confidence that the results of a study are actually caused solely by the influence of the independent variable. In the example above, the children in the study were randomly assigned to the two different writing instruction methods. This random assignment is the reason why the researchers can say with a great degree of certainty that the result, i.e., the effect on the dependent variable (the level of reading comprehension skills measured) was caused by the independent variable (the writing instruction methods used). In this study, researchers learned that one method of writing instruction did not influence children’s reading comprehension levels more than the other. (*) Note: Although the research design used in Study B is solid, early childhood professionals using a holistic approach to caring for and educating young children could reasonably question the approach to reading suggested here.

Quasi-experimental designs

Where in this discussion does the third major quantitative design, the quasi-experimental design, fit? If you consider the main requirement for an experimental design, namely random assignment, and then think of the realities of doing research with children and families, you realize that in the early childhood field random assignment is not easily accomplished, must be designed with care, and sometimes might not even be practical or desirable. As a researcher, you may have access to a specific group of children or families or settings. These might be the ideal participants for your research purpose, yet for a variety of reasons you are unable to randomly assign them to different groups: Perhaps they already exist in established groups that you cannot break up; even if you could randomly assign the participants, you might encounter some ethics issues; or perhaps the focus of your study is such that doing research in an unchanged natural environment is more important than establishing causal relationships between variables. Under any of these circumstances, you might choose a quasi-experimental design. The main difference between it and a true experiment is simply the absence of random selection/assignment of the participants.

A word about hypotheses

A great number of studies, though not all, using quantitative designs state a hypothesis. Some studies contain even more than one hypothesis. Experimental studies always include hypotheses. What then is a hypothesis? In what ways is a hypothesis different from a research question? And what are some ways to state a hypothesis?

What is a hypothesis?

In its most basic definition, a hypothesis is an explanation of something observed or a prediction of something we expect. People use hypotheses regularly in their daily lives. For example, we observe someone shouting at someone else and say “That person must be angry!” In this instance, we just hypothesized about the shouter’s emotional state. In another context we may read about someone’s difficult upbringing, for example the abuse the person experienced. Asked to speculate what might happen to such a person, someone might say: “That person will have a difficult time trusting other people.” In the first case, we hypothesize what might be the cause for the observed behavior. In the second case, we make a prediction of future behavior based on some bits of information. Our “hypotheses” may or may not be correct, and others may or may not ask us to support our statements. But in everyday life, we usually do not have to officially support our assertions with detailed data. Researchers, however, must always provide data that based on which a stated hypothesis is either accepted or rejected.

What makes hypotheses different from research questions?

Researchers ask all kinds of questions about their field of interest. They refine those that are of greatest interest to them, into more specific research questions. Research questions are still broad and not precisely defined. They are a good way to begin research; yet, for a study using quantitative designs, the question needs to be restated in a more specific way because any study outcome needs to be measurable in some way. The main purpose of formulating a hypothesis, therefore, is to narrow the larger research question so that one can actually observe something very specific, collect data, and draw an informed conclusion about whether the hypothesis is supported by the data or is not supported.

Take a look at this research question: “Does the personality of a mother have any influence on how her child behaves in day care or preschool?” Although interesting, it is too broad a question for a quantitative study. You don’t know what is meant by “personality,” and you do not know what is meant by “influence,” or what “behavior” the researcher has in mind. To specifically measure the influence of one variable on another, the research question needs to be narrower, more defined. To that purpose, researchers formulate a hypothesis based on the research question.

In what ways are hypotheses stated?

Before a hypothesis can be stated, the terms used have to be defined. In our example, the researchers, based on what is already known, decided that maternal depression might be an important variable that influences child behavior. They further decided to focus on problematic behaviors of toddlers in day care, and define them as behavior problems measured by the XYX scale. Finally, they predict, i.e., hypothesize, that there is a relation between maternal depression and children’s behavior problems. They are now ready to state a hypothesis: “There is a strong relationship between maternal depression and toddler’s behavior problems as defined by the XYZ scale.”

This is just one way to state the hypothesis. Sometimes researchers have information from other researchers that leads them to suggest a specific directional hypothesis. In the example above, they could state the hypothesis this way: “The more depressed mothers are, the worse the behavior problems of their toddlers in day care are.” Or -and this is an unlikely hypothesis- they could state: “The more depressed mothers are, the less the behavior problems of their toddlers in day care are.” Some researchers prefer to start out by stating that there is no connection between variables, and they then reject the hypothesis when their data suggest that in fact there is a relationship between variables. In the example here, such a neutral hypothesis (or “null hypothesis”) looks like this: “There is no relationship between maternal depression and toddler behavior problems in day care.”

To sum it up: Hypotheses are based on research questions. They contain terms that are clearly defined and can be measured. Hypotheses can be stated in a variety of forms: they can be directional or neutral. The decision with regard to what kind of hypothesis to state is based, among other things, on the information already available about a research topic. Hypotheses predict the outcome of a study. Based on the analysis of the collected data, hypotheses are either accepted or rejected.

Four common assumptions in quantitative research designs

Quantitative research, and therefore any quantitative research design, is based on the same assumptions. The four main assumptions of ethical quantitative research designs are the focus of this next section. Understanding each of these will help you be a more critical and competent consumer of research.

1. Objectivity

How can a consumer of research be sure that the results of a study are not just a reflection of the researcher’s opinion? That all other possible explanations for the results have either been excluded or are at least mentioned in the research report? These and similar questions form the basis for the demand of objectivity in research. The most effective way to achieve objectivity with quantitative designs is to control the circumstances as much as possible. In other words, researchers using quantitative designs try to limit the influences on the main variables as much as they can. This is usually accomplished by either controlling the condition under which the study is performed, by controlling the sampling procedure, by investigating additional variables, or by a combination of these options. Let’s take a closer look at each of these three controls that help establish the objectivity of a study.

Controlling the condition means that the researcher controls or manipulates the circumstances of the study. In study B above, the researchers made sure that the participating children were exposed to only two specific writing instruction methods. The teachers who administered these methods had agreed beforehand to not use any other methods. Therefore, the researchers controlled the experimental condition of the study. They also randomly assigned the students to one or the other group. In this way, they had total control over their participants. Once a child was assigned to one group, that child did not switch to a different group midway through the study. That was another way the researchers controlled the condition of the study.

Controlling the sampling is important because in their quest for objectivity, researchers attempt to eliminate or neutralize as many differences in their participants as possible so that the results are more strongly related to the chosen independent variable(s). The most effective way to control sampling is through randomization in true experimental studies. For studies using a quasi-experimental design or other non-experimental designs, however, there are other ways to limit external influences. One way is to study a specific group that is as similar as possible in all kinds of characteristics. Under these circumstances, researchers can more reasonably say that the results are due to the independent variable they introduced and not due to differences in participants. This type of control is not as strong as random sampling but is often good enough within the specific research context of a study.

The third device to make a study objective is called statistical control. When a researcher cannot use random assignment, or if using a sample of similar participants is not feasible, this is the control of choice. It means that, in addition to the variable(s) that constitute(s) the main focus of the study, researchers include other possibly influential variables, assess them, and analyze their influence on the dependent variable. Consider study A above: If the researchers of that study had used statistical control, they might have collected information on the following variables: the mothers’ marital status; their education; their age; their geographic location (urban or rural) – to name just a few. In their analysis, they might have used statistical procedures to show the effect of any of these variables, in addition to the main independent variable, i.e., depression, on the dependent variable, i.e., toddlers’ behavior problems.

2. Reductionism

Behind this formal term is a simple concept: One key element of quantitative designs is that something is being measured, e.g., a certain type of behavior, a skill, or a specific effect of an intervention. Only something specific can be measured. Therefore, a general concept must be reduced to a measurable element or elements. For example, you cannot measure a child’s “aggression,” but you can measure how often a child hits another child, or pushes someone, or shouts at someone, or takes a toy away from another child. In other words, a researcher needs to clearly define what is being measured. These specific definitions are called “operational definitions.” The aggression example is very instructive as it shows so clearly that different researchers may define a term in different ways. As a consumer of research you need to keep in mind that your definition of a term might not be the same as the definition for the same term as it is used in a specific research study. Good research will always clearly state the definitions used within the study. Through reductionism, i.e., through careful definitions of the terminology used in a study, researchers make their research process more transparent. Additionally, reductionism makes it possible for other researchers to duplicate a study and obtain results that are comparable because everyone used the same definitions of terms.

3. Reliability and Validity

These assumptions both refer to the measures used in a research study. Reliability and validity are important elements of a research study for the following reasons: When measures are reliable, their results are consistent regardless where they are used or who uses them. For example, if the measure is a scale that assesses very specific behavior problems in toddlers, it is said to be reliable if it consistently produces stable results. Ideally, such a measure is also valid, i.e., it actually measures what it says it measures. This means that if the measure says to measure toddler “aggression” as defined in specific ways, it actually measures that. Researchers who develop measuring instruments use complex procedures to make sure their instruments are reliable and valid. A researcher who wants to use a certain measuring instrument chooses whenever possible one with already established reliability and validity.

4. Generality

This assumption is sometimes called “external validity.” When a study has generality or external validity, the results can be generalized to a larger population than just the study sample, to different settings, or even different conditions. If the assumption of generality were applicable in the study about the effects of writing instruction methods on kindergarten students’ reading comprehension level, results should be the same if the experiment were repeated in a different part of the country, in different schools, with different kindergarten students. It is easy to see that achieving generality is a complex task. Researchers need to ask themselves to what extent they need to generalize the results of their study. For example, if the study about the effects of writing methods on reading comprehension is only important for a particular school district, then the results of the study need only be generalized to that population. A review of the sampling process for that study shows that the two participating schools were randomly chosen from all schools in the district. Through that process, the researchers guaranteed that the results of their study can be generalized to all schools in the same district. There are different ways to achieve or control for generality. As always, responsible researchers will address this assumption in their research report. Critical consumers of research should keep in mind that the results of any study may only be valid under specific circumstances. A study is not better or worse depending on how widely the results can be generalized. The only important thing is that researchers be open about this assumption, describe their sampling process clearly, and discuss any limitations they perceive with regard to the assumption of generality.

Where do statistics fit in?

Once researchers have collected the relevant information, i.e., data, for their study, they use a variety of mathematical devices called statistical procedures, to make sense of the data. These procedures clarify whether any results are significant or not. Statistically significant results show that by applying specific mathematical procedures to the collected data, one can actually observe an effect of the independent variable on the dependent variable. The question remains, however, what is the importance of this effect in a larger context beyond the study? A responsible researcher will take such a statistically significant result and discuss their practical implications. These discussions are usually found in the discussion and/or conclusion portion of a research report. As a consumer of research, keep in mind that not all

statistically significant results have also practical significance.

Why do you need to understand the concept of variables? The main reason for learning about variables now is that you will more fully understand the principles that govern quantitative research when you understand what variables are about. Quantitative research consists primarily of investigating relationships between variables, i.e., on finding out in what ways one set of variables influences or changes another set of variables.

A discussion of the nature of variables, the types of variables, and their place in research could fill a book of its own. Within the parameters of this course, however, you only need to know the main types of variables as well as ways to recognize and define them. This article will introduce you to the nature of three main types of variables, i.e., dependent variables, independent variables, and extraneous variables, as well as to practical applications of each.

Definition and types of variables

In its simplest form, a variable can be thought of as something that is subject to variation. Weather, for example, is variable; so are economic conditions, marital satisfaction, or the mood of a two year old child. In research, that which changes as a result of some influence is called a dependent variable.

Looking back at two examples in the previous paragraph, it is obvious that the stated conditions change for a reason: Weather may change because of many different reasons, among them a change in the jet stream, the warming of the atmosphere, or the change in seasons. An individual’s economic condition may change for reasons such as loss of a job, high medical expenses, or bad investments. In short, there are numerous variables that might influence or change one specific variable.

In quantitative research designs, the focus is usually on one specific independent variable, or variable of influence, that seems to affect change in a dependent variable. (See examples below) Specifically, a variable that affects changes is called independent variable. What the independent variable represents is sometimes called “treatment,” and the person or group that is under the influence of the independent variable is the called “treatment group.” Other variables that might also affect change but are not the main focus in a study are generally called extraneous variables.

Three examples of variables*

Here are three research examples to help you practice identifying dependent and independent variables.

1. A researcher might pose the question for her/his study: “Do toddler-age children of depressed mothers have more behavior problems than those children whose mothers are not depressed?”

In this study, the researcher is seeking to find out whether the difference in a mother’s emotional state impacts the toddler’s behavior. The researcher is basing her/his study on the premise that mothers’ depression is expected to influence or change toddlers’ behavior problems. Therefore, “mothers’ depression” is the independent variable and “toddlers’ behavior problems” is the dependent variable.

2. A researcher might ask: “Do young children whose parents have a very controlling parenting style show more anxious behaviors in social situations than children whose parents have a more democratic parenting style?”’

In this study, the two different parenting styles (“very controlling” and “democratic”) are expected to influence the amount of anxious behavior in young children in different ways. Therefore, “anxious behavior in social situations” is the dependent variable, and “parenting style” is the independent variable. You notice in this example that the independent variable occurs in two variations: controlling style and democratic style. It is not unusual in research studies to incorporate several variations or “levels” of the independent variable.

3. A researcher might ask: “If, before a child’s spelling test, parents are very anxious, and if they show their anxiety in front of the child, is that child’s performance on a spelling test better, worse, or the same as that of a child whose parents show no anxiety?”

In this study, whether or not parents show their anxiety is expected to influence children’s spelling test performance. Therefore, “spelling test performance” is the dependent variable, and “parental anxiety” is the independent variable.

*Please note: For the purpose of this exercise, the examples contain only research questions and not final hypotheses. You will study specific information about variables and hypotheses in the article “Overview of Quantitative Designs.” There, you will also learn about the importance of defining the terms used very clearly. Although in the present examples the terms used are not yet specifically defined, keep in mind that in an actual research context, this would not be sufficient.

A word about extraneous variables

It was stated earlier that other variables which are not the main focus in a study might also affect change in the dependent variable. Those variables are generally called extraneous variables. Let’s briefly re-visit the three previous examples to provide you with a taste of extraneous variables.

1. Although researchers in the first example focus on maternal depression as the independent variable, other factors might also have an impact on children’s behavior problems. Lack of sleep, differences in parenting styles, or even nutritional imbalances might influence children’s behaviors. These factors are commonly called extraneous variables. However, as stated above, this research was studying the influence of maternal depression on children’s behavior.

2. The independent variable in the second example is “parenting style,” and the researcher expected the type of parenting style to have an influence on a child’s anxious behavior in social situations. However, it is quite possible that some social situations make children more anxious than others and that the anxious behavior they exhibit is more likely a result of the specific situation rather than the parenting style. In that case, “type of social situation” is an extraneous variable.

3. In the third example, the independent variable, i.e., the variable that is expected to influence spelling test performance, is parental anxiety. What other explanations might account for differences in spelling test scores? Could it be that some children were less prepared for the test than others? Might children have taken different spelling tests? These are some of the other possible influences, and therefore are examples of extraneous variables.

These examples demonstrate that extraneous variables can provide significant challenges for researchers: Considering that the objective of a study using a quantitative research design is generally to investigate the effect of one main independent variable on a dependent variable, what do researchers need to do to account for possible effects of extraneous variables? To obtain relatively unambiguous results, researchers need to neutralize these influences as much as possible. You will read more about ways to do that in the article “Overview of Quantitative Designs.”