BIZZNA TUTOR

QUANTITATIVE RESEARCH DESIGNS

Objective: Following completion of this course, the student will understand the basic differences between descriptive research, pre-experimental designs, true experimental designs, and quasi-experimental designs.

Descriptive research

Quantitative research, requiring statistical analysis, involves a variety of designs that dictate the names of subcategories that you saw in the preceding section. The first, descriptive, includes surveys or questionnaires which can be structured to give a picture of perceptions, knowledge, or attitudes of one or more groups of subjects. There is no experiment involved. Rather, in descriptive research one is simply reporting on a present status, not experimenting. This is the design that you will employ in your research in this class. Descriptive research is popular in education and establishes a foundation upon which more sophisticated follow up studies can be built. There is no cause and effect implied through descriptive research. It just provides a snapshot of the way things are. This design is not limited to surveys. One can quantify anything that can be measured and report on the findings. For example, you could produce a research report that compares reading levels of third graders in urban and rural schools. You might be interested in learning how grade point averages of athletes and nonathletes compare. In these instances and an infinite number of others, you would obtain important information, but there would be no discussion of cause and effect. Rather, your findings might lead to more complex research aimed at explaining observed differences or it could lead to remedial action. For example, a survey might reveal that there is a disconnect between teachers’ and parents’ understanding of a new program aimed at reducing or eliminating bullying. Looking at the means of survey item responses, we find that there were no significant differences between the means of several items, but on five the parents’ means were significantly lower according to t-test results. In this case, rather than leading to a complex experimental design to learn more about the problem, it would be more logical simply to address deficiencies in parents’ understanding by production of a brochure to mail out to parents and perhaps to do some elaboration in the parent-teacher organization meetings. This is but one example of how surveys can reveal problems and how this type of action research can help solve those problems.

Pre-experimental designs

Pre-experimental research includes three designs, all of which are worthless except for the fact that they are quick and easy to do. If you see any of these in a publication, you should dismiss them as at least very questionable. They include the one-shot study where some experience, experimental treatment, or intervention is followed by some kind of measurement. The lack of a control group is a red flag, a major sign that the study is weak. A ridiculous

Table 1. One-shot study design.

No random assignment of subjects

Experimental group

Intervention

Posttest

example for purposes of illustration would be a project that looked at the effect of reading instruction on subjects’ weight. The intervention, reading instruction, occurred over a period of three years and at the end of those three years of learning to read, the subjects’ average weight was 120 pounds, an indication that for this age group they are obese. So we now know that learning to read makes you fatter. Right? Wrong! What was their weight at the start? We don’t have a clue because there was no pretest. We know that obesity in young children is of epidemic proportions and they were might have been too fat before the study began.

Another of the pre-experimental designs is the single group pre & posttest design. Immediately upon seeing that there is no control group, this is discounted as worthless. At least they do a pretest that they could compare to posttest scores, but without the control group the

Table 2. Single group pre & posttest design.

No random assignment of subjects

Experimental group

Pretest

Intervention

Posttest

results are worthless. An example could be a study that looked at the effect of students using a supplementary software package for learning fractions. The researcher obtains pretest scores on fraction mastery and then the teacher goes through her period of instruction on fractions and has the students use the software as a supplementary tool. On the posttest, we find a significant improvement in mastery. Was it a result of the software use? They might have done just as well without the software! There’s no way to know unless there had been a control group that got the identical traditional instruction but did not use the software. This might seem to be common sense, but there are still some folks who don’t understand the necessity for having a control group. Fortunately, that does not include students in this class!

The third and last of these pre-experimental designs is the static group comparison design in which we finally have a control group! This design is set up so that one group gets some experimental treatment but the other doesn’t and measurements are taken afterwards to see if differences exist that could be attributed to the different experience that one group had. Let’s consider the approach taken in the last-mentioned example where fractions software was used to

Table 3. Static group comparison design.

No random assignment of subjects	Experimental group	Intervention	Posttest
	Control group		Posttest

supplement tradition instruction. In this static group comparison design, we would have two groups, both having traditional instruction and one the supplemental software. Obviously, we’ve solved a problem. However, a serious concern remains and makes this unacceptable. There was no pretest, so the groups’ basic math understandings might have been different to begin with. In addition, what guarantee do we have that the subjects in the two groups are equivalent in other ways, in intelligence for example and maybe in parental support for doing homework? There is too little control over these factors and we’d like to see equivalent groups by randomly assigning subjects and we’d like to see on a pretest that math mastery to this point was not significantly different between the two groups. The bottom line on all of these pre-experimental designs is that they are almost worthless. Use of control groups and randomization of assignment of subjects to groups would make for a more rigorous design, one with which some serious attention could be given to findings. Look for the differences between these pre-experimental designs and others that follow in the text.

True experimental designs

There are several true experimental designs and they are all characterized by random assignment of subjects to experimental and control groups. We will consider three of these designs. These are strong designs that, unlike descriptive research, indicate cause and effect. That is, the experimental treatment either does or does not influence the dependent variable.

The most commonly used of these experimental designs is probably the pretest-posttest with control group. Remember that subjects are randomly assigned, so it is unlikely that there will be any important differences between the groups prior to pretesting. If there is some uncertainty, pretest results could be tested statistically to ensure that they are equivalent. If they

Table 4. Pretest-posttest with control group design.

Random assignment of subjects	Experimental group	Pretest	Intervention	Posttest
	Control group	Pretest		Posttest

aren’t significantly different, then that would dictate the approach to take in the eventual statistical analysis. Because there is a control group, any events happening coincidently along with the experimental treatment would likely affect both groups in a similar fashion and thus would have no effect on the outcome. This is a rather simple design and is excellent in terms of control.

If one accepts as reasonable that random assignment of subjects to the control and experimental groups yields two groups that are equivalent at the outset, one could choose to eliminate the pretests and the design would be a posttest only with control group. This would be

Table 5. Posttest only with control group design.

Random assignment of subjects	Experimental group	Intervention	Posttest
	Control group		Posttest

an obvious alternative if pretesting isn’t possible or if there is a possibility that the very act of pretesting might influence subjects’ interaction with the experimental intervention. While it is an excellent design, some traditionalists decry the absence of pretesting and avoid its use.

Merging the pretest-posttest with control group with the posttest only with control group, we have the most rigorous of the experimental designs, the Solomon four-group design. This can reveal the effect of pretesting, but at the cost of requiring a much larger sample size. The

Table 6. Solomon four-group design.

Random assignment of subjects	Group 1	Pretest	Intervention	Posttest
	Group 2	Pretest		Posttest
	Group 3		Intervention	Posttest
	Group 4			Posttest

researcher must make the decision as to whether the added expense in terms of time and effort is worth learning about possible pretesting effects.

All three of these true experimental designs are considered to be excellent. The random assignment of subjects and the use of control groups avoid many of the pitfalls that can interfere with validity of a study. Also, remember another advantage of experimental designs over descriptive research – cause and effect are involved.

Quasi-experimental designs

We will look at three examples of quasi-experimental designs. These are intermediate in strength, far better than the pre-experimental designs, but not quite up to the standards of true experimental designs. In addition, they are a step up from descriptive research because one does look at cause and effect.

The first that we will consider looks very similar to one of the true experimental designs. In fact, the nonrandomized pretest-posttest with control group is identical except that as the name implies there is no random assignment of subjects. Obviously, the subjects in the two

Table 7. Nonrandomized pretest-posttest with control group design.

No random assignment of subjects	Experimental group	Pretest	Intervention	Posttest
	Control group	Pretest		Posttest

groups might not be equivalent at the outset, but being able to look at pretest scores for equivalence and then doing statistical testing accordingly buffers the danger involved by not randomizing subjects’ assignments to either the experimental or control groups. This is a design frequently used in education as a response to being unable to randomly assign subjects. Imagine that you are a middle school science teacher and want to examine the effect of technology in your instruction and ask your principal before school starts to pool all 7th graders and then randomly assign thirty to one of your classes in which you’ll use computers and other technology and assign thirty more to another class which you will teach by older traditional instructional techniques. Your request will be turned down and your principal will likely consult the school psychologist for expert advice on your sanity. This just isn’t practical and you would have to compromise by using whatever students are assigned to your two classes, assuming you’d get approval for the project. Your principal might turn it down on ethical grounds; withholding the use of technology by one class might seem to not only be unethical, but it would bring irate parents down on him or her. Nobody should ever tell you that doing research in the schools is either simple or easy!

Another example where the nonrandomized pretest-posttest with control group would be preferred over a true experimental design is when differences among curricular materials are being tested. In such cases, rather than random assignment of subjects, intact classes in different schools could be used, each being exposed to different curricula. This is sometimes called cluster sampling. Attempts would be made to ensure that the classes would be as similar as possible. For example, you wouldn’t have one rural class and one urban. You wouldn’t have one from a low-income area school and one from an affluent neighborhood school. Even so, with your best efforts at choosing classes with students that are fairly similar, one would want to look at pretest scores very carefully just to be on the safe side. As mentioned earlier, if pretest scores are different the selection of the proper statistical test can counter the problem.

One other quasi-experimental design will be presented, one of several time-series designs. This family of designs is characterized by a series of repeated measures on the pretest followed by the intervention and a series of repeated measures by posttest. For example, one might be interested in testing a strategy to reduce anxiety during testing. Because anxiety often fluctuates, we would want to do the pretest on several consecutive occasions to get a baseline value on anxiety which we would obtain through using heart rate as an indication of stress. We would then intervene with some sort of program designed to lessen test-produced anxiety after which we would observe heart rates during several post-testing sessions. If the heart rates are lower during post-testing, we would consider that the intervention worked and that now testing is less stressful. Here is one time-series design with abbreviations because of space limitations. In this example, there are three pre-tests and three post-tests. That is not a magic number and it

Table 8. Time series with control group design.

Groups not randomly assigned	Exp group	Pre	Pre	Pre	Intervention	Post	Post	Post
	Control gp	Pre	Pre	Pre		Post	Post	Post

might require four or five or six pre-tests before you, the researcher, feel confident that a baseline value has been determined. Other time-series designs forego the control group or manipulate observations and interventions in various ways.

Ex post facto design

Researchers are characterized by their creativity and there are many other designs or variations of the most popular and fundamental designs presented here. We will finish this section by looking at the ex post facto design. Ex post facto means after the fact and involves analysis of data that already exists. This is also sometimes called causal comparative research. Why would you want to analyze pre-existing data? In some cases, ethical concerns prevent setting up experiments for certain conditions. For example, examination of the coronary arteries of soldiers killed in the Korean War revealed that a large number of these young men had some evidence of fatty streaks in the arteries, the beginning of the process of arterial occlusion. We now know very well that coronary artery disease begins early in life, one of the reasons we’re so concerned about childhood obesity today. For obvious ethical reasons, this information could not have been obtained on living subjects, thus the ex post facto, or causal comparative research design. I don’t like the causal comparative label because as often as not, cause is suggested rather than shown and the best we can do is describe degree of association.

If we wanted to examine the factors contributing to race-related violence in schools, we could set up situations designed to trigger such violence and then study the process, an ethical no-no, or we could analyze information gathered at a number of schools after such terrible incidents had occurred (After the fact, ex post facto). One of the gold mines of potential research is the mass of existing data in most school systems. A school psychologist might like to compare grade point averages of students referred for behavior problems with a sample of others. There is no greater database than that generated in special education. There are a myriad of ex post facto studies that could be done in special education. For these and other students, how are standardized test scores used? What use could be made of them in looking for possible causes of improvements or of lack of improvement? In other situations, where one group of students has undergone some experience that others haven’t, the ex post facto design applies. Some special education students are mainstreamed; some are not. Some students participate in extracurricular activities; others do not. In these and an infinite number of situations, experiences differ and one can examine the effects of said experiences. The ex post facto design is identical to the previously maligned static group comparison, one of the pre-experimental designs discussed.

Table. Ex post facto design.

No random assignment of subjects	Experimental group	Experience	Posttest
	Control group		Posttest

Is it a very strong design? No, for random assignment of subjects to groups is the hallmark of the best, the true experimental designs. There are validity questions in ex post facto research as well, but it has a solid place in educational research as a method of avoiding ethical issues and a practical way to make use of pre-existing information in looking at the effects of certain experiences. Do a search in the Magale Library, using ex post facto as your key words and you’ll see some interesting examples of such research.

Additional reading

Campbell, D. T. & Stanley, J. C. (1963). Experimental and Quasi-experimental Designs for Research. Boston: Houghton Mifflin.

Myers, J. L. (2010). Research design and statistical analysis. New York: Routledge.