Research Designs and Algorithms

CHAPTER 8

Clarifying Quantitative Research Designs

Susan K. Grove

A research design is a blueprint for conducting a study. Over the years, several quantitative research designs have been developed for conducting descriptive, correlational, quasi-experimental, and experimental studies. Descriptive and correlational designs are focused on describing and examining relationships of variables in natural settings. Quasi-experimental and experimental designs have been developed to examine causality, or the cause and effect relationships between interventions and outcomes. The designs focused on causality were developed to maximize control over factors that could interfere with or threaten the validity of the study design. The strengths of the design validity increase the probability that the study findings are an accurate reflection of reality. Well-designed studies, especially those focused on testing the effects of nursing interventions, are essential for generating sound research evidence for practice (Melnyk, Gallagher-Ford, & Fineout-Overholt, 2017).

Being able to identify a study design and evaluate its strengths and weaknesses are an important part of critically appraising studies. Therefore, this chapter introduces you to the different types of quantitative study designs and provides an algorithm for determining whether a study design is descriptive, correlational, quasi-experimental, or experimental. Algorithms are also provided so that you can identify specific types of designs in published studies. The concepts relevant for understanding quantitative research designs are defined. The different types of validity—construct, internal, external, and statistical conclusion—are described. Guidelines are provided for critically appraising designs in quantitative studies. The chapter concludes with an introduction to randomized controlled trials (RCTs), with a flow diagram provided to examine the quality of these trials conducted in nursing.

Identifying quantitative research designs in nursing studies

A variety of quantitative research designs are implemented in nursing studies; the four most common types are descriptive, correlational, quasi-experimental, and experimental. These designs are categorized in different ways in textbooks (Kerlinger & Lee, 2000; Shadish, Cook, & Campbell, 2002). Sometimes, descriptive and correlational designs are referred to as noninterventional or nonexperimental designs because the focus is on examining variables as they naturally occur in environments and not on the implementation of an intervention by the researcher.

Some of the noninterventional designs include a time element, such as the cross-sectional design, which involves data collection on variables at one point in time. For example, cross-sectional designs might involve examining a group of study participants simultaneously in various stages of development, levels of education, severity of illness, or stages of recovery to describe changes in a phenomenon across stages. The assumption is that the stages are part of a process that will progress over time. Selecting participants at various points in the process provides important information about the totality of the process, even though the same subjects are not monitored throughout the entire process (Gray, Grove, & Sutherland, 2017). For example, researchers might describe the depression levels of three different groups of women with breast cancer who are prechemotherapy, receiving chemotherapy, or postchemotherapy treatment to understand depression levels based on the phase of treatment. Longitudinal design involves collecting data from the same study participants at multiple points in time and might also be referred to as repeated measures. Repeated measures might be included in descriptive, correlational, quasi-experimental, or experimental study designs. With a longitudinal design, a sample of women with breast cancer could be monitored for depression before, during, and after their chemotherapy treatment.

Quasi-experimental and experimental studies are designed to examine causality or the cause and effect relationship between a researcher-implemented intervention and selected study outcomes. The designs for these studies are sometimes referred to as interventional or experimental because the focus is on examining the differences in dependent variables thought to be caused by independent variables or interventions. For example, the researcher-implemented intervention might be a home monitoring program for patients initially diagnosed with hypertension, and the dependent or outcome variables could be systolic and diastolic blood pressure values measured at 1 week, 1 month, and 6 months. This chapter introduces you to selected interventional designs and provides examples of these designs from published nursing studies. Details on other study designs can be found in a variety of methodology sources (Campbell & Stanley, 1963; Creswell, 2014; Gray et al., 2017; Kerlinger & Lee, 2000; Shadish et al., 2002).

The algorithm shown in Fig. 8.1 may be used to determine the type of design (e.g., descriptive, correlational, quasi-experimental, experimental) used in a study. This algorithm includes a series of yes or no responses to specific questions about the design. The algorithm starts with the question, “Is there an intervention?” The answer leads to the next question, with the four types of designs being identified in the algorithm. For example, if researchers conducted a study to identify the characteristics of nurses who either passed or failed their registered nurse (RN) licensure on the first try, Fig. 8.1 indicates that a descriptive design would be used. If the researchers examined the relationships among the nurses’ characteristics and their score on the RN licensure examination, a correlational design would be implemented. If researchers tested the effectiveness of a relaxation intervention on graduates’ RN licensure examination scores, either a quasi-experimental or experimental design would be implemented. Experimental designs have the greatest control because (1) a tightly controlled intervention is implemented and (2) study participants are randomly assigned to either the intervention or control group (see Fig. 8.1).

FIG 8.1 Algorithm for determining the type of quantitative study design.

Understanding concepts relevant to quantitative research designs

Concepts relevant to quantitative research designs include causality, multicausality, probability, bias, prospective, retrospective, control, and manipulation. These concepts are described to provide a background for understanding noninterventional and interventional research designs.

Causality

Causality basically means that things have causes, and causes lead to effects. In a critical appraisal, you need to determine whether the purpose of the study is to examine causality, examine relationships among variables (correlational designs), or describe variables (descriptive designs). You may be able to determine whether the purpose of a study is to examine causality by reading the purpose statement and propositions within the framework (see Chapter 7). For example, the purpose of a causal study may be to examine the effect of an early ambulation program after surgery on the length of hospital stay. The framework proposition may state that early physical activity following surgery improves recovery time. However, the early ambulation program is not the only factor affecting the length of hospital stay. Other important factors or extraneous variables that affect the length of hospital stay include the diagnosis, type of surgery, patient’s age, physical condition of the patient before surgery, and complications that occurred after surgery. Researchers usually design quasi-experimental and experimental studies to examine causality or the effect of an intervention (independent variable) on a selected outcome (dependent variable), using a design that controls for relevant extraneous variables.

Multicausality

Very few phenomena in nursing can be clearly linked to a single cause and a single effect. A number of interrelating variables can be involved in producing a particular effect. Therefore studies developed from a multicausal perspective will include more variables than those using a strict causal orientation. The presence of multiple causes for an effect is referred to as multicausality. For example, patient diagnosis, age, presurgical condition, and complications after surgery are interrelated causes of the length of a patient’s hospital stay. Because of the complexity of causal relationships, a theory is unlikely to identify every element involved in causing a particular outcome. However, the greater the proportion of causal factors that can be identified and examined or controlled in a single study, the clearer the understanding will be of the overall phenomenon. This greater understanding is expected to increase the ability to predict and control the effects of study interventions.

Probability

Probability addresses relative rather than absolute causality. A cause may not produce a specific effect each time that a particular cause occurs, and researchers recognize that a particular cause probably will result in a specific effect. Using a probability orientation, researchers design studies to examine the probability that a given effect will occur under a defined set of circumstances. The circumstances may be variations in multiple variables. For example, while assessing the effect of multiple variables on length of hospital stay, researchers may choose to examine the probability of a given length of hospital stay under a variety of specific sets of circumstances. One specific set of circumstances may be that the patient had undergone a knee replacement, had no chronic illnesses, and experienced no complications after surgery. Sampling criteria could be developed to control most of these extraneous variables. The probability of a given length of hospital stay could be expected to vary as the set of circumstances are varied or controlled in the design of the study.

Bias

The term bias means a slant or deviation from the true or expected. Bias in a study distorts the findings from what the results would have been without the bias. Because studies are conducted to determine the real and the true, quantitative researchers place great value on identifying and removing sources of bias in their study and controlling their effects on the study findings. Any component of a study that deviates or causes a deviation from a true measurement of the study variables contributes to distorted findings. Many factors related to research can be biased; these include attitudes or motivations of the researcher (conscious or unconscious), components of the environment in which the study is conducted, selection of the study participants, composition of the sample, groups formed, measurement methods, data collection process, and statistical analyses (Gray et al., 2017; Grove & Cipher, 2017). For example, some of the participants for the study might be taken from a unit of the hospital in which the patients are participating in another study involving quality nursing care or a nurse, selecting patients for a study, might include only those who showed an interest in the study (Gray et al., 2017). Researchers might use a scale with limited reliability and validity to measure a study variable (Waltz, Strickland, & Lenz, 2017). Each of these situations introduces bias to nonintervention and intervention studies.

An important focus in critically appraising a study is to identify possible sources of bias. This requires careful examination of the methods section in the research report, including the strategies for obtaining study participants, methods of measurement, implementation of a study intervention, and data collection process. However, not all biases can be identified from the published study report. The article may not provide sufficient detail about the methods of the study to detect possible biases.

Prospective Versus Retrospective

Prospective is a term that means looking forward, whereas the term retrospective means looking backward, usually in relation to time. In research, these terms are used most frequently to refer to the timing of data collection. Are the data obtained in real time, with measurements being obtained by the research team, or are the study’s data obtained from information collected at a prior time? Data collection in noninterventional research can be either prospective or retrospective because, by definition, it lacks researcher intervention. Many noninterventional studies in health care use retrospective data obtained from national electronic databases and clinical and administrative databases of healthcare agencies. Secondary analysis of data from a previous study to address a newly developed study purpose is also considered retrospective. However, prospective data collection is usually more accurate than retrospective data collection, especially when researchers are passionate about their phenomenon of study and are rigorous in the measurement of study variables and the implementation of the data collection process.

Data collection in interventional research, however, must be prospective because the researcher enacts an intervention in real time. This is not to say that the research team does not access current data from the health record for real-time studies. A researcher collecting arterial blood pressure data in critically ill infants might collect data over a 24-hour period for several days. Nurses on the various shifts would record arterial blood pressure at least hourly, as is common practice, and the research team would retrieve that information during daily data collection. Although information retrieval of the infants’ electronic chart data does look back in time over the preceding 24-hour period, this study would be considered prospective because data are generated and recorded at the same time that the infants are hospitalized.

Control

One method of reducing bias is to increase the amount of control in the design of a study. Control means having the power to direct or manipulate factors to achieve a desired outcome. For example, in a study of an early ambulation program, study participants may be randomly selected and then randomly assigned to the intervention group or control group. The researcher would control the duration of and the assistance during the ambulation program or intervention. The time that the ambulation occurred in relation to surgery would also be controlled, as well as the environment in which the patient ambulated. Measurement of the length of hospital stay could be controlled by ensuring that the number of days, hours, and minutes of the hospital stay is calculated exactly the same way for each participant. Limiting the characteristics of the study participants, such as diagnosis, age, type of surgery, and incidence of complications, would also be a form of control. The greater the researcher’s control over the study situation, the more credible (or valid) the study findings.

Manipulation

Manipulation is a form of control generally used in quasi-experimental and experimental studies. Controlling an intervention is the most common manipulation in these studies. In descriptive and correlational studies, little or no effort is made to manipulate factors regarding the circumstances of the study. Instead, the purpose is to examine the phenomenon and its characteristics as they exist in a natural environment or setting. However, when quasi-experimental and experimental designs are implemented, researchers must manipulate the intervention under study. Researchers need to develop quality interventions that are implemented in consistent ways by trained individuals (Eymard & Altmiller, 2016). This controlled manipulation of a study’s intervention decreases the potential for bias and increases the validity of the study findings.

Examining the design validity of quantitative studies

Study validity is a measure of the truth or accuracy of the findings obtained from a study. The validity of a study’s design is central to obtaining accurate trustworthy results and findings from a study. Design validity encompasses the strengths and threats to the quality of a study design. Critical appraisal of studies requires that you identify the design strengths and think through the threats to validity or the possible weaknesses in a study’s design. Four types of design validity relevant to nursing research include construct validity, internal validity, external validity, and statistical conclusion validity (Gray et al., 2017; Kerlinger & Lee, 2000; Shadish et al., 2002). Table 8.1 describes these four types of design validity and summarizes the threats common to each. Understanding these types of validity and their possible threats are important in critically appraising quantitative study designs.

Table 8.1

Types of design validity critically appraised in studies

Types of design validity Description Threats to design validity

Construct validity Validity is concerned with the fit between the conceptual and operational definitions of variables and that the instrument measures what it is supposed to in the study.

Inadequate definitions of constructs: Constructs or concepts examined in a study lack adequate conceptual or operational definitions, so the measurement method is not accurately capturing what it is supposed to in a study.

Mono-operation bias: Only one measurement method is used to measure the study variable.

Experimenter expectancies (Rosenthal effect): Researchers’ expectations or bias might influence study outcomes, which could be controlled by researchers designating research assistants to collect study data. Another option is blinding researchers and data collectors to the group receiving the study intervention.

Internal validity Validity is focused on determining if study findings are accurate or are the result of extraneous variables.

Participant selection and assignment to group concerns: The participants are selected by nonrandom sampling methods and are not randomly assigned to groups.

Participant attrition: The percentage of participants withdrawing from the study is high or more than 25%, which can affect the findings of any quantitative study.

History: An event not related to the planned study occurs during the study and could have an impact on the findings.

Maturation: Changes in participants, such as growing wiser, more experienced, or tired, which might affect study results.

External validity Validity is concerned with the extent to which study findings can be generalized beyond the sample used in the study.

Interaction of selection and intervention: The participants included in the study might be different than those who decline participation. If the refusal to participate is high, this might alter the study results.

Interaction of setting and intervention: Bias exists in study settings and organizations that might influence implementation of a study intervention and data collection process. For example, some settings are more supportive and assist with a study, and others are less supportive and might encourage patients not to participate in a study.

Interaction of history and intervention: An event, such as closing a hospital unit, changing leadership, or high nursing staff attrition, might affect the implementation of the intervention and the measurement of study outcomes, which would decrease generalization of findings.

Statistical conclusion validity Validity is concerned with whether the conclusions about relationships or differences drawn from statistical analysis are an accurate reflection of the real world.

Low statistical power: This refers to concluding that there are no differences between samples when one exists (Type II error), which is usually caused by small sample size.

Unreliable measurement methods: Scales or physiological measures used in a study are not consistently measuring study variables. Reliability or consistency of scales is determined using the Cronbach alpha, which should be greater than 0.70 in a study (see Chapter 10).

Intervention fidelity concerns: The intervention in a study is not consistently implemented because of lack of study protocol or training of individuals implementing the intervention.

Extraneous variances in study setting: Extraneous variables in the study setting influence the scores on the dependent variables, making it difficult to detect group differences.

Construct Validity

Construct validity examines the fit between the conceptual and operational definitions of variables. Theoretical constructs or concepts are defined within the study framework when a framework is identified. When the researchers do not identify a specific study framework, the variables may be defined according to how they have been defined in other studies. These abstract statements about the variables are the conceptual definitions, which provide the basis for the operational definitions of the variables. Operational definitions (methods of measurement) must accurately reflect the theoretical constructs or concepts. Construct validity is the extent of the congruence or consistency between the conceptual definitions and operational definitions (see Chapter 5). The process of developing construct validity for an instrument often requires years of scientific work, and researchers need to discuss the construct validity of the instruments that they used in their study (see Chapter 10; Shadish et al., 2002; Waltz et al., 2017). The threats to construct validity are related to previous instrument development and to the development of measurement techniques as part of the methodology of a particular study. Threats to construct validity are described here and summarized in Table 8.1.

Inadequate Definitions of Constructs

Measurement of a construct stems logically from a concept analysis of the construct by the theorist who developed the construct or by the researcher. Ideally, the conceptual definition should emerge from the concept analysis, which is an in-depth study of the meanings of a construct or concept provided by theorists and researchers. The method of measurement (operational definition) should clearly reflect both the framework concept and study variable. A deficiency in the conceptual or operational definition leads to low construct validity.

Mono-Operation Bias

Mono-operation bias occurs when only one method of measurement is used to assess a construct. When only one method of measurement is used, fewer dimensions of the construct are measured. Construct validity greatly improves if the researcher uses more than one instrument (Waltz et al., 2017). For example, if pain were a dependent variable, more than one measure of pain could be used, such as a pain rating scale, verbal reports of pain, physical responses (e.g., increased pulse, blood pressure, respirations), and observations of behaviors that reflect pain (e.g., crying, grimacing, guarding of painful area, pulling away). It is sometimes possible to apply more than one measurement of the dependent variable with little increase in time, effort, or cost. Using multiple methods of measuring a construct increases the construct validity (see Chapter 10).

Experimenter Expectancies (Rosenthal Effect)

The expectancies of the researcher can bias the data. For example, experimenter expectancy occurs if a researcher expects a particular intervention to relieve pain. The data that he or she collects may be biased to reflect this expectation. If another researcher who does not believe that the intervention would be effective had collected the data, results could have been different. The extent to which this effect actually influences studies is not known. Because of their concern about experimenter expectancy, some researchers choose not to be involved in the data collection process. In other studies, data collectors do not know which study participants were assigned to the intervention and control groups, which means that they were blinded to group assignment. Using nonbiased data collectors or those who are blinded to group assignment increases the construct design validity of a study.

Internal Validity

Internal validity is the extent to which the effects detected in the study are a true reflection of reality, rather than the result of extraneous variables. Internal validity is a concern in all studies, but is a major focus in studies examining causality. When examining causality, the researcher must determine whether the dependent variables may have been influenced by a third, often unmeasured, variable (an extraneous variable). The possibility of an alternative explanation of cause is sometimes referred to as a rival hypothesis (Shadish et al., 2002). Any study can contain threats to internal design validity, and these validity threats can lead to false-positive or false-negative conclusions (see Table 8.1). The researcher must ask, “Is there another reasonable (valid) explanation (rival hypothesis) for the finding other than the one I have proposed?” Some of the common threats to internal validity, such as study participant selection and assignment to groups, participant attrition, history, and maturation, are discussed in this section.

Participant Selection and Assignment to Groups

Selection addresses the process whereby participants are chosen to take part in a study and how they are grouped within a study. A selection threat is more likely to occur in studies in which randomization is not possible (Gray et al., 2017; Shadish et al., 2002). In some studies, people selected for the study may differ in some important way from people not selected for the study. In other studies, the threat is a result of the differences in participants selected for study groups. For example, people assigned to the control group could be different in some important way from people assigned to the intervention group. This difference in selection could cause the two groups to react differently to the intervention; in this case, the groups’ outcomes would not be due to the intervention, but to the differences in the individuals selected for the two groups. Random selection of participants for nursing studies is often not possible, and the number of participants available for studies is limited. The random assignment of participants to groups decreases the possibility of their selection being a threat to internal validity.

Participant Attrition

Attrition involves participants dropping out of a study before it is completed. Participant attrition becomes a threat (1) when those who drop out of a study are a different type of person from those who remain in the study or (2) there is a difference in the number and types of people who drop out of the intervention group and the people who drop out of the control or comparison group (see Chapter 9). If the attrition in a study is high (> 25%), this could affect the accuracy of the study results (Cohen, 1988; Gray et al., 2017).

History

History is an event that is not related to the planned study but that occurs during the time of the study. History could influence a participant’s response to the intervention or to the variables being measured and alter the outcome of the study. For example, if researchers studied the effect of an emotional support intervention on a participant’s completion of his or her cardiac rehabilitation program, and several nurses quit their job at the rehabilitation center during the study, this historical event would create a threat to the study’s internal design validity. Study participants who had worked closely with the nurses who quit may decide to stop participating in the study or working with different nurses might change the study outcomes.

Maturation

In research, maturation is defined as growing older, wiser, stronger, hungrier, more tired, or more experienced during the study. Such unplanned and unrecognized changes are a threat to the study’s internal validity and can influence the findings of the study. Maturation is more likely to occur in longitudinal studies with repeated measures of study variables.

External Validity

External validity is concerned with the extent to which study findings can be generalized beyond the sample used in the study (Gray et al., 2017). With the most serious threat, the findings would be meaningful only for the group studied. To some extent, the significance of the study depends on the number or types of people and situations to which the findings can be applied. Sometimes, the factors influencing external validity are subtle and may not be reported in research reports; however, the researcher must be responsible for these factors. Generalization is usually narrower for a single study than for multiple replications of a study using different samples, perhaps from different populations in different settings. Some of the threats to the ability to generalize the findings (external validity) in terms of study design are described here and summarized in Table 8.1.

Interaction of Selection and Intervention

Finding individuals who are willing to participate in a study can be difficult, particularly if the study requires extensive amounts of time and energy. Researchers must report the number of persons who were approached and refused to participate in the study (refusal rate) so those examining the study can identify any threat to external validity. If the refusal rate for a study is high, there is a greater potential for threats to the external design validity. For example if 39% of the persons approached to participate in a study declined, the sample actually selected will be limited in ways that might not be evident at first glance to the researchers. Only the researcher knows the participants well. They might be volunteers, “do-gooders,” or those with nothing better to do. In this case, generalizing the findings to all members of a population, such as all nurses, all hospitalized patients, or all persons experiencing diabetes, is not easy to justify.

Studies should be planned to limit the demands on people and increase their interest in participation. For example, researchers might select instruments that are valid and reliable but have fewer items to decrease participant burden. The study intervention must be skillfully developed and clearly communicated to individuals to increase their participation in the study. Sufficient data need to be collected on the participants to allow researchers to be familiar with their characteristics and, to the greatest extent possible, the characteristics of those who decline to participate (see Chapter 9).

Interaction of Setting and Intervention

Bias exists in regard to the types of settings and organizations that agree to participate in studies. This bias has been particularly evident in nursing studies. For example, some hospitals, such as those seeking Magnet designation, welcome nursing studies and encourage employed nurses to conduct studies. Others are resistant to the conduct of nursing research. These two types of hospitals may be different in important ways; thus there might be an interaction of setting and intervention that limits the generalizability of the findings.

Different settings may also serve different types of patients or potential study participants. For example, a low-income clinic may have patients with lower health literacy, whereas a clinic that only accepts patients with insurance might have a larger portion of college-educated patients. The intervention and measurement methods to be implemented may interact with the ability to read and comprehend written materials by the patients in different settings and cause variations in the study outcomes. Researchers must consider the characteristics of the settings and patients they serve when making statements about the population to which their findings can be generalized.

Interaction of History and Intervention

The circumstances occurring when a study is conducted might influence the intervention implemented or the outcomes measured, which could affect the generalization of the findings. For example, study participants receiving a supportive intervention to facilitate their dialysis process would probably have altered outcomes if three patients suddenly died as a result of dialysis equipment failure (historical event) during the study. Logically, one can never generalize to the future; however, replicating the study during various time periods strengthens the usefulness of findings over time. In critically appraising studies, you need to consider the effects of nursing practice and societal events that occurred during the period of the reported findings.

Statistical Conclusion Validity

The first step in inferring cause is to determine whether the independent and dependent variables are related. You can determine this relationship through statistical analysis. Statistical conclusion validity is concerned with whether the conclusions about relationships or differences drawn from statistical analysis are an accurate reflection of the real world (Grove & Cipher, 2017). The second step is to identify differences between and among groups. There are reasons why false conclusions can be drawn about the presence or absence of a relationship or difference. The reasons for the false conclusions are called threats to statistical conclusion validity (see Table 8.1). This text discusses some of the more common threats to statistical conclusion validity that you might identify in studies, such as low statistical power, unreliable measurement methods, limited intervention fidelity, and extraneous variances in the study setting. Shadish et al. (2002) have provided a more detailed discussion of statistical conclusion validity.

Low Statistical Power

Low statistical power increases the probability of concluding that there is no significant relationship between variables or significant difference between groups when actually there is one, a Type II error. A Type II error is most likely to occur when the sample size is small or when the power of the statistical test to determine differences is low (Cohen, 1988; Grove & Cipher, 2017). You need to ensure that the study has adequate sample size and power to detect relationships and differences. The concepts of sample size, statistical power, and Type II error are discussed in detail in Chapters 9 and 11.

Reliability or Precision of Measurement Methods

The technique of measuring variables must be reliable to reveal true differences. A measure is reliable if it gives the same result each time that the same situation or variable is measured. If a scale used to measure depression is reliable, it should give similar scores when depression is repeatedly measured over a short time period (Waltz et al., 2017). Physiological measures that consistently measure physiological variables are considered precise. For example, a thermometer would be precise if it showed the same reading when tested repeatedly on the same patient within a limited time (see Chapter 10). You need to examine the measurement methods in a study and determine if they are reliable.

Fidelity of the Intervention Implementation

Intervention fidelity ensures that the research intervention is standardized by a protocol and is applied consistently each time it is implemented in a study (Bova et al., 2017; Eymard & Altmiller, 2016). If the method of administering a research intervention varies from one person to another, the chance of detecting a true difference decreases. For example, one data collector might implement the study intervention to the first 20 participants and spend more time with them than was designated by intervention protocol; another data collector might then implement the intervention protocol exactly as was planned to the next 20 participants. Data collectors must be trained to ensure consistent or reliable implementation of a study intervention to prevent threats to the statistical conclusion validity.

Extraneous Variances in the Study Setting

Extraneous variables in complex settings (e.g., clinical units) can influence scores on the dependent variable. These variables increase the difficulty of detecting differences between the experimental and control groups. Consider the activities that occur on a nursing unit. The numbers and variety of staff, patients, health crises, and work patterns merge into a complex arena for the implementation of a study. Any of the dynamics of the unit can influence manipulation of the independent variable or measurement of the dependent variable. You might review the methods section of the study and determine how extraneous variables were controlled in the study setting. This discussion of design validity was presented to assist you in critically appraising the designs in the quantitative studies presented as examples in the next sections.

Descriptive designs

Descriptive studies are designed to gain more information about concepts, variables, or elements in a particular field of study. The purpose of these studies is to provide a picture of a situation as it naturally happens. A descriptive design may be used to develop theories, identify problems with current practice, make judgments about practice, or identify trends of illnesses, illness prevention, and health promotion in selected groups. No manipulation of variables is involved in a descriptive design. Protection against bias in a descriptive design is achieved through: (1) conceptual and operational definitions of variables; (2) sample selection and size; (3) valid and reliable measurement methods; and (4) data collection procedures that might partially control the environment or setting. Descriptive studies differ in level of complexity. Some contain only two variables; others may include multiple variables that are studied over time. You can use the algorithm shown in Fig. 8.2 to determine the type of descriptive design used in a published study. Simple descriptive and comparative descriptive designs are discussed in this chapter. Gray and colleagues (2017) have provided details about additional descriptive designs.

FIG 8.2 Algorithm for determining the type of descriptive design.

Simple Descriptive Design

A simple descriptive design is used to examine variables in a single sample (Fig. 8.3). This descriptive design includes identifying the variables within a phenomenon of interest, measuring these variables, and describing them. The description of the variables leads to an interpretation of the theoretical meaning of the findings and the development of possible relationships or hypotheses that might guide future correlational or quasi-experimental studies.

FIG 8.3 Simple descriptive design.

Critical appraisal guidelines

Descriptive and Correlational Designs

The critical appraisal guidelines presented in this section will be applied to the designs from example descriptive and correlational studies. You need to address the following questions when critically appraising descriptive and correlational study designs:

1. Is the study design descriptive or correlational? Review the algorithm in Fig. 8.1 to determine the type of study design.

2. If the study design is descriptive, use the algorithm in Fig. 8.2 to identify the specific type of descriptive design implemented in the study.

3. If the study design is correlational, use the algorithm in Fig. 8.5 to identify the specific type of correlational design implemented in the study.

4. Does the study design address the study purpose and/or objectives or questions?

5. Was the sample appropriate for the study?

6. Were the study variables measured with quality (reliable and valid) measurement methods (see Chapter 10)?

7. Was the data collection process implemented consistently and without bias?

Spratling (2017, p. 62) conducted a descriptive study to expand the understanding of healthcare utilization by children with tracheostomies who required medical technology. This study included a simple descriptive design; key aspects of this study’s design are presented in Research Example 8.1.

Research example 8.1

Simple Descriptive Design

Research Study Excerpt

Methods

In this study, a retrospective electronic health record (EHR) review was completed to identify common health problems that led to ED [emergency department] visits and hospitalization, and to create a data abstraction form… Charts were reviewed by the researcher and two trained Graduate Research Assistants [GRAs]…; all had experience with EHRs and medical terminology. The structured data abstraction form was created for the EHR review, and this was reviewed by an expert in technology dependent children and an expert in measurement prior to use (Spratling & Powers, 2017).…

The study sample included the EHRs on 171 children who require medical technology at an outpatient technology dependent pulmonary clinic that were reviewed over a three year period (January 2010–December 2012). Inclusion criteria were active clinic patients (newborn to age 21) who had at least one clinic visit since discharge from the hospital with a tracheostomy…

The study identified common health problems that led to ED visits and hospitalizations, used expert review to categorize these ED visits and hospitalizations as avoidable or unavoidable by expert review, and examined sociodemographic and clinical characteristics that affected the children’s ED visits and hospitalizations. Expert review included… a clinic nurse practitioner [NP] with 15 years of experience, and the researcher who has both clinical and research expertise with children who require medical technology.

(Spratling, 2017, p. 63)

Critical Appraisal

Spratling (2017) accurately identified their study as descriptive, with a retrospective design that involved EHR review. This simple descriptive design was appropriate to address the purpose and research questions of this study. The sample criteria were relevant to reduce the influence of extraneous variables. The sample size obtained over 3 years was strong (n = 171 children) and without attrition, which increases the representativeness of the sample and internal design validity. However, the setting was limited to only one pulmonary clinic in the South, which decreases the ability to generalize the findings. Consistency in data collection was facilitated by training the GRAs who used a quality data abstraction form developed and reviewed by experts. The structured unbiased collection of data strengthened the construct and statistical conclusion design validity of this study. Determining if ED visits and hospitalizations were avoidable or unavoidable was operationalized by experts in research and in the care of children who required medical technology, strengthening construct design validity (Shadish et al., 2002).

“The findings from this study noted an increased utilization of health care by these children, and identified common symptoms and medical technologies for which caregivers may need interventions, focusing on education in managing symptoms and medical technology prior to presentation to the ED or hospital” (Spratling, 2017, p. 62).

Comparative Descriptive Design

A comparative descriptive design is used to describe variables and examine differences in variables in two or more groups that occur naturally in a setting. The groups might be formed using gender, age, ethnicity and race, educational level, medical diagnosis, and/or severity of illness (Gray et al., 2017). Fig. 8.4 presents a comparative descriptive design’s structure.

FIG 8.4 Comparative descriptive design.

Mosleh, Eshah, and Almalik (2017, p. 418) conducted a descriptive study “to identify the differences in perceived learning needs between cardiac patients who have undergone major coronary interventions and their nurses.” The study participants were obtained by a sample of convenience and included 365 cardiac patients who had either a percutaneous coronary angioplasty (PTCA) or coronary artery bypass graft (CABG) and 166 cardiac nurses. Research Example 8.2 includes key elements of this comparative descriptive design.

Research example 8.2

Comparative Descriptive Design

Research Study Excerpt

Methods

A descriptive comparative design was used to examine the difference in perceived learning needs between cardiac patients who underwent major coronary interventional procedures and the nurses who provided care for them. The survey data were collected from three major hospitals…

Research assistants provided the self-report questionnaire along with verbal information about the study purpose… To ensure accuracy and consistency among the six research assistants, a workshop was held by the main researchers, wherein a full explanation of the study was provided. The steps of data collection were discussed with the assistants and they were trained in dealing with patients’ inquiries during the data collection process…

The Patient Learning Needs Scale (PLNS) was used to identify the learning needs of patients who have undergone PTCA or CABG… The PLNS comprises 40 items rated on a five-point Likert scale ranging from 1 (not important) to 5 (extremely important). This scale comprises several subscales corresponding to different types of learning needs, including wound care, medications, daily physical activities, diet, postintervention complications, postintervention care, and risk-factor management… The Cronbach’s alpha for PLNS in this study was 0.85; and the Cronbach’s alpha for the subscales ranged from 0.72-0.94… A pilot study including 10 patients and five cardiac nurses was conducted to confirm the stability and clarity of the scale’s items.

(Mosleh et al., 2017, pp. 420–421)

Critical Appraisal

Mosleh and colleagues (2017) clearly identified their study design, which addressed their research purpose and questions. The sample sizes were strong for both patients and nurses and included relevant sample criteria for these populations. The researchers chose to have research assistants collect the data, which reduced the potential for bias and strengthened construct validity (Shadish et al., 2002). The training of the research assistants improved the consistency of the data collected, strengthening the statistical conclusion validity. The learning needs were measured with the PLNS, which was reported as valid from previous research and was reliable for this study population, supporting statistical conclusion validity (Gray et al., 2017). In addition, the pilot study supported the use of this scale with these study participants. In summary, this comparative descriptive design included several strengths that promoted the trustworthiness of the study results and findings.

Mosleh et al. (2017) did find a disparity between the perceptions of patients and nurses on essential learning needs following a cardiac intervention. The researchers recommended that the nurses focus on information about wound care, medications, and postintervention complications before discharge because these were the priority needs of the patients. Education on diet and physical activity needs to be presented later in the patients’ recovery process.

Correlational designs

The purpose of a correlational design is to examine relationships between or among two or more variables in a single group in a study. This examination can occur at any of several levels— descriptive correlational, in which the researcher can seek to describe a relationship; predictive correlational, in which the researcher can predict relationships among variables; or the model testing design, in which the relationships proposed by a theory are tested simultaneously.

In correlational designs, a large range in the variable scores is necessary to determine the existence of a relationship. Therefore the sample should be large to reflect the full range of scores possible on the variables being measured (Grove & Cipher, 2017). Some study participants should have very high scores and others very low scores, and the scores of the rest should be distributed throughout the possible range.

The algorithm in Fig. 8.5 can be used to identify the specific correlational design included in a study. Sometimes, researchers combine elements of different designs to accomplish their study purpose. For example, researchers might conduct a cross-sectional, descriptive, correlational study design to examine the relationship of body mass index (BMI) to blood lipid levels in early adolescence (ages 13 − 16 years) and late adolescence (ages 17 − 19 years). It is important that researchers clearly identify the specific design that they are using in their research report. More details on correlational designs in this algorithm are available from other research sources (Gray et al., 2017; Kerlinger & Lee, 2000).

FIG 8.5 Algorithm for determining the type of correlational design.

Descriptive Correlational Design

The purpose of a descriptive correlational design is to describe variables and examine relationships among these variables. Using this design facilitates the identification of many interrelationships in a situation. The study may examine variables in a situation that has already occurred or is currently occurring. Researchers make no attempt to control or manipulate the situation. As with descriptive studies, variables must be clearly identified and defined conceptually and operationally (see Chapter 5). As shown in Fig. 8.6, the variables are measured, described, and examined for relationships. The findings from descriptive correlational studies are interpreted and provide the basis for further research.

FIG 8.6 Descriptive correlational design.

Branson, Loftin, Hadley, Hartin, and Devkota (2016, p. 185) conducted a correlational study to “explore the relationship between attendance and course grade in a prenursing course.” The sample included 445 prenursing students’ records for those enrolled in a required skills and safety course in a Texas university. Research Example 8.3 includes key aspects of this descriptive correlational design.

Research example 8.3

Descriptive Correlational Design

Research Study Excerpt

Methods

For this project, a descriptive-correlational design was used. A retrospective analysis of prenursing student sign-in-sheets collected over a 4-year period was accomplished, allowing us to explore the relationship between the final course outcome and attendance. For the purposes of this study, course outcome was operationalized as the final course grade, and course attendance was operationalized as the percentage of classes attended during the semester that course content was presented. At this university, each semester consists of 15 weeks, and course content was delivered via a variety of means during 13 of the 15 weeks. The first week of each semester was generally considered an introductory week with no course content presented, and the final week was devoted to the final course examination… Thus, there were 13 course sign-in sheets analyzed per class for each semester and included in this study… This course is available to sophomore-level prenursing students. Neither attendance nor participation during class was considered mandatory for this course, and neither was considered for grading purposes.

(Branson et al., 2016, p.186)

Critical Appraisal

Branson and colleagues (2016) identified their specific study design, which was relevant for their study purpose and research questions. A nonrandom sample of convenience (students in a prenursing course) was used, which is common for descriptive and correlational studies (Gray et al., 2017). Nonrandom sampling methods decrease the sample’s representativeness of the population; however, the sample size was strong (n  = 445) and produced significant results (no Type II error; Grove & Cipher, 2017). The variables of final grade and course attendance were clearly operationalized, promoting construct design validity (Shadish et al., 2002). The course grade came from the student record, and the process for obtaining course attendance was consistently implemented for one course over 4 years, supporting statistical conclusion validity. The study was limited to one setting, but data collection over several years increased the internal validity of the design. The potential for bias was decreased by not requiring course attendance or including it as part of the student’s grade. The design of this study was strong, and the results and findings generated seem to be representative of the study population.

Branson and colleagues (2016) found a significant positive relationship between course grade and class attendance (r443 = 0.54; p < 0.001; Grove & Cipher, 2017). These study results support the long-held faculty belief that class attendance has a positive impact on final course grades. These researchers recommended that nursing advisors and faculty stress the importance of class attendance on final course grades and on successful program progression.

Predictive Correlational Design

The purpose of a predictive correlational design is to predict the value of one variable based on the values obtained for another variable or variables. Prediction is one approach for examining causal relationships between variables. Because causal phenomena are being examined, the terms dependent and independent are used to describe the variables. The variable to be predicted is classified as the dependent variable, and all other variables are independent or predictor variables. A predictive correlational design study attempts to predict the level of a dependent variable from the measured values of the independent variables. For example, the dependent variable of medication adherence could be predicted using the independent variables of age, number of medications, and medication knowledge of patients with heart failure. The independent variables that are most effective in prediction are significantly correlated with the dependent variable but are not highly correlated with other independent variables used in the study (Grove & Cipher, 2017). The predictive correlational design structure is presented in Fig. 8.7. Predictive designs require the development of a theory-based hypothesis proposing variables expected to predict the dependent variable effectively. Researchers then use regression analysis to test the hypothesis (see Chapter 11).

FIG 8.7 Predictive correlational design.

De Santis, Hauglum, Deleon, Provencio-Vasquez, and Rodriguez (2017) conducted a correlational study to determine if human immunodeficiency virus (HIV) risk perception and HIV knowledge are predictive of sexual risk behaviors in transgender women. Research Example 8.4 includes major elements of this predictive correlational design.

Research example 8.4

Predictive Correlational Design

Research Study Excerpt

Methods

Design and Sample

This pilot study used a descriptive correlation design to study health issues among a sample of MTF [male-to-female] transgender women living in South Florida (n = 50), an area with a large lesbian, gay, bisexual, and transgender (LGBT) population… Participants were recruited from agencies that serve transgender women such as HIV testing and counseling centers, mental health counseling centers, and a university-based gender reassignment surgery clinic…. After obtaining informed consent, participants were given the study’s self-administered instruments to complete on paper…

Measures

HIV risk perception was measured using the four-item Perceived Risk for HIV Infection scale… While the instrument has not been previously used with transgender women, it has an overall reliability coefficient of .78… HIV knowledge was measured using the 18-item HIV knowledge Questionnaire-18… Although the instrument has not been previously used with transgender women, reliability coefficients of .76 to .94 have been reported… Sexual risk behaviors were measured using Behavior Risk Assessment Tool (BRAT), a clinical tool that assesses an individual’s risk of HIV infection… This instrument has not been used in research with transgender women…

Analytic Strategy

Correlation coefficients were used to examine the relationship among the continuous variables of HIV risk perception, HIV knowledge, and sexual risk behaviors. A regression analysis was used to determine variables associated with sexual risk behaviors.

(De Santis et al., 2017, pp. 211 − 212)

Critical Appraisal

De Santis and colleagues (2017) did not identify their study design as predictive correlational. However, the focus of this study was to predict the sexual risk behaviors of transgender women, and the results were generated by regression analysis (Grove & Cipher, 2017). The sample size was small for a correlational study, which resulted in low statistical power and contributed to the nonsignificant findings (Type II error; Gray et al., 2017). Because sample size is often smaller in pilot studies, additional research is needed for this understudied population. The measurement methods were a threat to construct validity because they had not been used with a transgender population and a threat to statistical conclusion validity because the researchers did not provide the reliability coefficients for the scales in this study. Consistency in data collection is unknown because participants self-administered the study questionnaires, threatening construct validity. De Santis et al. (2017) found that HIV risk perception and HIV knowledge were neither significantly correlated with nor predictive of sexual risk behaviors in transgender women. The researchers recommended further study with a larger sample size to examine factors contributing to sexual risk behaviors in this population. The design validity needs to be strengthened in future studies.

Model Testing Design

Some studies are designed specifically to test the accuracy of a hypothesized causal model (see Chapter 7). The model testing design requires that all concepts relevant to the model be measured and the relationships among these concepts examined. A large heterogeneous sample is required. Correlational analyses and structured equation modeling are conducted to determine the relationships among the model concepts, and the results are presented in the framework model for the study. This type of design is very complex; this text provides only an introduction to a model testing design implemented by Battistelli, Portoghese, Galletta, and Pohl (2013).

Battistelli and colleagues (2013) developed and tested a theoretical model to examine turnover intentions of nurses working in hospitals. The concepts of work-family conflict, job satisfaction, community embeddedness, and organizational affective commitment were identified as predictive of nurse turnover intention. The researchers collected data on these concepts using a sample of 440 nurses from a public hospital. The analysis of study data identified significant relationships among all concepts in the model. The results of this study are presented in Fig. 8.8 and indicate the importance of these concepts in predicting nurse turnover intention.

FIG 8.8 Results of the structural equation modeling analysis of the hypothesized model of turnover intention on the cross-validation sample (n = 440, standardized path loadings; P <0.05; two-tailed). (From Battistelli, A., Portoghese, I., Galletta, M., & Pohl, S. [2013]. Beyond the tradition: Test of an integrative conceptual model on nurse turnover. International Nursing Review, 60[1], 109.)

Elements of designs examining causality

Quasi-experimental and experimental designs are implemented in studies to obtain an accurate representation of cause and effect by the most efficient means. That is, the design should provide the greatest amount of control, with the least error possible. The effects of some extraneous variables are controlled in a study by using specific sampling criteria, a structured independent variable or intervention, and a highly controlled setting. An RCT is a type of experimental design often considered to be one of the strongest designs to examine cause and effect (Schulz, Altman, & Moher, 2010). RCTs are discussed later in this chapter. The essential elements of research to examine causality are:

• Random assignment of study participants to groups

• Precisely defined independent variable or intervention

• Researcher-controlled manipulation of the intervention

• Researcher control of the experimental situation and setting

• Inclusion of a control or comparison group in the study

• Clearly identified sampling criteria (see Chapter 9)

• Carefully measured dependent or outcome variables (see Chapter 10; Waltz et al., 2017)

Examining Interventions in Nursing Studies

In studies examining causality, investigators develop an intervention that is expected to result in differences in posttest measures between the treatment and control groups. Interventions may be physiological, psychosocial, educational, or a combination of these. The therapeutic intervention implemented in a nursing study needs to be carefully designed, clearly described, and appropriately linked to the outcomes (dependent variables) in the study. The intervention needs to be provided consistently to all study participants. The research report should document intervention fidelity, which includes a detailed description of the essential elements of the intervention and its consistent implementation during the study (Bova et al., 2017; Eymard & Altmiller, 2016). Sometimes, researchers provide a table of the intervention content and/or the protocol used to implement the intervention to each participant consistently. Researchers should indicate who implemented the intervention and what training was conducted to ensure consistent intervention implementation. Some studies document the monitoring of intervention fidelity (completeness and consistency of the intervention implementation) during the conduct of the study (Carpenter et al., 2013). For example, Bova et al. (2017, p. 54) examined the intervention fidelity for their study of 191 parents of young children newly diagnosed with type 2 diabetes, as indicated in the following quote: “Intervention fidelity was measured for both the intervention and control condition by direct observation, self-report of interventionist delivery, and parent participant receipt of educational information. Intervention fidelity data were analyzed after 50%, 75%, and 100% of the participants had been recruited and compared by group (treatment and control) and research site.” This intervention monitoring allowed Bova et al., 2017 to make corrections as needed in their multisite study.

Spiva and colleagues (2017) conducted a quasi-experimental study to examine the effectiveness of an evidence-based practice (EBP) nurse mentor training program on clinical nurses’ delivery of EBP. These researchers implemented a multifaceted intervention to train the EBP nurse mentors to prepare clinical nurses to incorporate research into their clinical practice. The components of their formal educational program (intervention) are detailed in Research Example 8.5.

Research example 8.5

Nursing Intervention

Research Study Excerpt

Interventions

A project charter and education curriculum were created by the study researchers, chief nursing officer (CNO) and a chief learning officer all knowledgeable in the principles of EBP and leadership development… Clinical nurse leaders, clinical nurse specialists, and educators received training to prepare them to serve as EBP mentors. Mentor training included didactic instruction and discussion, in-person training, and online interactive webinars (Table 1) to prepare a foundation to support and foster EBP.… Clinical nurses training included four 30-minute online modules designed to equip nurses with the tools and resources to translate evidence to practice and implement an EBP project (Table 2).

(Spiva et al., 2017, pp. 186–187)

Table 1

Educational Intervention Objectives for Nurse Mentor Training

•  Introduction to EBP

• Define and discuss the origins of EBP, QI, and research.

•  Guidelines for implementation

• Describe the Johns Hopkins EBP model and PET (practice question, evidence, and translation).

• Describe how to develop an answerable practice (EBP) question.

•  Search for evidence

• Describe how to search for evidence and available resources.

•  Appraisal of evidence

• Review tools to critically appraise evidence (both research and nonresearch).

• Discuss essential components of a research article.

• Evaluate research and nonresearch articles using appraisal tools (independent study).

•  Summarizing the evidence and beyond

• Provide an overview of frameworks to conduct EBP, QI, and research.

• Describe how to create a plan for translation, secure resources, and common evaluation methods.

•  Evaluate outcomes

• Provide an EBP project example from start to finish, including dissemination.

• Describe how to move a project from practice to abstract to presentation to publication.

• Identify the steps needed for poster and podium presentation development.

• List components of an abstract, poster, and podium presentation.

• Identify publication options.

From Spiva, L., Hart, P. L., Patrick, S., Waggoner, J., Jackson, C., & Threatt, J. L. (2017). Effectiveness of an evidence-based practice nurse mentor training program. Worldviews on Evidence-Based Nursing, 14(3), 185.

EBP, Evidence-based practice; QI, quality improvement.

Table 2

Computer-based learning module intervention objectives for clinical nurse training

Computer-Based Learning Module I (January 2015)

• Overview of EBP

• How to develop an EBP question

Computer-Based Learning Module II (March 2015)

• Overview of frameworks to conduct EBP, QI, and research

• An overview of available evidence resources, how to search for evidence, and how to demonstrate appraisal and translation of evidence

Computer-Based Learning Module III (April 2015)

• An overview of frameworks to conduct EBP, QI, and research

• How to create a plan for translation, secure resources, and review common evaluation methods to evaluate outcomes

• Review of a completed EBP project

Computer-Based Learning Module IV (May 2015)

• How to move an EBP project from practice to abstract to presentation

From Spiva, L., Hart, P. L., Patrick, S., Waggoner, J., Jackson, C., & Threatt, J. L. (2017). Effectiveness of an evidence-based practice nurse mentor training program. Worldviews on Evidence-Based Nursing, 14(3), 185.

EBP, Evidence-based practice; QI, quality improvement.

Critical Appraisal

The educational programs were developed and implemented by experts in the area of EBP and leadership. Tables 1 and 2 detailed the content and protocol for training the nurse mentors and clinical nurses. The implementation of the educational programs was structured and consistently implemented by experts. The structure of the nurse mentor training and the computer-based modules for the clinical nurses promoted intervention fidelity. Spiva et al. (2017) used a quasi-experimental design, which is presented as an example later in this chapter.

Experimental and Control or Comparison Groups

The group of participants who received the study intervention is referred to as the intervention, treatment, or experimental group. The group that is not exposed to the intervention is referred to as the control or comparison group. In some disciplines, the control groups receive no care or action, but this is not possible in most nursing studies because patients must receive care. For example, it would be unethical not to provide preoperative education to a patient in the control group of a study. Furthermore, in many studies, just spending time with patients or having them participate in activities that they consider beneficial may cause a change in the dependent variable. Therefore nursing studies often include a comparison group receiving a nursing action so that both groups (intervention and comparison) receive time and attention. This design structure allows the researchers to differentiate between the effect of time and attention and the effect of the intervention (Gray et al., 2017).

The standard nursing action is the care that the patients would receive if a study were not being conducted. Researchers should describe in detail the standard nursing care that the control or comparison group receives so that the study can be adequately appraised. Because the quality of this standard care is likely to vary considerably among study participants, variance in the control or comparison group is likely to be high and needs to be considered in the discussion of findings. Some researchers provide the experimental group with both the intervention and standard nursing care to control the effect of standard care in the study.

Quasi-experimental designs

Use of a quasi-experimental design facilitates the search for knowledge and examination of causality in situations in which complete control is not possible. This type of design was developed to control as many threats to validity (see Table 8.1) as possible in a situation in which some of the components of true experimental design were lacking (Shadish et al., 2002). Most studies with quasi-experimental designs have samples that were not selected randomly, and there is less control of the study intervention, extraneous variables, and setting. Most quasi-experimental studies include a sample of convenience, in which the participants are included in the study because they are in the right place at the right time (see Chapter 9). The participants selected are usually randomly assigned to receive the experimental intervention or standard care. The group that receives standard care is usually referred to as a comparison group versus a control group, which would receive no treatment or standard care (Shadish et al., 2002). However, the terms control group and comparison group are frequently used interchangeably in nursing studies.

Random assignment of participants from the original sample to either the intervention or comparison group promotes internal design validity. Occasionally, comparison and interventional groups may evolve naturally. For example, groups may include study participants who choose to be in the intervention group and those who choose not to receive the intervention as the comparison group. These groups cannot be considered equivalent because the participants who choose to be in the comparison group probably differ in important ways from those who select to be in the intervention group. For example, if researchers were implementing an intervention of a structured exercise program to promote weight loss, the participants should not be allowed to select whether they are in the intervention group receiving the exercise program or the comparison group receiving standard care. Participants’ self-selecting to be in the intervention or comparison group is a threat to the internal design validity of a study.

Quasi-experimental designs have varying levels of control of the sampling process, group assignment, intervention, setting, and extraneous variables. The algorithm in Fig. 8.9 identifies a variety of quasi-experimental designs so you can identify the type of quasi-experimental design in a study. More details about specific designs identified in this algorithm are available from other sources (Gray et al., 2017; Shadish et al., 2002).

FIG 8.9 Algorithm for determining the type of quasi-experimental design.

Pretest and Posttest Designs With Comparison Group

Quasi-experimental study designs vary widely. The most frequently used design in social science research is the untreated comparison group design, with a pretest and posttest (Fig. 8.10). With this design, the researchers have a group of participants who received the experimental intervention and a comparison group of participants who received standard care.

FIG 8.10 Quasi-experimental pretest and posttest design with a comparison group.

Another commonly used design is the posttest-only design with a comparison group, shown in Fig. 8.11. This design is used in situations in which a pretest is not possible. For example, if the researcher is examining differences in the amount of pain that a study participant feels during a painful procedure, and a nursing intervention is used to reduce pain for participants in the experimental group, it might not be possible (or meaningful) to pretest the amount of pain before the procedure. This design has a number of threats to validity because of the lack of a pretest (Shadish et al., 2002).

FIG 8.11 Quasi-experimental posttest-only design with a comparison group.

Critical appraisal guidelines

Quasi-Experimental and Experimental Designs

When critically appraising the design of a quasi-experimental or experimental study, you need to address the following questions:

1. Is the study design quasi-experimental or experimental? Review the algorithm in Fig. 8.1 to determine the type of study design.

2. Identify the specific type of quasi-experimental or experimental design used in the study. Review the algorithm in Fig. 8.9 for the types of quasi-experimental study designs and the algorithm in Fig. 8.12 for the types of experimental designs.

FIG 8.12 Algorithm for determining the type of experimental design.

3. What were the strengths and threats to validity (construct validity, internal validity, external validity, and statistical conclusion validity) in the study (see Table 8.1)? Review the methods section and limitations identified in the discussion section of the study report for ideas.

4. Which elements were controlled and which elements could have been controlled to improve the study design? Review the sampling criteria, sample size, assignment of participants to groups, and study setting.

5. Was the study intervention described in detail? Was a protocol developed to ensure consistent or reliable implementation of the intervention with each participant throughout the study? Did the study report indicate who implemented the intervention? If more than one person implemented the intervention, how were they trained to ensure consistency in the delivery of the treatment? Was intervention fidelity achieved in the study (Bova et al., 2017; Eymard & Altmiller, 2016; Murphy & Gutman, 2012)?

6. Were the study dependent variables measured with reliable and valid measurement methods (Waltz et al., 2017)?

Spiva and colleagues (2017) conducted a quasi-experimental study to examine the effects of an EBP nurse mentor training program (see Table 1) and clinical nurse module intervention (see Table 2) on the outcomes related to EBP. These two interventions were introduced in the previous section, and we encourage you to locate this article on the website for this text and critically appraise the design of this study. The critical appraisal of this study was conducted using the Guidelines for Critically Appraising Quasi-Experimental and Experimental Designs. The study design of Spiva et al. (2017) was very complex, so only selected content from the methods section of their study is presented in Research Example 8.6.

Research example 8.6

Quasi-Experimental Pretest-Posttest Design With a Comparison Group

Research Study Excerpt

Methods

Design

A two-group, pretest-posttest, quasi-experimental, interventional design was used…

Measures

The Evidence-Based Nursing Questionnaire was used to measure conditions that impede or sustain evidence-based nursing. The questionnaire has undergone validity and reliability testing. In our sample, subscales used included Cronbach’s alpha coefficients of .87 total scale, .80 for organizational support (readiness), .92 nurses’ beliefs and attitudes of research evidence, .79 EBP skills, and .80 nurses’ knowledge of research language and statistics…

Confidence scale was used… to measure nurses’ perceived confidence in their knowledge and ability to implement EBP. Higher mean scores indicate a higher perception of confidence. Content validity was established by five expert nurses knowledgeable in the field of EBP and staff education… The content validity index was .90. In a previous pilot study, the Cronbach’s alpha coefficient was .94 and in this sample .96.….

The 29-item Barriers to Research Utilization Scale was used to measure perceived barriers to research utilization… This scale has undergone extensive testing and is deemed reliable and valid. In our study, Cronbach’s alpha coefficients were total scale .96, .85 communication, .90 adopter, .89 organization, and .87 innovation.

(Spiva et al., 2017, p. 184)

Sample and Recruitment

A convenience sample of registered nurses and nurse mentors working in a five-hospital integrated nonprofit healthcare system located in the Southeast were recruited. Recruitment pool included 1,916 nurses… Sixty-six (66) mentors participated and completed the surveys… Initially, 793 [clinical nurses] completed the presurvey and module one. The final sample included 367 who completed the pre- and post-surveys and all modules.

(Spiva et al., 2017, p. 187)

Critical Appraisal

Spiva and colleagues (2017) identified the specific quasi-experimental design used in their study, which addressed the study aims. However, the study would have been stronger if hypotheses had been developed to direct the study. The sample of convenience decreased the representativeness of the population but the setting included five hospitals, which increased representativeness. The sampling criteria were not clearly addressed in the study, and the nurses were not randomized into groups, resulting in threats to internal design validity. No attrition occurred from the nurse mentor group, but 426 (54%) of the clinical nurses dropped out of the study after the first module, resulting in possible threats to internal and statistical conclusion validity.

The instruments used in this study had reliability and validity from previous research and had strong reliability (most Cronbach alphas were > 0.8) in this study, resulting in construct and statistical conclusion design validity. The structured interventions for the nurse mentors and clinical nurses were developed by experts and documented in the research report (see Tables 1 and 2). A historical event of implementing a new electronic medical record during the study delayed the training of the nurse mentors, which in turn delayed the training of the clinical nurses. Spiva et al. (2017) thought this delay probably caused the high attrition for the clinical nurses. The researchers noted that the threats to design validity might have altered the true effects of the interventions, and further study is needed. However, Spiva et al. (2017, p. 183) concluded that “EBP mentors are effective in educating and supporting nurses in evidence-based care. Leaders should use a multifaceted approach to build and sustain EBP, including developing a critical mass of EBP mentors to work with point of care staff.” This type of study promotes EBP in nursing to ensure quality and safe care (Quality and Safety Education for Nurses [QSEN, 2014]; Sherwood & Barnsteiner, 2017).

Experimental designs

A variety of experimental designs, some relatively simple and others very complex, have been developed for studies focused on examining causality. In some cases, researchers may combine characteristics of more than one design to meet the needs of their study. Names of designs vary from one text to another. When reading and critically appraising a published study, determine the author’s name for the design (some authors do not name the specific design used), and/or read the description of the design to determine the type of design used in the study. Use the algorithm shown in Fig. 8.12 to determine the type of experimental design used in a study. More details about the specific designs identified in Fig. 8.12 are available from other sources (Gray et al., 2017; Shadish et al., 2002).

Classic Experimental Pretest and Posttest Designs With Experimental and Control Groups

A common experimental design used in healthcare studies is the pretest-posttest design with experimental and control groups (Campbell & Stanley, 1963; Shadish et al., 2002). This design is shown in Fig. 8.13; it is similar to the quasi-experimental design in Fig. 8.10, except that the experimental study is more tightly controlled in the areas of intervention, setting, measurement, and/or extraneous variables, resulting in fewer threats to design validity. The experimental design is stronger if the initial sample is randomly selected; however, most healthcare studies do not include a random sample but do randomly assign participants to the experimental and control groups. Most studies in nursing use the quasi-experimental designs shown in Fig. 8.9 because of the inability to control selected extraneous and environmental variables.

FIG 8.13 Experimental pretest-posttest control group design.

Multiple groups (both intervention and control) can be used to great advantage in experimental designs. For example, one control group might receive no treatment, another control group might receive standard care, and another control group might receive a placebo or intervention with no effect, like a sugar pill in a drug study. Each one of multiple experimental groups can receive a variation of the intervention, such as a different frequency, intensity, or duration of nursing care actions. For example, a different frequency, intensity, or duration of massage treatments might be implemented in a study to determine their effect(s) on patients’ back pain. These additions greatly increase the generalizability of study findings when the sample is representative of the target population and the sample size is strong.

Posttest-Only With Control Group Design

The experimental posttest-only control group design is also frequently used in healthcare studies when a pretest is not possible or appropriate. This design is similar to the design in Fig. 8.13, with the pretest omitted. The characteristics of the experimental and control groups are usually examined at the start of the study to ensure that the groups are similar. The lack of a pretest does, however, increase the potential for error that might affect the findings. Additional research is recommended before generalization of findings.

McWilliams, Malecha, Langford, and Clutter (2017, p. 154) conducted an experimental study to examine “the effectiveness of cooperative team learning compared with independent learning when used with nursing students who are learning intravenous (IV) catheter insertion using a haptic IV simulator.” Two convenience samples (n = 180) of junior-level nursing students attending the fall and spring semester (2015 − 2016) at a university in southeast Texas were randomized into four groups (A, B, C, and D). Participants from groups A, B, and C were used to make up each cooperative learning team, and group D participants included the independent learners. The study attrition was small (n = 6; 3.3%). The researchers conducted their study with a posttest-only experimental design that is presented in Research Example 8.7.

Research example 8.7

Experimental Posttest-Only Control Group Design

Research Study Excerpt

Methodology

A posttest-only experimental research design was used to evaluate the effectiveness of cooperative team learning compared with independent learning with nursing students while using the haptic IV simulator. The initial performance score and the number of attempts to earn a passing performance score on the haptic IV simulator (score of 85 or better) were used to examine differences between the four-student group assignments.

To increase the reliability of the study, a researcher-designed procedural checklist, including a script, was utilized. The goal of the checklist was to ensure that the primary investigator (PI) was consistent with all interactions with the students from random assignment and sequencing to instructions regarding procedures, use of assigned usernames/passwords, reiterating that grades earned on the simulator were not connected to their course, and the language needed to describe how the cooperative learning teams were to work together.

The Virtual Intravenous Simulator by Laerdal Corporation was designed to support learning of IV catheter insertion. The simulator includes an IV catheter/hub assembly and an interface that allows students to palpate a vein, stretch the skin, and feel resistance during venipuncture. Additionally, during the simulated cannulation, a computer screen provides immediate feedback related to bleeding, bruising, and swelling.

(McWilliams et al., 2017, p. 156)

Procedure

On the day of their haptic IV simulation, the IV simulation cluster arrived in the nursing skills laboratory, and each learner was given an envelope with a group assignment (A, B, C, or D) listed on the outside… An independent learner (group assignment D) was brought to the first haptic IV simulator. After signing into the IV simulator, the independent learner was instructed to follow the instructions in his/hers envelope and to view the IV tutorial. The tutorial informed the learner how to use the IV simulator… The cooperative team of learners (group assignments A, B, C) was escorted to the second haptic IV simulator. The PI reviewed the instructions in the team’s envelope and presented information on how the team must work together on the IV simulator until all team members completed the task…

For this study, the initial performance score earned by each learner on the haptic IV simulator, and the number of attempts to earn a passing performance score were recorded as the dependent variables. The initial performance scores were obtained from the haptic IV simulator’s computer printout. The number of attempts to earn a passing performance score was obtained by logging into the computer system of the simulator and counting the number of attempts required by each learner.

(McWilliams et al., 2017, p. 157)

Critical Appraisal

McWilliams and colleagues (2017) identified their specific experimental design which addressed the study purpose and hypotheses. The sample of convenience and the use of only one university for the study setting decreased the sample’s representativeness of the nursing student population. However, the sample size was strong, attrition was low (3.3%), and the results were significant, indicating strength in internal and statistical conclusion validity. The random assignment of participants to the cooperative and independent learning groups was presented in detail, strengthening internal and external design validity (see Table 8.1; Shadish et al., 2002).

The researchers promoted intervention fidelity by using a procedural checklist and script (statistical conclusion validity; Eymard & Altmiller, 2016), and implemented the intervention using a quality IV simulator (external validity). McWilliams et al. (2017) clearly operationalized their study dependent variables—performance score earned and number of attempts to earn a passing score—that were measured using the simulator’s computer. These computer-generated measures ensured the accuracy of the data collected and the construct validity of the design. The detailed control of the intervention, setting, and data collection process are consistent with implementing a quality, experimental study design (Gray et al., 2017; Shadish et al., 2002).

McWilliams and colleagues (2017, p. 154) found that the “cooperative team members performed better with fewer attempts than independent learners when using an IV simulator…. This study provided empirical evidence that supports the efficacy of simulation as a means of learning a psychometric skill.”

Randomized Controlled Trials

Currently, in nursing and medicine, the randomized controlled trial (RCT) is noted to be the strongest methodology for testing the effectiveness of an intervention because of the elements of the experimental design that limit the potential for bias and error. Participants are randomized to the intervention and control groups to reduce selection bias (Carpenter et al., 2013; Schulz et al., 2010). In addition, blinding or withholding of study information from data collectors, participants, and their healthcare providers can reduce the potential for bias. RCTs, when appropriately conducted, are considered the gold standard for determining the effectiveness of healthcare interventions. RCTs may be carried out in a single setting or in multiple geographic locations to increase sample size and obtain a more representative sample.

The initial RCTs conducted in medicine demonstrated inconsistencies and biases. Consequently, a panel of experts—clinical trial researchers, medical journal editors, epidemiologists, and methodologists—developed guidelines to assess the quality of RCT reports. This group initiated the Standardized Reporting of Trials (SORT) statement that was revised and became the CONsolidated Standards for Reporting Trials (CONSORT). This current guideline includes a checklist and flow diagram that might be used to develop, report, and critically appraise published RCTs (CONSORT, 2010). Nurse researchers should follow the CONSORT 2010 statement recommendations in conducting and reporting RCTs (Schulz et al., 2010). You might use the flow diagram in Fig. 8.14 to critically appraise the RCTs reported in nursing journals. An RCT needs to include the following elements:

1. The study was designed to be a definitive test of the hypothesis that the intervention caused the defined dependent variables or outcomes.

2. The intervention is clearly described and consistently implemented to ensure intervention fidelity (Bova et al., 2017; CONSORT, 2010; Schulz et al., 2010; Yamada, Stevens, Sidani, Watt-Watson, & De Silva, 2010).

3. The study is conducted in a clinical setting, not in a laboratory.

4. The design meets the criteria of an experimental study (Schulz et al., 2010).

5. Study participants are drawn from a reference population through the use of clearly defined criteria. Baseline values are comparable in all groups included in the study. Selected participants are then randomly assigned to treatment and comparison groups (see Fig. 8.14), hence the term randomized controlled trial (CONSORT, 2010; Schulz et al., 2010).

6. The study has high internal validity. The design is rigorous and involves a high level of control of potential sources of bias that will rule out possible alternative causes of the effect (Shadish et al., 2002). The design may include blinding to accomplish this purpose. With blinding, the patient, those providing care to the patient, and/or the data collectors are unaware of whether the patient is in the experimental group or in the control group.

7. Dependent variables or outcomes are measured consistently with quality measurement methods (Waltz et al., 2017).

8. The intervention is defined in sufficient detail so that clinical application can be achieved (Schulz et al., 2010).

9. The participants lost to follow-up are identified with their rationale for not continuing the study. The attrition from the experimental and control groups needs to be addressed, as well as the overall sample attrition.

10. The study has received external funding sufficient to allow a rigorous design with a sample size adequate to provide a definitive test of the intervention.

FIG 8.14 CONSORT 2010 statement showing a flow diagram of the progress through the phases of a parallel randomized trial of two groups: enrollment, intervention allocation, follow-up, and data analysis. (From CONSORT. [2010]. The CONSORT flow diagram. Retrieved June 26, 2017, from http://www.consort-statement.org/consort-statement/flow-diagram; and Schulz, K. F., Altman, D. G., & Moher, D. [2010]. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomized trials. Annals of Internal Medicine, 152[11], 726 − 733.)

Hallas, Koslap-Petraco, and Fletcher (2017, p. 33) conducted an RCT “to examine the effectiveness of an office-based educational program to improve maternal confidence and social-emotional development of toddlers.” The sample of convenience included mother and toddler dyads obtained from five pediatric primary healthcare offices and clinics in New York. The details of the sample size, randomization to groups, and attrition are presented in Fig. 4 in Research Example 8.8. This study was funded by a grant from the New York University Research Fund. Research Example 8.8 briefly presents the experimental design of this study.

FIG. 4 Participants. (From Hallas, D., Koslap-Petraco, M., & Fletcher, J. [2017]. Social-emotional development of toddlers: Randomized controlled trial of an office-based intervention. Journal of Pediatric Nursing, 33[1], 37.)

Research example 8.8

Randomized Controlled Trial (RCT)

Research Study Excerpt

Methods

Trial Design

A prospective, double blind, randomized controlled trial using pretest/posttest experimental design was used to test the effectiveness of a videotape (DVD) parenting skills intervention on the social-emotional development of toddles and on maternal confidence of mothers caring for their toddlers… Two treatment intervention DVDs were designed: one for teenage mothers and one for all other mothers… The control group intervention was a standardized DVD on toddler nutrition…. The DVDs were available in English and Spanish. The wrapping for the DVD was the same as the one for the intervention DVD, thus the identification of the DVD contents (treatment and control) were concealed from the mothers, the RNs, and research assistants (RAs).…

After enrollment, each mother-toddler dyad was given a folder that had been randomized into either the treatment or control group using a computer-generated random numbers list… All RNs and RAs were blinded to the folder contents, as all folders were identical and the DVDs were all labeled with a code…

Outcome Assessments…

The Toddler Care Questionnaire (TCQ) is a measure of maternal confidence for all mothers of toddlers between the ages of 12 and 36 months… The TCQ instrument reliability is reported to be between 0.91 and 0.96 and test-retest reliability is 0.87… The Brigance Toddler Screen was used for toddlers between the ages of 12 to 33 months old… to measure their social-emotional skills… All participants completed the same study instruments, thus no information about the group assignments was known to the RNs or RAs. Since all participants watched a DVD privately, the participants did not know if they were assigned to the treatment or control group (Fig. 4).

(Hallas et al., 2017, pp. 35–36)

Critical Appraisal

Hallas and colleagues (2017) identified the specific RCT design they used to address their study purpose and hypotheses. The sampling process, randomization to groups, and limited attrition are detailed in Fig. 4, indicating support for internal, external, and statistical conclusion design validity (Shadish et al., 2002). The researchers detailed their study intervention (DVDs of parenting skills) and controlled its implementation using checklists and scripts (Bova et al., 2017; Yamada et al., 2010). The steps taken to promote intervention fidelity by blinding mothers, RNs, and RAs to group assignment reduced the potential for bias and added to the internal and external design validity. The scales used to collect data were reliable in previous research and consistently administered in this study. However, no reliability values were provided for the scales in this study, causing a threat to statistical conclusion validity. In summary, Hallas et al. (2017) closely followed the CONSORT (2010) guidelines in conducting and reporting their RCT, which reduced the potential for bias and promoted trustworthy study results. The researchers concluded that the DVDs were a significant and efficient way to educate mothers of toddlers in office waiting rooms in order to improve their toddlers’ social-emotional development.

Key Points

•  A research design is a blueprint for conducting a quantitative study that maximizes control over factors that could interfere with the validity of the findings.

•  Four common types of quantitative designs conducted in nursing include descriptive, correlational, quasi-experimental, and experimental designs.

•  The concepts important in understanding quantitative research designs include causality, multicausality, probability, bias, prospective, retrospective, control, and manipulation.

•  Elements central to the study design include the presence or absence of an intervention, method of sampling, number of groups in the sample, number and timing of measurements to be performed, time frame for data collection, planned comparisons, and control of extraneous variables.

•  Study validity is a measure of the truth or accuracy of the findings obtained from a study. Four types of validity are covered in this text: construct, internal, external, and statistical conclusion.

•  Descriptive and correlational designs, called nonexperimental or noninterventional designs, focus on the description and examination of relationships among variables.

•  Cross-sectional design involves examining a group of participants simultaneously in various stages of development, levels of educational, severity of illness, or stages of recovery to describe changes in a phenomenon across stages.

•  Longitudinal design involves collecting data from the same participants at different points in time and might also be referred to as repeated measures.

•  Correlational designs are of three different types: (1) descriptive correlational, in which the researcher can seek to describe a relationship; (2) predictive correlational, in which the researcher can predict relationships among variables; and (3) model testing design, in which all the relationships proposed by a theory are tested simultaneously.

•  Interventions or treatments are implemented in quasi-experimental and experimental studies to determine their effect on selected dependent variables. Interventions may be physiological, psychosocial, educational, or a combination of these.

•  The essential elements of experimental research are: (1) the random assignment of participants to groups; (2) the researcher’s manipulation of the independent variable; and (3) the researcher’s control of the experimental situation and setting, including a control or comparison group.

•  Critically appraising a design involves examining the study setting, sample, intervention, measurement of variables, and data collection procedures.

•  RCT design is noted to be the strongest methodology for testing the effectiveness of an intervention because the elements of the design limit the potential for bias.