PICOT Statement Paper

UnderstandingtheEvidence.pdf

Home >Nursing homework help > PICOT Statement Paper

36 | Spring 2010 • Vol. 34 .No. 1

G E N E R AT IO N S – Journal of the American Society on Aging

Copyright © 2010 American Society on Aging; all rights reserved. This article may not be duplicated, reprinted or distributed in any form without written permission from the publisher: American Society on Aging, 71 Stevenson St., Suite 1450, San Francisco,CA 94105-2938; e-mail: info@asaging.org.

Practitioners’ understanding of research methodology and researchers’ understand- ing of clinical issues are both needed to in- crease the development and use of evidence- based practice. Taking an evidence-based approach to practice means integrating the best available research with clinical expertise. “Evidence” refers to research findings that link a particular type of treatment to a particular outcome for a particular population, indicating that a treatment “works.” Evidence for a particular treatment may or may not exist, and existing evidence may or may not be good. This article discusses aspects of research methodol- ogy that can be used to determine whether evidence exists and whether or not it is suffi- cient. Dynamic, continuous interaction between researchers and practitioners is required to advance evidence-based practice. Critical factors in promoting evidence-based practice include commitment on the part of both practitioners and researchers to the importance of this work, willingness to become educated about the process, and development of skills needed to make it happen (Nelson and Steele, 2007). The suggestion is not for researchers to

become clinicians or clinicians to become researchers. Researchers maintain their scien- tific perspective, and practitioners maintain their clinical autonomy. Yet, by learning about important issues in practice, researchers can ask questions that are clinically informed, and thus useful for practice. For their part, practi- tioners who are committed to conducting evidence-based practice can learn about research methodology to assist them in identi- fying appropriate practices and utilizing treatment guidelines to provide services that are more scientifically based.

An understanding of research methodology can also enable practitioners to better imple- ment, document, and show how well specific evi- dence-based practices work in their particular settings. This clinical experience can then inform the design of further research to test additional types of interventions. These interac- tive, dynamic processes occur over time between researchers and practitioners to ensure the use of effective strategies to meet desired outcomes in terms of quality of life and quality of care for older adults and their families. The goal is to integrate scientific knowledge and effective

By Joann P. Reinhardt

Research Methods in Evidence-Based Practice: Understanding the Evidence

Dynamic, continuous interaction between researchers and practitioners is required to advance evidence-based practice.

Evidence-Based Practice: Critical Thinking About Translating Research to Practice

Spring 2010 • Volume 34 .Number 1 | 37

Pages 36–42

©American Society on Aging

interventions into usual care practices. To promote this integrative effort, this paper defines and discusses research terminology, types of research designs, and methodological issues such as sampling and attrition, signifi- cance, and replication.

Research Terminology and Design Researchers study two types of variables,

independent and dependent. An independent variable is manipulated or observed, and a dependent variable is the outcome of interest. Often, a study uses multiple independent and dependent variables. Such studies are designed to determine the effect of the independent variables on the dependent variable. For example, what is the effect of coping strategies and social support (independent variables) on well-being (a dependent variable)? To under- stand what this study is looking at, the defini- tions and specific measures for coping strate- gies and social support must be examined

closely. For example, social support measures may assess support that is received or per- ceived, affective or instrumental, provided by family or friends. Similarly, measures of well- being vary and may include variables that are positive (high scores indicate better well-being) such as life satisfaction, happiness, or positive affect, or variables that are negative (low scores indicate better well-being) such as depressive symptomatology, anxiety, or negative affect. Study findings may differ based on how a variable is defined and measured. For example, a study of older adults adapting to age-related vision loss showed that perceived, affective support was associated with better adaptation

to vision loss and lower depressive symptoms, yet received, instrumental support was associ- ated with higher depressive symptoms (Rein- hardt, Boerner, and Horowitz, 2006).

Research designs can be experimental, in which the researcher manipulates the conditions that are being measured, or observational, in which the researcher measures an already existing situation. Each of these designs can make contributions to evidence-based practice, although the experimental design is most common when we think about “evidence.”

In an experiment, a researcher manipulates an independent variable (often a treatment, program, or other type of intervention), and then observes its effect on a dependent, or outcome, variable in a group of participants. There are various types of experimental designs, but most contain two groups of participants, one group that receives a treatment, and a comparison group that does not receive the treatment. The way these two groups are selected determines

the type of design. In an experimental design, the treatment and control groups are randomly assigned. In a quasi-experimental design, the two groups cannot be assigned randomly for ethical or practical reasons, and thus preexisting groups are selected for comparison. The researcher is better

able to establish a causal inference with the experimental design, because the two groups are selected randomly.

Randomized Controlled Trials The gold standard design for intervention

research is the randomized clinical, or con- trolled, trial (RCT). A key feature of this design is the random assignment—assignment by chance alone—of study participants to either a group that receives the treatment being studied (the “treatment group”) or a group that does not receive the treatment (the “control group”). Because the assignment of subjects to treatment groups is random, factors like medication use,

An understanding of research methodology can also enable practitioners to better show how well specific evidence-based practices work in their particular settings.

G E N E R AT IO N S – Journal of the American Society on Aging

38 | Spring 2010 • Volume 34 .Number 1

Pages 36–42

©American Society on Aging

age, and other characteristics of individuals (potential confounding factors) should be equally distributed between the two groups, so the researcher can infer that any differences in outcomes between the two groups are the result of the treatment. Such an experiment is referred to as having “internal validity,” meaning that the effect on the dependent variable is caused by the independent variable, not by some other factor that is not controlled in the experiment. The RCT is considered a true experiment and one of the most powerful tools in clinical research because it provides the potential to show a causal relationship between the treat- ment variable and the outcome—that is, that the treatment that was tested affected the outcome. For example, a researcher may want to examine whether or not a particular type of psychothera- py affects the level of depressive symptoms in older adults. With the focus in this case on seeing a change (reduction) in depressive symptoms over time, the population targeted for this study should be older adults who are depressed. In this type of design, a pre-test of the outcome is conducted for all participants who have provided informed consent to partici- pate in the study. The pre-test is a measure of the variables of interest (e.g., level of depressive symptoms) before a program or intervention is administered. After this initial assessment is completed, study participants are randomly assigned to either the treatment or the control group. The treatment is then administered to the treatment group only. After the treatment is administered, a post-test is carried out for all participants, to see if there is a significant change in the outcome variable. If the treatment works, there should be a significant change in the outcome variable for the treatment group only. An important aspect of intervention research is documenting fidelity, that is, show- ing that the specified intervention procedures were actually followed. If the intervention procedures are followed and documented and the treatment does not have a significant effect

on a particular outcome variable, one may conclude that the intervention was not effective. If, on the other hand, the procedures were not documented, and the intervention did not work, the latter may instead be due to improper implementation. Study findings would be inconclusive in this case.

Interestingly, the more we exert experimen- tal control to determine whether or not a particular treatment affects a particular out- come, the more we move away from real-world circumstances. In practice, older adults may have a range of physical health conditions that could also be associated with their mental health status, yet researchers studying a particular depression intervention would likely exclude participants with specific coexisting physical conditions.

A criticism of randomized controlled trials is that the strict eligibility criteria that are often applied make the samples in such cases unrepre- sentative of the general population affected by the condition being studied (Adams, LeCroy, and Matto, 2009; Gartlehner et al, 2006). Older adults seen in practice settings would rarely fit the eligibility criteria used in controlled trials, and thus we may wonder if particular treatments shown to be effective in these trials are truly generalizable to real-world situations or appli- cable to these older adults. A study is said to have “external validity” if the results shown are generalizable to other people and other situa- tions. To know if a particular treatment works, controlled experiments must be conducted, yet these treatments may eventually be applied to usual care, where the recipients differ from those on whom the treatment was tested.

Efficacy, effectiveness, and feasibility trials Interventions can be tested via either efficacy

or effectiveness trials. Trials may be conducted to show whether or not interventions show “efficacy” in research settings, or “effectiveness” in practice settings. In efficacy studies, treat- ments are studied under carefully controlled

Evidence-Based Practice: Critical Thinking About Translating Research to Practice

Spring 2010 • Volume 34 .Number 1 | 39

Pages 36–42

©American Society on Aging

circumstances, to see if they work in this ideal situation (Gartlehner et al., 2006). The goal is to minimize factors that could alter a treatment effect, so researchers are confident that the treatment being tested has produced the desired effects. Potential respondents may be excluded if their characteristics deviate from the protocol being studied. Findings provide information about whether or not a particular treatment works for a certain population under ideal circumstances. As noted above, such controlled conditions do not exist in clinical practice. Compliance with the treatment, adverse events, other treatments, and coexisting conditions can all affect the efficacy of an intervention being tested in practice. Thus, highly controlled efficacy research may be considered less appli- cable to practitioners’ work.

In effectiveness trials, on the other hand, steps are taken to ensure generalizability, yet these steps can actually compromise the trial’s internal validity (assurance that the experiment actually measures what it intends to measure)

(Gartlehner et al., 2006). Ideally, both satisfac- tory internal validity and high degree of gener- alizability would be present for a given inter- vention. Effectiveness trials are studies that examine whether or not treatment effects change under real-life conditions or clinical settings. Study designs and hypotheses for effectiveness trials are formulated based on routine clinical practice and outcomes that are important for making clinical decisions (Gartlehner et al., 2006). Effectiveness trials may have less stringent participant eligibility, longer study duration, and more clinically relevant treatment modalities compared to efficacy trials (Gartlehner et al., 2006).

Another distinguishing feature is compliance, which is adherence to a recommended course of treatment. Compliance with a particular treat- ment is more likely in a controlled setting and thus can be demonstrated as evidence of the fidelity of an intervention in an efficacy trial. However, in an effectiveness trial, compliance may or may not be obtained. In fact, compliance can be an additional outcome measure in an effectiveness trial. Poor compliance can actually render an efficacious treatment ineffective (Gartlehner et al., 2006). Another important aspect of measuring efficacious and effective treatment is comparative effectiveness research, which largely refers to medical research and focuses on determining which one of several treatments is better for a particular disorder. This information is important for both practitioners and individuals in making decisions on the best treatment for specific disorders with the ultimate goal of improving health.

Overall, interventions can be shown to be efficacious within a clinical trial, and effective in a real-world setting. While it is ideal to have evidence of both qualities, as Bowen and col- leagues (2009) point out, given resource con- straints, not all interventions can be tested for both. These writers suggest that via feasibility studies (which, as noted, measure the probability of success of a particular intervention), we can

G E N E R AT IO N S – Journal of the American Society on Aging

40 | Spring 2010 • Volume 34 .Number 1

Pages 36–42

©American Society on Aging

determine which treatments are the most promising candidates for efficacy testing and thus the best investment. For example, feasibil- ity studies might assess different recruitment strategies for particular populations, or improve on the design of a previous intervention that was not successful with a particular sample.

Another type of experimental design is a single-subject design in which a subject serves as his or her own control. The subject is exposed to both a nontreatment phase and a treatment phase, and performance in each of these is measured and compared (McMillan, 2004). The logic is that the person’s baseline behavior should match his or her behavior during and after an intervention phase unless the interven- tion affects or changes it. If there is a clear distinction between the baseline and the intervention, and then the scores return to the same trends/level during reversal, a relationship between the variables is inferred. This design can be replicated with additional subjects.

Observational research Observational research, such as survey

research and qualitative studies, can also be used to inform practice. In an observational study, the researcher measures (but cannot control) an independent variable and assesses whether or not that independent variable has a significant association with some dependent, or “outcome,” variable in a particular group of participants. For example, in an effort to understand racial and ethnic disparities in advance care planning (the dependent variable) among patients with cancer, researchers examined as independent variables acknowledgment of terminal illness, religious- ness, and treatment preferences (Smith et al., 2008). If these independent variables were found to be significantly associated with the use of advance care planning, that information could serve as a starting point for a targeted interven- tion study with the goal of reducing disparities in advance care planning. In fact, findings showed that although racial and ethnic dispari-

ties were found with regard to less use of advance care planning by African American and Hispanic patients in this sample, the indepen- dent variables studied did not have an effect on this outcome variable. The researchers suggest- ed using additional measures of these indepen- dent variables in future studies, in addition to studying other potential factors that could affect the outcome, such as literacy and the use of video images in understanding advanced disease. Overall, it is important to distinguish two major types of research methods, quantitative versus qualitative methods. Qualitative studies, which utilize an inductive approach, are perhaps most useful in an area where little research exists, so the researcher initially can learn what areas or variables are important to an understanding of certain phenomena (also referred to as develop- ing a “grounded theory”). Quantitative studies, on the other hand, are deductive, and their purpose is to predict and test causal relation- ships among variables, that is, to test particular hypotheses. Quantitative studies begin with theories and hypotheses, while the goal of many qualitative studies is to develop hypotheses to test. The data gathered via each approach also differ. In qualitative research, assessments capture subjects’ responses to questions accord- ing to their own subjective experience. Data are in the form of text, recorded as the subjects relay their responses. Quantitative data are more structured as they are based on participants’ responses to predetermined statements or questions, typically in the form of ratings of choices on a scale. Quantitative research often utilizes widely accepted measures with estab- lished reliability and validity, and data are subject to more rigorous statistical analyses compared to qualitative data.

Methodological Issues: Sampling, Attrition, Significance, Replication

Some important methodological issues in research are sampling, attrition, significance, and replication. Regarding sampling, it is

Evidence-Based Practice: Critical Thinking About Translating Research to Practice

Spring 2010 • Volume 34 .Number 1 | 41

Pages 36–42

important to look at the way participants were selected for the study (the eligibility criteria), the response rate, and the sample descriptives. If there was more than one point of data collection (a longitudinal study design), it is also essential to examine the rate of attrition (study drop-outs and missing cases) and how attrition was

handled in the analyses. These factors are impor- tant because they affect the ability to generalize from the research sample to the population of interest. The more unique a sample is, the less generalizable it will be. That is, results may only be generalizable to people who are very similar to those in the particular sample. It is helpful if there is a comparison of the characteristics (e.g., sex, age, marital status) of participants to those of individuals who declined to take part in the study. Similarly, at study completion, it is important to compare the characteristics of those participants who remained in a longitudi- nal study over multiple time points with those who dropped out at a follow-up time point. In a best-case scenario, these differences are minor. However, often it is the most physically and mentally healthy who are most likely to remain in a study. If analyses are conducted only on those participants who remained in the study for all time points, the “survivors,” then it should be acknowledged that this group is most likely in better physical and mental health compared to those participants who dropped out, thus reducing the generalizability of study results.

Another important distinction is between statistical and clinical significance. Statistical significance refers to the likelihood that a particular study result is due to chance. We want this likelihood to be extremely small, the smaller the better. For example, we want to say that a treatment caused or was associated with a particular outcome, or that two or more vari- ables are indeed, significantly associated with

each other, not that we obtained our results by chance. A common percentage used to demon- strate a statistically significant finding is 0.95, which indicates that a finding has a 95% chance of being true. However, for reporting purposes, a statistically significant finding is described as having a five percent chance of not being true (p<.05). In contrast, clinical significance refers to whether or not study results are clinically meaningful with regard to the specific popula- tion under study. The question here is ‘Are the results clinically important, from a practical perspective?’ A reduction of a point or two on a particular depression scale may be statistically significant because of a large sample size, but clinically the reduction does not represent a major change in symptoms experienced.

Finally, the importance of replication of research supporting evidence-based practices must be stressed. To feel most confident in the use of a particular intervention, a practitioner would want to be sure that study findings supporting this evidence were replicated or repeated in numerous studies by similar and different groups of researchers. The more study findings have been replicated, the greater the ability to generalize to various groups of participants and the greater the external validity of the study. External validity is important for practitioners who want to utilize evidence-based practices with their own particular client populations.

Conclusions and Future Directions Although practitioners and researchers in

aging work in very different arenas—research- ers develop and test hypotheses that contribute to generalizable knowledge and practitioners deliver particular treatments or interventions designed to promote an individual’s well- being—their shared overall goal is to improve the quality of life for older adults. Both groups want the best possible treatments, programs, and other interventions for patients and clients who need them. Unfortunately, a large gap

Interventions can be tested via either efficacy or effectiveness trials.

G E N E R AT IO N S – Journal of the American Society on Aging

42 | Spring 2010 • Volume 34 .Number 1

Pages 36–42

exists between research evidence, on the one side, and application in clinical practice, on the other. This gap is largely due to a lack of translation of research findings that are rel- evant for practice into treatment guidelines that are accessible to clinicians. In part, the lack of translation may be caused by confusion regarding whose job it is, and how to do it. One hopeful note is that the increasing availability of both public and private funding initiatives that focus specifically on methodology for the dissemination and implementation of evidence- based approaches in applied settings. The

National Institutes of Health and the Rosalynn Carter Institute for Caregiving are two exam- ples (see articles by Tilly and by Birkel, this issue). With researchers and practitioners working as partners on these translational goals, evidence-based practice will become the business of usual care.

Joann P. Reinhardt, Ph.D., is Director of Research at the Jewish Home Lifecare Research Institute on Aging and Associate Professor, Brookdale Department of Geriatrics and Palliative Medicine, Mount Sinai School of Medicine, New York, NY.

References Adams, K. B., LeCroy, C. W., and Matto, H. C. 2009. “Limitations of Evidence-Based Practice for Social Work Education: Unpacking the Complexity.” Journal of Social Work Education 45(2):165-186.

Bowen, D. J., et al. 2009. “How We Design Feasibility Studies.” American Journal of Preventive Medicine 36(5):452-457.

Gartlehner G., et al. 2006. A Simple and Valid Tool to Distinguish Efficacy from Effectiveness Studies. Journal of Clinical Epidemiology 59(10):1040-1048.

McMillan, J. H.2004. Educational Research: Fundamentals for the Consumer, (4th ed.). Boston: Allyn and Bacon.

Nelson, T. D., and Steele, R. G. 2007. Predictors of practitioner self-reported use of evidence- based practices: Practitioner training, clinical setting, and attitudes toward research. Administrative Policy , Mental Health and Mental Health Services Research 34:319-330.

Reinhardt, J. P., Boerner, K., and Horowitz, A. 2006. “Good to Have but Bad to Use: Differential Impact of Perceived and Received Support on Well-Being.” Journal of Social and Personal Relation- ships 23:117-129.

Smith et al. 2008. “Racial and Ethnic Differences in Advance Care Planning Among Patients with Cancer: Impact of Terminal Illness Acknowledgment, Reli- giousness, and Treatment Prefer- ences.” Journal of Clinical Oncol- ogy 26(25):4131-4137.

emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.

However, users may print, download, or email articles for individual use.